II.

LITERATURE SURVEY

In 2012, Hong Su et. al. [15] described ‘A Partial Least Squares Regression-Based Fusion Model
for Predicting the Trend in Drowsiness’. They proposed a new technique of modeling driver
drowsiness with multiple eyelid movement features based on an information fusion technique—
partial least squares regression (PLSR), with which to cope with the problem of strong collinear
relations among eyelid movement features and, thus, predicting the tendency of the drowsiness.
The predictive precision and robustness of the model thus established are validated, which show
that it provides a novel way of fusing multi-features together for enhancing our capability of
detecting and predicting the state of drowsiness.

In June, 2014, Bin Yang et. al. described ‘Camera- based Drowsiness Reference for Driver State
Classification under Real Driving Conditions’. They proposed that measures of the driver’s eyes
are capable to detect drowsiness under simulator or experiment conditions. The performance of
the latest eye tracking based in-vehicle fatigue prediction measures are evaluated. These
measures are assessed statistically and by a classification method based on a large dataset of 90
hours of real road drives. The results show that eye-tracking drowsiness detection works well for
some drivers as long as the blinks detection works properly. Even with some proposed
improvements, however, there are still problems with bad light conditions and for persons
wearing glasses. As a summary, the camera based sleepiness measures provide a valuable
contribution for a drowsiness reference, but are not reliable enough to be the only reference.

In 2016, M.J. Flores et. al. described ‘Driver drowsiness detection system under infrared
illumination for an intelligent vehicle’. They proposed that to reduce the amount of such
fatalities, a module for an advanced driver assistance system, which caters for automatic driver
drowsiness detection and also driver distraction, is presented. Artificial intelligence algorithms
are used to process the visual information in order to locate, track and analyze both the driver’s
face and eyes to compute the drowsiness and distraction indexes. This real- time system works
during nocturnal conditions as a result of a near-infrared lighting system. Finally, examples of
different driver images taken in a real vehicle at nighttime are shown to validate the proposed
algorithms.

In June, 2017, A. Cheng et. al. described 'Driver Drowsiness Recognition Based on Computer
Vision Technology’. They presented a nonintrusive drowsiness recognition method using eye-
tracking and image processing. A robust eye detection algorithm is introduced to address the
problems caused by changes in illumination and driver posture. Six measures are calculated with
percentage of eyelid closure, maximum closure duration, blink frequency, average opening level
of the eyes,
opening velocity of the eyes, and closing velocity of the eyes. These measures are combined
using Fisher’s linear discriminated functions using a stepwise method to reduce the correlations
and extract an independent index. Results with six participants in driving simulator experiments
demonstrate the feasibility of this video-based drowsiness recognition method that provided 86%
1
accuracy.

In 2018, G. Kong et. al. described ‘Visual Analysis of Eye State and Head Pose for Driver
Alertness Monitoring’. They presented visual analysis of eye state and head pose (HP) for
continuous monitoring of alertness of a vehicle driver. Most existing approaches to visual
detection of non-alert driving patterns rely either on eye closure or head nodding angles to
determine the driver drowsiness or distraction level. The proposed scheme uses visual features
such as eye index (EI), pupil activity (PA), and HP to extract critical information on non-
alertness of a vehicle driver. A support vector machine (SVM) classifies a sequence of video
segments into alert or non-alert driving events. Experimental results show that the proposed
scheme offers high classification accuracy with acceptably low errors and false alarms for people
of various ethnicity and gender in real road driving conditions.

In June, 2019, Eyosiyas et. al. Described ‘Driver Drowsiness Detection through \HMM based
Dynamic Modeling’. They proposed a new method of analyzing the facial expression of the
driver through Hidden Markov Model (HMM) based dynamic modeling to detect drowsiness.
They have implemented the algorithm using a simulated driving setup. Experimental results
verified the effectiveness of the proposed method
.
In August 2019, García et. al. described ‘Driver Monitoring Based on Low-Cost 3-D Sensors’.
They proposed a solution for driver monitoring and event detection based on 3-D information
from a range camera is presented. The system combines 2-D and 3-D techniques to provide head
pose estimation and regions-of-interest identification. Based on the captured cloud of 3-D points
from the sensor and analyzing the 2-D projection, the points corresponding to the head are
determined and extracted for further analysis. Later, head pose estimation with three degrees of
freedom (Euler angles) is estimated based on the iterative closest points algorithm. Finally,
relevant regions of the face are identified and used for further analysis, e.g., event detection and
behavior analysis. The resulting application is a 3-D driver monitoring system-based on low-cost
sensors. It represents an interesting tool for human factor research studies, allowing automatic
study of specific factors and the detection of special event related to the driver, e.g., driver

2
drowsiness, inattention, or head pose.

3
 DEPARTMENT OF COMPUTER
APPLICATIONS

PROJECT REPORT
ON
DRIVER’S DROWSINESS
DETECTION SYSTEM

BACHELOR OF COMPUTER APPLICATION

Presented By

Karthikeyan R. 19351065
Sriram R. 19351053
Balasurya S. 19351056
Hemanand E S. 19352001

UNDER THE GUIDANCE OF

Associate prof .Dr. A. Rama Prasath

4
5
 ABSTRACT
Driver fatigue is one of the major causes of accidents in the world. Detecting the
drowsiness of the driver is one of the surest ways of measuring driver fatigue. In
this project we aim to develop a prototype drowsiness detection system. This
system works by monitoring the eyes of the driver and sounding an alarm when
he/she is drowsy.

The system so designed is a non-intrusive real-time monitoring system. The

priority is on improving the safety of the driver without being obtrusive. In this
project the eye blink of the driver is detected. If the drivers eyes remain closed
for more than a certain period of time, the driver is said to be drowsy and an
alarm is sounded. The programming for this is done in OpenCV using the
Haarcascade library for the detection of facial features

.Driver drowsiness is one of the major causes of traffic accidents. It is a serious

highway safety problem. If drivers could be warned before they became too
drowsy to drive safely, some of these crashes could be prevented. In order to
reliably detect the drowsiness, it depends on the presentation of timely warnings
of drowsiness. To date, the effectiveness of drowsiness detection methods has
been limited by their failure to consider individual differences. Based on the
type of data used, drowsiness detection can be conveniently separated into the
two categories of intrusive and non-intrusive methods. During the survey, non-
intrusive methods detect drowsiness by measuring driving behavior and
sometimes eye features, through which camera based detection system is the
best method and so are useful for real world driving situations. This paper
presents the review of existed drowsiness detection techniques that will be used
in this system like Circular Hough Transform, FCM, Lab Color Space etc.
2
Keywords:- Drowsy Driving; Drowsiness Recognition; Driver Monitoring;
Circular Hough Transform; FCM; Lab Color Space; Varying Luminance
Conditions.

3
 CONTENTS
Chapter Page No.
1. Introduction 07
2. Why OpenCV 11
2.1 Eye blink detection using Matlab 12
2.2 What is OpenCV? 12
2.3 What is Computer Vision? 13
2.4 The origin of OpenCV 13
2.5 OpenCV structure and content 14
2.6 Why OpenCV? 14
3. Machine Learning 16
3.1 What is Machine Learning? 17
3.2 Training and Test Set 17
3.3 OpenCV ML Algorithms 18
3.4 Using Machine Learning in Vision 21
3.5 Variable Importance 22
3.6 Tools of Machine Learning 23
3.7 Learning Objects 25
3.7.1 Dataset Gathering 25
3.7.2 Building Vector Output File 25
3.7.3 Object Detection 26
3.7.4 Training 26
3.7.5 Creation of .xml Files 27

4
 Chapter Page No.
4. Algorithm and Implementation 28
4.1 Algorithm 29
4.2 Image Acquisition 30
4.3 Face Detection 30
4.4 Setting Image Region of Interest 32
4.5 Eye Detection 33
5. Result 34
5.1 Summary 35
5.2 Sample Images 36
5.3 Table and Analysis 44
5.4 Limitations 45
5.5 Future Works 46
5.6 Conclusion 47
6. References 48

5
 CHAPTER-1
INTRODUCTION

6
 INTRODUCTION
Driver fatigue is a significant factor in a large number of vehicle accidents. Recent
statistics estimate that annually 1,200 deaths and 76,000 injuries can be attributed to
fatigue related crashes.

The development of technologies for detecting or preventing drowsiness at the wheel

is a major challenge in the field of accident avoidance systems. Because of the
hazard that drowsiness presents on the road, methods need to be developed for
counteracting its affects.

The aim of this project is to develop a prototype drowsiness detection system. The
focus will be placed on designing a system that will accurately monitor the open or
closed state of the driver’s eyes in real-time.

By monitoring the eyes, it is believed that the symptoms of driver fatigue can be
detected early enough to avoid a car accident. Detection of fatigue involves the
observation of eye movements and blink patterns in a sequence of images of a face.
[1]

Initially, we decided to go about detecting eye blink patterns using Matlab. The
procedure used was the geometric manipulation of intensity levels. The algorithm
used was as follows.

First we input the facial image using a webcam. Preprocessing was first performed
by binarizing the image. The top and sides of the face were detected to narrow down
the area where the eyes exist. Using the sides of the face, the center of the face was
found which will be used as a reference when computing the left and right eyes.
Moving down from the top of the face, horizontal averages of the face area were
7
calculated. Large changes in the averages were used to define the eye area. There
was little change in the horizontal average when the eyes were closed which was
used to detect a blink.
However Matlab had some serious limitations. The processing capacities required
by Matlab were very high. Also there were some problems with speed in real time
processing. Matlab was capable of processing only 4-5 frames per second. On a
system with a low RAM this was even lower. As we all know an eye blink is a
matter of milliseconds. Also a drivers head movements can be pretty fast. Though
the Matlab program designed by us detected an eye blink, the performance was
found severely wanting.

This is where OpenCV came in. OpenCV is an open source computer vision library.
It is designed for computational efficiency and with a strong focus on real time
applications. It helps to build sophisticated vision applications quickly and easily.
OpenCV satisfied the low processing power and high speed requirements of our
application.

8
We have used the Haartraining applications in OpenCV to detect the face and eyes.
This creates a classifier given a set of positive and negative samples. The steps were
as follows:-

• Gather a data set of face and eye. These should be stored in one or more
directories indexed by a text file. A lot of high quality data is required for the
classifier to work well.
• The utility application createsamples() is used to build a vector output file.
Using this file we can repeat the training procedure. It extracts the positive
samples from images before normalizing and resizing to specified width and
height.
• The Viola Jones cascade decides whether or not the object in an image is
similar to the training set. Any image that doesn’t contain the object of interest
can be turned into negative sample. So in order to learn any object it is
required to take a sample of negative background image. All these negative
images are put in one file and then it’s indexed.
• Training of the image is done using boosting. In training we learn the group
of classifiers one at a time. Each classifier in the group is a weak classifier.
These weak classifiers are typically composed of a single variable decision
tree called stumps. In training the decision stump learns its classification
decisions from its data and also learns a weight for its vote from its accuracy
on the data. Between training each classifier one by one, the data points are
reweighted so that more attention is paid to the data points where errors were
made. This process continues until the total error over the dataset arising from
the combined weighted vote of the decision trees falls below a certain
threshold. [6]

This algorithm is effective when a large number of training data are available.

9
For our project face and eye classifiers are required. So we used the learning objects
method to create our own haarclassifier .xml files. Around 2000 positive and 3000
negative samples are taken. Training them is a time intensive process. Finally
face.xml and haarcascade-eye.xml files are created.
These xml files are directly used for object detection. It detects a sequence of objects
(in our case face and eyes). Haarcascade-eye.xml is designed only for open eyes. So
when eyes are closed the system doesn’t detect anything. This is a blink. When a
blink lasts for more than 5 frames, the driver is judged to be drowsy and an alarm is
sounded.

10
11
12
 II. LITERATURE SURVEY

In 2012, Hong Su et. al. [15] described ‘A Partial Least Squares Regression-Based
Fusion Model for Predicting the Trend in Drowsiness’. They proposed a new
technique of modeling driver drowsiness with multiple eyelid movement features
based on an information fusion technique—partial least squares regression (PLSR),
with which to cope with the problem of strong collinear relations among eyelid
movement features and, thus, predicting the tendency of the drowsiness. The
predictive precision and robustness of the model thus established are validated,
which show that it provides a novel way of fusing multi-features together for
enhancing our capability of detecting and predicting the state of drowsiness.

In June, 2014, Bin Yang et. al. described ‘Camera- based Drowsiness Reference
for Driver State Classification under Real Driving Conditions’. They proposed that
measures of the driver’s eyes are capable to detect drowsiness under simulator or
experiment conditions. The performance of the latest eye tracking based in-vehicle
fatigue prediction measures are evaluated. These measures are assessed statistically
and by a classification method based on a large dataset of 90 hours of real road
drives. The results show that eye-tracking drowsiness detection works well for
some drivers as long as the blinks detection works properly. Even with some
proposed improvements, however, there are still problems with bad light conditions
and for persons wearing glasses. As a summary, the camera based sleepiness
measures provide a valuable contribution for a drowsiness reference, but are not
reliable enough to be the only reference.

In 2016, M.J. Flores et. al. described ‘Driver drowsiness detection system under
infrared illumination for an intelligent vehicle’. They proposed that to reduce the
amount of such fatalities, a module for an advanced driver assistance system, which
caters for automatic driver drowsiness detection and also driver distraction, is
presented. Artificial intelligence algorithms are used to process the visual
information in order to locate, track and analyze both the driver’s face and eyes to
compute the drowsiness and distraction indexes. This real- time system works
during nocturnal conditions as a result of a near-infrared lighting system. Finally,
examples of different driver images taken in a real vehicle at nighttime are shown
to validate the proposed algorithms.

13
In June, 2017, A. Cheng et. al. described 'Driver Drowsiness Recognition Based on
Computer Vision Technology’. They presented a nonintrusive drowsiness
recognition method using eye-tracking and image processing. A robust eye
detection algorithm is introduced to address the problems caused by changes in
illumination and driver posture. Six measures are calculated with percentage of
eyelid closure, maximum closure duration, blink frequency, average opening level
of the eyes,
opening velocity of the eyes, and closing velocity of the eyes. These measures are
combined using Fisher’s linear discriminated functions using a stepwise method to
reduce the correlations and extract an independent index. Results with six
participants in driving simulator experiments demonstrate the feasibility of this
video-based drowsiness recognition method that provided 86% accuracy.

In 2018, G. Kong et. al. described ‘Visual Analysis of Eye State and Head Pose for
Driver Alertness Monitoring’. They presented visual analysis of eye state and head
pose (HP) for continuous monitoring of alertness of a vehicle driver. Most existing
approaches to visual detection of non-alert driving patterns rely either on eye
closure or head nodding angles to determine the driver drowsiness or distraction
level. The proposed scheme uses visual features such as eye index (EI), pupil
activity (PA), and HP to extract critical information on non-alertness of a vehicle
driver. A support vector machine (SVM) classifies a sequence of video segments
into alert or non-alert driving events. Experimental results show that the proposed
scheme offers high classification accuracy with acceptably low errors and false
alarms for people of various ethnicity and gender in real road driving conditions.

In June, 2019, Eyosiyas et. al. Described ‘Driver Drowsiness Detection through
\HMM based Dynamic Modeling’. They proposed a new method of analyzing the
facial expression of the driver through Hidden Markov Model (HMM) based
dynamic modeling to detect drowsiness. They have implemented the algorithm
using a simulated driving setup. Experimental results verified the effectiveness of
the proposed method
.
In August 2019, García et. al. described ‘Driver Monitoring Based on Low-Cost 3-
D Sensors’. They proposed a solution for driver monitoring and event detection
based on 3-D information from a range camera is presented. The system combines
2-D and 3-D techniques to provide head pose estimation and regions-of-interest
identification. Based on the captured cloud of 3-D points from the sensor and
analyzing the 2-D projection, the points corresponding to the head are determined
and extracted for further analysis. Later, head pose estimation with three degrees of
freedom (Euler angles) is estimated based on the iterative closest points algorithm.
14
Finally, relevant regions of the face are identified and used for further analysis,
e.g., event detection and behavior analysis. The resulting application is a 3-D driver
monitoring system-based on low-cost sensors. It represents an interesting tool for
human factor research studies, allowing automatic study of specific factors and the
detection of special event related to the driver, e.g., driver drowsiness, inattention,
or head pose .

15
16
CHAPTER 2
Why OpenCV?

17
2.1 Eye blink detection using Matlab:-
In our four year B.Tech career, of all the programming languages we had obtained the most
proficiency in Matlab. Hence it was no surprise that we initially decided to do the project using
Matlab. The procedure used was the geometric manipulation of intensity levels. The algorithm
used was as follows.

First we input the facial image using a webcam. Preprocessing was first performed by binarizing
the image. The top and sides of the face were detected to narrow down the area where the eyes
exist. Using the sides of the face, the center of the face was found which will be used as a reference
when computing the left and right eyes. Moving down from the top of the face, horizontal averages
of the face area were calculated. Large changes in the averages were used to define the eye area.
There was little change in the horizontal average when the eyes were closed which was used to
detect a blink. [1]

However Matlab had some serious limitations. The processing capacities required by Matlab were
very high. Also there were some problems with speed in real time processing. Matlab was capable
of processing only 4-5 frames per second. On a system with a low RAM this was even lower. As
we all know an eye blink is a matter of milliseconds. Also a drivers head movements can be pretty
fast. Though the Matlab program designed by us detected an eye blink, the performance was found
severely wanting.

2.2 What Is OpenCV?

OpenCV [OpenCV] is an open source (see http://opensource.org) computer vision library
available from http://SourceForge.net/projects/opencvlibrary.
OpenCV was designed for computational efficiency and having a high focus on real-time image
detection. OpenCV is coded with optimized C and can take work with multicore processors. If we
desire more automatic optimization using Intel architectures
[Intel], you can buy Intel’s Integrated Performance Primitives (IPP) libraries [IPP]. These consist
of low-level routines in various algorithmic areas which are optimized. OpenCV automatically
uses the IPP library, at runtime if that library is installed.

18
One of OpenCVs goals is to provide a simple-to-use computer vision infrastructure which helps
people to build highly sophisticated vision applications fast. The OpenCV library, containing over
500 functions, spans many areas in vision. Because computer vision and machine learning oft en
goes hand-in-hand,
OpenCV also has a complete, general-purpose, Machine Learning Library (MLL).
This sub library is focused on statistical pattern recognition and clustering. The MLL is very useful
for the vision functions that are the basis of OpenCV’s usefulness, but is general enoughto be
used for any machine learning problem. [4]

2.3 What Is Computer Vision?

Computer vision is the transforming of data from a still, or video camera into either a
representation or a new decision. All such transformations are performed to achieve a particular
goal. A computer obtains a grid of numbers from a camera or from the disk, and that’s that.
Usually, there is no built in pattern recognition or automatic control of focus and aperture, no cross-
associations with years of experience. For the most part, vision systems are still fairly naïve. [6]

2.4 The Origin of OpenCV

OpenCV came out of an Intel Research initiative meant to advance CPU-intensive applications.
Toward this end, Intel launched various projects that included real-time ray tracing and also 3D
display walls. One of the programmers working for Intel at the time was visiting universities. He
noticed that a few top university groups, like the MIT Media Lab, used to have well-developed as
well as internally open computer vision infrastructures—code that was passed from one student to
another and which gave each subsequent student a valuable foundation while developing his own
vision application. Instead of having to reinvent the basic functions from beginning, a new student
may start by adding to that which came before. [6]

19
2.5 OpenCV Structure and Content
OpenCV can be broadly structured into five primary components, four of which are shown in the
figure. The CV component contains mainly the basic image processing and higher-level computer
vision algorithms; MLL the machine learning library includes many statistical classifiers as well
as clustering tools. HighGUI component contains I/O routines with functions for storing, loading
video & images, while CXCore contains all the basic data structures and content. [6]

2.6 Why OpenCV?

Specific

OpenCV was designed for image processing. Every function and data structure has been designed
with an Image Processing application in mind. Meanwhile, Matlab, is quite generic. You can get
almost everything in the world by means of toolboxes. It may be financial toolboxes or specialized
DNA toolboxes.

20
Speedy

Matlab is just way too slow. Matlab itself was built upon Java. Also Java was built upon C. So
when we run a Matlab program, our computer gets busy trying to interpret and compile all that
complicated Matlab code. Then it is turned into Java, and finally executes the code.

If we use C/C++, we don’t waste all that time. We directly provide machine language code to the
computer, and it gets executed. So ultimately we get more image processing, and not more
interpreting.

After doing some real time image processing with both Matlab and OpenCV, we usually got very
low speeds, a maximum of about 4-5 frames being processed per second with Matlab. With
OpenCV however, we get actual real time processing at around 30 frames being processed per
second.

Sure we pay the price for speed – a more cryptic language to deal with, but it’s definitely worth
it. We can do a lot more, like perform some really complex mathematics on images using C and
still get away with good enough speeds for your application.

Efficient

Matlab uses just way too much system resources. With OpenCV, we can get away with as little
as 10mb RAM for a real-time application. Although with today’s computers, the RAM factor isn’t
a big thing to be worried about. However, our drowsiness detection system is to be used inside a
car in a way that is non-intrusive and small; so a low processing requirement is vital.

Thus we can see how OpenCV is a better choice than Matlab for a real-time drowsiness detection
system.

21
 Chapter 3
Machine Learning

22
3.1 What Is Machine Learning
The goal of machine learning is to turn data into information. After having learned from a
gathering of data, we want a machine that is able to answer any question about the data:
• What are the other data that are similar to given data? Is there a face in the image?
• What kind of ad will influence the user?
• There is usually a cost component, so the question may become:
Of the many products that we can make the money from, which one will most
likely be bought by the user if an ad is shown for it?
Machine learning converts data into information by detecting rules or patterns from that data. [6]

3.2 Training and Test Set

Machine learning works on data like temperature values or stock prices or color intensities, and
in our case face and eye detection. The data is usually preprocessed into features. We might take
a database containing 1,000 face images then perform an edge detector on all the faces, and then
obtain features such as edge direction, edge intensity, also offset from the face center for every
face. We may obtain up to 500 such values for every face or a feature vector of 500 entries. We
may then use machine learning to construct some sort of model from the obtained data. If we want
to see how the faces fall into various groups (narrow, wide, etc.), after that a clustering algorithm
can be the preferred choice. In case we want to learn how to guess the age of a woman from the
pattern of the edges that are detected on her face, then we can use a classifier algorithm.To reach
our goals, machine learning algorithms can analyze our obtained features and hence adjust the
weights, the thresholds, and all the other parameters for maximizing performance set by those
goals. This method of parameter adjustment for meeting a goal is what is called learning.
It is very important to understand how efficiently machine learning methods can work. This might
be a delicate task. Usually, we break up the input data set into a very large training set (i.e. 900
faces, in our project) and a relatively small test set (i.e. remainder 100 faces). Then we can run our
classifier on the training set in order to learn for a age prediction model, data feature

23
vectors given. Once done, we can then test our age prediction classifier obtained on the remainder
of the images left in the set.
The test set is not applied for training; also we don’t allow the classifier to see the age labels of the
test set. We then run the classifier on all of the 100 faces present in the test set and then record
how well the ages predicted by the feature vector match the real ages. When the classifier performs
poorly, we may try to add new features to the data or consider other types of classifiers. There are
many types of classifiers and various algorithms used for training them.
When the classifier performs well, we have a possibly lucrative model that can be used on data in
the actual world. This system may be used to set the performance of a video game according to
age.
After the classifier has been setup, it sees faces that it could not see before and it makes decisions
with regards to what it had learned during training. Finally, when we develop a classification
system, we usually use a validation data set.
At times, testing the entire system at the finish is a big step to take. We usually want to change
parameters in the way before preparing our classifier for the final testing. We may do this by
splitting the initial 1,000-face data set into 3 parts: i.e. a training set consisting of 800 faces and
also a validation set with 100 faces, as well as a test set containing 100 faces. While we’re running
across the training data, we can perform pretests on validation data in order to see our progress. If
we are happy with our output for the validation set, we can run the classifier up on the test set for
the final result. [6]

3.3 OpenCV ML Algorithms

The machine learning algorithms that are included in OpenCV are given as follows. All the
algorithms are present in the ML library apart from Mahalanobis and K-means, which are present
in CVCORE, and the algorithm of face detection, which is present in CV. [6]

Mahalanobis: It is a measure of distance that is responsible for the stretchiness of the data. We
can divide out the covariance of the given data to find this out. In case of the covariance being the
identity matrix (i.e. identical variance), this measure will be identical to the Euclideandistance. [6]

24
K-means: It is an unsupervised clustering algorithm which signifies a distribution of data w.r.t.
K centers, K being chosen by the coder. The difference between K-means and expectation
maximization is that in K-means the centers aren’t Gaussian. Also the clusters formed look
somewhat like soap bubbles, as centers compete to occupy the closest data points. All these cluster
areas are usually used as a form of sparse histogram bin for representing the data. [6]

Normal/Naïve Bayes classifier: It is a generative classifier where features are often assumed to
be of Gaussian distribution and also statistically independent from one another. This assumption
is usually false. That’s why it’s usually known as a ―naïve Bayes‖ classifier. That said, this
method usually works surprisingly well. [6]

Decision trees: It is a discriminative classifier. The tree simply finds a single data feature and
determines a threshold value of the current node which best divides the data into different classes.
The data is broken into parts and the procedure is recursively repeated through the left as well as
the right branches of the decision tree. Even if it is not the top performer, it’s usually the first thing
we try as it is fast and has a very high functionality. [6]

Boosting: It is a discriminative group of classifiers. In boosting, the final classification decision

is made by taking into account the combined weighted classification decisions of the group of
classifiers. We learn in training the group of classifiers one after the other. Each classifier present
in the group is called a weak classifier. These weak classifiers are usually composed of single-
variable decision trees known as ―stumps‖. Learning its classification decisions from the given
data and also learning a weight for its vote based on its accuracy on the data are things the decision
tree learns during training. While each classifier is trained one after the other, the data points are
re-weighted to make more attention be paid to the data points in which errors were made. This
continues until the net error over the entire data set, obtained from the combined weighted vote of
all the decision trees present, falls below a certain threshold. This algorithm is usually effective
when a very large quantity of training data is available. [6]

Random trees: It is a discriminative forest of a lot of decision trees, each of which is built down
to a maximal splitting depth. At the time of learning, every node of every tree is allowed a choice

25
of splitting variables, but only from a randomly generated subset of all the data features. This
ensures that all the trees become statistically independent and a decision maker. In the run mode,
all the trees get an unweighted vote. Random trees are usually quite effective. They can also
perform regression by taking the average of the output numbers from every tree. [6]

Face detector (Haar classifier): It is an object detection application. It is based on a smart use
of boosting. A trained frontal face detector is available with the OpenCV distribution. This works
remarkably well. We can train the algorithm for other objects by using the software provided. This
works wonderfully for rigid objects with characteristic views. [6]

Expectation maximization (EM): It is used for clustering. It is a generative unsupervised

algorithm It fits N multidimensional Gaussians to the data, N being chosen by the user. It can act
as an efficient way for representing a more complex distribution using only a few parameters (i.e.
means and variances). Usually used in segmentation, it can be compared with K-means. [6]

K-nearest neighbors: It is one of the simplest discriminative classifier. The training data is simply
stored using labels. Then, a test data point is classified in accordance to the majority vote of the K
nearest data points. K-nearest neighbours is probably th simplest algorithm we can use. It is usually
effective but it can be slow. It also requires a lot of memory. [6]

Neural networks or Multilayer perceptron (MLP): It is a discriminative algorithm which

almost always contains hidden units in between the output and the input nodes for better
representation of the input signal. It is slow to train, however it is quite fast to run. It remains the
best performer for applications like letter recognition. [6]

Support vector machine (SVM): It is a discriminative classifier that is also capable of doing
regression. Here, a distance function in between two data points is defined in a higher- dimensional
space. (Projecting data onto higher dimensions helps in making the data more likely for linear
separation.) Support vector machine (SVM) learns separating hyperplanes which maximally
separate all the classes in the higher dimension. This tends to be the best when there

26
is limited data. However, when large data sets are available, boosting or random trees are preferred.
[6]

3.4 Using Machine Learning in Vision

Usually, most algorithms take a data vector having many features as input. Here, the number of
features may number in the thousands. If our task is recognizing a certain kind of object—take
for example, a person’s face. The first problem that we encounter is obtaining and labeling the
training data which falls into positive (i.e. there is a face in the window) and negative (i.e. no face)
cases. We soon realize that faces can appear at various scales: i.e. their image might consist of only
a few pixels, or we might be looking at an ear which is filling the whole screen. Worse still, faces
are usually occluded. We have to define what we actually mean when we say that a face is in the
window. [6]

After having labeled the data that was obtained from various sources, we should decide which
features we need to extract from these objects. Also, we must know what objects we are after. If
the faces always appear upright, there is no reason for using rotation-invariant features and also no
reason for trying to rotate the objects before processing.
In general, we must try to find features that express a little invariance in the objects. These can
be scale-tolerant histograms of gradients or colors or even the popular SIFT features.
When we have requisite background window information, we can first remove it in order to help
other objects stand out. Then we perform our image processing. This may consist of normalizing
the image and then computing the various features. The resulting data vectors are all given the
label that is associated with the object, action, or window. [6]
Once the data is obtained and converted into feature vectors, we break up the data into training
sets, validation sets and test sets. It is advisable to do our learning, validation, and testing using a
cross-validation framework. Here, the data is split into K subsets and we run various training
(maybe validation) as well as test sessions. Each session consists of various sets of data that take
on the roles of training (validation) and test. The test results obtained from these separate

27
sessions are used for averaging to get the final performance result. A more accurate picture of how
the classifier performs when deployed in operation can be given by cross-validation.

Now that our data is ready, we must choose a classifier. Usually the choice of the classifier is
determined by computational, data, and memory requirements. For certain applications, like online
user preference modeling, we need to train the classifier quickly. In such a case, nearest neighbors,
normal Bayes, or decision trees should be a good choice. When memory is the primary
consideration, decision trees or neural networks are used for their space efficiency. When we have
time to train our classifier but it needs to run quickly, neural networks can be a good choice, as
with normal Bayes classifiers & support vector machines. When we have time to train but require
high accuracy, then boosting and random trees are good choices. When we just want an easy and
understandable check weather our features are chosen well or not, thendecision trees or nearest
neighbors should be used. For a good out of the box classification performance, boosting or random
trees are tried. [6]

3.5 Variable Importance

This is the importance of a particular variable in a dataset. One of the uses of variable importance
is for reducing the number of features the classifier needs to consider. After starting with a number
of features, we train our classifier to find the importance of each feature in relation to all the other
features. We then get rid of unimportant features. Eliminating unimportant features helps in
improving speed performance (since it can eliminate the processing taken for computing those
features) and also makes training and testing faster. When we don’t have sufficient data, which is
regularly the case, then eliminating unimportant variables helps in increasing classification
accuracy, which in turn yields faster processing and better results. [6]

Breiman’s algorithm for variable importance is as follows.

1. A classifier is trained on the training set.

2. A validation or test set is used for determining the accuracy of the classifier.

28
3. For each data point and a chosen feature, a new value for that feature is randomly chosen
from among the values that the feature has in the remainder of the data set (known as ―sampling
with replacement‖). This helps in ensuring that the distribution of that feature remains the same as
in the original data set, however, the actual structure or meaning of that feature is removed (as its
value is chosen at random from the remainder of the data).
4. The classifier is trained on the altered set of the training data and then the accuracy of
classification measured on the changed test or validation data set. When randomizing of a feature
hurt accuracy a lot, then it is implied that the feature is vital. When randomizing of a feature does
not hurt accuracy much, then the feature is of little importance and is a suitable candidate for
removal.
5. The original test or validation data set is restored and the next feature is tried until we are
finished. The result obtained orders each feature by its importance. This procedure used above is
built into random trees and decision trees and boosting. Thus, we can use these algorithms to decide
which variables we will finally use as features; then we can use the optimized feature vectors to
train the classifier. [6]

3.6 Tools of Machine Learning

There are a few basic tools which are used in machine learning to know the results. In supervised
learning, one of the fundamental problems is knowing just how well the algorithm has performed:
i.e. How accurate is it at fitting or classifying the data?
For real problems, we must take into account noise, fluctuations and errors in sampling, and so on.
In other words, our test or validation data set may not reflect accurately the actual distribution of
data. To come closer to predicting the actual performance of the classifier, we use the technique
of cross-validation and/or the closely related technique of bootstrapping.
In its most rudimentary form, cross-validation involves the division of data into K different subsets.
We train on K – 1 of these subsets and then test on the final subset of data (i.e. the
―validation set‖) that we didn’t train. We do this K times, in which each of the K subsets gets its
turn at becoming the validation set. Then we average the results.

29
Bootstrapping is somewhat similar to cross-validation, but here the validation set is randomly
selected from the training data. The selected points for the round are then used only for testing, not
training. Then the process is started again from scratch. We do this N times. Each time we
randomly select a new set of validation data and then average the results in the end. This means
some and/or many of the data points are reused in various validation sets, hence the results are
usually superior compared to cross-validation. [6]
There are two other very useful ways of assessing, tuning and characterizing classifiers. One is to
plot the receiver operating characteristic (ROC) and the other is filling in a confusion matrix; see
the figure. The ROC curve is used to measure the response of the performance parameter of the
classifier through the full range of settings of that parameter. Here, the parameter is a threshold.
Just to make this more relevant to our project, suppose we are want to recognize blue eyes in an
image and we are using a threshold on the color blue as our detector. Setting the blue threshold
extremely high would result in the classifier failing to recognize any blue eyes, thereby yielding
a false positive rate of 0 but also having a true positive rate also at 0 (i.e. lower left part of the
curve in the figure). On the other side, if the blue threshold is set to 0 then all signals count as a
recognition. This means that all true positives (the blue eyes) are recognized including all the false
positives (brown and green eyes); thus we end up having a false positive rate of 100% (i.e. up for
every right part of the curve in the figure). The best possible ROC curve would be the one that
follows the y-axis up to 100% and then cuts horizontally over to the up for every right corner.
Failing that, the closer that the curve comes to the up for every left corner, the better. We can
compute the fraction of area under the ROC curve versus the total area of the ROC plot. It can be
a statistic of merit: The closer that the ratio is to 1, the better the classifier.
The given figure also shows a confusion matrix. It is just a chart of true and false positives along
with true and false negatives. It is a quick way to evaluate the performance of a classifier. In an
ideal case, we would see 100% along the NW-SE diagonal and 0% elsewhere. If we have a
classifier that is capable of learning more than one class, then the confusion matrix is generalized
for many classes and we just have to keep track of the class to which each labeled data point was
assigned. [6]

30
3.7 Learning Objects
A trained classifier cascade stored in an XMLfile can be used by the cvLoad() function to load
it and then cvHaarDetectObjects() to find objects similar to the ones it was trained on to train our
own classifiers to detect other objects such as faces and eyes.
We do this with the OpenCV haartraining application, which creates a classifier
from a training set of positive and negative samples. Steps are described below. [2][3]

3.7.1 Dataset gathering:-

Collect a data set consisting of examples of the object you want to learn These may be stored in
one or more directories indexed by a text file in the following format:
<path>/img_1 count_1 x11 y11 w11 h11 x12 y12 . . .
<path>/img_2 count_2 x21 y21 w21 h21 x22 y22 . . .
Each of these lines contains the path (if any) and file name of the image containing the
object(s). This is followed by the count of how many objects are in that image and then a list of
rectangles containing the objects. The format of the rectangles is the x- and y-coordinates of the
upper left corner followed by the width and height in pixels.
For optimum performance of the classifier you have to gather a lot of positive samples with very
less unnecessary variance in data. The output of this process finally creates a file in the format.idx
(e.g face.idx). [2][3]

3.7.2 Building vector output file:-

The utility application createsamples is used to build a vector output file of the positive samples.
Using this file, you can repeat the training procedure below on many runs, trying different
parameters while using the same vector output file. For example:
createsamples –vec faces.vec –info faces.idx –w 30 –h 40
This takes the faces.idx file and outputs a formatted training file, faces.vec. Then createsamples
extracts the positive samples from the images before normalizing and resizing them to the specified
width and height (here, 30-by-40). [2][3]

31
3.7.3 Object detection:-
The Viola-Jones cascade is a binary classifier: It simply decides whether or not the object in an
image is similar to the training set. Any image that doesn’t contain the object of interest can be
turned into a negative sample. It is useful to take the negative images from the same type of data.
That is, if we want to learn faces in online videos, for best results we should take our negative
samples from comparable frames .However, comparable results can still be obtained usingnegative
samples taken from any other source. Again we put the images into one or more directories and
then make an index file consisting of a list of image filenames, one per line. For example, an image
index file called background.idx. [2][3]

3.7.4 Training:-
Training call cn be initialized either by a batch file or using the following command lines in
command prompt:
Haartraining /
–data face_classifier /
–vec faces.vec –w 30 –h 40 /
–bg backgrounds.idx /
–nstages 20 /
–nsplits 1 /
[–nonsym] /
–minhitrate 0.998 /
–maxfalsealarm 0.5
Now the resulting classifier will be stored in face_classifier.xml. Here faces.vec is the set of
positive samples (sized to width-by-height = 30-by-40), and random images extracted from
backgrounds.idx will be used as negative samples. The cascade is having 20 (-nstages) stages,
where every stage is trained to have a detection rate (-minhitrate) of 0.998 or higher. The false hit
rate (-maxfalsealarm) is set at 50% (or lower) each stage to allow for the overall hit rate of 0.998.
The weak classifiers are specified in this case as ―stumps‖, which means they can have

32
only one split (-nsplits);. For more complex object detection as many as six splits can be used,
but mostly you no more than three splits are used.
Even on a fast machine, training may take a long time in the range of several hours to a day,
depending on the size of the data samples. The training procedure must test approximately 100,000
features within the training window over all positive and negative samples. [2][3]

3.7.5 Creation of .xml files:-

For our project face and eye classifiers are required. So we used the learning objects method to
create our own haarclassifier .xml files. The Haarcascade files are one for the face and one for
the eyes.
Around 1000 positive and 1500 negative samples were taken. It took a long time to train them.
Finally face.xml and Haarcascade_eye.xml files are created.
These xml files are directly used for object detection using haardetectobjects() function. It detects
a sequence of objects (in our case our face and eyes). Haarcascade-eye.xml is designed only for
open eyes. So when eyes are closed the system doesn’t detect anything. This is a blink. When a
blink lasts for more than 5 frames, the driver is judged to be drowsy and an alarm is sounded. [2][3]

33
 Chapter 4
Algorithm and Implementation

34
 Image acquisition 4.1 Algorithm
from webcam
Drowsiness of a person can be
measured by the extended period of
time for which his/her eyes are in
closed state. In our system, primary
attention is given to the faster
Face detected for detection and processing of data.
individual frame
The number of frames for which eyes
are closed is monitored. If the number
of frames exceeds a certain value,
then a warning message isgenerated
on the display showing that the driver
Is no. of
NO is feeling drowsy.
faces > 0
In our algorithm, first the image is
acquired by the webcam for
processing. Then we use the
YES Haarcascade file face.xml to search
and detect the faces in each
Set the image ROI to individual frame. If no face is
a fixed area inside
detected then another frame is
face
acquired. If a face is detected, then
a region of interest in marked within
the face. This region of interest
contains the eyes. Defining a region
Detect eyes from of interest significantly reduces the
ROI computational requirements of the
system. After that the eyes are
detected from the region of interest
by using Haarcascade_eye.xml.
Is no. of YES K=0
eyes > 0 If an eye is detected then there is no
blink and the blink counter K is set to
NO ‘0’. If the eyes are closed in a
particular frame, then the blink
K=K+1 counter is incremented and a blink is
detected. When the eyes are closed
for more than 4 frames then it is
deducible that the driver is feeling
If drowsy. Hence drowsiness is
You are feeling drowsy detected and an alarm sounded.
K>4
After that the whole process is
repeated as long as the driver is
29
NO driving the car.
Blink!
4.2 Image acquisition:-
The function cvCaptureFromCAM allocates and initialized the CvCapture structure for reading a
video stream from the camera.
CvCapture* cvCaptureFromCAM( int index );
Index of the camera to be used. If there is only one camera or it does not matter what camera to
use , -1 may be passed.

cvSetCaptureProperty Sets camera properties For example

cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_WIDTH, 280 );

cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_HEIGHT, 220 );

The function cvQueryFrame() grabs a frame from a camera or video file, decompresses it
and returns it. This function is just a combination of GrabFrame and RetrieveFrame, but
in one call. The returned image should not be released or modified by the user. In the event
of an error, the return value may be NULL. [4][5][7]

4.3 Face detection:-

Cascade is loaded by:

cascade = (CvHaarClassifierCascade*)cvLoad( cascade_name,
0, 0, 0 );
storage is allocated:
CvMemStorage* storage = cvCreateMemStorage(0);
CvSeq* cvHaarDetectObjects(
const CvArr* image,
CvHaarClassifierCascade* cascade,
CvMemStorage* storage,
double scale_factor = 1.1,
int min_neighbors = 3,
int flags = 0,
30
CvSize min_size = cvSize(40,40)
);

CvArr image is a grayscale image. If region of interest (ROI) is set, then the function will
respect that region. Thus, one way of speeding up face detection is to trim down the image
boundaries using ROI. The classifier cascade is just the Haar feature cascade that we loaded with
cvLoad() in the face detect code. The storage argument is an OpenCV ―work buffer‖ for the
algorithm; it is allocated with cvCreateMemStorage(0) in the face detection code and cleared for
reuse with cvClearMemStorage(storage). The cvHaarDetectObjects() function scans the input
image for faces at all scales. Setting the scale_factor parameter determines how big of a jump
there is between each scale; setting this to a higher value means faster computation time at the
cost of possible missed detections if the scaling misses faces of certain sizes. The min_neighbors
parameter is a control for preventing false detection. Actual face locations in an image tend to
get multiple ―hits‖ in the same area because the surrounding pixels and scales often indicate a
face. Setting this to the default (3) in the face detection code indicates that we will only decide a
face is present in a location if there are at least three overlapping detections. The flags parameter
has four valid settings, which (as usual) may be combined with the Boolean OR operator. The
first is CV_HAAR_DO_CANNY_PRUNING. Setting flags to this value causes fl at regions (no
lines) to be skipped by the classifier. The second possible flag is CV_HAAR_SCALE_IMAGE,
which tells the algorithm to scale the image rather than the detector (this can yield some
performance advantages in terms of how memory and cache are used). The next flag option,
CV_HAAR_FIND_BIGGEST_OBJECT, tells OpenCV to return only the largest object found
(hence the number of objects returned will be either one or none).* The final flag is
CV_HAAR_DO_ROUGH_SEARCH, which is used only with
CV_HAAR_FIND_BIGGEST_OBJECT. This flag is used to terminate the search at whatever
scale the first candidate is found (with enough neighbors to be considered a ―hit‖). The final
parameter, min_size, is the smallest region in which to search for a face. Setting this to a larger
value will reduce computation at the cost of missing small faces.

CvRect *face = (CvRect*)cvGetSeqElem(face, 0);

31
The above fnction gets a sequence of objects from the image given and stores them as a
rectangular region in the image.
cvRectangle(
img,

cvPoint(face->x, face->y),
cvPoint(
face->x + face->width,
face->y + face->height
),
CV_RGB(0, 255, 0),
1, 8, 0
)
The above function draws a rectangle in the image for the corresponding corner points .the other
parameter are for drawing colour and thickness and type of lines in the rectangle. [4][5][7]

4.4 Set Image Region of interest:-

cvSetImageROI(
img, /* the source image */
cvRect(
face->x, /* x = start from leftmost */
face->y + (face->height)/5, /* y = a few pixels from the top */
face->width, /* width = same width with the face */
(face->height)/3 /* height = 1/2 of face height */
)
It sets the ROI for the corresponding image. We have taken from the left most point in the face to
a few pixels from the top to half of face height. Width of the region is same as that of the face. The
rectangular region is now used to get eyes. [4][5][7]

32
4.5 Eye Detection:-
CvSeq *eyes = cvHaarDetectObjects(
img,
cascade,
storage,
1.1,
5,
0 /*CV_HAAR_DO_CANNY_PRUNNING*/,
cvSize( 10, 5 ) );
It is the same function as that of the face detection .Here eye cascade is used and minimum size of
the object is decreased and optimized to detect eyes of various sizes in the image.
As described earlier cvRectangle() is used to draw rectangle for sequences of eyes detected in a
given frame .
then cvResetImageROI(img) is used to reset the ROI so that the whole image can be used to be
displayed in a window using cvShowImage( "video", img ) function.
Since the cascade is constructed for only open eyes ,so when eyes are closed even for momentarily
they are not detected. The closing state of eyes are detected as blink and if the closing state
continues for more than 7 consecutive frames then drowsiness condition is displayed using
cvPutText() function . We can also use batch file command system(―abc.mp3‖)to play any audio
file at the instant drowsiness is detected .
The detection process is continued until a key is pressed .When the key is pressed the program
stops executing the detection function and stops capture form camera, releases memory and closes
video window using the functions given below. [4][5][7]
cvReleaseCapture( &capture );
cvDestroyWindow( "video" );
cvReleaseMemStorage( &storage );

33
CHAPTER-5
RESULT

34
5.1 Summary:-
To obtain the result a large number of videos were taken and their accuracy in determining eye
blinks and drowsiness was tested.

For this project we used a 5 megapixel USB webcam connected to the computer. The webcam had
inbuilt white LEDs attached to it for providing better illumination. In real time scenario, infrared
LEDs should be used instead of white LEDs so that the system is non-intrusive. An external
speaker is used to produce alert sound output in order to wake up the driver when drowsiness
exceeds a certain threshold.

The system was tested for different people in different ambient lighting conditions( daytime and
nighttime). When the webcam backlight was turned ON and the face is kept at an optimum
distance, then the system is able to detect blinks as well as drowsiness with more than 95%
accuracy. This is a good result and can be implemented in real-time systems as well.

Sample outputs for various conditions in various images is given below. Two videos were taken;
one in which only the eyes were detected and the other in which both face and eyes were detected.
Though the two processes have relatively equal accuracy, the computational requirements of the
former and lesser than that of the latter.

35
5.2 Sample Images:-
Sample 1
Only eyes detected face and eyes detected

36
Sample 2

Only eyes detected face and eyes detected

37
38
Sample 3
Only eyes detected face and eyes detected

39
Sample 4
Only eyes detected face and eyes detected

40
41
Sample 5
Only eyes detected face and eyes detected

42
Sample 6
Only eyes detected face and eyes detected

43
Sample 7
Only eyes detected face and eyes detected

44
Sample 8
Only eyes detected face and eyes detected

45
 SOURCE CODE

Drowsiness Detection
import numpy
from pygame import mixer
import time
import cv2
from tkinter import *
import tkinter.messagebox

# In[4]:

root=Tk()
root.geometry('500x570')
frame = Frame(root, relief=RIDGE, borderwidth=2)
frame.pack(fill=BOTH,expand=1)
root.title('Driver Cam')
frame.config(background='light blue')
label = Label(frame, text="Driver Cam",bg='light blue',font=('Times 35 bold'))
label.pack(side=TOP)
filename = PhotoImage(file="demo.png")
background_label = Label(frame)
background_label.pack(side=TOP)

def hel():
help(cv2)

def Contri():
tkinter.messagebox.showinfo("Contributor","Stuti\n")

def anotherWin():
tkinter.messagebox.showinfo("About",'Driver Cam version v1.0\n Made Using\n-OpenCV\n-
Numpy\n-Tkinter\n In Python 3')

menu = Menu(root)
root.config(menu=menu)

46
subm1 = Menu(menu)
menu.add_cascade(label="Tools",menu=subm1)
subm1.add_command(label="Open CV Docs",command=hel)

subm2 = Menu(menu)
menu.add_cascade(label="About",menu=subm2)
subm2.add_command(label="Driver Cam",command=anotherWin)
subm2.add_command(label="Contributor",command=Contri)

def exitt():
exit()

def web():
capture =cv2.VideoCapture(0)
while True:
ret,frame=capture.read()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF ==ord('q'):
break
capture.release()
cv2.destroyAllWindows()

def webrec():
capture =cv2.VideoCapture(0)
fourcc=cv2.VideoWriter_fourcc(*'XVID')
op=cv2.VideoWriter('Sample1.avi',fourcc,11.0,(640,480))
while True:
ret,frame=capture.read()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
cv2.imshow('frame',frame)
op.write(frame)
if cv2.waitKey(1) & 0xFF ==ord('q'):
break
op.release()
capture.release()
cv2.destroyAllWindows()

def webdet():
capture =cv2.VideoCapture(0)
face_cascade = cv2.CascadeClassifier('lbpcascade_frontalface.xml')
eye_glass = cv2.CascadeClassifier('haarcascade_eye_tree_eyeglasses.xml')

while True:
47
 ret, frame = capture.read()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray)

for (x,y,w,h) in faces:

font = cv2.FONT_HERSHEY_COMPLEX
cv2.putText(frame,'Face',(x+w,y+h),font,1,(250,250,250),2,cv2.LINE_AA)
cv2.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)
roi_gray = gray[y:y+h, x:x+w]
roi_color = frame[y:y+h, x:x+w]

eye_g = eye_glass.detectMultiScale(roi_gray)
for (ex,ey,ew,eh) in eye_g:
cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)

cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xff == ord('q'):
break
capture.release()
cv2.destroyAllWindows()

def webdetRec():
capture =cv2.VideoCapture(0)
face_cascade = cv2.CascadeClassifier('lbpcascade_frontalface.xml')
eye_glass = cv2.CascadeClassifier('haarcascade_eye_tree_eyeglasses.xml')
fourcc=cv2.VideoWriter_fourcc(*'XVID')
op=cv2.VideoWriter('Sample2.avi',fourcc,9.0,(640,480))

while True:
ret, frame = capture.read()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray)

for (x,y,w,h) in faces:

font = cv2.FONT_HERSHEY_COMPLEX
cv2.putText(frame,'Face',(x+w,y+h),font,1,(250,250,250),2,cv2.LINE_AA)
cv2.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)
roi_gray = gray[y:y+h, x:x+w]
roi_color = frame[y:y+h, x:x+w]

eye_g = eye_glass.detectMultiScale(roi_gray)
for (ex,ey,ew,eh) in eye_g:
48
 cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)
op.write(frame)
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xff == ord('q'):
break
op.release()
capture.release()
cv2.destroyAllWindows()

def alert():
mixer.init()
alert=mixer.Sound('beep-07.wav')
alert.play()
time.sleep(0.1)
alert.play()

def blink():
capture =cv2.VideoCapture(0)
face_cascade = cv2.CascadeClassifier('lbpcascade_frontalface.xml')
eye_cascade = cv2.CascadeClassifier('haarcascade_eye.xml')
blink_cascade = cv2.CascadeClassifier('CustomBlinkCascade.xml')

while True:
ret, frame = capture.read()
gray = cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray)

for (x,y,w,h) in faces:

font = cv2.FONT_HERSHEY_COMPLEX
cv2.putText(frame,'Face',(x+w,y+h),font,1,(250,250,250),2,cv2.LINE_AA)
cv2.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)
roi_gray = gray[y:y+h, x:x+w]
roi_color = frame[y:y+h, x:x+w]

eyes = eye_cascade.detectMultiScale(roi_gray)
for(ex,ey,ew,eh) in eyes:
cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)

blink = blink_cascade.detectMultiScale(roi_gray)
for(eyx,eyy,eyw,eyh) in blink:
cv2.rectangle(roi_color,(eyx,eyy),(eyx+eyw,eyy+eyh),(255,255,0),2)
alert()
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF ==ord('q'):
break

49
 capture.release()
cv2.destroyAllWindows()

but1=Button(frame,padx=5,pady=5,width=39,bg='white',fg='black',relief=GROOVE,command=
web,text='Open Cam',font=('helvetica 15 bold'))
but1.place(x=5,y=104)

but2=Button(frame,padx=5,pady=5,width=39,bg='white',fg='black',relief=GROOVE,command=
webrec,text='Open Cam & Record',font=('helvetica 15 bold'))
but2.place(x=5,y=176)

but3=Button(frame,padx=5,pady=5,width=39,bg='white',fg='black',relief=GROOVE,command=
webdet,text='Open Cam & Detect',font=('helvetica 15 bold'))
but3.place(x=5,y=250)

but4=Button(frame,padx=5,pady=5,width=39,bg='white',fg='black',relief=GROOVE,command=
webdetRec,text='Detect & Record',font=('helvetica 15 bold'))
but4.place(x=5,y=322)

but5=Button(frame,padx=5,pady=5,width=39,bg='white',fg='black',relief=GROOVE,command=
blink,text='Detect Eye Blink & Record With Sound',font=('helvetica 15 bold'))
but5.place(x=5,y=400)

but5=Button(frame,padx=5,pady=5,width=5,bg='white',fg='black',relief=GROOVE,text='EXIT',c
ommand=exitt,font=('helvetica 15 bold'))
but5.place(x=210,y=478)

root.mainloop()

50
 Face And Eye Detection

from scipy.spatial import distance

from imutils import face_utils
import numpy as np
import pygame #For playing sound
import time
import dlib
import cv2

#Initialize Pygame and load music

pygame.mixer.init()
pygame.mixer.music.load('audio/alert.wav')

#Minimum threshold of eye aspect ratio below which alarm is triggerd

EYE_ASPECT_RATIO_THRESHOLD = 0.3

#Minimum consecutive frames for which eye ratio is below threshold for alarm to be triggered
EYE_ASPECT_RATIO_CONSEC_FRAMES = 50

#COunts no. of consecutuve frames below threshold value

COUNTER = 0

#Load face cascade which will be used to draw a rectangle around detected faces.
face_cascade = cv2.CascadeClassifier("haarcascades/haarcascade_frontalface_default.xml")

#This function calculates and return eye aspect ratio

def eye_aspect_ratio(eye):
A = distance.euclidean(eye[1], eye[5])
B = distance.euclidean(eye[2], eye[4])
C = distance.euclidean(eye[0], eye[3])

ear = (A+B) / (2*C)

return ear

#Load face detector and predictor, uses dlib shape predictor file
detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor("models/shape_predictor_68_face_landmarks.dat")

#Extract indexes of facial landmarks for the left and right eye
(lStart, lEnd) = face_utils.FACIAL_LANDMARKS_IDXS['left_eye']
(rStart, rEnd) = face_utils.FACIAL_LANDMARKS_IDXS['right_eye']

#Start webcam video capture

video_capture = cv2.VideoCapture(0)
51
#Give some time for camera to initialize(not required)
time.sleep(2)

while(True):
#Read each frame and flip it, and convert to grayscale
ret, frame = video_capture.read()
frame = cv2.flip(frame,1)
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

#Detect facial points through detector function

faces = detector(gray, 0)

#Detect faces through haarcascade_frontalface_default.xml

face_rectangle = face_cascade.detectMultiScale(gray, 1.3, 5)

#Draw rectangle around each face detected

for (x,y,w,h) in face_rectangle:
cv2.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)

#Detect facial points

for face in faces:

shape = predictor(gray, face)

shape = face_utils.shape_to_np(shape)

#Get array of coordinates of leftEye and rightEye

leftEye = shape[lStart:lEnd]
rightEye = shape[rStart:rEnd]

#Calculate aspect ratio of both eyes

leftEyeAspectRatio = eye_aspect_ratio(leftEye)
rightEyeAspectRatio = eye_aspect_ratio(rightEye)

eyeAspectRatio = (leftEyeAspectRatio + rightEyeAspectRatio) / 2

#Use hull to remove convex contour discrepencies and draw eye shape around eyes
leftEyeHull = cv2.convexHull(leftEye)
rightEyeHull = cv2.convexHull(rightEye)
cv2.drawContours(frame, [leftEyeHull], -1, (0, 255, 0), 1)
cv2.drawContours(frame, [rightEyeHull], -1, (0, 255, 0), 1)

#Detect if eye aspect ratio is less than threshold

if(eyeAspectRatio < EYE_ASPECT_RATIO_THRESHOLD):
COUNTER += 1
#If no. of frames is greater than threshold frames,
if COUNTER >= EYE_ASPECT_RATIO_CONSEC_FRAMES:
52
 pygame.mixer.music.play(-1)
cv2.putText(frame, "You are Drowsy", (150,200), cv2.FONT_HERSHEY_SIMPLEX,
1.5, (0,0,255), 2)
else:
pygame.mixer.music.stop()
COUNTER = 0

#Show video feed

cv2.imshow('Video', frame)
if(cv2.waitKey(1) & 0xFF == ord('q')):
break

#Finally when video capture is over, release the video capture and destroyAllWindows
video_capture.release()
cv2.destroyAllWindows()

53
5.3 Table and Analysis:-
Various samples with various accuracies were taken and a table plotted for them.
Input Eye blink accuracy Eye blink accuracy Drowsiness Drowsiness
for only eyes for face and eyes accuracy for only accuracy for face
eyes and eyes
Sample 1 100% 100% 100% 100%
Sample 2 100% 95% 100% 100%
Sample 3 95% 95% 100% 100%
Sample 4 95% 100% 100% 100%
Sample 5 65% 45% 50% 33%
Sample 6 100% 100% 100% 100%
Sample 7 90% 95% 100% 100%
Sample 8 100% 100% 100% 100%
Total 93.125% 91.25% 93.75% 91.6%

Each volunteer was asked to blink 20 times and become drowsy 6 times during the testing
process. The accuracy for eye blink was calculated by the formula

Accuracy = 1 - |total no. of blinks -no. of blinks detected| / total no. of blinks.

The same formula was used for calculating accuracy of drowsiness detection.

It can be seen from the above table that if sample 5 is not taken into consideration then the
system has an accuracy of nearly 100%. That said; the high amount of errors in sample 5 shows
that the system is prone to error and has certain limitations which we will discuss in the next
section. In sample 5 we did not use the backlight of the webcam. The resulting poor lighting
conditions gave a highly erroneous output.

54
5.4 Limitations:-
The limitations of the system are as follows.

1. Dependence on ambient light:- With poor lighting conditions even though face is easily
detected, sometimes the system is unable to detect the eyes. So it gives an erroneous result
which must be taken care of. In real time scenario infrared backlights should be used to
avoid poor lighting conditions.
2. Optimum range required:- when the distance between face and webcam is not at
optimum range then certain problems are arising.

When face is too close to webcam(less than 30 cm) , then the system is unable to detect the
face from the image. So it only shows the video as output as algorithm is designed so as to
detect eyes from the face region.

This can be resolved by detecting eyes directly using haardetectobjects functions from the
complete image instead of the face region. So eyes can be monitored even if faces are not
detected.

When face is away from the webcam(more than 70cm) then the backlight is insufficient
to illuminate the face properly. So eyes are not detected with high accuracy which shows
error in detection of drowsiness.

This issue is not seriously taken into account as in real time scenario the distance between
drivers face and webcam doesn’t exceed 50cm. so the problem never arises.

Considering the above difficulties, the optimum distance range for drowsiness detection
is set to 40-70 cm

3. Hardware requirements:- Our system was run in a PC with a configuration of 1.6GHz

and 1GB RAM Pentium dual core processor.
Though the system runs fine on higher configurations, when a system has an inferior
configuration, the system may not be smooth and drowsiness detection will be slow.

55
 The problem was resolved by using dedicated hardware in real time applications, so there
are no issues of frame buffering or slower detection.
4. Delay in sounding alarm:- When drowsiness level exceeds a certain threshold, an alarm
is produced by a system speaker. It requires a media player to run the audio file. There is
a significant delay between when drowsiness is detected and when the media player starts
and generates the alarm. But in real time, drowsiness is a continuous phenomenon rather
than a one off occurrence. So the delay is not that problematic.
5. Orientation of face:- when the face is tilted to a certain extent it can be detected, but
beyond this our system fails to detect the face. So when the face is not detected, eyes are
also not detected.
This problem is resolved by using tracking functions which track any movement and
rotation of the objects in an image. A trained classifier for tilted face and tilted eyes can
also be used to avoid this kind of problem.
6. Poor detection with spectacles:- When the driver wears glasses the system fails to detect
eyes which is the most significant drawback of our system. This issue has not yet been
resolved and is a challenge for almost all eye detection systems designed so far.
7. Problem with multiple faces:- If more than one face is detected by the webcam, then our
system gives an erroneous result. This problem is not important as we want to detect the
drowsiness of a single driver.

5.5 Future works:-

In the real time driver fatigue detection system it is required to slow down a vehicle automatically
when fatigue level crosses a certain limit. Instead of threshold drowsiness level it is suggested to
design a continuous scale driver fatigue detection system. It monitors the level of drowsiness
continuously and when this level exceeds a certain value a signal is generated which controls the
hydraulic braking system of the vehicle.
Hardware components required-
Dedicated hardware for image acquisition processing and display

56
Interface support with the hydraulic braking system which includes relay, timer, stepper motor
and a linear actuator.
Function
When drowsiness level exceeds a certain limit then a signal is generated which is communicated
to the relay through the parallel port(parallel data transfer required for faster results).The relay
drives the on delay timer and this timer in turn runs the stepper motor for a definite time period
.The stepper motor is connected to a linear actuator.
The linear actuator converts rotational movement of stepper motor to linear motion. This linear
motion is used to drive a shaft which is directly connected to the hydraulic braking system of the
vehicle. When the shaft moves it applies the brake and the vehicle speed decreases.

Since it brings the vehicle speed down to a controllable limit , the chances of accident occurrence
is greatly reduced which is quite helpful for avoiding crashes caused by drowsiness related cases.

5.6 Conclusion:-

Thus we have successfully designed a prototype drowsiness detection system using OpenCV
software and Haar Classifiers. The system so developed was successfully tested, its limitations
identified and a future plan of action developed.

57
 References
[1] http://www.ee.ryerson.ca/~phiscock/thesis/drowsy-detector/drowsy-detector.pdf
[2] http://www.cnblogs.com/skyseraph/archive/2011/02/24/1963765.html
[3] http://www.scribd.com/doc/15491045/Learning-OpenCV-Computer-Vision-with-the-
OpenCV-Library
[4]http://opencv.willowgarage.com/documentation/reading_and_writing_images_and_vid
eo.html
[5] http://www.scribd.com/doc/46566105/opencv
[6] Learning OpenCV by Gary Bradski and Adrian Kaehler
[7] http://note.sonots.com/SciSoftware/haartraining.html

58