Drowsiness Detection System Using Haar Classifiers

Shubham Kumar Singh, Smit Soni, Sonu Suman, Prof. Soumyalatha Naveen
School of Computing and IT
REVA UNIVERSITY
Bangalore 560064, India
[email protected], [email protected] , [email protected] , [email protected]

Abstract— The image processing algorithm is used to create a systems is the recent development of technology to detect or
non-intrusive device that can detect any human being's prevent drowsiness on the wheel.
exhaustion and that can provide a timely alert. Studies by the
expert show that about one fifth of all serious motorway incidents While the former method is more effective, it is not practical
are due to tired drivers in need of rest, which suggests that because highly sensitive electrodes will have to be connected
drowsiness causes more road accidents than drunk driving. Using directly to the driver’s body, which can be bothersome and
a camera, this device can track driver eyes and, by creating an distracting to the driver. It is very necessary and reasonable to
algorithm, we can predict driver fatigue symptoms early enough use image processing in the following step, as it provides one
to prevent the person from sleeping. Right now, consider certain of the best solutions for early detection of drowsiness and
parameters like the discovery of the face, position of the head and saves time to work to avoid accidents. In this approach the
the flickering of the eye. Picture Processing Algorithms are processing of images is used to process the images collected
utilized to guarantee legitimate identification of tiredness to keep from the person driving the vehicle. Furthermore, long work
away from mishaps. So, this project will help detect driver will result in convection on the sensors, reducing their
fatigue in advance and will offer alarm and pop-up warming accurate monitoring capability. The second method is to
performance. monitor physical changes (i.e. open / closed eyes for detecting
fatigue) because it is not invasive to detect changes in the real
Keywords—Drowsiness, Face detection, Flickering of eye.
world. Therefore, micro sleeps that are brief sleep cycles of 2
I. INTRODUCTION to 3 minutes are strong markers of exhaustion. Thus, one can
detect the driver's sleepy condition by constantly watching the
Drowsiness is a psychological disorder no one takes to heart. driver's eyes, and a timely alert is provided.
This characteristic, however, can have serious and fatal
consequences if it is not considered and behaves when driving A. DROWSINESS
on roads. Drowsiness is defined as the state of
unconsciousness. Real Time Drowsiness behaviors that are Drowsiness is a state of sleepiness. By implication, this is
sleepy are in the form of eye closing, head shaking or brain abnormal when it is
activity. A human being requires at least 6-hour sleep a day to  Persistent
function properly and carry out his / her day — activities.  Occur at inappropriate times, or is
When this factor is ignored and for whatever reason a person  Excessive in nature.
does not get enough sleep it contributes directly to a state of
drowsiness. Therefore, we can either assess changes in Somnolence is a concern more often because it interferes with
physiological signals such as brain waves, heart rate and eye regular daytime activities.
blinking to track drowsiness or we can find physical
adjustments such as loosening stance, driver's head leaning
and eye open / closed state. Driver somnolence detection is a II. LITERATURE SURVEY
device that guarantees the safety of the vehicle, which can in Within this section we addressed numerous methodologies
effect help avoid mishap such as accidents when the driver that researchers have proposed within recent years for the
feels asleep. There are several other factors that can cause car detection of drowsiness and blink detection.
accidents such as road conditions, atmospheric conditions, and
mechanical error / fault of the vehicle. Regardless of driver
violations involving drinking and driving, fatigue and Sukrit Mehta, Sharad Dadhich, Sahil Gumber, Arpita
drowsiness, 80 percent of the mishaps occur. Factors influence Jadhav Bhatt in 2019, has proposed a real time system that
the ability of the driver to regulate the vehicle such as vision, monitors and detects the loss of attention of drivers of vehicles
natural reflexes and understanding. Reducing these factors is proposed. The driver's face was identified by recording
will lead to accidents. Our paper attempts to evaluate the facial landmarks and the driver was warned to prevent crashes
driver's daily behaviors in order to determine the extent of in real time. Non-intrusive measures were chosen over
drowsiness. Various reports have indicated that tiredness invasive measures to prevent interference of the driver by the
accounts for around 20 percent of all road accidents. Driver sensors attached to his body. The proposed approach uses Eye
exhaustion is a significant factor in many automobile Aspect Ratio and Eye Closure Ratio with adaptive
collisions. A major problem in the field of accident prevention thresholding to detect driver’s drowsiness in real-time. This is
useful in situations when the drivers are used to strenuous
workload and drive continuously for long distances. The marked as "closed" otherwise the eye status will be "open."
proposed system works with the collected data sets under We also invoked frame timer to know whether the person is
different conditions. The facial landmarks identified by the feeling drowsy. It is achieved by taking person's mean blink
program are processed, and algorithms for recognition were period 100-400 milliseconds (i.e. 0.1-0.4 of a second) in
used for machine learning. The framework offers 84 per cent consideration.
best case accuracy for random forest classifier.
III. IMAGE PROCESSING
Numerous methods have been suggested to detect drowsiness Digital image processing in computing science involves using
in drivers. This section summarizes the methods available to digital computers to process digital images through an
diagnose drowsiness. Rateb et al. (R. Jabbar, K. Al-Khalifa, algorithm. Digital image processing has enormous advantages
M. Kharbeche, W. Alhajyaseen, M. Jafari, and S. over analog image processing as a subcategory or field of
Jiang,2018) detected real-time driver drowsiness using deep digital signal processing. This avow the application of a
neural networks. They developed an Android application. spacious mixture of algorithms for input data and can eliminate
Shailesh et al.( S. Sangle, B. Rathore, R. Rathod, A. Yadav, problems such as noise creation and distortion during process.
and A. Yadav,2018) used a camera fixed on the dashboard to Because images are depicted in two (perhaps more)
capture and send images to Raspberry Pi server installed in dimensions, digital image processing can be fashioned as
the vehicle, to detect faces using Harr classifier and facial multidimensional systems.
points using the Dlib Library. Vibin Varghese (V. Varghese, Working Principle:  Sleep deprivation driver monitoring
A. Shenoy, S. Ks, and K. P. Remya,2018) detected system built using non-invasive computer vision concepts. The
landmarks for every frame captured to compute the EAR program uses a real-time camera that straight points to the
(between height and width of eye) using the landmark points driver's face and monitors the driver's head speed to detect
of face. After computing the EAR; (V. Varghese, A. Shenoy, fatigue. In such a locale, an SOS beep is given to alert the
S. Ks, and K. P. Remya,2018) determined the driver as driver when lethargy is detected. The generated algorithm
drowsy if the EAR was less than the limit for 2 or 3 seconds resembles any officially published article that is the main goal
(because the eye blink lasts approximately 100-400ms). of the project. The system pinpoints the eye, nose and mouth in
Ashish Kumar (A. Kumar and R. Patra,2018) used Mouth a sever chunk of the image. If these are not visible, then for
Opening Ratio as a criterion for identifying yawning during five consecutive frames, the software user will fall asleep.
somnolence. Viola Jones Algorithm: The Viola-Zone Object Detection
In 2016, Manu B.N proposed a system that uses Haar feature- System was the first object detection system introduced by
based cascade classifiers to detect the face. Firstly, the Paul Viola and Michael Jones in 2001 to provide competitive
algorithm needs plenty of positive pictures (face pictures) and object detection levels in real time. It is mainly inspired by the
negative pictures (faceless pictures) to prepare the classifier to catechism of facial recognition. The program is built using the
identify the object. So along with the Haar feature-based library OpenCV.
classifiers, cascaded AdaBoost classifier is utilized to Feature types and evaluation: The Face Recognition-
recognize the face region and then the contracted image is Objective is only meant to identify the features (the
divided into numbers of rectangle regions, at any location and classification is the first step in the identification phase). The
scale within the main image. The hair-like appearance is algorithm consists of three steps:
effective for face detection in real time because of the  Haar feature selection
difference in facial appearance. This can be calculated based  AdaBoost Training
on the difference in the quantity of pixels quantities in the  Cascading Classifiers
rectangle region and during the procedure the AdaBoost The goal of the monitoring system is to equalize the number of
algorithm will allow all face samples and reject non-face image points in the rectangular areas. This is because they
picture samples. buck some kinship to each function routinely used in image-
object detection. Additionally, the concept used by Viola and
Amna Rahman in 2015 suggested a system for detecting Jones depend on more than a single rectangular area, they are
somnolence by using Eye blinking technique to detect eye usually more complicated. The diagram on the right defines
condition. Next, the image is transformed to a gray scale in four types of attributes used throughout the method. Whatever
this process and the corners are identified using the Harris the specific function is generated, the number of pixels cut
corner detection algorithm, which identifies the corner on both from the pixel rectangles to the dark rectangles in the plain
sides and at the bottom of the eye lid. After tracing the lines, it rectangles. Such rectangular features can be simplified when
will make a straight line between the top two lines, finding the opposed to alternatives such as steerable filters. Figure 1
midpoint by line measurement, and connecting the midpoint to illustrates about the Haar features and Figure 2 illustrates
the lower point. Now it will perform the same process for each implementing Haar features on face.
image and it measures the distance 'd' from the midpoint to the
lower point to evaluate eye condition. Finally, the eye-state
decision is made based on measured distance ' d.' Whether the
difference is zero or close to zero, the eye status will be
 below the black rectangle under the white rectangle.

Figure.1:Feature Selection

Figure.2: Implementing Features on Human Face Figure.3: Flow Chart for Drowsiness State Detection
A. HARR FEATURES Figure 3 illustrates the flow chart for drowsiness state
Some special features of the human face: the eye area is darker detection. The AdaBoost classifier with features like HAR is
than the upper cheeks. The area of the nasal bridge is lighter used to identify the face area. Subsequently, within the main
than the lips. Characteristics of facial features: Position and image, the compensation image breaks down into increasing
shape: Bridge of eyes, mouth, nose: gradient in intensity of numbers and the number of rectangles in the scale. Because of
dependent pixels. The face photograph looks for four features the different features of the face, a hair-like feature can be
that fit this pattern. used to detect the face in real time. Each poor classification is
fitted using the AdaBoost algorithm. Going through the
IV. DETECTION METHODS AdaBoost classification in the target pattern cascade you will
A. FACE DETECTION find the area on the nose. Almost all face samples can be
turned off, though non-face samples can be done.
In 2016, Manu B.N. proposed “Rapid Object Detection in
Boosted Cascades of Simple Features”. The proposed method B. EYE DETECTION
of object detection proposed in 2016 is used for face We used facial landmark assessment to identify the eyes in the
recognition. This is a technique of machine learning, which system. Facial milestones are used to identify and describe
prepares cascade function from both positive and negative facial features, for example:
pictures. It can be used in other images to identify objects. We  Eyes
focus on facial recognition here. To train the classification, the  Eyebrow
algorithm initially requires very positive images (facial images)
 Nose
and negative images (facial images). After this we remove the
 Mouth
facilities. The Haar properties shown in the diagram below can
be used for this purpose. We are like our kernels in  Jawline
punishment. This function is the same value obtained by
subtracting the number of pixels from the number of pixels Facial areas have been effectively applied for orientation, head
position estimation, facial lighting, blink detection and more.
Using this algorithm, we are going to identify some facial noted that a variety of landmark face predictors can be tested
features that we can use to find EAR. Identifying facial spots in the Helen dataset, including the 194-point model. Even
is an old-fashioned process: using a dataset, the same DLIB system can be used to train the
• Identify the face in the image. shape predictor over the input training data.
• Face Identify the main facial structures ROI facial
structures C. EYE ASPECT RATIO CALCULATION
Identify the face in the image: Her feature-based cascade The eye coordinates are identified for each video frame. The
classifiers detect the face image, which is discussed in the first eye-aspect ratio (EAR) is measured between eye height and
stage of our algorithm, i.e. face recognition. Find the main width.
facial features on the face of ROI: There are many face
monitors, however all approaches aim to classify the following EAR = ||p2 − p6|| + ||p3 − p5||
facial areas: ||p1 − p4||
• Right eyebrow
• The mouth where p1, . . ., p6 are the 2D landmark point. If an eye is open,
• Left eyebrow the EAR is mostly constant and gets down to zero when
• Right eye closing the eye. It is partially disrespectful person and head
• Left eye position. The open eye ratio is different than closed eye ratio.
Since both eyes blink at same time, we find the mean EAR of
• Nose
both the eyes. Figure 5 illustrates the identification of all the
This method starts using:
eye landmarks.
1. A training set of milestones labeled on the image.
Such images are manually labeled, defining the
regions (X, Y) surrounding each facial structure.
2. Priors, more precisely, are the probabilities of the
distance between pairs of input pixels. A pre-learned
face mark predictor is used in the dlib library to
calculate the state of the 68(x, y) -coordinates, which
plot the ground on the face.
In the image Figure 4 below you can see the indexes of the 68
coordinates:

Figure.5: Selecting Points For EAR

Figure.4: Co-ordinates of Facial Landmarks

• The following facial landmark index shows that we

can detect and control both regions of the eye
• The right eye using [37, 42]. Figure.6: Graph showing Threshold Time Vs EAR
• The left eye with [43, 48].

Such annotations are part of the 68-point iBUG 300-W dataset

on which the dlib phase mark predictor is learned. It should be
 V. DROWSINESS DETECTION SYSTEM State, how many times the eyes have blinked based on the
To design the proposed system to detect passivity following number of frames captured.
components are used. Software Requirements are Windows,
OpenCV Library, QT as editor. Hardware Requirements are Limitations:
Camera, Raspberry Pi Board, Buzzer.  Use of spectacles: If the consumer uses spectacle
then the state of the eye is hard to detect. Since it
The final step of the proposed design is to find the condition depends heavily on light, spectacle reflection may
of the person based on a pre-drowsiness condition. A person give the performance as an open eye for a closed eye.
blink for a duration of approx. 100-milliseconds. Therefore, if For this reason, the camera must be close to the eye
a person's eye closure is somnolent, it will go past this to avoid light.
interval. We set a 5-second time. When the eyes stay closed  Multiple face problem: When several faces appear in
for five seconds or longer, somnolence is observed and this is the frame, the camera can be able to identify more
caused by alerting pop. Figure 7 illustrates the detection of unwanted output faces. Since different faces are in
face and Figure 8 illustrates the drowsiness alert and EAR poor shape. So, we will only take driver’s face in the
value. frame of the camera. The detection speed may
decrease due to activity on multiple faces.

VII. RESULTS AND DISCUSSION

The average vertical gap estimation algorithm yielded the
strongest outcomes, with an overall accuracy of 79%. The
slopes estimation algorithm was the highest, with an average
accuracy of 44.5%.

Figure.7: Face Detection

Table.1: Different Algorithms and Their Accuracy

Table 1 shows the cumulative test runs. From all three

algorithms a dip in precision is observed while the person is
wearing glasses. A growing number of high-end vehicles now
have sophisticated screening devices for drowsiness. We have
found that several pairs of eyes are identified if two or more
faces are present in the picture. This blinds the system and the
results are unpredictable. With just one pair of eyes in the
picture, a potential challenge will be to generate a result.
Figure.8: Drowsiness Alert
VIII. CONCLUSION
VI. EXPECTED NEW APPLICATIONS
It proposed an algorithm for real-time eye blink detection. We
Our model is built in the form of an audio alarm to detect have quantitatively shown that Haar feature-based cascade
drowsy eye-state and set up alarm buzz. But if the driver classifiers and regression-based facial landmark detectors are
responds lately to the alarm buzz, accident is probably to sufficiently accurate to accurately estimate the positive images
occur. Therefore, we can build and install a motor driven of the face and a degree of eye openness. While they are
device in accelerator to slow down the vehicle to prevent this resilient to poor image quality (in large measure poor image
and synchronize it with the alarm buzz so that the vehicle can resolution) and in the wild.
slow down automatically after the warning signal is received.
We can also provide the user with an Android application
which will provide his / her drowsiness level information
during any ride. The user would know Normal State, Drowsy IX. REFERENCES
[1] Sukrit Mehta, Sharad Dadhich, Sahil Gumber, Arpita
Jadhav Bhatt, “Real-Time Driver Drowsiness Detection
System Using Eye Aspect Ratio and Eye Closure Ratio”,
International Conference on Sustainable Computing in
Science, Technology and Management, Volume No 2,
Issue No 9, , Page No: 1333-1339,2019.
[2] S. Sangle, B. Rathore, R. Rathod, A. Yadav, and A.
Yadav, “Real Time Drowsiness Detection System,” pp.
87–92, 2018.
[3] V. Varghese, A. Shenoy, S. Ks, and K. P. Remya, “Ear
Based Driver Drowsiness Detection System,”, pp. 93–96,
2018.
[4] A. Kumar and R. Patra, “Driver drowsiness monitoring
system using visual behaviour and machine learning,”
ISCAIE 2018 - 2018 IEEE Symposium on Computer
Applications and Industrial Electronics, pp. 339–344,
2018.
[5] R. Jabbar, K. Al-Khalifa, W. Alhajyaseen, M. Jafari, M.
Kharbeche and S. Jiang, "Real-time Driver Drowsiness
Detection Using Deep Neural Network Techniques for
Android Device," Procedia Computer Science, vol. 130,
pp. 400–407,2018.
[6] Manu B.N, “Facial Features Monitoring for Real Time
Drowsiness Detection”, 12th International Conference on
Innovations in Information Technology (IIT), Vol. No 2,
Issue No 4, , Page No. 78-81,2016.
[7] Amna Rahman, “Real Time Drowsiness Detection using
Eye Blink Monitoring”, National Software Engineering
Conference (NSEC 2015), Vol No 4, Issue No 2, , Page
No. 988-1001 ,2015.
[8] Rukhsar Khan, Shruti Menon, Shivraj Patil, Suraj Anchan,
Saritha L. R, “Human Drowsiness Detection System “An
IOT cum Android based project”, International Journal of
Engineering and Advanced Technology (IJEAT) ISSN:
2249 – 8958, Volume-8 Issue-4, April, 2019
[9] Seemar , P., Chandna, A., “Drowsy driver detection using
image processing “, International Journal of Engineering
science and Research Technolgy, pp. 234-241, 2017.