See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/355651503

Real-Time Driver Drowsiness Detection using Computer Vision

Article in International Journal of Engineering and Advanced Technology · October 2021

DOI: 10.35940/ijeat.A3159.1011121

CITATIONS READS

28 2,851

1 author:

Mahek Jain
R.V. College of Engineering
3 PUBLICATIONS 74 CITATIONS

SEE PROFILE

All content following this page was uploaded by Mahek Jain on 26 October 2021.

The user has requested enhancement of the downloaded file.

 International Journal of Engineering and Advanced Technology (IJEAT)
ISSN: 2249-8958 (Online), Volume-11 Issue-1, October 2021

Real-Time Driver Drowsiness Detection using

Computer Vision
Mahek Jain, Bhavya Bhagerathi, Sowmyarani C N

Abstract: The proposed system aims to lessen the number of This paper is focused on designing a drowsiness detection
accidents that occur due to drivers’ drowsiness and fatigue, which system that functions efficiently and is affordable. The
will in turn increase transportation safety. This is becoming a method that is needed in the present scenario detects
common reason for accidents in recent times. Several faces and drowsiness based on the eyes and mouth geometric features.
body gestures are considered such as signs of drowsiness and
fatigue in drivers, including tiredness in eyes and yawning. These
This paper proposes to achieve the same, by developing a
features are an indication that the driver’s condition is improper. drowsiness detection system to monitor and prevent a
EAR (Eye Aspect Ratio) computes the ratio of distances between destructive outcome from the negligence of fatigue. There are
the horizontal and vertical eye landmarks which is required for an increasing number of accidents on roads today and driver
detection of drowsiness. For the purpose of yawn detection, a drowsiness is an important factor and this has been accepted
YAWN value is calculated using the distance between the lower lip extensively. The exact number of accidents caused by
and the upper lip, and the distance will be compared against a
threshold value. We have deployed an eSpeak module (text to
driver’s drowsiness is difficult to decide because it is
speech synthesizer) which is used for giving appropriate voice underestimated most of the time. The transition from being
alerts when the driver is feeling drowsy or is yawning. The fatigued to dozing off is subtle and generally goes unnoticed
proposed system is designed to decrease the rate of accidents and by the driver. This explains why it is important to do more
to contribute to the technology with the goal to prevent fatalities work in this field with an aim to reduce the occurrence of
caused due to road accidents. accidents related to drowsiness and motivate ourselves to
Keywords: Drowsiness, eSpeak module, Eye aspect ratio, Yawn
develop a driver drowsiness detection system.
Detection.

I. INTRODUCTION II. RELATED WORK

The survey which is done includes the present
Driver drowsiness and fatigue are one of the most technologies and research available related to the topic of our
project. It is an attempt to better understand the efforts that
common reasons for accidents. The number of fatalities due
to such accidents is increasing worldwide each year. This have gone into this field of study, and also to understand
paper aims to lessen the number of accidents due to driver where our efforts should be focused on while developing this
drowsiness and fatigue. This will in turn increase project. This literature review has been carried out on the
transportation safety. Driver drowsiness detection is a topic of the current drowsiness detection technologies for
technology in vehicles that is useful in preventing accidents facial landmark detection [7], blink detection, and yawn
and saving the lives of drivers when they are getting drowsy. detection. across many techniques to carry out drowsiness
This project uses computer vision for the detection of drivers’ detection, which includes deep CNN [13], Computer Vision
drowsiness. With the constant improvement and novelty in [15], Behavioral measures, and machine learning techniques
technology, there is an advancement in transportation modes. all with different advantages, challenges, and different levels
Our dependencies on it have started increasing at a high rate. of accuracy. Research has been done on EAR and
It has greatly affected our lives in many ways. Considering MAR-based technologies for blink detection and yawn
any social status, there are some rules which should be detection respectively. From [1] titled “Computer Vision
followed by any vehicle driver. One is to stay alert and the based drowsiness detection for motorized vehicles with Web
other one is being active while driving. The existing Push Notifications”. In this paper, they have designed a
technologies to detect driver drowsiness are either very costly drowsiness system that is based on computer vision for
systems that apply to the high-end car models or systems that
vehicles with alarm sounds and web push notifications. These
are affordable but are not robust.
notifications will alert the driver to prevent the accident from
occurring. The system is also designed to generate an alert
that shows nearby coffee shops which will increase the
Manuscript received on September 22, 2021. alertness of the driver. As a result, the system detected the
Revised Manuscript received on September 27, 2021. driver’s drowsiness and showed successful performance in
Manuscript published on October 30, 2021. the trial run. The Eye Aspect Ratio was used to detect if the
*Correspondence Author
Mahek Jain*, Department of Computer Science and Engineering,
eyes are closed or open. Then, a buzzer was used to generate
Rashtreeya Vidyalaya College, Bangalore (Karnataka), India. an alert and a web push notification was sent to the user
Bhavya Bhagerathi, Department of Computer Science and Engineering, which shows the nearby coffee shops.
Rashtreeya Vidyalaya College, Bangalore (Karnataka), India.
Dr. Sowmyarani C N, Associate Professor, Department of Computer
Science and Engineering. Bengaluru (Karnataka), India.

© The Authors. Published by Blue Eyes Intelligence Engineering and

Sciences Publication (BEIESP). This is an open access article under the CC
BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)

Retrieval Number: 100.1/ijeat.A31591011121 Published By:

DOI: 10.35940/ijeat.A3159.1011121 Blue Eyes Intelligence Engineering
Journal Website: www.ijeat.org and Sciences Publication (BEIESP)
109 © Copyright: All rights reserved.
 Real-Time Driver Drowsiness Detection using Computer Vision

The limitations of the paper are that the Raspberry Pi real-time. The other objectives include designing a system
camera was used and because of this, the system could not that detects drowsiness of drivers by monitoring the eyes of
work during the nighttime. A night-vision camera should the driver regularly, especially the retina. The system should
have been used. From [2] titled “Real-time monitoring of give an alert to the driver when the driver yawns frequently or
driver drowsiness on mobile platforms using 3D neural when the driver’s eyes remain closed for a few seconds. The
networks”. In this paper, they have used depth-wise separable system works even when a driver is wearing spectacles. The
3D convolution operations to detect drowsiness in drivers system is not affected by bad lighting conditions.
from real-time face video and also, they have identified micro
sleeps and alerted the drivers. The results obtained show that IV. METHODOLOGY
the method can decide which features are important and it
Firstly, the face is localized in the image using facial
does not depend on the developer to pre-specify a set of
landmark detection. Then, shape prediction methods are used
features because they might miss some features like nose
to detect important features on the face. Face detection [14] is
wrinkles, eyelid movement, and other facial gestures. The
done by OpenCV built in HAAR cascades, which are
limitations of the paper are that the dataset which is used
pre-trained. In the next step, to estimate the location of 68 (x,
consists of just 18 persons and also, the frames were not
y)-coordinates that map to facial structures, a facial landmark
labeled properly.
detector which is pre-trained and included in the dlib library
From [3] titled “The detection of drowsiness using a driver
is used. It is also trained on the iBUG 300-W dataset. The
monitoring system”. In this paper, they have made use of a
EAR [8] is computed using the ratio of distances between the
driver monitoring system (DMS) to detect drowsiness along
horizontal and vertical eye landmarks for drowsiness
with different kinds of sensors. They have also collected data
detection. For the purpose of yawn detection [9], a YAWN
in the form of signals from other vehicle-based sensors. The
value will be calculated using the distance between the upper
results obtained show that the models were effective at
and the lower lip, and the distance will be compared against a
dividing the drowsiness into three levels - low, moderate, and
threshold value. eSpeak module (text to speech synthesizer)
severe drowsiness. But, while differentiating between
is used for giving appropriate voice alerts when the driver is
moderate and severe levels, the model was not efficient
feeling drowsy or is yawning.
enough. The limitations of the paper are that the model was
not effective while differentiating moderate drowsiness from
V. SOFTWARE REQUIREMENTS SPECIFICATION
severe drowsiness. Another limitation is that the size of the
sample used in this paper is small. The proposed system must be able to detect drowsiness
From [4] titled “Driver Drowsiness Detection System given a proper real-time driving environment. The
Using Computer Vision”. The objective of the paper is to performance will depend upon the quality of the camera as
detect driver drowsiness by analyzing human eye blinks well. The proposed system due to its well-designed and
using a recent facial landmark detection and to make use of easy-to-use interface can be used by both day-time and
E.A.R(eye aspect ratio) for easy, fast, and efficient blink night-time drivers. Users can follow up the interface step by
detection. The results showed that the system was successful step for their purpose. The proposed system must be available
in driver drowsiness detection by providing a reliably precise for use to the user as and when needed provided that the
enough estimation of the level of eye openness. This alert user’s system meets the specified requirements. The
system can be used in real-time due to a very negligible proposed system must be able to recover from failure in case
performance cost experienced in facial landmark detection. of the application crashing abruptly and become ready-to-use
The limitations of the paper are that a fixed blink duration is after recovery. The prototype of the drowsiness detection
assumed even though everyone’s blink duration lasts system will be implemented on the Raspberry
differently. EAR is estimated from two-dimensional data Pi microcontroller board, along with the necessary peripheral
which cannot account for out-of-plane head orientation and hardware, and Python 3 will be used to implement the
the model depends only on eyes for drowsiness detection. software functionality of drowsiness detection.
From [5] titled “Drowsiness Detection Based on Eye
Closure and Yawning Detection”. In this paper, driver’s eye VI. SYSTEM DESIGN
and mouth movements are tracked using Haar-cascade A. System Architecture
classifiers. This will help to detect the closing of eyes and
When the driver is driving, the driver’s face is captured by
frequency of yawning. The system also generates alarm
a camera and it is converted into a video stream. The
sounds if the driver is drowsy or already asleep. As a result, in
application then analyzes the video to detect drowsiness and
85% of the cases, the system detects faces and facial features
fatigue and also checks the level of drowsiness. In this stage,
which are required, accurately. The system is prompt in
the main parts which should be considered for analysis are:
detecting drowsiness once the feature detection of the face is
the driver’s face tracking, driver’s fatigue state, and
positive. The limitation of the paper is that it is observed that
recognition of key regions of the face based on eye closure
the system's accuracy decreases in bad lighting conditions.
and yawning [10]. Finally, if the drowsiness is detected, a
warning voice alert is given.
III. OBJECTIVES
The main objective is to develop a system that is accurate
to detect a driver’s drowsiness based on eyelid movement and
yawning and is reliable to give appropriate voice alerts [6] in

Retrieval Number: 100.1/ijeat.A31591011121 Published By:

DOI: 10.35940/ijeat.A3159.1011121 Blue Eyes Intelligence Engineering
Journal Website: www.ijeat.org and Sciences Publication (BEIESP)
110 © Copyright: All rights reserved.
 International Journal of Engineering and Advanced Technology (IJEAT)
ISSN: 2249-8958 (Online), Volume-11 Issue-1, October 2021

Fig. 1 depicts High-Level System Architecture consisting covering all the key facial landmarks including eyes, mouth,
of the input to the model and the preprocessing and nose, etc. [19] But with regards to this project our region of
evaluation in stages. Stage 1 involves pre-processing of video interest (ROI) includes only the eyes and lips region which
stream for human face tracking. Stage 2 involves extraction can be extracted using python slicing techniques.
of facial key regions such as eyes and mouth. Stage 3
Table- I: iBUG 300w Dataset split up
involves detection of drowsiness symptoms like eye closure,
blinking, and yawning. Total Images Number of Annotated faces
involved
293 1
53 2
53 [3,7]

B. Performance Analysis
To achieve the expected results, a large number of pictures
were taken and their accuracy in drowsiness (using EAR
calculation) and yawn detection [11] was analyzed.
Sample 1: Face improperly aligned

Fig. 1. High-Level System Architecture

B. Detailed Design
The system has been designed such that the face and hence
the eyes and mouth of the driver are always monitored and if
the predefined levels of alertness are observed to be defaulted
and compromised, then an appropriate alarm is set off, and
Fig. 3. Face improperly aligned
accordingly, action is taken to prevent any fatalities. Fig. 2
depicts the System Design of Driver Drowsiness and Yawn
Sample 2: While wearing a hat (wearable obstacle)
Detection System. It can be seen that the camera is used for
monitoring the driver’s face continuously and upon detection
of drowsiness or fatigue, the system in the dashboard
generates a voice alert type warning to the driver.

Fig. 4. Wearing obstacle (Hat)

Sample 3: Face properly aligned

Fig. 2. System Design

VII. EXPERIMENTAL RESULTS

A. Experimental Dataset
The experimental dataset used in our project is the Fig. 5. Face properly aligned and eyes closed
iBUG-300w Dataset, which consists of 300 indoor and as Performance analysis of the system under different
many outdoor images. The dataset covers a large variety of conditions is shown in the table below:
identities, face size, lighting conditions, pose, etc. The
images in the 300-W dataset cover more expressions than the
common neutral smile such as surprise or scream.
Annotations on the image were done using 68-point markup

Retrieval Number: 100.1/ijeat.A31591011121 Published By:

DOI: 10.35940/ijeat.A3159.1011121 Blue Eyes Intelligence Engineering
Journal Website: www.ijeat.org and Sciences Publication (BEIESP)
111 © Copyright: All rights reserved.
 Real-Time Driver Drowsiness Detection using Computer Vision

Table- II: Performance Analysis drowsiness. [18] The outer factors may be weather
I/P Drowsiness Yawn Detection
conditions, state of the vehicle, time of sleeping and,
Detection mechanical data. Driver drowsiness is among the major
Sample 1 Not detected Not detected threats to road safety, and in the case of commercial motor
(improper face (improper face vehicle operators, the problem is particularly severe. The
alignment) alignment) factors that contribute to this serious safety issue are
Sample 2 96% 98% twenty-four-hour services, unpredictable conditions of the
Sample 3 99% 99% environment, high annual mileage, and an increase of work
The following parameters are considered for performance schedules that are demanding. One important step of
analysis: preventive measures that are needed to solve this problem is
• Drowsiness Detection Accuracy = total no. of times alert by continuously observing the driver’s drowsiness state [17]
raised when eyes closed / (total no. of times alert raised and giving information about their state to the driver so that
when eyes closed + total no of times alert didn’t raise they can take necessary action. Presently, no adjustment can
when the eyes are closed) be done concerning the zoom or direction of the camera
• Yawn Detection Accuracy = total no. of times alarm during the system operation. [20] In the future, more work
raised when user yawns / (total no. of times alarm raised can be done to automate the zoom on the eyes after they are
when user yawns + total no of times alarm didn’t raise localized.
when the user yawns).
C. System Testing LIMITATIONS
The following table represents five test cases that were The accuracy of the model degrades if the eye frames are
conducted while doing this project for drowsiness and yawn not captured clearly due to any kind of obstacles such as
detection of the driver. goggles or spectacles having reflection). Camera operations
Table- III: System Testing such as auto adjustments with respect to zoom and rotation
are not considered in conducting experiments. Once the eyes
Test Cases Eyes Eye Yawn
Result are localized, zooming in automatically will help increase the
Detected closure detected
accuracy. The accuracy of detection of eyes and mouth
Case 1 No No No No result
reduces when the driver is not facing the camera.
Case 2 Yes No No No result
Case 3 Yes Yes No Voice alert
ACKNOWLEDGMENT
Case 4 Yes No Yes Voice alert
Case 5 Yes Yes Yes Voice alert We thank Anirudh Devpura for his assistance in checking the
According to the methodology of the system, when the case results of the experiment. We thank the professors of RV
eyes are closed for more than the set threshold number of College and Engineering for their advice and suggestions.
frames or when the driver is yawning [12], then the driver is
feeling tired. Henceforth, one of these distinguished cases REFERENCES
arises, and the corresponding result happens. The accuracy as 1. Rahul Atul Bhope, “Computer Vision based drowsiness detection for
measured in the performance analysis phase is almost found motorized vehicles with Web Push Notifications”, IEEE 4th
to be 100% in the case when the face is properly aligned and International Conference on Internet of Things, IEEE, Ghaziabad,
India, 2019.
no wearable obstacle is present. Accuracy drops down a little 2. Jasper S. Wijnands, Jason Thompson, Kerry A. Nice, Gideon D. P,
when an obstacle is present (e.g., Hat). Ambient lighting Aschwanden & Mark Stevenson, “Real-time monitoring of driver
conditions are very essential for getting proper results. In drowsiness on mobile platforms using 3D neural networks”, Neural
case, the user’s eye closure and yawn occur simultaneously, a Computing and Applications, 2019.
3. Chris Schwarz, John Gaspar, Thomas Miller & Reza Yousefian, “The
voice alert is raised but the system behaves in an erroneous detection of drowsiness using a driver monitoring system” , in Journal
and unsynchronized fashion. Hence, such a situation should of Traffic Injury Prevention (Taylor and Francis Online), 2019.
be avoided to prevent any inconsistent results. 4. Aditya Ranjan, Karan Vyas, Sujay Ghadge, Siddharth Patel, Suvarna
Sanjay Pawar, “Driver Drowsiness Detection System Using Computer
Vision.”, in International Research Journal of Engineering and
VIII. CONCLUSION AND FUTURE SCOPE Technology(IRJET), 2020.
5. B.Mohana, C.M.Sheela Rani, “Drowsiness Detection Based on Eye
The model is capable of detecting drowsiness by Closure and Yawning Detection”, in International Research Journal of
monitoring the eyes and mouth. Shape prediction methods Engineering and Technology(IRJET), 2019.
[16] are used to detect important features on the face. The 6. Driver Alert Control (DAC). (2016, Feb 10). Retrieved from
http://support.volvocars.com/uk/cars/Pages/owners-manual.
inputs given to these methods are facial landmarks which are aspx?mc=Y555&my=2015&sw=14w20&article=2e82f6fc0
obtained from facial landmark detection. This module deals d1139c2c0a801e800329d4e
with the EAR function which computes the ratio of distances 7. Z. Mardi, S. N. Ashtiani, and M. Mikaili, “EEG-based drowsiness
detection for safe driving using chaotic features and statistical tests,”
between the horizontal and vertical eye landmarks. An Journal of Medical Signals and Sensors, vol. 1, pp. 130–137, 2011.
eSpeak module (text to speech synthesizer) is also deployed 8. T. Danisman, I.M. Bilasco, C. Djeraba and N. Ihaddadene, “Drowsy
which is used for giving appropriate voice alerts when the driver detection system using eye blink patterns,” Universite Lille 1 &
Telecom Lille 1, Marconi, France, 2010.
driver is feeling drowsy or is yawning. The whole project is
designed to decrease the rate of accidents and to contribute to
the technology with the goal to prevent fatalities caused due
to road accidents. The future work of this paper can be
focused on the use of outer factors for measuring fatigue and

Retrieval Number: 100.1/ijeat.A31591011121 Published By:

DOI: 10.35940/ijeat.A3159.1011121 Blue Eyes Intelligence Engineering
Journal Website: www.ijeat.org and Sciences Publication (BEIESP)
112 © Copyright: All rights reserved.
 International Journal of Engineering and Advanced Technology (IJEAT)
ISSN: 2249-8958 (Online), Volume-11 Issue-1, October 2021

9. B. Hariri, S. Abtahi, S. Shirmohammadi, and L. Martel, “A yawning

measurement method to detect driver drowsiness,” Distributed and
Collaborative Virtual Environments Research Laboratory, University
of Ottawa, Ottawa, ON, Canada, 2011.
10. L. Li, Y. Chen and Z. Li, "Yawning detection for monitoring driver
fatigue based on two cameras", Proceedings of the 12th International
IEEE Conference Intelligent Transportation Systems., pp. 1-6, Oct.
2009.
11. S. Abtahi, B. Hariri and S. Shirmohammadi, "Driver drowsiness
monitoring based on yawning detection", Proceedings of the IEEE
International Control, Measurement and Instrumentation (CMI), IEEE,
pp. 1-4, May 2011.
12. X. Fan, B. Yin, and Y. Sun, "Yawning detection for monitoring driver
fatigue", Proceedings of the International Conference on Machine
Learning and Cybernet, vol. 2, pp. 664-668, Aug. 2007.
13. F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally and
K. Keutzer, "Squeezenet: Alexnet-level accuracy with 50x fewer
parameters and <0.5Mb model size", Proceedings of the International
Conference of Learning Representations, pp. 1-13, 2017.
14. K. Zhang, Z. Zhang, Z. Li and Y. Qiao, "Joint face detection and
alignment using multitask cascaded convolutional networks", IEEE
Signal Processing Letters, vol. 23, no. 10, pp. 1499-1503, Oct. 2016.
15. D. B. Lucas and T. Kanade, "An iterative image registration technique
with an application to stereo vision", Proceedings of the 7th
International Joint Conference on Artificial Intelligence, vol. 2, pp.
674-679, 1981.
16. D. S. Bolme, J. R. Beveridge, B. A. Draper and Y. M. Lui, "Visual
object tracking using adaptive correlation filters", Proceedings of the
IEEE Computer Society Conference on Computer Vision Pattern
Recognition., pp. 2544-2550, Jun. 2010.
17. F. J. Henriques, R. Caseiro, P. Martins and J. Batista, "Exploiting the
circulant structure of tracking-by-detection with kernels", Proceedings
of the European Conference Computer Vision, pp. 702-715, 2012.
18. Y. Li and J. Zhu, "A scale adaptive kernel correlation filter tracker
with feature integration", Proceedings of the European Conference
Computer Vision, pp. 254-265, 2015.
19. M. Danelljan, G. Häger, F. S. Khan and M. Felsberg, "Discriminative
scale space tracking", IEEE Transactions on Pattern Analysis and
Machine Intelligence, vol. 39, no. 8, pp. 1561-1575, Aug. 2017.
20. N. Wang and D.-Y. Yeung, "Learning a deep compact image
representation for visual tracking", Proceedings of the 26th
International Conference on Neural Information Processing Systems.,
vol. 1, pp. 809-817, 2013.

AUTHORS PROFILE

Mahek Jain, is currently a final year undergraduate

student in the Computer Science and Engineering
department at Rashtreeya Vidyalaya College of
Engineering, Bangalore, Karnataka, India. She is an
IT head and member of Ashwa Racing student
organization at RVCE, which builds racing cars and
competes at global level competitions. Her current
research interests include applying deep learning
algorithms in autonomous vehicle modules for
pattern recognition and computer vision problems.

Bhavya Bhagerathi, is currently a final year

undergraduate student in the Computer Science and
Engineering department at Rashtreeya Vidyalaya
College of Engineering, Bangalore, Karnataka,
India. Her research interests include computer
vision, image processing, image analysis and
intelligent systems. She has actively taken part in
research internships and intermediate-level projects.
Notably, her recent works have focused on projects
related to computer vision and image segmentation using machine learning.

Dr. Sowmyarani C N, is currently working in

Rashtreeya Vidyalaya College of Engineering as an
Associate Professor in the Department of Computer
Science and Engineering. She has secured her M.E.
degree from UVCE, Bengaluru with 2nd rank in
university and secured her Ph.D. degree from VTU,
Belagavi. She has published more than forty
international Journals Conference papers and book
chapters. She is an active member of the IEEE association, ISTE, and ACM.
Her areas of interest are Data Privacy, Computer Networks, Smart
technologies, and Education Technology.

Retrieval Number: 100.1/ijeat.A31591011121 Published By:

DOI: 10.35940/ijeat.A3159.1011121 Blue Eyes Intelligence Engineering
Journal Website: www.ijeat.org and Sciences Publication (BEIESP)
View publication stats
113 © Copyright: All rights reserved.