2022 International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES) | 978-1-6654-7413-9/22/$31.

00 ©2022 IEEE | DOI: 10.1109/ICSES55317.2022.9914261

Driver Drowsiness Detection and Alert System

using YOLO
Yogesh R, Vaddi Ritheesh, Suneel Reddy and Rajesh George Rajan*

Computer Science and Engineering Department, KCG College of Technology, Chennai, Tamil Nadu, India

Abstract- Fatigue detection is a safety technology that done and checked. According to the given data,
prevents accidents that are caused by drivers who feel almost 50% percent of the accidents are caused due
fatigued while driving. Every year the rate of injuries to driver drowsiness and therefore driver fatigue
and accidents are increases due to fatigue around the detection is necessary.
world. In this paper, a module for Driver Fatigue
Detection is presentedto limit the incidence of
accidents caused by fatigued drivers. Here we propose Physiology and Behavioral DetectionMethods:
an YOLO algorithm to find the drivers face and eye Techniques for detecting physiology and behaviour
detection using Dlibs and alert when the driver is evaluate the state of the driver by implanting an
drowsy. Second, using the Dilib toolbox, and the invasive sensor in the vehicle and gathering
landmarks and coordinates of the facial regions, Face information on the driver's full mental as well as
Feature Triangle is a geometric area that we created other operations. These methods are known as
(FFT). We make a Face Feature Vector (FFV) that "physiology and behaviour detection." EEG signal
having all of the data about each FFT's area and detection [7, 8], ECG signal detection [9, 10], and
centroid. We utilise FFV as a metric to detect whether EMG signal detection are among these detection
or not a driver is fatigued. Last, we create a sliding
methods.
window to calculate the entropy of face data.

Keywords—Fatigue, YOLO, drowsy, coordinates, Machine Vision-Based DetectionMethods: The

entropy. amount of weariness experienced by the driver is
evaluated using this approach, which considers the
I.INTRODUCTION particular qualities of the vehicle's motion and the
driver's performance. The detection approach that is
Road traffic accidents claim the lives of thousands of based on machine vision has quickly become the one
people every year. The World Health Organization that is used the most often since it does not involve
(WHO) cites driver fatigue as one of the primary any intrusive procedures and provides more accurate
contributors to the prevalence of vehicular results[13,12] This approach uses vital technologies
accidents.NSF states that almost in a month, 32 such as Face identification, eye location, and
percent of drivers had experienced at least one tiredness measurement, all possible thanks to this
drowsy driving incident event. Driver Drowsiness technique. Yang et al. [1] used PERCLOS to
causes very harmful effects to the driver, Almost all identify the driver's degree of weariness and the
Countries many laws have been enacted around the mask to detect the driver's eye position by collecting
world to address the issue of drowsiness. For a facial picture. Both of these methods were
example "Drivers should not be permitted to drive for employed to acquire data. However, the manufacture
more than four hours, and a minimum of 20 minutes of the mask makes a major role in the performance
break is necessary for each ride," according to of the model. This approach is more successful when
Chinese Traffic Safety Laws.. In countries like used for persons who have noticeable characteristics.
Europe, their law states that "Drivers should not In[4],the authors segmented the face image seen in
continue drive for continuous for 4.5 hrs”. In the the sequence into blocks that were identical to one
United States, the law states that "Driver should another but did not overlap. After that, they
continuously drive more than eleven hours, and the successfully obtained multiscale features by using
driving time should not be less than 10 hours[2,14]. the Gabor wavelet transform. In order to choose the
As stated above driver drowsiness depends upon the most well-known, they made use of the AdaBoost
driving duration. For a driver to check whether he is algorithm. So this approach can classify various
drowsy or not a reliable data analysis is needed to be gestures even when the lighting circumstances

Authorized licensed use limited to: Government Engineering College Thrissur. Downloaded on January 19,2024 at 05:20:12 UTC from IEEE Xplore. Restrictions apply.
 change. Bergasa and Nuevo[10]used the "bright eye precision have been verified, indicating that it may be
effect" in order to pinpoint the location of the used to combine many features to improve our
subject's eyes using active near-IR active device. A capacity to identify tiredness.
finite-state machine was used so that it could be
ascertained whether or not the eye was closed. They In [9]the real time driving experience under various
also used a fuzzy system to determine the level of experimental conditions, developed assessments of
fatigue. Bergara's method, on the other hand, is the driver's eyes be used to detect weariness. Eye
substantially hardware-dependent, but the "bright tracking's most recent performance is based on in-
eye effect's" efficiency is wholly dependent on vehicle sleepiness prediction measurements, which
ambient light circumstances.Even when the targeted are reviewed the metrics are then rationally analysed
locations were hidden, they employed the CAM , and to classify them, a classification algorithm
Shift approach to identify them. Feature points of based on a larger dataset of ninety hours of road
eyes were then determined based on the proportional driving is used. Based on these data, eye blink
connection of the face organs. Finally, PERCLOS detection may help determine whether or not a driver
was used to find the amount of driver weariness. is sleepy, provided that it is performed accurately.
Poor lighting and difficulty seeing for those who
The fundamental goal is to create a system capable need to wear corrective lenses are two issues that the
of determining whether or not a driver is drowsy by proposed changes will not address.
continuously monitoring the driver's facial
expressions using the notion of facial landmark In 2011, M.J. Flores et al.[5] published their findings
detection. on a driver tiredness detection system for an
advanced vehicle that used infrared radiance. They
The following is a list of the objective of the project: recommended proposing an extension for an
1. To determine whether or not the driver's eyes are enhanced driver fatigue support system that provides
closed by determining the driver's eye aspect automated driver tiredness identification and driver
ratio (EAR) value. distraction to cut down on the number of deaths
2. Find out how big of a gap there is between the caused by these types of accidents. In order to
driver's teeth by using the Mouth Aspect Ratio monitor and analyse the driver's face and eyes to
(MAR) number. This is also known as compute tiredness and distraction indices, artificial
"yawning." intelligence algorithms are employed to interpret
3. Determine the angle of the driver's head by visual data. This allows the algorithms to track and
computing the Euler angles about the rotational analyse the driver's face and eyes. This system
motion of the head in order to determine the operates at night and is powered by a lighting system
head tilt. that emits light in the near-infrared spectrum.
4. Using canny edge detection and the Hough
transform, identify the lane lines present in the Driver Fatigue Recognition Using Computer Vision
input picture and then apply this notion to the Technology was an investigation that A. Cheng
production of video files. et.al.[6], demonstrated a technique for identifying
5. To sound an alarm if sleepiness is detected in the discrete weariness that was based on image
driver. processing and eye-tracking technology. An eye
detection algorithm has been implemented to address
II.RELATED WORKS the challenges that have arisen as a result of changes
in driving position. After that, the percentage of eye
Hong Su et al.[5] published a paper in 2008 titled " A closure movement, the maximum length of eye
Partial Least Squares Regression Model Based on the closure, the blink frequency, the average opening
Fusion Model." have presented a new method for level of the eyes, the eyes opening velocity, and the
detecting driver weariness using numerous eyelid eyes' closing velocity will be computed. After that,
movements, which is based on the PLSR information Fisher's linear differentiated functions are used in a
fusion technology, with which to deal with the issues stepwise method to combine these measures, which
of significant relationships between eye movement is done to eliminate correlations and get an
features and therefore simply forecasting the independent index. The findings obtained from the
fatigueless tendency. The model's robustness and trials carried out on a driving simulator with these six
participants demonstrate that techniques of tiredness

Authorized licensed use limited to: Government Engineering College Thrissur. Downloaded on January 19,2024 at 05:20:12 UTC from IEEE Xplore. Restrictions apply.
 identification based on video may be used and yield eating, and head position. The captured image is
an accuracy of 86 percent. transferred to the processor for processing. The eye
closure is determined by calculating the Eye Aspect
Ratio (EAR). When an EAR value drops below a
Start threshold value, it is determined that the driver is
sleepy, and a warning is sent to them. The yawn
detection is accomplished by calculating the Mouth
Aspect Ratio (MAR). When the MAR value exceeds
Camera the threshold, yawning is confirmed. Calculating the
Euler angles detects head tilt. Head tilt is detected
when the roll value falls below or exceeds certain
thresholds.

IV.METHODOLOGY
Divide frames
Figure 1 depicts our project's overall concept. The
algorithm of the project is made up of the four
modules listed below.Positioning your face. The
data is derived from real-time camera video
Dilib using CNN processing. The advanced YOLOv3 architecture,
which is based on deep learning technology, is used
to extract suspect points in face regions.Extraction of
Feature Vectors For recognition, FFT is a geometry
area that extracts the face area from every frame
Face and Eye Detection containing facial features.

We obtain face landmarks and develop FFV by

finding the centroid of the driver's FFT and the area
using the Dlib tool and the extracted coordinates
If the driver is from the suspected face region. Creating a Data Set
Inactive the driver state data set is organised chronologically
based on the FFV extracted from a specific time
period.Fatigue Evaluation. When analysing multiple
sequential FFVs, we must construct a sliding
window that checks each time. That corresponds to
Alert Voice Speaker On
the successive frames that project the FFV onto the
face projection dataset.Later, it loops with all FFVs
and provides face information, and compares to the
facial set of feature points Then we'll compare
End themto determine the driver's fatigue state, facial
motion information is combined with a threshold.
Fig.1. Architecture for Driver Drowsiness Accept the discovery of a driver is referred to as
detection location. drowsiness detection, and the
III.PROPOSED METHOD results are highly accurate and have an impact on the
algorithm's performance.
A web camera takes still images of the driver's face
at regular intervals throughout the monitoring As a result, the driving drowsiness detection
process. The pictures that were collected are then algorithm's finite task is to detect faces quickly and
processed. Recognizing the subject's face as it accurately.Viola and Jones [18] came up with using
appears in the video acquired by the camera is the Haar to identify faces in images. However, there is a
first stage in detecting sleepiness. The Facial possibility that image characteristics may be lost if
landmark detection concept then detects facial the face is posed incorrectly, there is not enough
expressions such as eye closure, yawning, speaking,

Authorized licensed use limited to: Government Engineering College Thrissur. Downloaded on January 19,2024 at 05:20:12 UTC from IEEE Xplore. Restrictions apply.
 light, there is noise interference, or the face is only or not. To detect the drowsiness of a person we have
partly visible. This will lower the wellness and developed a platform where the driver can be
reliability of the approach that was specified. Recent detected whether the person is drowsy or not. This
years have seen the development of deep learning also includes an alert system that alerts when the
algorithms that use a wide array of detection and person is in a fatigue state. We have used the YOLO
decomposition techniques. After that, it will be split v3 algorithm which detects the face and eye of a
into two distinct categories: the first will be used to driver and compares with the already present datasets
transfer the target detection model for the face image and alerts if the driver is drowsy. If the driver is not
identification and decomposition process, while the Drowsy this keeps on running until the driver feels
second will be used for cascading techniques such as Drowsy.
Cascade CNN and MTCNN. When compared to
traditional methods, the face detection based on We analysed the results of a comparison between the
CNN will extract the features autonomously rather capabilities of the better YOLOv3-tiny and those of
than through manual intervention. The YOLO benchmarked datasets known as WIDER FACE in
algorithm model is a deep learning-based fast face order to establish the validity of this statement.Based
and eye detection model. It's a unique end-to-end on this, a design comparison experiment is done as
algorithm that turns face recognition into a reversing shown in figures 4,5&6 respectively to see how
challenge. In particular, we can use the regression accurate the algorithm for detecting tired driving is
method and CNN to replace the sliding window in based on facial motion information's
traditional target detection.This feature's extraction entropy.Specifics of implementation compared to the
method is very little affected by external factors. The mouth change that occurs during the tired state,
YOLO algorithm prototype is a fast face and eye which can be readily recognized when circumstances
detection system that is based on DL and the are normal, the amplitude of the mouth change that
environment advantages of quickly extracting target occurs during the driving process is more diminutive.
features. However, since specifying the change in the state of
the eye that happens during blinking is challenging,
Table 1. Eye Closure Detection - Accuracy the state of the eye is stated as estimating the closure
No of of eye using computer vision. After binarizing the
Sample No of time
times Accuracy ROI, the resulting binary picture is finished using
Number blink detected
blinked expansion and erosion methods.The status of the eyes
1 30 29 96.66% then determined. The ROI may be represented by a
2 30 28 93.33% black area, and the total number of pixels in the ROI
3 30 29 96.66% and number of black pixels are counted. This
4 30 28 93.33% instance, the degree of eye closure is calculated by
5 30 28 93.33% multiplying the two numbers together, the eye
condition assessed by0.15 threshold. The eye is open
We will reduce the number of calculations by if the ratio is greater than 0.15. Tables 1 and 2 exhibit
converting the multiple objective regression into a the accuracy of yawn and eye recognition,
single target regression, as motivated by the YOLO respectively. The eye is closed if the value is equal to
algorithm. Then,improve YOLOv3's detection of or less than 0.15. While the training is being done,
suspected face regions. Because it is a reduced the batch size is set to 32, and the optimization
version of the YOLO architecture, it will perform strategy is Adam having aLR of 0.001). All of the
better in real-time than YOLOv3. Darknet 53, a training sets have been completed when an epoch is
feature detection network based on the YOLOv3 reached. The warm-up set is only done once, whereas
architecture, is reduced to seven convolutional the training set is done a hundred times. All the
layers, six Max Pooling layers, and one Up Sample experiments are using Python 3.6.
layer.
The accuracy of eye closure detection is calculated
V.RESULTS AND DISCUSSION by =

The aim and motivation of the project to find the

drowsiness of a person whether the person is fatigue

Authorized licensed use limited to: Government Engineering College Thrissur. Downloaded on January 19,2024 at 05:20:12 UTC from IEEE Xplore. Restrictions apply.
 It has been shown that the detection of eye closure
has an accuracy of 94.66 percent.

Eye Closure Accuracy

31
30
29
28
Fig 4.Active State
27
1 2 3 4 5

No of times blinked
No of times blink detected
Fig 2.Graphical Representation of eye blinking
accuracy

In figure 2 and 3 represents the eye blinking

accuracy and Yawn detection accuracy respectively.

Table 2. Yawn detection- Accuracy

No of the No time Fig 5. Sleepy State
Sample
times yawn Accuracy
Number
yawned detected
1 30 29 96.66%
2 30 29 96.66%
3 30 29 96.66%
4 30 28 93.33%
5 30 28 96.66%

The accuracy of yawn detection is found to be 95.99

%. Fig 6. Drowsy State
Yawn Detection Accuracy
VI.CONCLUSION
31
30 Thesystem which we have proposed is a real-time
project and thus system has a very goodoperation
29 speed. From the above given results, the projectcan
28 be used at different places and it gives steady
performance when compared. In real-world driver
27 drowsiness detection is mandatory to reduce the
1 2 3 4 5 speed of the vehicle when the driver reaches certain
drowsiness level.It is offered to develop a
No of times Yawned continuous driver drowsiness detection system
No of times Yawn detected
instead of a threshold. The system continuously
monitors the amount of drowsiness, and when it
Fig 3.Graphical Representation of Yawn Detection surpasses a predetermined expected value, a signal
Accuracy and an alert being issued. The blink of an eye and a

Authorized licensed use limited to: Government Engineering College Thrissur. Downloaded on January 19,2024 at 05:20:12 UTC from IEEE Xplore. Restrictions apply.
 yawn are recorded for multiple samples to determine [9] L. Jiang, Bin Yang et al, H. Wang, S. Gao, and S. Jiang,
“Research of the automotive driver fatigue driving
sleepiness. The results of the eye blink and yawn earlywarningsystem,”CommunicationsinComputerandInformati
tests are tabulated. The detection accuracy of eye onScience,Springer-Verlag,Berlin,Germany,pp.383–391,2011
[10] L. M. Bergasa and J. Nuevo, “Real-time system for monitoring
blinks is around 94.66 percent. The detection of driver vigilance,” in Proceedings of the IEEE International
yawns is around 95.99 percent accurate. Most of the Symposium on Industrial Electronics, 2005. ISIE 2005, pp.
time, our technology accurately identifies eye blinks 1303–1308, Dubrovnik, Croatia, June 2005.
[11] M. Zhou, Y. Yu, and X. Qu, “Development of an efficient
and yawns. The technology also performs well in driving strategy for connected and automated vehicles at
low-light situations. The system is quick, and once it signalized intersections: a reinforcement learning
begins recording frames, it continues to recognize approach,”IEEE Transactions on Intelligent Transportation
Systems, vol. 21, no. 1, pp. 433–443, 2019.
the face and conduct detection until it is turned off. [12] P. R and R. G. Rajan, "Autonomous Application in
Requirements Analysis of Information System Development for
Producing a Design Model," 2021 2nd International Conference
REFERENCES on Communication, Computing and Industry 4.0 (C2I4), 2021,
pp. 1-8, doi: 10.1109/C2I454156.2021.9689440.
[1] Ahmad et al., “Driving fatigue detection with fusion of EEG [13] R. G. Rajan and M. Judith Leo, "American sign language
and forehead EOG,” in Proceedings of the International Joint alphabets recognition using hand crafted and deep learning
Conference on Neural Networks, pp. 897–904, Vancouver, features", Proc. Int. Conf. Inventive Comput. Technol. (ICICT), pp.
Canada, July 2016.
430-434, Feb. 2020.
[2] Dhanya, V.G., Jacob, M.S., Dhanalakshmi, R. (2021). Twitter-
[14] R. G. Rajan and P. S. Rajendran, "Gesture recognition of RGB-
Based Disaster Management System Using Data Mining. In: D and RGB static images using ensemble-based CNN
Pandian, A., Fernando, X., Islam, S.M.S. (eds) Computer
architecture," 2021 5th International Conference on Intelligent
Networks, Big Data and IoT. Lecture Notes on Data
Computing and Control Systems (ICICCS), 2021, pp. 1579-1584,
Engineering and Communications Technologies, vol 66.
doi: 10.1109/ICICCS51141.2021.9432163.
Springer, Singapore. https://doi.org/10.1007/978-981-16-
[15] R. P. Balandong, R. F. Ahmad, M. N. Mohamad Saad, and A. S.
0965-7_16
Malik, “A review on EEG-based automatic sleepiness detection
[3] F. Chen, M. Song, and X. Ma, “Investigation on the injury
systems for driver,” Ieee Access, vol. 6, pp. 22908–22919,
severity of drivers in rear-end collisions between cars using a
2018.
random parameters bivariate ordered probit model,”
[16] S. Yang, P. Luo, C. C. Loy, and X. Tang, “Wider face: a face
International Journal of Environmental Research and Public
detection benchmark,” in Proceedings of the IEEE Conference
Health, vol. 16, no. 14, p. 2632, 2019.
on Computer Vision and Pattern Recognition (CVPR),pp. 5525–
[4] G. Niu and C. Wang, “Driver fatigue features extraction,”
5533, IEEE Computer Society, Las Vegas, NV, USA,June
Mathematical Problems in Engineering, vol. 2014, Article
2016.
ID860517, 10 pages, 2014.
[17] Viola and Jones, X. Zhu, and R. Zhang, “Energy recovery
[5] G. Sun, Y. Jin, Z. Li, , M.J. Flores et al ,F. Zhang, and L. Jia, “A
strategynumerical simulation for dual axle drive pure electric
vision-based head status judging algorithm for driving fatigue
vehiclebased on motor loss model and big data calculation,”
detection system, ”Advances in Transportation Studies, Complexity,vol.2018,ArticleID4071743,14pages,2018.
vol.2015,pp.51–64, 2015.
[18] X. Qu, M. Zhou, Y. Yu, C. T. Lin, and X. Wang,
[6] Hong.su et al, F. You, X. Chu, X. Li, and X. Sun, “Research
“Jointlydampening traffic oscillations and improving energy
oncustomer marketing acceptance for future automatic driving-a consumptionwith electric, connected, and automated vehicles:
case study in China city,” IEEE Access, vol. 7, pp. 20938–
areinforcement learning-based approach,” Applied Energy,vol.
20949,2019.
257, Article ID 114030, 2019.
[7] J.Herna´ndez-Orallo, “ROCcurvesfor regression, ”Pattern
Recognition,vol.46,no. 12,pp.3395–3411,2013.
[8] J. Yan, H. Kuo, Y. Lin, and T Liao, "Real-time driver
drowsiness detection system based on PERCLOS and grayscale
image processing," in Proceedings of the 2016 International
Symposium on Computer, Consumer and Control (IS3C), pp.
243-246, Xi'an, China, July 2016.

Authorized licensed use limited to: Government Engineering College Thrissur. Downloaded on January 19,2024 at 05:20:12 UTC from IEEE Xplore. Restrictions apply.