ISSN: 2366-1313

Driver Drowsiness Monitoring System Using Visual Behaviour

and Machine Learning
1
Mrs.B Vijitha, 2A Sai Deepak, 3A Sai Abhishek, 4Ch Teja Kiran

1
Assistant Professor, Dept.of CSE, Teegala Krishna Reddy Engineering College, Meerpet, Hyderabad,

vijitha.boppena@tkrec.ac.in

BTech student, Dept.of CSE, Teegala Krishna Reddy Engineering College, Meerpet, Hyderabad,
2

deepaksai1192001@gmail.com

BTech student, Dept.of CSE, Teegala Krishna Reddy Engineering College, Meerpet, Hyderabad,
3

abhishek.akkirala100@gmail.com

BTech student, Dept.of CSE, Teegala Krishna Reddy Engineering College, Meerpet, Hyderabad,
4

tejakiranch1143@gmail.com

Abstract: Drowsy driving is one of the major causes of road accidents and death. Hence,
detection of driver’s fatigue and its indication is an active research area. Most of the
conventional methods are either vehicle based, or behavioral based or physiological based.
Few methods are intrusive and distract the driver, some require expensive sensors and data
handling. Therefore, in this study, a low-cost, real-time driver’s drowsiness detection system
is developed with acceptable accuracy. In this system, a webcam records the video and the
driver’s face is detected in each frame employing image processing techniques. Facial
landmarks on the detected face are pointed and subsequently the eye aspect ratio and nose
length ratio are computed and depending on their values, drowsiness is detected based on
developed adaptive threshold. Machine learning algorithms have been implemented as well
in an offline manner.
Keywords: drowsiness detection, visual behaviour, eye aspect ratio, mouth opening ratio,
nose length ratio.
I. INTRODUCTION Driver drowsiness is an overcast nightmare
Drowsy driving is one of the major causes to passengers in every country. Every year,
of deaths occurring in road accidents. The a large number of injuries and
truck drivers who drive for continuous deaths occur due to fatigue related road
long hours (especially at night), bus accidents. Hence, detection of driver’s
drivers of long-distance route or overnight fatigue and its indication is an active area
buses are more susceptible to this problem.

Volume VIII Issue I FEBRUARY www.zkginternational.com 339

 ISSN: 2366-1313
of research due to its immense practical Electroencephalogram (EEG), heartbeat,
applicability. pulse rate etc. are monitored and from
The basic drowsiness detection system has these metrics, drowsiness or fatigue level
three blocks/modules; acquisition system, is detected. This is intrusive measurement
processing system and warning system. as the sensors are attached on the driver
Here, the video of the driver’s frontal face which will distract the driver. Depending
is captured in acquisition system and on the sensors used in the system, system
transferred to the processing block where it cost as well as size will increase. However,
is processed online to detect drowsiness. If inclusion of more parameters/features will
drowsiness is detected, a warning or alarm increase the accuracy of the system to a
is sent to the driver from the certain extent. These factors motivate us to
warning system. Generally, the methods to develop a low-cost, real time driver’s
detect drowsy drivers are classified in drowsiness detection system with
three types; vehicle based, behavioural acceptable accuracy. Hence, we have
based and physiological based. In vehicle- proposed a webcam-based system to detect
based method, a number of metrics like driver’s fatigue from the face image only
steering wheel movement, accelerator or using image processing and machine
brake pattern, vehicle speed, lateral learning techniques to make the system
acceleration, deviations from lane position low-cost as well as portable.
etc. are monitored continuously. Detection PROBLEM STATEMENT
of any abnormal change in these values is driver drowsiness monitoring system using
considered as driver drowsiness. This is a visual behavior and machine learning aims
nonintrusive measurement as the sensors to detect when a driver is feeling drowsy
are not attached on the driver. In or fatigued while driving and alert them to
behavioural based method, the visual take a break. This can help reduce the risk
behavior of the driver i.e., eye blinking, of accidents caused by drowsy driving.
eye closing, yawn, head bending etc. are The system would analyze the driver's
analyzed to detect drowsiness. This is also visual behavior, such as eye closure
nonintrusive measurement as simple duration and frequency, blink rate, and
camera is used to detect these features. In gaze direction, using a camera or other
physiological based method [8,9], the visual sensor. It would then use machine
physiological signals like learning algorithms to analyze this data
Electrocardiogram (ECG), and make predictions about the driver's
Electooculogram (EOG), level of drowsiness. If the system

Volume VIII Issue I FEBRUARY www.zkginternational.com 340

 ISSN: 2366-1313
determines that the driver is feeling and should be energy efficient to minimize
drowsy, it would alert them with a visual the impact on the vehicle's battery.
or auditory warning to take a break and II. LITERATURE SURVEY
rest. This system could be implemented in W. L. Ouet.al [3] An intelligent video-
a variety of vehicles, including cars, trucks, based drowsy driver detection system,
buscraft. Some additional details that could which is unaffected b various illuminations,
be included in the problem statement for a is developed in this study. Even if a driver
driver drowsiness monitoring system using wears glasses, the proposed system detects
visual behavior and machine learning are: the drowsy conditions effectively. By a
The system should be able to operate in a near-infrared-ray (NIR) camera, the
variety of lighting conditions, including proposed system is divided into two
low light and bright sunlight. The system cascaded computational procedures: the
should be able to handle changes in the driver eyes detection and the drowsy driver
driver's appearance and facial features, detection. The average open/closed eyes
such as facial hair or glasses, without detection rates without/with glasses are
negatively affecting its performance. 94% and 78%, respectively, and the
The system should be able to accurately accuracy of the drowsy status detection is
detect drowsiness even when the driver is up to 91%. By implementing on the
not looking directly at the camera or FPGAbased embedded platform, the
sensor. The system should be able to processing speed with the 640×480 format
distinguish between drowsiness and other video is up to
factors that may affect the driver's visual frames per second (fps) after software
behavior, such as distractions or the optimizations.
presence of passengers in the Vehicle. W. B. Hornget.alA [4] vision-based real-
The system should be able to alert the time driver fatigue detection system is
driver in a non-intrusive manner, such as proposed for driving safely. The driver's
through a gentle vibration or a subtle face is located, from colour images
auditory warning. The system should be captured in a car, by using the
able to adapt to the individual characteristic of skin colours. Then, edge
characteristics of each driver and become detection is used to locate the regions of
more accurate over time as it is used. The eyes. In addition to being used as the
system should be able to run on a variety dynamic templates for eye tracking in the
of hardware platforms, including next frame, the obtained eyes' images are
embedded systems and mobile devices, also used for fatigue detection in order to

Volume VIII Issue I FEBRUARY www.zkginternational.com 341

 ISSN: 2366-1313
generate some warning alarms for driving In the last stage the accuracy of 93% of the
safety. The system is tested on a Pentium outputs were evaluated. The intelligent
III 550 CPU with 128 MB RAM. The detection and usage of various criteria in
experiment results seem quite encouraging long-term time frame are of the advantages
and promising. The system can reach 20 of the present study, comparing to other
frames per second for eye tracking, and the researches. This is helpful in early
average correct rate for eye location and detection of sleepiness and prevents the
tracking can achieve 99.1% on four test irrecoverable losses by alarming [6].
videos. The correct rate for fatigue In existing system, the driver drowsiness
detection is l00%, but the average detection system involves controlling
precision rate is 88.9% on the test videos. accident due to unconsciousness through
Alshaqaqi,et.al [5]. Drowsiness detection Eye blink. Here one eye blink sensor is
has many implications including reducing fixed in vehicle were if driver loses
roads traffic accidents importance. Using consciousness, then it alerts the driver
image processing techniques is amongst through buzzer to prevent vehicle from
the new and reliable methods in sleepy accident. In future we can implement
face. The present pilot study was done to Drowsiness Detection System in aircraft in
investigate sleepiness and providing order to alert causes irritation in the eye,
images of drivers' face, employing virtual- May damage retina highly expensive and
reality driving simulator. In order to distract the driver
detecting level of sleepiness according to
the signal, information related to 25 III. PROPOSED WORK
drivers was recorded with imaging rate of A block diagram of the proposed driver
10 fps, Moreover, on average 3000 frames drowsiness monitoring system has been
were analyzed for each driver. The frames depicted in Fig1. At first, the video is
were investigated by transforming in grey recorded using a webcam. The camera will
scale space and based on the Cascade and be positioned in front of the driver to
Viola & Jones techniques and the images capture the front face image. From the
characteristics were extracted using Binary video, the frames are extracted to obtain 2-
and Histogram methods. The MPL neural D images using OpenCV. Face and facial
network was applied for analysing landmarks like positions of eye and nose
data.70% of information related to each are marked on the images detected using
driver were inserted to the network of Viola-Jones algorithm after detecting the
which 15% for test and 15% for validation. face and facial landmarks Eye Aspect

Volume VIII Issue I FEBRUARY www.zkginternational.com 342

 ISSN: 2366-1313
Ratio is used for determining if the eye is image processing techniques to enhance
opened or closed. the visibility of the driver's eyes. Feature
SYSTEM ARCHITECTURE extraction: Machine learning algorithms
will need to be trained on data that
includes features that are relevant for
detecting drowsiness. These features could
include the duration and frequency of eye
closures, the blink rate, and the gaze
direction. Machine learning model: The
machine learning model will be trained on
a labeled dataset that includes examples of
drowsy and non-drowsy drivers. The
model will learn to recognize patterns in
the data that are indicative of drowsiness.
Alert generation: If the machine learning
model predicts that the driver is drowsy,
the system will generate an alert to alert
the driver to take a break and rest. This
alert could be a visual warning, such as a
flashing light, or an auditory warning, such
Fig.1 System architecture as a beep or spoken message.
System evaluation: The performance of
ARCHITECTURE OF DISCRIPTION the system should be regularly evaluated to
Visual sensor: This could be a camera or ensure that it is accurately detecting
other type of sensor that captures images drowsiness and generating appropriate
or video of the driver's face and eyes. alerts. This could be done using data from
Data pre-processing: The data collected real-world driving situations or through
by the visual sensor will need to be pre- simulations.
processed to extract features that are IV. IMPLEMENTATION
relevant for detecting drowsiness. This MODUL DESCRIPTION
may include cropping the images to focus Modules
on the driver's face and eyes, converting  Face Detection
the images to grayscale, and applying  Eye Detection

Volume VIII Issue I FEBRUARY www.zkginternational.com 343

 ISSN: 2366-1313
 Closed eye state Detection: METHODOLOGY
 Alert System Data Acquisition
Modules Description The video is recorded using webcam (Sony
Face Detection CMU-BR300) and the frames are extracted
The first step is to extract the face region and processed in a laptop. After extracting
from the real time video stream for which the frames, image processing techniques
the Viola-Jones algorithm is used. The are applied on these 2D images. Presently,
algorithm presented by Paul Viola and synthetic driver data has been generated.
Mike Jones was the first of ts kind real- The volunteers are asked to look at the
time face detector. The algorithm has 4 webcam with intermittent eye blinking,
steps HAAR Feature Selection, Creating eye closing, yawning and head bending.
an Integral Image, Adaboost Training and The video is captured for 30 minutes
then Cascading Classifiers. duration using OPenCV.
Eye Detection Face Detection
After face detection the region of the face After extracting the frames, first the
is defined, now we can search for eyes in human faces are detected. Numerous
this defined region soon this region again online face detection algorithms are there.
the Viola-Jones Cascade classifier is In this Face and facial landmarks like
applied to detect positions of eye and nose are marked on
eyes. the images detected using Viola-Jones
Closed eye state Detection: algorithm after detecting the face and
The EAR or the Eye Aspect Ratio is used facial landmarks Eye Aspect Ratio is used
for determining if the eye is opened or for determining if the eye is opened or
closed. The EAR is a constant value which closed. In this method, after training, the
rapidly falls to 0 when the eye is closed. classifier is tested on the labeled data and
The EAR is calculated for 20 consecutive the false positive sample feature values are
frames and if the average EAR is less than used again for training purpose. For the
the threshold which is 0.25 the trigger is test image, the fixed size window is
generated. translated over the image and the classifier
Alert System computes the output for each window
The alert system displays the message on location. Finally, the maximum value
the console and running video stream of output is considered as the detected face
application, it can give sound alert also. and a bounding box is drawn around the
face. This non-maximum suppression step

Volume VIII Issue I FEBRUARY www.zkginternational.com 344

 ISSN: 2366-1313
removes the redundant and overlapping
bounding boxes.
Facial Landmark Detection
Marking After detecting the face, the next
task is to find the locations of different
facial features like the corners of the eyes
and mouth, the tip of the nose and so on.
Prior to that, the face images should be
normalized in order to reduce the effect of
distance from the camera, non-uniform
illumination and varying image resolution.
Therefore, the face image is resized to a
width of 500 pixels and converted to
grayscale image. After image Fig.2 The facial landmark points
normalization, ensemble of regression Feature Extraction
trees [11] is used to estimate the landmark After detecting the facial landmarks, the
positions on face from a sparse subset of features are computed as described below.
pixel intensities. In this method, the sum of Eye aspect ratio (EAR): From the eye
square error loss is optimized using corner points, the eye aspect ratio is
gradient boosting learning. Different priors calculated as the ratio of height and width
are used to find different structures. Using of the eye as given by where represents
this method, the boundary points of eyes, point marked as i in facial landmark and is
mouth and the central line of the nose are the distance between points marked as i
marked and the number of points for eye, and j. Therefore, when the eyes are fully
mouth and nose are given in Table I. The open, EAR is high value and as the eyes
facial landmarks are shown in Fig 2. The are closed, EAR value goes towards zero.
red points are the detected landmarks for Thus, monotonically decreasing EAR
further processing. values indicate gradually closing eyes and
Table.1 Facial landmark points it’s almost zero for completely closed eyes
(eye blink). Consequently, EAR values
indicate the drowsiness of the driver as eye
blinks occur due to drowsiness. Mouth
opening ratio (MOR): Mouth opening ratio

Volume VIII Issue I FEBRUARY www.zkginternational.com 345

 ISSN: 2366-1313
is a parameter to detect yawning during Classification
drowsiness. Similar to EAR, it is After computing all the three features, the
calculated as defined, it increases rapidly next task is to detect drowsiness in the
when mouth opens due to yawning and extracted
remains at that high value for a while due frames. In the beginning, adaptive
to yawn (indicating that the mouth is open) thresholding is considered for
and again decreases rapidly. Towards zero. classification. Later, machine learning
As yawn is one of the characteristics of algorithms are used to classify the data.
drowsiness, MOR gives a measure For computing the threshold values for
regarding driver drowsiness. Head each feature, it is assumed that initially the
Bending: Due to drowsiness, usually driver is in complete awake state. This is
driver’s head tilts (forward or backward) called setup phase. In the setup phase, the
with respect to vertical axis. So, from the EAR values for first three hundred (for 10s
head bending angle, driver drowsiness can at 30 fps) frames are recorded. Out of
be detected. As the projected length of these three hundred initial frames
nose on the camera focal plane is containing face, average of 150 maximum
proportional to this bending, it can be used values is considered as the hard threshold
as a measure of head bending. In normal for EAR. The higher values are considered
condition, our nose makes an acute angle so that no eye closing instances will be
with respect to focal plane of the camera. present. If the test value is less than this
This angle increases as the head moves threshold, then eye closing (i.e.,
vertically up and decreases on moving drowsiness) is detected. As the size of eye
down. Therefore, the ratio of nose length can vary from person to person, this initial
to an average nose length while awake is a setup for each person will reduce this
measure of head bending and if the value effect. Similarly, for calculating threshold
is greater or less than a particular range, it of MOR, since the mouth may not be open
indicates head bending as well as to its maximum in initial frames (setup
drowsiness. From the facial landmarks, the phase) so the threshold is taken
nose length is calculated and it is defined experimentally from the observations. If
as 28 25 nose length (p p ) NLR average the test value is greater than this threshold
nose length − = The average nose length is then yawn (i.e., drowsiness) is detected.
computed during the setup phase of the Head bending feature is used to find the
experiment as described in the next sub- angle made by head with respect to
section. vertical axis in terms of ratio of projected

Volume VIII Issue I FEBRUARY www.zkginternational.com 346

 ISSN: 2366-1313
nose lengths. Normally, NLR has values values out of 300 frames in the setup phase.
from 0.9 to 1.1 for normal upright position Then offset is determined heuristically and
of head and it increases or decreases when the threshold is obtained as offset
head bends down or up in the state of subtracted from the average value. Driver
drowsiness. The average nose length is safety is at risk when EAR is below this
computed as the average of the nose threshold. This EAR threshold value
lengths in the setup phase assuming that no increases slightly with each yawning and
head bending is there. After computing the head bending upto a certain limit. As each
threshold values, the system is used for yawning and head bending is distributed
testing. The system detects the drowsiness over multiple frames, so yawning and head
if in a test frame drowsiness is detected for bending of consecutive frames are
at least one feature. considered as single yawn and head
To make this Thresholding more realistic, bending and added once in the adaptive
the decision for each frame depends on the threshold. In a test frame, if EAR value is
last 75 frames. If at least 70 frames (out of less than this adaptive threshold value,
those 75) satisfy drowsiness conditions for then drowsiness is detected and an alarm is
at least one feature, then the system gives given to the driver. Sometimes it may
drowsiness detection indication and the happen that when the head is too low due
alarm. To make this thresholding adaptive, to bending, the system is unable to detect
another single threshold value is computed the face. In such situation, previous three
which initially depends on EAR Threshold frames are considered and if head bending
value. The average of EAR values is was detected in those three frames,
computed as the average of 150 maximum drowsiness alarm will be shown
.
V. RESULTS

Volume VIII Issue I FEBRUARY www.zkginternational.com 347

 ISSN: 2366-1313

Fig.3 Image uploaded

Fig.4 drowsy face recognition

VI. CONCLUSION
In this project, a low-cost, real-time driver machine learning. Here, visual behavior
drowsiness monitoring system has been features like eye aspect ratio, and nose
developed based on visual behavior and length ratio are computed from the

Volume VIII Issue I FEBRUARY www.zkginternational.com 348

 ISSN: 2366-1313
streaming video, captured by a webcam. [4] S. Singh, N. P. papanikolopoulos,
An adaptive threshold technique has been “Monitoring Driver Fatigue using Facial
developed to detect driver drowsiness in Analysis
real time. Machine learning algorithms Techniques”, IEEE Conference on
have been used for visual behaviour Intelligent Transportation System, pp 314-
features and classification. From the video, 318.
the frames are extracted to obtain 2-D [5] B. Alshaqaqi, A. S. Baquhaizel, M. E.
images using OpenCV. Face and facial A. Ouis, M. Bouumehed, A. Ouamri, M.
landmarks like positions of eye and nose Keche,
are marked on the images detected using “Driver Drowsiness Detection System”,
Viola-Jones algorithm after detecting the IEEE International Workshop on Systems,
face and facial landmarks Eye Aspect Signal
Ratio is used for determining if the eye is Processing and their Applications, 2013.
opened or closed this give better accuracy, [6] M. Karchani, A. Mazloumi, G. N.
work will be carried out to implement Saraji, A. Nahvi, K. S. Haghighi, B. M.
them in the developed system to do the Abadi, A. R.
classification (i.e., drowsiness detection. Foroshani, A. Niknezhad, “The Steps of
REFERENCES Proposed Drowsiness Detection System
[1] W. L. Ou, M. H. Shih, C. W. Chang, X. Design based on Image Processing in
H. Yu, C. P. Fan, "Intelligent Video-Based Simulator Driving”, International Research
Drowsy Journal of Applied and Basic Sciences, vol.
[2] Driver Detection System under Various 9(6), pp 878-887, 2015.
Illuminations and Embedded Software [7]R. Ahmad, and J. N. Borole, “Drowsy
Implementation", 2015 international Conf. Driver Identification Using Eye Blink
on Consumer Electronics - Taiwan, 2015. Detection,”
[3] W. B. Horng, C. Y. Chen, Y. Chang, C. IJISET - International Journal of Computer
H. Fan, “Driver Fatigue Detection based Science and Information Technologies, vol.
on Eye 6, no.
Tracking and Dynamic Template 1, pp. 270-274, Jan. 2015.
Matching”, IEEE International Conference [8] A. Abas, J. Mellor, and X. Chen,
on Networking, “Non-intrusive drowsiness detection by
Sensing and Control, Taipei, Taiwan, employing Support Vector Machine,”
March 21-23, 2004. 2014 20th International Conference on
Automation and Computing (ICAC),

Volume VIII Issue I FEBRUARY www.zkginternational.com 349

 ISSN: 2366-1313
Bedfordshire, UK, 2014, pp. 188- 193. [8]
A. Sengupta, A. Dasgupta, A. Chaudhuri,
A.
[9] George, A. Routray, R. Guha; "A
Multimodal System for Assessing
Alertness Levels Due to Cognitive
Loading", IEEE Trans. on Neural Systems
and Rehabilitation Engg., vol. 25 (7), pp
1037-1046, 2017.
[10] Prasadu Peddi (2019), "Data Pull out
and facts unearthing in biological
Databases", International Journal of
Techno-Engineering, Vol. 11, issue 1, pp:
25-32.

Volume VIII Issue I FEBRUARY www.zkginternational.com 350