2019 Fifth International Conference on Image Information Processing (ICIIP)

AD3S: Advanced Driver Drowsiness Detection

System using Machine Learning
Sukrit Mehta, Parimal Mishra, Arpita Jadhav Bhatt*, Parul Agarwal*
Department of Computer Science and Engineering
Jaypee Institute of Information Technology, Noida, India
[email protected], [email protected], [email protected], [email protected]
*corresponding author

Abstract—Drivers drowsiness is one of the prime reasons for Literature studies have identified that there are several
road accidents around the globe. Persistent monotonous driving ways to detect drowsiness of drivers and can be categorized as
for an extended period of time without rest leads to drowsiness physiological [4], vehicular [5] and behavioral based methods
and fatal mishaps. Automatic detection of driver’s drowsiness [6],[7]. Physiological techniques such as heartbeat sensing,
can prevent a large number of road accidents and thus, can save usage of electrocardiogram and sensors are mostly intrusive.
valuable lives. In this work, an advanced system namely AD3S Vehicular-based techniques include increase in driver's speed
(Advanced Driver Drowsiness Detection System) using Android and steering movements. Behavioral methods incorporate
application has been developed. The system is capable of techniques such as eye blinking, eye closure, mouth opening
capturing real-time facial landmarks of the drivers. The facial
ratio etc. In this work, behavioral-based methods are used and
landmarks are further utilized to compute several parameters
are non-intrusive in nature. The highlight of AD3S is that no
namely Eye Aspect Ratio (EAR), Nose Length Ratio (NLR) and
Mouth Opening Ratio (MOR) based on adaptive threshold which
extra expenses would occur due to the non-usage of sensors or
are capable of detecting driver’s drowsiness. The highlighting external devices. Several parameters used in this approach are
feature of AD3S is that it is non-intrusive in nature and is cost Eye Aspect Ratio (EAR), Mouth Opening Ratio (MOR) and
effective. To test the efficacy of AD3S, machine learning and deep Nose Length Ratio (NLR) to detect the driver’s drowsiness.
learning techniques have been applied over a data set of 1200 The driver and the passenger would be alerted if the driver
application users. The empirical results demonstrate that the was drowsy.
proposed system is capable of detecting driver’s drowsiness with The rest of the paper is organized as follows. Section II
an accuracy of approximately 98% with Bagging classifier. describes the related work. Section III describes different
Keywords: Ensemble Learning, Machine Learning, Driver modules of AD3S system. Section IV details the performance
Drowsiness evaluation of AD3S. The paper is concluded in Section V.
I. INTRODUCTION
Road network is one of the most essential requirements for II. RELATED WORK
the economic growth of any country. India has spanned its
To detect if a person is fatigue or not, many approaches
road network in the past years. Although road infrastructure
have been put forward till now and this field is still a part of
has improved the connectivity between different cities,
ongoing research. This section describes the related work that
however, this has also increased vehicular traffic as well as
has been conducted by various researchers to determine
number of road accidents. According to 2018’s global status
driver’s drowsiness. Ashish Kumar et al. used the technique of
report on Road Safety of WHO (World Health Organization),
mouth opening to detect the drowsiness level of person [6].
a total of 1.35 million road accidents occurred in 2016 [1]. A
Rateb et al. recognized driver laziness by utilizing neural
critical analysis of a survey report revealed that the prime
systems. Taro et al. proposed an approach of measuring the
reason behind accidents is because of the driver’s fault that
estimation of wrinkle changes for the detection of drowsiness
accounts for 78% of road accidents in India [2]. Ministry of
of driver. Mkhuse et al. reviewed machine learning procedures
Transport Government of India identified that driver’s fatigue
that incorporated support vector machines, convolutional
is the prime reason behind road accidents.
Intense workload on the drivers tends to make them work neural systems and shrouded Markov models to determine
overtime. These long working hours make them fatigued driver’s drowsiness [8].Weiwei et al. presented a yawning
which in turn makes them feel drowsy while driving. The discovery framework that comprised of a face locator, a nose
major victims of this overburden are the truck drivers and bus finder, a nose tracker and a yawning locator. Profound
drivers who drive non-stop throughout day and night. This learning calculations were done for identifying driver’s face
imposes a major threat to his/her life and as well as to the life and nose area. A nose following calculation that joins Kalman
of the passengers. According to “Ministry of Road Transport channel with a committed open-source TLD (Track-Learning-
& Highways” every year 4,552 accidents are reported [3]. Detection) tracker was proposed by [9]. Sukrit et al. used EAR
Hence, it becomes of paramount importance that we address (Eye Aspect Ratio) and ECR (Eye Closure Ratio) to detect
this issue of fatigueness of drivers in these formidable drowsiness of the driver [10]. The following section details
situations. In this work, we propose an automatic advance AD3S system.
system capable of determining driver’s drowsiness.

978-1-7281-0899-5/19/$31.00
Authorized ©2019 of
licensed use limited to: Auckland University IEEE 108 on June 08,2020 at 00:52:55 UTC from IEEE Xplore. Restrictions apply.
Technology. Downloaded
 2019 Fifth International Conference on Image Information Processing (ICIIP)

III. AD3S: ADVANCED DRIVER DROWSINESS DETECTION serve as an input to machine learning models and ANN to
SYSTEM detect if the person is drowsy or not. Figure 1 describes a
snapshot of facial landmark points in accordance with the Dlib
The proposed approach of AD3S has been well
library followed by a description of parameters which have
demonstrated with the help of an algorithm and is
been captured to determine driver’s drowsiness.
implemented with the help of an Android application. The
application can be installed on an Android device. Once the
application gets installed on the device, it captures facial
landmarks at the backend by utilizing Dlib library. A server
has been implemented on Flask. When the application is
running, the images of the driver are continuously sent over
the server and are processed for capturing facial landmarks. A
data set of 1200 participants is collected and trained using
machine learning classifiers.
To determine the drowsiness of driver, EAR, NLR and
MOR values are used for training the models. The accuracy
results have been further utilized when a driver starts a new
ride. The important point that is taken into consideration by
AD3S is that the driver can be drowsy as soon as he/she starts
the ride. Therefore, to tackle this situation, AD3S captures
facial images of the driver as soon as he/she starts the ride. For
the first 10 seconds, the system checks whether the driver is fit
to drive or not, by sending the values to the pre-trained model.
The returned variable from the pre-trained-model determines
if the driver is fit to drive or not. If the driver is fit to drive,
then he/she can continue with the ride else system generates a
message that the driver is drowsy and suggests the driver to
take a break. The following sections describe various modules Fig. 1 Facial Landmark Points according to Dlib library
of AD3S system.
Eye Aspect Ratio (EAR): It is defined as the ratio of height
A. Data Procurement and width of the eye and has been computed using equation 1
The module captures facial landmarks of the real-time [10]. The details of parameters that have been referred in
user. The application provides different options for drivers and equation 1 have been depicted in Figure 2. From equation 1, it
passengers using the toggle option on the login page. In case can be inferred that when a person feels drowsy, the numerator
if the application user is driver, he/she performs registration, will decrease since the distance between the vertical points
followed by creating a new ride followed by setting up the will decrease. For instance, the difference of the parameters
origin and destination of the ride. In case if the application p2, p3 and p5, p6 will become zero when eyes are closed [11].
user is passenger, then they can add themselves in the ride
(| – | | – |)
created by the driver. Once the passengers have joined the EAR= (1)
ride, the driver can start the ride. ∗| |

Once driver and passengers are ready for the ride, AD3S
then captures photos of the driver continuously for entire ride
duration. Pictures are clicked each time the application gets a
response from the server. The system continuously captures
the pictures until the driver stops the ride. The images are
simultaneously sent on the server to be processed for feature
extraction. For testing the efficiency of the proposed approach,
facial landmark points of 1200 volunteers were gathered and
their EAR, NLR and MOR values were collected by AD3S.
Later, the accumulated results were additionally examined to
test the viability of the proposed framework utilizing Machine
learning classifiers and Artificial Neural Network (ANN). Fig 2. Landmarks of Eye in EAR

B. Feature Extraction
Nose Length Ratio (NLR): It is defined as the ratio of the
Dlib library support of Python has been employed to length of the nose to the unit value of the measurement and
obtain 68 facial landmark points from the image captured has been calculated using equation 2 [9]. The parameters using
during the entire duration while the ride is being carried out. in the equation have been detailed in Figure 3. It can be
The dlib library comprises of a pre-trained face detector and inferred from equation 2, that the length would increase when
uses Support vector machine (SVM ) for identifying objects the person sits upright but would decrease when the person
[8]. The Euclidean distance between the coordinate points is bends forward or backward, indicating drowsiness.
calculated. Three parameters namely are EAR, NLR, and
MOR have been taken into consideration [10]. These features

109 on June 08,2020 at 00:52:55 UTC from IEEE Xplore. Restrictions apply.
Authorized licensed use limited to: Auckland University of Technology. Downloaded
 2019 Fifth International Conference on Image Information Processing (ICIIP)

(| |) some pattern to permit correspondence between the units.

NLR = (2) These units are called neurons, basic processors which work in
parallel. In the work presented here, multilayer feed-forward
ANN is applied on driver drowsiness data. It consists of total 5
layers, out of which first is the input layer, second, third and
fourth are hidden layers, and last is the output layer. The
activation function for the hidden layers is ReLU [14]
(Rectified Linear Units), and for the output, layer is sigmoid
function [14] shown in Equation 4 and 5 respectively.
f(x) = (4)

x, if x 0
f(x) = max(0, x) (5)
0, if x 0
Fig 3. Landmarks of Nose in MOR
Where x is the value of the input parameter.
Mouth Opening Ratio (MOR): It is defined as the ratio of The optimizer selected for the model is Adam (Adaptive
the height of the extreme points of mouth to the width of the Moment Estimation). Adam is a stochastic optimization
mouth and is computed by referring figure 4. The height algorithm that is used instead of the classical stochastic
denotes the distance between vertical points of mouth when it gradient descent procedure for updating network weights
is open [12]. While the width is the horizontal distance iteratively based on training data [15].
between the mouth ends. MOR has been calculated using
equation 3, and the parameters used in the equation have been D. Proposed System
referred using figure 4. Figure 5 depicts the architecture of AD3. The prime focus of
work proposed here is to develop a non-intrusive method for
(| – | | |) driver’s drowsiness detection.
MOR = (3)
∗| |

Fig 4. Landmarks of Mouth in MOR

C. Machine Learning Classifiers Applied

After the data collection, various machine learning
classifiers were applied namely SVM [4], Naïve Bayes [8],
Random Forest [9], Bagging [10], Boosting [11], and Voting
[12] to test the accuracy of AD3S. The complete dataset was
divided into two parts: 70% for training data and 30% for
testing data. Naive Bayes Classifier is used to differentiate
objects by applying Bayes Algorithm. SVM (Support Vector
Machine) is a discriminative classifier that obtains a line to
differentiate between two classes. Random Forest Classifier
uses an ensemble technique and randomly generates several
decision trees and chooses the class label, for instance, having
the maximum vote. Bagging, boosting and voting use
ensemble techniques too.
Another machine learning classifier applied for detecting
driver’s drowsiness is an Artificial Neural Network (ANN)
[13]. ANN is a productive figuring framework which is
inspired by the analogy of biological neural systems. ANN Fig 5. Flowchart of AD3S.
obtains a large collection of units which are interconnected in

110 on June 08,2020 at 00:52:55 UTC from IEEE Xplore. Restrictions apply.
Authorized licensed use limited to: Auckland University of Technology. Downloaded
 2019 Fifth International Conference on Image Information Processing (ICIIP)

The highlighting feature of AD3S is that it does not require Precision = (9)
any external hardware for detecting driver’s drowsiness.
Previously, the work carried out in this field either required ∗ ∗
very high computational power or were mainly based on just F_measure = (10)
single parameter for determining the state of drowsiness of an
individual. AD3S overcomes these issues by applying several
machine learning models such as Naive Bayes, SVM, Random Where TPR (computed using Equation 6) is True Positive
Forest, Bagging and ANN for classifying the drivers into the Ratio, FPR (computed using Equation 7) is False Positive
categories of fit to drive or not fit to drive. Machine learning Ratio, TP is True Positive, FP is False Positive, TN is True
techniques were applied by collectively studying the impact of Negative, and FN is False Negative.
three parameters namely, EAR, MOR, and NLR. The Table I depicts the Accuracy, Precision and F_measure
parameters were calculated with the help of dlib support in (computed using equations 8, 9 and 10 respectively) of the
python which extracts 68 facial landmarks of an individual several optimizers applied on ANN. The batch size of 10 is
using Android application. These calculated values were then considered, and the number of epochs taken is 100. Out of all
fed into the trained machine learning models and ANN for optimizers, the maximum accuracy has been observed for
predicting the state of driver’s drowsiness. The proposed Adam optimizer, i.e. 86.77%.
system permits the driver to drive if the result is predicted as
fit to drive else, the driver cannot proceed with the current TABLE I. PERFORMANCE METRICS OF ANN OPTIMIZERS
state of drowsiness, and hence the ride has to stop.
Optimizer TPR FPR Accuracy F-measure Precision
IV. PERFORMANCE EVALUATION
In order to prove the performance efficacy of the proposed SGD 75.64 16.23 60.94 58.70 75.64
system, data collected is fed into various machine learning Adagrad 71.79 12.29 82.92 80.57 71.79
classifiers. The work uses an Android app which has been AdaDelta 65.38 14.67 85.21 76.11 65.38
built on Android studio to collect real-time data. Flask server RMSProp 65.57 14.97 85.37 77.41 65.57
is used in Android studio for obtaining the real-time images Adam 82.05 8.38 86.77 84.76 82.05
locally. The dataset collected has been classified by various
machine learning algorithms such as Naïve Bayes, SVM, The activation function for three hidden layers in all the
Random Forest, Bagging, Boosting and Voting. These above-mentioned optimizers is ReLU, and for the output layer
algorithms were applied using Weka (version: 3.8.3). is sigmoid. The maximum accuracy is observed for the above-
Additionally, another machine learning technique namely mentioned combination of the activation functions in all the
artificial neural networks (ANN) has also been applied. ANN optimizers.
is implemented in python using Keras, numpy, scikit-learn,
pandas libraries. TABLE II. SUMMARY OF PERFORMANCE METRICS WITH EAR

Performance evaluation of the above-defined classifiers is Classifier TPR FPR Accuracy F_measure Precision
made through an incremental approach. In this approach, Naive Bayes 77.3 27.9 77.33 77 77
results are computed for an individual feature, and after that
features are added to prove system’s robustness. Subsection SVM 74.4 29.6 74.42 74.4 74.3
discusses the experimental results and analysis Random 88.5 10.7 88.5 88.6 89.1
A. Experimental Results Forest
Bagging 89.5 9.6 89.5 89.6 90
To identify the best machine learning classifier for
detection of driver’s drowsiness on different parameters, Boosting 76.3 30 76.33 75.9 75.9
experiments are carried out. Performance of each classifier is Voting 77.4 27.8 77.41 77.1 77.1
contrasted on the basis of various evaluation metrics. Once
confusion matrix is formed, i.e. true positive, true negative,
false positive and false negative are identified from result then TABLE III. SUMMARY OF PERFORMANCE METRICS WITH EAR AND NLR
evaluation metrics are computed through the following
Equations: Classifier TPR FPR Accuracy F_measure Precision
Naive Bayes 84 22.5 84 83.5 84.3
SVM 77.3 27.9 77.33 77 77
TPR = (6) Random 9.87 2.5 97.75 97.8 97.8
Forest
Bagging 97.8 2.4 97.83 97.8 97.8
FPR = (7) Boosting 85.3 17.9 85.25 85.2 85.1
Voting 84 22.5 84 83.5 84.3

Accuracy = (8)

111 on June 08,2020 at 00:52:55 UTC from IEEE Xplore. Restrictions apply.
Authorized licensed use limited to: Auckland University of Technology. Downloaded
 2019 Fifth International Conference on Image Information Processing (ICIIP)

TABLE IV. SUMMARY OF PERFORMANCE METRICS WITH EAR, NLR AND

MOR

Classifier TPR FPR Accuracy F_measure Precision

Naive Bayes 88 88 88 87.7 88.3
SVM 81.3 24.0 81.25 80.9 81.1
Random 97.8 25 97.75 97.8 97.8
Forest
Bagging 97.8 2.4 97.83 97.8 97.8
Boosting 89.6 12.5 89.58 89.5 89.5
Voting 89.3 15 89.33 89.1 89.5
ANN 87.67 8.3 86.77 84.76 82.05

Table II, III and IV show values of performance metrics of

AD3S after applying different classifiers. It has been observed
that Bagging and random forest gives excellent performance Fig. 6 Comparison of Accuracy of different Classifiers
when all three parameters, i.e. EAR, NLR, and MOR are
considered. Bagging accuracy is slightly higher than that of
random forest. However, F_measure, precision, and FPR of
Bagging and random Forest are the same. Hence both
classifiers depict similar kind of performances.

B. Result Analysis And Discussion

Based on the literature review, EAR is the essential
parameter for detecting driver’s drowsiness and thus added as
first parameter. With the EAR parameter, the results of various
classifiers are depicted in Table II. It has been observed that
Boosting technique depict highest accuracy of 89.5% while
Naïve Bayes, SVM, Random Forest, Bagging, and Voting
shows 77.33, 74.42%, 88.5%, 76.33%, and 77.41% accuracy
respectively.

Fig. 7 Best case accuracy for each classifier

On addition of one more i.e. NLR parameter (shown in
Table III), the highest accuracy is depicted by Naïve Bayes
and Voting methods. However, overall average accuracy is
improved by 8.745 %. Table IV represents the performance V. CONCLUSION
efficacy of different classifiers on adding one more parameter
i.e. MOR. Best accuracy is illustrated by Bagging method. In this paper, an Advanced Driver Drowsiness Detection
However, overall average accuracy is improved by 3.452 %. System (AD3S) has been proposed. The system is capable of
determining driver’s drowsiness using facial features.
Literature studies have identified that the major reason behind
Analysis of results is further established through bar charts road accidents is because of driver’s fatigue. With the
shown in figure 6 and figure 7. Figure 6 depicts the accuracies economic growth and expansion of road networks, there is an
of different classifiers. upsurge in the requirements of drivers. Moreover, driving for a
prolonged period of time makes them susceptible to accidents.
To tackle the above situation and determine drowsiness of
It has been observed that boosting technique and random the drivers, an automated and advanced system AD3S has
forest gives approximately similar results while SVM is worse been proposed. The highlight of AD3S is that it determines
among all. Figure 7 depicts the best case accuracies for each drowsiness of drivers using non-intrusive methods. Hence, no
classifier including ANN. Thus best results are obtained external devices or sensors are required to be attached to the
through boosting and random forest classifiers. body of the driver. The system uses dlib library to capture
facial coordinates of drivers. Based on the facial coordinates
captured by the system, Eye Aspect Ratio (EAR), Nose
Length Ratio (NLR) and Mouth Opening Ratio (MOR) have
been calculated from captured facial coordinates to determine
drowsiness of drivers. To test the efficacy of AD3S, several
machine learning algorithms have been incorporated to detect
driver’s drowsiness.

112 on June 08,2020 at 00:52:55 UTC from IEEE Xplore. Restrictions apply.
Authorized licensed use limited to: Auckland University of Technology. Downloaded
 2019 Fifth International Conference on Image Information Processing (ICIIP)

The proposed system can be integrated with cab hiring

services such as Uber and Ola and reduce the casualty counts.
The system provides same accuracy for the drivers wearing
spectacles.

REFERENCES

[1] D. MOHAN, “Road Accidents in India,” IATSS Res., vol. 33, no. 1,
pp. 75–79, 2014.
[2] A. Kumar and R. Patra, “Driver drowsiness monitoring system
using visual behaviour and machine learning,” ISCAIE 2018 - 2018
IEEE Symp. Comput. Appl. Ind. Electron., pp. 339–344, 2018.
[3] R. Jabbar, K. Al-khalifa, M. Kharbeche, and W. Alhajyaseen,
“ScienceDirect ScienceDirect Real-time Driver Drowsiness
Detection for Android Application Using Deep Neural Networks
Techniques,” Procedia Comput. Sci., vol. 130, pp. 400–407, 2018.
[4] M. Ngxande and M. Burke, “Driver drowsiness detection using
Behavioral measures and machine learning techniques : A review of
state-of-art techniques,” pp. 156–161, 2017.
[5] W. Zhang, Y. L. Murphey, T. Wang, and Q. Xu, “Driver Yawning
Detection based on Deep Convolutional Neural Learning and
Robust Nose Tracking,” 2015 Int. Jt. Conf. Neural Networks, pp. 1–
8, 2015.
[6] S. Mehta, S. Dadhich, S. Gumber, and A. J. Bhatt, “Real-Time
Driver Drowsiness Detection System Using Eye Aspect Ratio and
Eye Closure Ratio,” pp. 1333–1339, 2019.
[7] S. Abtahi, B. Hariri, and S. Shirmohammadi, “Driver drowsiness
monitoring based on yawning detection,” Conf. Rec. - IEEE
Instrum. Meas. Technol. Conf., vol. 352326, pp. 1606–1610, 2011.
[8] Y. Huang and L. Li, “Naive Bayes classification algorithm based on
small sample set,” 2011 IEEE Int. Conf. Cloud Comput. Intell.
Syst., no. 2, pp. 34–39.
[9] H. Nakahara, A. Jinguji, S. Sato, and T. Sasao, “A Random Forest
Using a Multi-valued Decision Diagram on an FPGA,” Proc. Int.
Symp. Mult. Log., pp. 266–271, 2017.
[10] N. C. Oza, “Online Bagging and Boosting,” pp. 2340–2345, 2006.
[11] C. W. Wang, “New ensemble machine learning method for
classification and prediction on gene expression data,” Annu. Int.
Conf. IEEE Eng. Med. Biol. - Proc., pp. 3478–3481, 2006.
[12] S.Site and S. K. Mishra, “Model for measuring accuracies of majority
voting of ensemble classifier with COB and genetic algorithm,”
2013 Int. Conf. Inf. Commun. Embed. Syst. ICICES 2013, pp. 99–
103, 2013.
[13] Y. Hamamoto, S. Uchimura, and S. Tomita, “On the behavior of
artificial neural network classifiers in high-dimensional spaces,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 18, no. 5, pp. 571–574,
1996.
[14] C. Nwankpa, W. Ijomah, A. Gachagan, and S. Marshall, “Activation
Functions: Comparison of trends in Practice and Research for Deep
Learning,” pp. 1–20, 2018.
[15] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic
Optimization,” pp. 1–15, 2014.

113 on June 08,2020 at 00:52:55 UTC from IEEE Xplore. Restrictions apply.
Authorized licensed use limited to: Auckland University of Technology. Downloaded