Driver monitoring algorithm for Advanced

Driver Assistance Systems
Aleksandra Simić, Ognjen Kocić, Milan Z. Bjelica and Milena Milošević

 (to locate the eyes region) and weighted matrix sum (to
Abstract —Fast expansion of Advanced Driver Assistance detect the iris).
Systems (ADAS) market and applications has resulted in a
high demand for various accompanying algorithms. In this II. RELATED WORK
paper we present an implementation of Driver monitoring Detection of driver’s fatigue level represents one of the
algorithm. Main goal of the algorithm is to automatically most researched topics when it comes to ADAS
asses if driver is tired and in that case, raise a proper alert. It applications [5]. In general, there are few different
is widely used as a standard component of rest approaches. First one combines readings from multiple
recommendation systems. Our approach is based on
combination of computer vision algorithms for face detection
sensors that are attached to driver. Conclusions are drawn
and eyes detection. Additionally, we have tested our using a fusion of collected data. For example, researchers
implementation in controlled environment on a real ADAS have used heart rate measurements for this purpose [7].
platform board. Some may found usage of sensors too intrusive for real
Keywords — Advanced Driver Assistance Systems, driver world use case. As often used alternative, driver is
monitoring, eyelid detection, face detection recorded with some camera equipment and computer
vision algorithms are applied in order to detect where the
I. INTRODUCTION driver is looking and if he/she pays attention to the road.

M AIN goal of Advance Driver Assistance System

(ADAS) is to provide safer environment for driver
and to contribute to traffic safety in general. ADAS is
Solution described in this paper can be classified into this
group.
Additionally, there are methods that consider driving
constantly providing new solutions for non-trivial real patterns as a main indicator of driver tiredness [8].
world problems like automated freeway driving, which is Finally, it might be important to note one major
addressed by lane detection algorithm [1], pedestrian difference between these approaches. When detecting long
detection [2], road signs recognition [3], automated term drowsiness it is more suitable to use sensor or pattern
parking [4], driver fatigue detection [5] and many more. methods. However, if we are trying to prevent accident that
Driver monitoring has a special place among these might occur as a result of imminent distraction, computer
algorithms, as it is addressing the driver’s fatigue problem, vision based methods are proven to be more beneficial.
which is responsible for serious number of road accidents.
This paper provides detailed description of an III. ALGORITHM IMPLEMENTATION
algorithm design and implementation for the The implemented algorithm can be divided into few
aforementioned driver monitoring. Driver’s face and eyes phases. These phases and their connections are shown in
position are used for detecting if driver is paying attention. Fig. 1.
Implemented algorithm is based on a Viola-Jones object
detection framework (2001) [6]. The algorithm is suitable
because it can be implemented in a real-time fashion and
was specially motivated by frontal face detection. It will
not found a face if the driver is turned to the side.
Moreover, after the face was found, eye center detection
algorithm is implemented using common face proportions

Aleksandra Simić is with the Faculty of Mathematics, University of

Belgrade, Studentski Trg 16, 11000 Belgrade, Serbia (phone: 381-65-
3835662; e-mail: [email protected]).
Ognjen Kocić is with the Faculty of Mathematics, University of
Belgrade, Studentski Trg 16, 11000 Belgrade, Serbia (phone: 381-65-
2235055; e-mail: [email protected]).
Dr. Milan Bjelica is with the Department of Computer Engineering
and Communications, Faculty of Technical Sciences, 21000 Novi Sad,
Serbia (email: [email protected])
Milena Miloševic is with RT-RK Institute for Computer Based
Systems, Novi Sad, Serbia (email: [email protected])
Fig. 1. High level algorithm diagram

978-1-5090-4086-5/16/$31.00 ©2016 IEEE

Authorized licensed use limited to: PES University Bengaluru. Downloaded on October 12,2023 at 11:18:17 UTC from IEEE Xplore. Restrictions apply.
 Phases “Grayscale & Crop” and “Resize” can be result of the first phase, complete set of features that will
grouped into one phase – named “Input image be used for face detection is determined. First phase can be
preprocessing”. Those steps are shown separated in Fig.1. done only one time, whereas we can run face detection
because it is important to see that resized image is input using that data many times. In general, this is a good
only for face detection (reasons will be given later). approach since there are no performance penalties in actual
face detection that are related to preparation phase of the
A. Input Image preprocessing algorithm. After features are selected, they shall be applied
Firstly, BGRA frame that is produced by camera is to the image.
converted to grayscale. The reason for doing this is In general, on a single image one can expect human
because a grayscale image is a requirement for Viola Jones faces in many sizes. For example, one face may fit in a
algorithm for face detection (described in phase B) and it 50x50 pixels bounding rectangle, while other might be
is also well suitable for iris detection. better placed in 30x30 pixels bounding rectangle. If the
Besides that, we took advantage of the fact that we standard detection window of 24x24 pixels is used, it is not
need to detect only one face, which by assumption difficult to see that some faces might not fit into it.
occupies large area in center of input frame. Since face Because of that, feature detection is run on an image
should be centered, we cropped 35% of frame’s width, pyramid with intuition that at some level in pyramid both
because we do not want to preprocess and later analyze faces will fit into 24x24 rectangle (of course not at the
parts of image where there is no chance for face or eyes to same level). Image pyramid consists of sequence of images
be found. As the result of these simple preprocessing steps, down sized by some factor. Pseudo-code is given as
image that is suitable for eyes and iris detection was follows:
successfully formed.
It is already mentioned that we search for a very large
face. Having that in mind, we concluded that input image
can be downscaled in purpose of getting face detection
algorithm that works faster and achieves the same
accuracy. This cannot be applied for eyes and iris detection
because iris occupies only small portion of image and
working with small image would damage detection quality.
B. Face detection
VJ framework can be divided in two phases. First
phase is a preparation phase and it consists from Haar
feature selection and AdaBoost training, while second is a
face detection phase. Haar features can be thought of as
convenient rectangle shaped filters (shown on Fig. below).

Fig. 3. Face detection pseudo-code

Fig. 2. Haar features
Output of this algorithm is an array of rectangles that
It was observed by Viola and Jones that the grayscale represent founded faces. It is possible that one face is
image of human face has some special characteristics. For founded in more than one phase (more than one image in
example, eye region of image is darker than the rest. image pyramid). Because of that, we need to group all
Furthermore, vertical nose region is distinctively brighter. rectangles that have at least 65% overlapping, which was
Filters shown in Fig. 2. are applied to a face image and determined empirically.
sums of pixels in brighter regions of rectangles are Finally, even after this step, more than one face can be
subtracted from the dark ones. Given value is then found (for example someone can watch the road over the
compared to some threshold which is used to determine if driver’s shoulder). In that case, the rectangle that is closest
current region is the region we are interested in. Some to the image center is chosen.
observations to be made are that there are 2 types of two- Output of face detection phase is one rectangle that
rectangle features, 2 types of three-rectangle features and a represents driver’s face, if the face was found or invalid
single type of four-rectangle features (considering rectangle otherwise.
symmetry in our deduction process). Smallest rectangle for
first feature left to right from Fig. 2. is 1x2 pixels and C. Eyes and iris detection
because of that width must be an even number, which also First step of eyes center detection algorithm is to crop
stands for height of the second feature. Similarly, for third input image to already detected face rectangle. Standard
feature left to right, width must be divisible by 3. When biometric proportions are then applied to further reduce
listed limitations for described features are included and processing to rectangles that contain only left and right
24x24 pixels base filter (it is standard detection window eye. These proportions are shown in Fig. 4. Blue rectangle
size) is used, there is about 160K features which is too represents face whereas green rectangles represent eye
much for a real time application to examine. AdaBoost regions.
training is used to select the best features that will provide
the most information about scanned part of image. As a

Authorized licensed use limited to: PES University Bengaluru. Downloaded on October 12,2023 at 11:18:17 UTC from IEEE Xplore. Restrictions apply.
 distinction to guide us, we constructed another weight
matrix with values vertically progressing from center of
matrix to the top and bottom. It should be clear that this
matrix is applied on top of eye center pixel which lies in
middle (Fig. 8). Additionally, matrix width is reduced by
50% (25% on each side) in order to eliminate border noise
(Fig. 8). As well as previously presented concentric matrix,
this matrix was also normalized.

Fig. 4. Biometric proportions

Fig. 7. (a) Eye close (b) Eye open
These regions are input to iris detection phase. We
used a fact that iris is darker then sclera around it and we Finally, a criterion if eye is open was formed by comparing
developed a custom algorithm that uses weighted matrix quotient of average pixel value in region of interest and
sums. Matrix used can be thought of as some kind of weighted matrix sum against the threshold. This threshold
convolution kernel. It is populated with floating point was established after series of experiments.
values that are concentrically reduced by factor k from
inside out. An example is shown in Fig. 5.

1 1 1 1 1 1 1
1 1.25 1.25 1.25 1.25 1.25 1
1 1.25 1.56 1.56 1.56 1.25 1
1 1.25 1.56 1.95 1.56 1.25 1
1 1.25 1.56 1.56 1.56 1.25 1
1 1.25 1.25 1.25 1.25 1.25 1 Fig. 8. Second weighted matrix position
1 1 1 1 1 1 1
Fig. 5 Weight matrix, k=1.25, dimension = 7 D. Drowsiness score calculation
Final phase of described algorithm is calculation of
Matrix dimension must be odd number and it depends on
drowsiness score. Drowsiness score is value in range
eye region size. Coefficient k, used to calculate matrix
between 0 and 10. This range is divided into three
values, is empirically determined value. Before applying
intervals:
this matrix on surroundings of each pixel, we have done
one more matrix modification - normalization. This means x 0-3 low
that we have divided each element of matrix with sum of x 3-7 medium
initial matrix that is not normalized (Fig. 6). Total sum of x 7-10 high
all coefficients in normalized matrix equals 1. New score is calculated based on previous score
PrevScore and output of face and eye detect phases for
0.0171 0.0171 0.0171 0.0171 0.0171 0.0171 0.0171 current frame.
0.0171 0.0214 0.0214 0.0214 0.0214 0.0214 0.0171 Difference between intervals is:
0.0171 0.0214 0.0267 0.0267 0.0267 0.0214 0.0171 x If PrevScore is in low interval, then score twice
0.0171 0.0214 0.0267 0.0334 0.0267 0.0214 0.0171 faster decreases than it rises
0.0171 0.0214 0.0267 0.0267 0.0267 0.0214 0.0171 x If PrevScore is in medium interval, then score
0.0171 0.0214 0.0214 0.0214 0.0214 0.0214 0.0171 decreases and rises at the same rate
0.0171 0.0171 0.0171 0.0171 0.0171 0.0171 0.0171 x If PrevScore is in high interval, then score twice
Fig. 6. Normalized weight matrix (from Fig.5.) faster rises, than it decreases
When score is in low interval, we can tell that driver is
For each considered pixel, weighted sum of its not distracted at all. In case of medium and high score,
ambience is calculated. When whole image has been driver is drowsy and system should eventually alert the
processed, iris should contain pixel for which the sum was driver.
minimal. Intuition behind algorithm is that we want to
punish more bright pixels near center of applied filter, then
pixels on the edge of filter. Moreover, these punishments IV. EVALUATION
should happen gradually because of fact that neighbor Described algorithm gives the best results if camera and
pixels are also very important for correctness of algorithm. light source are placed in front of the driver. Algorithm
When center of eye is determined, next task is to output used for verification of detected regions is shown in
deduce if eye is opened or closed. Again, our approach Fig. 9.
was associated with weight matrices. Let’s consider
grayscale image of closed eye for a moment. In the center
of this image we have a strong line of dark values that
correspond to eyelashes. However, when eye is opened,
such line cannot be established (Fig. 7). With this

Authorized licensed use limited to: PES University Bengaluru. Downloaded on October 12,2023 at 11:18:17 UTC from IEEE Xplore. Restrictions apply.
 The algorithm was not tested at night because it
requires infrared cameras which were not available at the
time.

V. CONCLUSION
In this paper we described one implementation of
driver monitoring algorithm, which is in early stage of
development. Implemented algorithm is not state of the art,
but we have achieved some solid results. There is more
than one way we can improve this algorithm:
(a) (b) x Using infrared cameras would give better input
images – we would have images with the same
illumination level, so light would not be a factor.
x Using advanced and more accurate methods for iris
detection which are based on Bayesian
classification of extracted features.
x Integration with data coming from the car itself,
like current speed, steering pattern etc.
x Creating more sophisticated drowsiness score
calculation scheme and testing it in actual vehicle.
First two ideas do not require usage of additional
(c) (d) equipment. Consequently, working on these ideas will be
Fig. 9. Algorithm output examples our first step. Upon successful completion of these steps,
further refinements can be implemented during the in car
Image (a) shows a case when a face is found, eyes are testing.
open and driver is looking ahead. In this case, drowsiness
score is decreasing depending on its current value. ACKNOWLEDGMENT
Image (b) shows that our algorithm also works well in This work was partially supported by the Ministry of
case when driver does not look ahead. It is natural for a Education, Science and Technological Development of the
driver to look left or right in order to check side mirrors or Republic of Serbia, under grant number: TR32041.
to address their attention to some other road event. It
should be clear that drowsiness score should not rise in this REFERENCES
case. [1] M. Aly, “Real time Detection of Lane Markers in Urban Streets” in
Image (c) shows driver with closed eyes and red IEEE Intelligent Vehicles Symposium, Eindhoven, The
rectangle shows that algorithm detected that eyes are not Netherlands, June 2008
[2] P. Dollar, C. Wojek, B. Schiele, and P. Perona, “Pedestrian
opened. This may be a serious problem if driver is
Detection: An Evaluation of the State of the Art” in IEEE
sleeping. However, it may be the case that camera took the Transactions on Pattern Analysis and Machine Intelligence
frame when they were blinking. No matter what happened (Volume: 34, Issue: 4, April 2012 )
drowsiness score will rise. [3] S. Hossain, Z. Hyder, “Traffic Road Sign Detection and
Image (d) shows case when driver turns head to some Recognition for Automotive Vehicles” in International Journal of
Computer Applications, Volume 120 - Number 24, 2015
side. In that case face detect algorithm will not find [4] H. Al-Absi, J. Devaraj, P. Sebastian, V. Yap, “Vision-based
driver’s face and that is sign that driver is not paying automated parking system” in 10th International Conference on
attention to the road. Not all rotations are sanctioned, Information Sciences Signal Processing and their Applications
whereas in this particular image head is completely turned (ISSPA), 2010
[5] Hang-Bong Kang, “Various Approaches for Driver and Driving
right. Of course, score that measures distraction will Behavior Monitoring: A Review” in Computer Vision Workshops
increase. (ICCVW), 2013 IEEE International Conference, 2-8 Dec. 2013.
The algorithm was tested on real ADAS platform [6] P. Viola, M. Jones, “Rapid object detection using a boosted
board. It works as real-time algorithm (about 20 frames per cascade of simple features” in Computer Vision and Pattern
Recognition, 2001. CVPR 2001.
second) with high level of accuracy. The main reason for [7] J. Vicente, P. Laguna, A. Bartra, R. Bailón, “Drowsiness detection
good driver monitoring performance is minimal impact of using heart rate variability” in Medical & Biological Engineering
wrongly classified frames due to high framerate and the & Computing, June 2016, Volume 54, Issue 6, pp 927–937
manner of drowsiness score calculation. The algorithm was [8] J. Kim, S. Kim, H. Jung, B. Lee, E. Chung, “Driver's Drowsiness
Warning System Based on Analyzing Driving Patterns and Facial
implemented only for demo purposes and detailed Images” in 23rd International Technical Conference on the
statistical evaluation will be part of future work. Enhanced Safety of Vehicles (ESV), 2013

Authorized licensed use limited to: PES University Bengaluru. Downloaded on October 12,2023 at 11:18:17 UTC from IEEE Xplore. Restrictions apply.