Design driver sleep warning system through image

recognition and processing in Python, Dlib, and OpenCV

Tran Thi Hien12*, Qiaokang Liang2, and Nguyen Thi Dieu Linh3

1
Nam Dinh University of Technology Education, Vietnam
2
Hunan University, Changsha 410082, China
3
Hanoi University of Industry, Vietnam
*
[email protected]
[email protected]
Abstract. A large number of road accidents today are related to the
drowsiness of the driver of a vehicle due to many causes such as fatigue,
drunkenness. However, detecting driver drowsiness and warning is com-
pletely possible. In this paper, the authors propose a design pattern for
driver drowsiness warning systems using Python, Dlib and OpenCV. Ex-
perimental results show that the system works in many different environ-
mental conditions, meeting the requirements such as: Flexibility and con-
venience, high accuracy, processing time and warning fast enough for a
real time request application.
Keywords: Drowsiness warning, OpenCV, Eyes closeness detection

1 Introduction

Today, along with the rapid development of the number of traffic vehicles, the num-
ber of traffic accidents is also increasing. One of the main causes of traffic accidents is
the lack of driver concentration, due to fatigue or sleepiness. The installation of a warn-
ing system, reminders when driving is the top concern of the driver in particular as well
as the transport industry in general.
Proposed solutions to detect drowsiness are usually based on the image of the driv-
er's eyes closed or open, when it detects the driver's drowsiness, then the system will
use the sound to alert driver. There have been many proposed methods, such as Das-
gupta [1] that have used the number of closed eyes and the interval with each close eyes
as a sign of sleepy. The circular Hough transform to detect the iris in an eye image to
classify the eye as being open or closed is used in [2]. In [3, 4] using template matching
method, in [5, 6] using local image features, and Hidden Markov Model in [7]. Teyeb
uses Head Posture Estimation, which measures head tilt and specific time intervals to
determine the driver's level of alertness [8].
In the article, the authors give a plan to identify, then study in depth the method of
monitoring and warning the driver's sleepiness in the human face recognition applica-
tion, by monitoring the state of closed / open eyes and yawning mouth of the driver.
2

Since then, researching and building equipment that can be applied into practice in Vi-
etnam with the majority of means of passenger transport, long-distance goods trans-
ported easily by traffic accidents due to fatigue and driver's sleepiness.

2 Design system

The requirement for the system is to monitor and detect the driver's sleepiness during
driving with high accuracy, real-time response, and compact.
The system is built with four main parts: Hardware, operating system, algorithms and
display functions of the system.
a) System hardware structure

Fig.1. System hardware structure

- Central processing unit: Raspberry Pi 4.
- Webcam captures the image of the driver: Use the BlueOver T3200 with compact
dimensions: 80x43x65mm; shooting angle up to 60 degrees; 30 fps shooting speed. Use
a USB connection. No need to install drivers, compatible with all types of operating
systems of PCs, laptops such as Windows, Ubuntu, MacOS ... Using 5-layer HD lens,
eliminating optical phenomenon causing image blur. Video resolution 640x480 pixels.
- The screen displays the results of the system: It is necessary to select a monitor
with HDMI port to connect to the Raspberry.
- Speaker: alarms when the system detects the driver's sleepy state.
- Mouse and keyboard: Help the driver to control the system.
b) Ubuntu operating system
The Ubuntu operating system is small in size, suitable for devices with limited
memory of the Raspberry Pi 4, as well as possessing a huge application store. In addi-
tion, it also has the advantages of: Hardware requirements are not too high, high secu-
rity, diverse and free application store, high compatibility, built-in necessary tools, and
community massive support.
c) Drowsiness detection algorithm
 3

In the study, the authors built algorithms based on the Python programming language
to help identify and detect driver drowsiness. The main tools used include:
- Dlib open source library: The algorithm that Dlib uses to extract image properties
is HOG (Histogram of Oriented Gradients) and SVM (Support Vector Machine).
- Facial Landmark Recognition Algorithm:
The mission of Facial Landmark is to locate the face in the photo and identify the
points that make up the structure of the face. To perform these two tasks, the facial
landmark recognition algorithm will determine 68 main points according to the coordi-
nates (x, y) that make up the human face. Thereby distinguishing: mouth, right eye-
brow, left eyebrow, right eye, left eye, nose, jaw.

Fig.2. 68 points on face

- Euclidean distance:
The distance between two points A and B is calculated according to the Pythagorean
formula used by scipy library to calculate the eucliden distance between the landmarks
on the face.
d) Display function of the system
This function displays the results of the OpenCV computer vision image processing
on an open source basis...

3 An algorithm to detect driver's drowsiness

This section will present algorithm to detect driver's drowsiness, as follows:

- Process video input data: Collecting image data while driving by Webcam. Frame
conversion using OpenCV's built-in function to convert input video from color image
to multi-level grayscale to to meet the processing speed of the system.
- Face detection: Use results from Dlib's algorithms combine with OpenCV to detect
the driver's face.
4

- Extract the parts of the face: Using Dlib and OpenCV, identify 68 facial points from
which extract facial parts, namely: 2 eyes and mouth
- Detection of drowsiness: Using Euclidean distance, calculate the distance of the co-
ordinates of the points on the face to calculate the eye aspect ratio (EAR), mouth aspect
rate (MAR).
Of the 68 facial points, each eye is denoted by 6 points with separate coordinates

Fig.3. Eye point coordinates

The formula for calculating EAR from point p1 to p6 :

p2  p6  p3  p5
EAR 
2 p1  p4

The results of the EAR calculation give us an approximate value that remains con-
stant at the time of the eye opening but will rapidly decrease to 0 when the eye closing.

Fig.4. Eye-frame rate values graph when eyes are closed and open
Looking at the figure above, this is the timed eye-frame rate for a video clip. The
eye-frame rate here fluctuated around 0.25 then quickly dropped to 0 then increased
again. That showed a blink of an eye had just taken place.
The mouth will be denoted by 12 distinct points.

Fig.5. The coordinates of the points on the mouth

Formula for calculating mouth frame rate
p2  p12  p3  p11  p4  p10  p5  p9  p6  p8
MAR 
5 p1  p7
 5

Look at the picture below, this is the oral frame rate over time for a video clip. The
oral frame rate here hovers around a 0.3 value when in the normal state then quickly
rises to more than 1 and then decreases. This shows that the state of the mouth has
changed from normal to open. So we can determine the driver's yawning state.

Fig.6. Mouth frame rate variation chart when close and open
Set threshold standards, to check driver drowsiness: EAR allowed value: 0.25. MAR
allowed values: 0.7.
- Warning: During driving, if the ratio of the eyes and mouth do not meet the above
two rates. The system will set a timer to provide safety warnings if the ratio of the eyes
and mouth.

4 Experiment and evaluate the results

a) Experiment
In order to evaluate our proposed method, we used image datasets that are incremen-
tally difficult. We outlined the accuracy of face detection, eye detection, mouth detec-
tion and then classification of the eyes as open or closed, yawn mouth or not . The
detection results of both the open/closed eyes and yawn mouth are shown in Figure 7
to figure 11.
MAR: 0.35 EAR: 0.38 MAR: 0.33 EAR: 0.33
counterMieng: 0.00 counterMat: 0.00 counterMieng: 00.00 counterMat: 00.00 MAR: 0.33 EAR: 0.35 EAR: 0.29
MAR: 0.32
NguGat: 0.00 NguGat: 1.00 counterMieng: 0.00 counterMat: 00.00 counterMat: 00.00
counterMieng: 0.00
NguGat: 0.00 NguGat: 2.00

Fig.7. Front face state, not wearing and wearing glasses, sufficient and underexposed
CANH BAO NGU GAT EAR: 0.14 CANH BAO NGU GAT! EAR: 0.18
CANH BAO NGU GAT! EAR: 0.16 CANH BAO NGU GAT! EAR: 0.15
MAR: 0.12 counterMat: 25.00 MAR: 0.34 counterMat: 29.00
MAR: 0.34 counterMat: 102.00 MAR: 0.36 counterMat: 5.00
counterMieng: 0.00 NguGat: 1.00 counterMieng: 00.00 NguGat: 2.00
counterMieng: 0.00 NguGat: 1.00 counterMieng: 0.00 NguGat: 3.00

Fig.8. Eyes closed for more than 1.5 seconds, not wearing and wearing glasses, suffi-
cient and underexposed
6

CANH BAO NGAP! EAR: 0.35 CANH BAO NGAP! EAR: 0.38 CANH BAO NGAP! EAR: 0.25
MAR: 1.31 counterMat: 0.00 MAR: 1.07 counterMat: 00.00 CANH BAO NGAP! EAR: 0.41
MAR: 0.92 counterMat: 2.00 MAR: 0.81
counterMieng: 79.00 NguGat: 0.00 counterMieng: 29.00 NguGat: 1.00 counterMat: 00.00
counterMieng: 45.00 NguGat: 1.00 counterMieng: 28.00 NguGat: 3.00

Fig.4. The state of yawn mouth more than 1.5 seconds, not wearing and wearing
glasses, sufficient and underexposed
MAR: 0.47 EAR: 0.41 MAR: 0.06 EAR: 0.4 MAR: 0.51 EAR: 0.43 MAR: 0.40 EAR: 0.28
counterMieng: 0.00 counterMat: 0.00 counterMieng: 0.00 counterMat: 0.00 counterMieng: 0.00 counterMat: 0.00 counterMieng: 0.00 counterMat: 0.00
NguGat: 0.00 NguGat: 0.00 NguGat: 0.00 NguGat: 3.00

Fig.9. The face is tilted at an angle of 300, not wearing and wearing glasses, sufficient
and underexposed
CANH BAO NGU GAT! EAR: 0.21 CANH BAO NGAP! EAR: 0.30 CANH BAO NGAP! EAR: 0.43 CANH BAO NGAP! EAR: 0.30
MAR: 0.32 counterMat: 103.00 MAR: 1.21 counterMat: 0.00 MAR: 1.10 counterMat: 0.00 MAR: 1.19 counterMat: 0.00
counterMieng: 0.00 NguGat: 2.00 counterMieng: 64.00 NguGat: 0.00 counterMieng: 25.00 NguGat: 2.00 counterMieng: 65.00 NguGat: 0.00

Fig.10. Eyes closed or yawning, the face is tilted at an angle of 300, not wearing and
wearing glasses, sufficient and underexposed

CHU Y!!! CHU Y!!!

Fig.11. State head down overslept or head up overslept make the system not recognize
b) Evaluate the results
The proposed method was evaluated with a real-time video of someone driving a car.
The video was recorded in the day time with variations in illumination due to differ-
ences in the direction of sunlight. The main purpose of the proposed algorithm is to
classify eyes in the detected face and eye images.
- When the driver sitting right posture, the system's exact identification rate is as follows: Face
is 97.5%; the eye is 88%; Mouth is 91%. So the system meets the requirements set out.
- When the driver sitting the face is tilted at an angle of 300, the system's exact
identification rate is as follows: Face is 97.5%; the eye is 65%; Mouth is 88%. Results
 7

are affected by ghosting that occurs when the driver wears glasses. However, the system
basically meets the requirements set out.
The above results show the feasibility of the proposed method for driver drowsiness
detection. It should be noted, however, that these results were achieved during the day
time. For night time, our proposed method can be used on top of face and eye detection
algorithms that work during night time.

5 Conclusion

By combining the hardware is the Raspberry Pi 4 processor, the Dlib algorithms with
the display function of OpenCV, we conduct design drowsiness warning system for
drivers. Experimental results above the image of the driver under light conditions, tilt
angle, wear different glasses. Synthesize and compare with the initial requirements of
the system, shows: The system works in many different environmental conditions, meet
the requirements set out such as: Flexible, onvenient, high precision, processing time
and alarms are fast enough for a real time request application.

References
1. A. Dasgupta, A. George, S. L. Happy, A. Routray: A visionbased system for monitoring the
loss of attention in automotive drivers. IEEE Transactions on Intelligent Transportation Sys-
tems, vol. 14, no. 4, pp. 1825–1838, (2013).
2. N. Alioua, A. Amine, M. Rziza, D. Aboutajdine: Eye state analysis using iris detection based
on Circular Hough Transform. Proceedings of the 2011 International Conference on Multi-
media Computing and Systems, ICMCS’11, IEEE, pp. 1–5, (2011).
3. X. Liu, X. Tan, S. Chen: Eyes closeness detection using appearance based methods. Pro-
ceedings of the International Conference on Intelligent Information Processing, vol. 385, pp.
398–408, Springer, (2012).
4. G. Fa-deng, H. Min-xian: Study on the detection of locomotive driver fatigue based on im-
age. Proceedings of the 2010 2nd International Conference on Computer Engineering and
Technology, pp. V7-612–V7-615, Chengdu, China, (2010).
5. L. Zhou, H. Wang: Open/closed eye recognition by local binary increasing intensity patterns.
Proceedings of the 2011 IEEE 5th International Conference on Robotics, Automation and
Mechatronics, RAM 2011, IEEE, pp. 7–11, China, (2011).
6. M. Tafreshi, A. M. Fotouhi: A fast and accurate algorithm for eye opening or closing detec-
tion based on local maximum vertical derivative pattern. Turkish Journal of Electrical Engi-
neering & Computer Sciences, vol. 24, no. 6, pp. 5124–5134, (2016).
7. H. Qin, J. Liu, T. Hong: An eye state identifcation method based on the embedded hidden
Markov model. Proceedings of the 2012 IEEE International Conference on Vehicular Elec-
tronics and Safety, ICVES 2012, IEEE, pp. 255–260, Turkey, (2012).
8. I. Teyeb, O. Jemai, M. Zaied, C. B. Amar: A drowsy driver detection system based on a new
method of head posture estimation. Proceedings of the International Conference on Intelli-
gent Data Engineering and Automated Learning, pp. 362–369, Springer, (2014).