Design driver sleep warning system through image

recognition and processing in Python, Dlib, and OpenCV

Tran Thi Hien12*, Qiaokang Liang2, and Nguyen Thi Dieu Linh3

1
Nam Dinh University of Technology Education, Vietnam
2
Hunan University, Changsha 410082, China
3
Hanoi University of Industry, Vietnam
*
[email protected]
Abstract. A large number of road accidents today are related to the
drowsiness of the driver of a vehicle due to many causes such as fatigue,
drunkenness. However, detecting driver drowsiness and warning is com-
pletely possible. In this paper, the authors propose a design pattern for
driver drowsiness warning systems using Python, Dlib and OpenCV. Ex-
perimental results show that the system works in many different environ-
mental conditions, meeting the requirements such as: Flexibility and con-
venience, high accuracy, processing time and warning fast enough for a
real time request application.
Keywords: Drowsiness warning, OpenCV, Eyes closeness detection

1 Introduction

Most of the methods proposed for imagebased drowsiness detection use symptoms
related to the level of the driver’s drowsiness. Detecting eyes that are closed or open is
used in many methods. Dasgupta [1] used the amount of eyelid closure as a cue for
drowsiness. Te presence of the iris in the image indicates that the eye is open. Detection
of the iris in an eye image using the circular Hough transform is used in [2] to classify
the eye as being open or closed. Other methods include template matching [3, 4], use
of local image features [5, 6], and using a Hidden Markov Model [7]. Head pose esti-
mation is used by Teyeb [8], where the level of driver alertness is measured by the head
being inclined to a certain degree and for a specifc time period. A threshold placed on
the changing rate of the mouth contour to detect yawning is used by Alioua [9] to detect
drowsiness of the driver. However, expert knowledge can be instrumental for accurate
and timely detection of drowsiness. Such knowledge is implemented by Rezaei and
Klette [10] using a fuzzy control fusion system.
In addition to these techniques, specialized machine learning-based methods have
been proposed with the recent emergence of deep learning. Park [11] proposed a deep
architecture, referred to as deep drowsiness detection. It is reliable for the exclusion of
backgrounds and environment variations and achieved 73.06% accuracy. Weng [12]
introduced a novel hierarchical temporal deep belief network for drowsiness detection.
Teir work highlights the detection of head positions and faces to detect drowsiness.
Huynh [13] used a three-dimensional convolutional neural network to extract features
2

in spatial-temporal domain. Te method is designed to solve issues with extreme head

poses, where it achieved 87.46% accuracy. Shih and Hsu [14] use a multistage spatial-
temporal network with a CNN to detect various states of drowsiness. Tey achieved
82.61% accuracy. Lyu [15] used random forest method to extract efective facial de-
scriptors to describe drowsiness based on face alignment, and to classify the driver’s
facial states where the claimed accuracy is 88.18%. Although, a deep learning-based
method achieves state-of-the-art performance, success still lies in the availability of
large amounts of data to train deep nets.
In the article, the authors give a plan to identify, then study in depth the method of
monitoring and warning the driver's sleepiness in the human face recognition applica-
tion, by monitoring the state of closed / open eyes and yawning mouth of the driver.
Since then, researching and building equipment that can be applied into practice in Vi-
etnam with the majority of means of passenger transport, long-distance goods trans-
ported easily by traffic accidents due to fatigue and driver's sleepiness.

2 Design system

The requirement for the system is to monitor and detect the driver's sleepiness during
driving with high accuracy, real-time response, and compact.
The system is built with four main parts: Hardware, operating system, algorithms and
display functions of the system.
a) System hardware structure

Fig.1. System hardware structure

- Central processing unit: Raspberry Pi 4.
- Webcam captures the image of the driver: Use the BlueOver T3200 with compact
dimensions: 80x43x65mm; shooting angle up to 60 degrees; 30 fps shooting speed. Use
a USB connection. No need to install drivers, compatible with all types of operating
systems of PCs, laptops such as Windows, Ubuntu, MacOS ... Using 5-layer HD lens,
eliminating optical phenomenon causing image blur. Video resolution 640x480 pixels.
 3

- The screen displays the results of the system: It is necessary to select a monitor
with HDMI port to connect to the Raspberry.
- Speaker: alarms when the system detects the driver's sleepy state.
- Mouse and keyboard: Help the driver to control the system.
b) Ubuntu operating system
The Ubuntu operating system is small in size, suitable for devices with limited
memory of the Raspberry Pi 4, as well as possessing a huge application store. In addi-
tion, it also has the advantages of: Hardware requirements are not too high, high secu-
rity, diverse and free application store, high compatibility, built-in necessary tools, and
community massive support.
c) Drowsiness detection algorithm
In the study, the authors built algorithms based on the Python programming language
to help identify and detect driver drowsiness. The main tools used include:
- Dlib open source library: The algorithm that Dlib uses to extract image properties
is HOG (Histogram of Oriented Gradients) and SVM (Support Vector Machine).
- Facial Landmark Recognition Algorithm:
The mission of Facial Landmark is to locate the face in the photo and identify the
points that make up the structure of the face. To perform these two tasks, the facial
landmark recognition algorithm will determine 68 main points according to the coordi-
nates (x, y) that make up the human face. Thereby distinguishing: mouth, right eye-
brow, left eyebrow, right eye, left eye, nose, jaw.

Fig.2. 68 points on face

- Euclidean distance:
4

The distance between two points A and B is calculated according to the Pythagorean
formula used by scipy library to calculate the eucliden distance between the landmarks
on the face.
d) Display function of the system
This function displays the results of the OpenCV computer vision image processing
on an open source basis...

3 An algorithm to detect driver's drowsiness

This section will present algorithm to detect driver's drowsiness, as follows:

- Process video input data: Collecting image data while driving by Webcam. Frame
conversion using OpenCV's built-in function to convert input video from color image
to multi-level grayscale to to meet the processing speed of the system.
- Face detection: Use results from Dlib's algorithms combine with OpenCV to detect
the driver's face.
- Extract the parts of the face: Using Dlib and OpenCV, identify 68 facial points from
which extract facial parts, namely: 2 eyes and mouth
- Detection of drowsiness: Using Euclidean distance, calculate the distance of the co-
ordinates of the points on the face to calculate the eye aspect ratio (EAR), mouth aspect
rate (MAR).
Of the 68 facial points, each eye is denoted by 6 points with separate coordinates

Fig.3. Eye point coordinates

The formula for calculating EAR from point p1 to p6 :

p2  p6  p3  p5
EAR 
2 p1  p4

The results of the EAR calculation give us an approximate value that remains con-
stant at the time of the eye opening but will rapidly decrease to 0 when the eye closing.
 5

Fig.4. Eye-frame rate values graph when eyes are closed and open
Looking at the figure above, this is the timed eye-frame rate for a video clip. The
eye-frame rate here fluctuated around 0.25 then quickly dropped to 0 then increased
again. That showed a blink of an eye had just taken place.
The mouth will be denoted by 12 distinct points.

Fig.5. The coordinates of the points on the mouth

Formula for calculating mouth frame rate
p2  p12  p3  p11  p4  p10  p5  p9  p6  p8
MAR 
5 p1  p7

Look at the picture below, this is the oral frame rate over time for a video clip. The
oral frame rate here hovers around a 0.3 value when in the normal state then quickly
rises to more than 1 and then decreases. This shows that the state of the mouth has
changed from normal to open. So we can determine the driver's yawning state.
6

Fig.6. Mouth frame rate variation chart when close and open
Set threshold standards, to check driver drowsiness: EAR allowed value: 0.25. MAR
allowed values: 0.7.
- Warning: During driving, if the ratio of the eyes and mouth do not meet the above
two rates. The system will set a timer to provide safety warnings if the ratio of the eyes
and mouth.

4 Experiment and evaluate the results

a) Experiment
In order to evaluate our proposed method, we used image datasets that are incremen-
tally difficult. We outlined the accuracy of face detection, eye detection, mouth detec-
tion and then classification of the eyes as open or closed, yawn mouth or not . The
detection results of both the open/closed eyes and yawn mouth are shown in Figure 7
to figure 11.
MAR: 0.35 EAR: 0.38 MAR: 0.33 EAR: 0.33
counterMieng: 0.00 counterMat: 0.00 counterMieng: 00.00 counterMat: 00.00 MAR: 0.33 EAR: 0.35 EAR: 0.29
MAR: 0.32
NguGat: 0.00 NguGat: 1.00 counterMieng: 0.00 counterMat: 00.00 counterMat: 00.00
counterMieng: 0.00
NguGat: 0.00 NguGat: 2.00

Fig.7. Front face state, not wearing and wearing glasses, sufficient and underexposed
CANH BAO NGU GAT EAR: 0.14 CANH BAO NGU GAT! EAR: 0.18
CANH BAO NGU GAT! EAR: 0.16 CANH BAO NGU GAT! EAR: 0.15
MAR: 0.12 counterMat: 25.00 MAR: 0.34 counterMat: 29.00
MAR: 0.34 counterMat: 102.00 MAR: 0.36 counterMat: 5.00
counterMieng: 0.00 NguGat: 1.00 counterMieng: 00.00 NguGat: 2.00
counterMieng: 0.00 NguGat: 1.00 counterMieng: 0.00 NguGat: 3.00

Fig.8. Eyes closed for more than 1.5 seconds, not wearing and wearing glasses, suffi-
cient and underexposed
 7

CANH BAO NGAP! EAR: 0.35 CANH BAO NGAP! EAR: 0.38 CANH BAO NGAP! EAR: 0.25
MAR: 1.31 counterMat: 0.00 MAR: 1.07 counterMat: 00.00 CANH BAO NGAP! EAR: 0.41
MAR: 0.92 counterMat: 2.00
counterMieng: 79.00 NguGat: 0.00 counterMieng: 29.00 NguGat: 1.00 MAR: 0.81 counterMat: 00.00
counterMieng: 45.00 NguGat: 1.00
counterMieng: 28.00 NguGat: 3.00

Fig.4. The state of yawn mouth more than 1.5 seconds, not wearing and wearing
glasses, sufficient and underexposed
MAR: 0.47 EAR: 0.41 MAR: 0.06 EAR: 0.4 MAR: 0.51 EAR: 0.43 MAR: 0.40 EAR: 0.28
counterMieng: 0.00 counterMat: 0.00 counterMieng: 0.00 counterMat: 0.00 counterMieng: 0.00 counterMat: 0.00 counterMieng: 0.00 counterMat: 0.00
NguGat: 0.00 NguGat: 0.00 NguGat: 0.00 NguGat: 3.00

Fig.9. The face is tilted at an angle of 300, not wearing and wearing glasses, sufficient
and underexposed
CANH BAO NGU GAT! EAR: 0.21 CANH BAO NGAP! EAR: 0.30 CANH BAO NGAP! EAR: 0.43 CANH BAO NGAP! EAR: 0.30
MAR: 0.32 counterMat: 103.00 MAR: 1.21 counterMat: 0.00 MAR: 1.10 counterMat: 0.00 MAR: 1.19 counterMat: 0.00
counterMieng: 0.00 NguGat: 2.00 counterMieng: 64.00 NguGat: 0.00 counterMieng: 25.00 NguGat: 2.00 counterMieng: 65.00 NguGat: 0.00

Fig.10. Eyes closed or yawning , the face is tilted at an angle of 300, not wearing and
wearing glasses, sufficient and underexposed

CHU Y!!! CHU Y!!!

Fig.11. State head down overslept or head up overslept make the system not recog-
nize
b) Evaluate the results
The proposed method was evaluated with a real-time video of someone driving a car.
The video was recorded in the day time with variations in illumination due to differ-
ences in the direction of sunlight. The main purpose of the proposed algorithm is to
classify eyes in the detected face and eye images.
- When the driver sitting right posture, the system's exact identification rate is as
follows: Face is 97.5%; the eye is 88%; Mouth is 91%. So the system meets the require-
ments set out.
8

- When the driver sitting the face is tilted at an angle of 300, the system's exact
identification rate is as follows: Face is 97.5%; the eye is 65%; Mouth is 88%. Results
are affected by ghosting that occurs when the driver wears glasses. However, the system
basically meets the requirements set out.
The above results show the feasibility of the proposed method for driver drowsiness
detection. It should be noted, however, that these results were achieved during the day
time. For night time, our proposed method can be used on top of face and eye detection
algorithms that work during night time.

5 Conclusion

By combining the hardware is the Raspberry Pi 4 processor, the Dlib algorithms with
the display function of OpenCV, we conduct design drowsiness warning system for
drivers. Experimental results above the image of the driver under light conditions, tilt
angle, wear different glasses. Synthesize and compare with the initial requirements of
the system, shows: The system works in many different environmental conditions, meet
the requirements set out such as: Flexible, onvenient, high precision, processing time
and alarms are fast enough for a real time request application.

References
1. A. Dasgupta, A. George, S. L. Happy, A. Routray: A visionbased system for monitoring the
loss of attention in automotive drivers. IEEE Transactions on Intelligent Transportation Sys-
tems, vol. 14, no. 4, pp. 1825–1838, (2013).
2. N. Alioua, A. Amine, M. Rziza, D. Aboutajdine: Eye state analysis using iris detection based
on Circular Hough Transform. Proceedings of the 2011 International Conference on Multi-
media Computing and Systems, ICMCS’11, IEEE, pp. 1–5, (2011).
3. X. Liu, X. Tan, S. Chen: Eyes closeness detection using appearance based methods. Pro-
ceedings of the International Conference on Intelligent Information Processing, vol. 385, pp.
398–408, Springer, (2012).
4. G. Fa-deng, H. Min-xian: Study on the detection of locomotive driver fatigue based on im-
age. Proceedings of the 2010 2nd International Conference on Computer Engineering and
Technology, pp. V7-612–V7-615, Chengdu, China, (2010).
5. L. Zhou, H. Wang: Open/closed eye recognition by local binary increasing intensity patterns.
Proceedings of the 2011 IEEE 5th International Conference on Robotics, Automation and
Mechatronics, RAM 2011, IEEE, pp. 7–11, China, (2011).
6. M. Tafreshi, A. M. Fotouhi: A fast and accurate algorithm for eye opening or closing detec-
tion based on local maximum vertical derivative pattern. Turkish Journal of Electrical Engi-
neering & Computer Sciences, vol. 24, no. 6, pp. 5124–5134, (2016).
7. H. Qin, J. Liu, T. Hong: An eye state identifcation method based on the embedded hidden
Markov model. Proceedings of the 2012 IEEE International Conference on Vehicular Elec-
tronics and Safety, ICVES 2012, IEEE, pp. 255–260, Turkey, (2012).
8. I. Teyeb, O. Jemai, M. Zaied, C. B. Amar: A drowsy driver detection system based on a new
method of head posture estimation. Proceedings of the International Conference on Intelli-
gent Data Engineering and Automated Learning, pp. 362–369, Springer, (2014).
 9

9. N. Alioua, A. Amine, M. Rziza: Driver’s fatigue detection based on yawning extraction.

International Journal of Vehicular Technology, Article ID 678786, 7 pages, (2014).
10. M. Rezaei, R. Klette: Look at the driver, look at the road: No distraction! No accident!.
Proceedings of the 27th IEEE Conference on Computer Vision and Pattern Recognition,
CVPR 2014, pp. 129–136, USA, (2014).
11. S. Park, F. Pan, S. Kang, C. D. Yoo: Driver drowsiness detection system based on feature
representation learning using various deep networks. Proceedings of the Computer Vision –
ACCV 2016 Workshops, vol. 10118 of Lecture Notes in Computer Science, pp. 154–164,
Springer International Publishing, Cham, Switzerland, (2016).
12. C. Weng, Y. Lai, S. Lai: Driver drowsiness detection via a hierarchical temporal deep belief
network. Proceedings of the Computer Vision – ACCV 2016 Workshops, vol. 10118 of
Lecture Notes in Computer Science, pp. 117–133, Springer International Publishing, Cham,
Switzerland, (2016).
13. X.-P. Huynh, S.-M. Park, Y.-G Kim: Detection of driver drowsiness using 3D deep neural
network and semi-supervised gradient boosting machine. Proceedings of the Computer Vi-
sion – ACCV 2016 Workshops, vol. 10118 of Lecture Notes in Computer Science, pp. 134–
145, Springer International Publishing, Cham, Switzerland, (2016).
14. T. Shih, C. Hsu: Multistage spatial-temporal network for driver drowsiness detection. Pro-
ceedings of the Computer Vision – ACCV 2016 Workshops, vol. 10118 of Lecture Notes in
Computer Science, pp. 146–153, (2016).
15. J. Lyu, H. Zhang, Z. Yuan: Joint shape and local appearance features for real-time driver
drowsiness detection. Proceedings of the Computer Vision – ACCV 2016 Workshops, vol.
10118 of Lecture Notes in Computer Science, pp. 178–194, Springer International Publish-
ing, Cham, Switzerland, (2016).
16. F. Y. Song, X. Y. Tan, X. Liu, S. C. Chen: Eyes closeness detection from still images with
multi-scale histograms of principal oriented gradients. Pattern Recognition, vol. 47, no. 9,
pp. 2825–2838, (2014).