DRIVER DROWSINESS DETECTION SYSTEM

USING FACIAL FEATURES

A Project Report
Submitted in the partial fulfillment of the requirements for
the award of the degree of

Bachelor of Technology

in

Department of Computer Science Engineering

By

2000032082 AKKIREDDY DURGA PRATHYUSHA

20000031679 LAVANYA DHIRISHALA

2000031581 SUNKARI VENKATA NARYANA

2000031750 VULLURI SUMANTH SREERAM

under the supervision of

Dr.NILU SINGH

Associate Professor

Department of Computer Science Engineering

K L E F, Green Fields,
Vaddeswaram- 522502, Guntur(Dist), Andhra Pradesh, India.

1
March 2023

2
 Declaration

The Project Report entitled “DRIVER DROWSINESS DETECTION SYSTEM USING FACIAL
FEATURES” is a record of bonafide work of A.Durga Prathyusha(2000032082), D.Lavanya
(2000031679),S.Venkat Narayana (2000031581),V.Sumanth Sreeram(2000031750) submitted in partial
fulfillment for the award of B. Tech in Computer Science and Engineering to the K L University. The
results embodied in this report have not been copied from any otherdepartments.

Signature of the Students

2000032082
2000031679
2000031581
2000031750

3
 Certificate

This is to certify that the Project Report entitled “DRIVER DROWSINESS DETECTION
SYSTEM USING FACIAL FEATURES” is being submitted by A.Durga
Prathyusha(2000032082), D.Lavanya (2000031679),S.Venkat Narayana (2000031581),V.Sumanth
Sreeram(2000031750) submitted in partial fulfillment for the award of B.Tech in Computer Science
and Engineering to the KL University is a record of bonafide work carried out under our guidance
and supervision.
The results embodied in this report have not been copied from any other departments/
University/Institute.

Signature of the Supervisor

Dr. NILU SINGH

Associate Professor

Signature of the HOD Signature of the External Examiner

4
 Table of contents

1. Abstract ..................................................................................... 5

2. Introduction ............................................................................... 5

3. Requirements… ..........................................................................9

4. Literature Survey ........................................................ 11

5. Theoretical Analysis ................................................................ 18

6. Experimental Investigation ................................................. ,21

7. Experimental Results ...................................................... 33

8. Discussion of Results ................................................... 36

9. Conclusion................................................................................. 38

10. Reference/bibliography .................................................. 30

5
1. ABSTRACT
Drowsy driving impacts people who are doing the driving and puts everyone else on the road in
danger, making it a significant social problem. A drowsiness detection system is one of the safety
technologies that can stop accidents from being caused by tired drivers who fall asleep at the wheel.
In today's culture, people are telling vehicle passengers not to drive when inebriated, yet the
restrictions are still being broken in many areas by sleepy individuals. Therefore, a sensing system
has been included into the vehicle itself to address this issue. In order to extract the driver's state
of action, presentation, and evaluate if they are driving safely, these sensors of the detector must
continuously watch the driver's facial expressions and recognize landmarks in the face region using
the suitable algorithm. If the individual is drowsy, it captures the eyes, lips, and driver status in the
facial areas and informs the driver with an alarm beep sound. This research uses machine learning
and deep learning methods to predict a driver's emotional state, which will increase traffic safety.
To efficiently identify the driver's condition, the model is built on OpenCV and the Haar-based
cascade classifier, which can detect eye movement.

Keywords: facial areas, face detection, drowsiness, haar cascade, machine learning

2. INTRODUCTION

A big number of individuals all around the world wish to acquire automobiles. It is worth noting
that the risk of road accidents is quickly growing as the number of cars on the road increases.
Driving is one of the delicate professions that demands strong mental health and attention. Drowsy
drivers may lack the balance in response time and decision-making when operating a vehicle,
which increases the likelihood of accidents. The other vehicles and persons on the road are also
impacted by this collision in addition to the driver. In nations with densely populated streets and
highways, the number of traffic accidents is quite high. Drowsiness is one of the leading causes of
real vehicle accidents in our daily lives. According to the National Highway Traffic Safety
Administration, drunk driving contributes to about 150 homicides per year in the United States.

6
Many problems of the body can be revealed through facial expressions. When it comes to face
detection, which is the first step in the process, there are several algorithms and strategies to choose
from. The primary goal of this initiative is to lower the number of accidents on the road that are
caused by driver fatigue. Face image analysis is a major research topic, with applications ranging
from face recognition to person identification and tracking for security systems. The main goal of
this project is to localize the eyes and mouth, which entails examining the full facial areas image
and using the algorithm for image processing to determine where the eyes and mouth are located.
Once the location of the eyes has been determined, the system is meant to detect exhaustion and
sleepiness by determining whether the eyes and mouth are open or closed.

In this research, we offer a straightforward way to detect sleepiness that takes the use of vision-
based approaches. The main difficulties with the suggested method are (a) creating a real-time
system, face detection, and iris identification in a variety of situations, including with and without
glasses, illumination, etc. Creating a real-time system that will precisely track the open or closed
state of the driver's eyes was given top priority, along with blink detection and economy. By
keeping an eye on the consciousness, an automobile accident may be avoided. In order to identify
signs of exhaustion, it is necessary to watch eye movements and blink patterns in a series of
photographs of a face that were taken from a live stream. It is clear that the system is divided into
five independent modules: video capture, frame division, face recognition, object classification,
and drowsiness detection.

2.1 Video Capture:

The primary goal of video capture is to get the driver's live video feed. The acquisition of video is
accomplished by the employment of a camera. OpenCV has a wealth of features for gathering and
processing live video. By specifying the appropriate settings, it's also possible to decide whether
the video should be taken from an external camera or the computer's built-in webcam.

7
2.2 Frame Division:

This module's function is to take live video as an input and turn it into a collection of frames or
images that can then be processed. We initially load the cascade file in order to detect faces. After
that, send the captured image to an edge detection algorithm, which will find all potential objects
in the frame, regardless of size. Instead of identifying items of all conceivable sizes, because the
face of the vehicle driver covers a substantial portion of the image, we may configure the edge
detector to detect only objects of a certain size, which is determined by the Haar cascade file,
whereby each Haar cascade file will be built for a specific size.

2.3 Face Recognition:

The face detection takes one frame at a time from the images given by the frame grabber and
attempts to detect the vehicle driver's face in each frame. This is accomplished by employing a
pre-defined set of Haar cascade samples.

2.4 Object Classification:

The eyes identification function seeks to identify the car driver's eyes once the face recognition
component has identified the person's face. A collection of pre-defined Haarcascade samples is
used to achieve this. The following factors are taken into consideration while determining the area
of interest:

• The eyes are only present in the upper section of the identified face.

• The eyes are only located a few pixels down from the top angle of the detected face.

8
2.5 Drowsiness Detection:

The drowsiness detection feature first recognizes the driver's eyes and then determines whether or
not the driver is tired by analyzing the condition of the eyes—whether they are open or closed and
the blink speed. We set a time limit of 5 seconds , if the eyelids remain closed for five seconds or
more, sleepiness is recognized, and a warning pop is triggered. The advantages of detection are:

• Better differentiation between colors, as it uses multichannel colors.

• Consumes very less memory.
• Capable of achieving blink detection, even when the automobile driver is wearing
spectacles.

Figure 1. Process of driver drowsiness detetction system

9
3. REQUIREMENTS

3.1 SOFTWARE REQUIREMENTS:

These are the Software Configurations that are required for the proposed system.

• Operating System: Linux, Windows, Mac OS

• Language: Python 3
• IDE: Pycharm

3.2 HARDWARE REQUIREMENTS

These are the Hardware Configurations that are required for the proposed system.
• Processor : Intel Core i5
• Ram: 8GB
• Web cam / In-built laptop camera
• Speaker

3.3 FUNCTIONAL REQUIREMENTS

OPEN CV (Computer Vision)

Real-time applications are the primary emphasis of the well-known OpenCV computer vision
library. OpenCV is a massive open-source library for image processing, computer vision, and
machine learning. It can analyze photos and video content to recognize items, people.

10
KERAS

In order to create and assess deep learning models, Keras is a robust and user-friendly free open-
source Python framework. It covers efficient numerical calculation libraries and enables the
definition and training of neural network models in a few of lines of code.

NUMPY

NumPy is the abbreviation for numerical Python. NumPy could be a Python library for
manipulating arrays. It also includes routines for dealing with matrices and the Fourier transform
in the field of linear algebra. The numpy array is a widespread data shape for handling images,
filter kernels and function factors.

PYYGAME:

The Pygame library is an open-source Python module designed to help you create games and other
multimedia applications. By utilizing the Pygame module, you can manage the logic of your games
and applications with backend requirements for working with audio and video. It includes sound
and cg libraries and it’s intended for use with python.

The emphasis will be on developing a system that will continuously monitor and analyze detected
eye images to accurately monitor the open or closed state of the driver's eyes. The goal of this
research is to create a sleepiness detecting system. The current study basically used the same
general approach of starting with basic facial feature detection such as face and eyes, but our
research differs in that it simplifies the process further in order to run efficiently in real time
situations, it use additional techniques such as template matching to keep tracking the detected
eye, and it also used the pre-processed eye image.

11
12
4. LITERATURE SURVEY

The survey is taken from the different research papers on studying how they developed the
methodology of the detection system for gaining knowledge about the model developed. Many
scholars have studied driver tiredness and drowsiness in the literature and have recommended
various approaches based on a variety of metrics. In addition, authors developed different models
of their own driver fatigue and drowsiness systems in an effort to increase the quality and security
of their model and decrease losses brought on by sleepiness.

S.N PAPER NAME DESCRIPTION PUBL AUTHOR NAME

O ISHE
D
YEA
R
1 Drunkenness or exhaustion is a 2021 ANKIT KUMAR
Early Identification leading cause of car accidents, with
and Detection of severe implications for road safety.
Driver Drowsiness by More fatal accidents could be
Hybrid Machine avoided if fatigued drivers were
Learning warned ahead of time. Several
drowsiness detection technologies to
monitor for signs of inattention while
driving.

2. Real-Time Driver- The face, an important part of the 2019 WANGHUA

Drowsiness body, conveys a lot of information. DENG
Detection_System When a driver is in a state of fatigue,
Using Facial Features the facial expressions, the frequency
of blinking and yawning, are
different from those in the normal
state. In this paper, we propose a
system called DriCare, which detects
the drivers’ fatigue status, such as
yawning, blinking, and duration of
eye closure, using video images,
without equipping their bodies with
devices. Owing to the shortcomings
of previous algorithms, we introduce
a new face-tracking algorithm to
improve the tracking accuracy.

13
3 Driver’s real-time According to National Highway 2021 JANKI
Drowsiness Detection Traffic Safety Administration CHANDIWALA
using Adaptable Eye (NHTSA), drowsy driving leads to
Aspect Ratio and over 71,000 injuries, 1,500 deaths,
Smart Alarm System and $12.5 billion in monetary losses
per year. The main issue that needs to
be addressed is that the driver should
be warned well before time to take
precautionary measures. This paper
suggests a radical solution to the
problem by proposing a system. This
paper describes consideration of the
Eye Aspect Ratio (EAR) and Mouth
Aspect Ratio (MAR) to quantify
drowsiness by locating, monitoring,
and analyzing both eyes and mouth,
4 Driver Drowsiness In this paper, driver drowsiness 2021 AICHA
Detection Based on detection algorithm based on the GHOURABI
Joint Monitoring of state of eyes of the driver which is
Yawning, Blinking determined by his iris visibility has
and Nodding been implemented. If eyes remain in
one state either open or closed longer
than expected time as well as if the
driver is not looking straight front, it
is an indication that driver is drowsy
and then the system warns the driver.
5. Driver Drowsiness Driving a car is a complex and 2018 SUBRAMANIAM
Detection System potentially risky activity in people’s GANESAN
Based on Binary Eyes everyday life, and it requires the full
Image Data involvement of physiological and
cognitive resources. Any loss of
these resources can cause traffic
accidents. The developed framework
allows the extraction of drowsiness-
related measures by analysing the
driver’s face with a standard camera.
First, a face detection stage
identifies the driver face in a video
frame. Then, a set of facial
landmarks locations are identified.
These landmark points are used to
estimate the head orientation and to
detect when a blink occurs.

14
6. Evaluation of Driver The research work about driving 2020 CIRO NATALE
Drowsiness based on drowsiness detection algorithm has
Real-Time Face great significance to improve traffic
Analysis safety. Presently, there are many
fruits and literature about driving
drowsiness detection method.
However, most of them are devoted
to find a universal drowsiness
detection method, while ignore the
individual driver differences. This
paper proposes a real-time driving
drowsiness detection algorithm that
considers the individual differences
of driver.
7. Multimodal System to Sleepiness detection system that 2019 NAVEEN
Detect Driver Fatigue evaluates driver’s sleepiness level SENNIAPPAN
Using EEG, has always been the primary interest KARUPPUSAMY
Gyroscope, and of researchers. There are a number
Image Processing of systems like
electroencephalography-based
sleepiness detection system (ESDS),
vehicle-based sleepiness estimator
system, image acquisition
technology and bio-mathematical
models to detect drowsiness of
drivers. In order to overcome the
above limitation, we propose a
neural network-based hybrid
multimodal system that detects
driver fatigue using
electroencephalography (EEG) data,
gyroscope data and image
processing data.
In this paper, a module for 2013 BELAL
8. Driver drowsiness Advanced Driver Assistance System ALSHAQAQI
(ADAS) is presented to reduce the
detection system number of accidents due to drivers
fatigue and hence increase the
transportation safety; this system
deals with automatic driver
drowsiness detection based on
visual information and Artificial
Intelligence. We propose an
algorithm to locate, track, and
analyze both the drivers face and
eyes to measure PERCLOS, a

15
 scientifically supported measure of
drowsiness associated with slow eye
closure.

9. Driver drowsiness detection has been 2014 KARTIK

Drowsy driver considered a major potential area so DWIVEDI
detection using as to prevent a huge number of sleeps
representation induced road accidents. This paper
learning proposes a vision based intelligent
algorithm to detect driver
drowsiness. Previous approaches are
generally based on blink rate, eye
closure, yawning, eye brow shape
and other hand engineered facial
features. The proposed algorithm
makes use of features learnt using
convolutional neural network so as to
explicitly capture various latent
facial features and the complex non-
linear feature interactions.
10. Real-Time Driver’s This system makes it possible to 2019 ABDELLAH
Hypovigilance detect drowsiness by the AARAB
Detection using Facial identification of MicroSleep
Landmarks corresponding to a sleepiness of
more than 2 seconds through the
analysis of eye-closure, and to
identify fatigue by the analysis of the
movement of the mouth to detect
yawning.

11. The system is made illumination 2016 B.N. MANU

Facial features invariant by segmenting the skin part
monitoring for real alone and considering only the
time drowsiness chromatic components to reject most
detection of the non face image backgrounds
based on skin color. The tracking of
eyes and yawning detection are done
by correlation coefficient template
matching.

16
12. Driver Drowsiness Real time driver’s drowsiness 2018 ASHISH KUMAR
Monitoring System detection system is developed with
using Visual acceptable accuracy. In the
Behaviour and developed system, a webcam records
Machine Learning the video and driver’s face is
detected in each frame employing
image processing techniques. Facial
landmarks on the detected face are
pointed and subsequently the eye
aspect ratio, mouth opening ratio and
nose length ratio are computed and
depending on their values,
drowsiness is detected based on
developed adaptive thresholding.
Machine learning algorithms have
been implemented as well in an
offline manner.
Detection and indication of driver 2020 SANJAY DEY
13. Real Time Driver fatigue is an active area of research.
Fatigue Detection In this concussion, both inside and
Based on Facial outside individuals of the car are
Behaviour along with victimized. However, prevention of
Machine Learning drowsiness requires a technique to
Approaches detect the deterioration legitimately
along with a warning mechanism of
the vehicle operator. Although the
existing solutions are created using
some distinctive methods, there are
some problems such as costly
sensors and handling of information.
The objective of this research is to
create an improved, innovative, cost
efficient and real time strategy for
solving this problem of blinking,
yawn, and head bending.
14. The aim of this work is to create a 2019 ADITYA NAIR
Driver Assistant system which will help to assist in
System using Haar controlling such errors and
Cascade and preventing crashes. The proposed
Convolution Neural system uses machine learning
Networks(CNN) algorithms to detect the state of the
driver based on which decisions are
made on whether the driver should
drive the vehicle or not. First, the
detection of the facial features of the
driver is done using a camera(place

17
 in front of the driver) and image
processing. After capturing the
features they are fed to a trained
neural model to obtain driver state.

15. Driver Dangerous The real-time driving behavior 2019 IGOR LASHKOV
State Detection Based monitoring plays a significant role in
on OpenСV & Dlib intelligent transportation systems.
Libraries Using Such monitoring increases traffic
Mobile Video safety by reducing and eliminating
Processing the risk of potential traffic accidents.
Vision-based approach including
video cameras for dangerous
situation detection is undoubtedly
one of the most perspective and
commonly used in sensing driver
environment. In this case, the images
of a driver, captured with video
cameras, can describe its facial
features, like head movements, eye
state, mouth state, and, afterwards,
identify a current level of fatigue
state.

[1] Ankit Kumar In this research, they used machine learning approaches to perform real-time
picture segmentation and sleepiness. An emotion recognition system based on Support Vector
Machines (SVM) has been constructed utilizing facial expressions in the proposed study. The
algorithm was evaluated under varied brightness circumstances and surpassed existing studies in
terms of accuracy.

[2] Wanghua Deng In this study, the authors suggest a system called DriCare that uses video
pictures to identify drivers' signs of weariness without fitting their bodies with gadgets, such as
yawning, blinking, and length of eye closure. Because earlier algorithms had flaws, we developed
a new face-tracking algorithm to increase tracking accuracy. Furthermore, based on 68 critical
features, they developed a new detection approach for face areas. The drivers' condition is then
assessed using these face areas.

18
[3] Kahlon and Ganesan In this paper research they developed a face detection system based on
binary eye data and visuals. Additionally, the researchers have experimented with a variety of
fusion techniques. This research is based on the category that utilizes the condition of the eyes to
identify drowsiness. The driver's iris visibility is used to determine his or her state of eyes. The
image produced by a camera was processed using Matlab and image processing tools. To identify
things like the nose, mouth, or upper area, Matlab constructs a System Object using the Viola-
Jones method.

[4] Janki Chandiwala and Shrushti Agarwal In this paper they propose a novel solution to the
problem by presenting a system that detects tiredness in the driver and issues a warning signal. In
order to measure sleepiness, this work offers a non-intrusive method that locates, tracks, and
examines both eyes and mouth, also in the dark. This method considers the eye aspect ratio (EAR)
and mouth aspect ratio (MAR).

[5] Giovanni Salzillo and Ciro Natale In this article of research they focus on the most promising
method for application in actual automobiles for detecting driver drowsiness non-intrusive
measurements like the behavioral approach. By analyzing the driver's face using a regular camera,
the created framework enables the extraction of drowsiness-related metrics. In the first stage, the
driver's face in a video frame is recognized by a face detection stage and collection of face
landmarks are established. These actions are used to identify the blinks of the facial areas. A Fuzzy
Inference System monitors correctly defined ocular variables in order to determine the level of
driver fatigue.

19
[6] Ashish Kumar and Rusha Putra In this paper they discussed the implementation of detection
system that is developed to produce more accurate results and with low cost. They suggested
developing a webcam-based system to identify driver weariness just from the facial image utilizing
image processing and machine learning methods to make the system affordable as well as
effective.

[7] V. Uma Maheswari et al. In this study, an integrated strategy that is dependent on the
PERCLOS (Eye and Mouth Closure Status) and the computation of the new suggested vector FAR
(Facial Aspect Ratio), much like EAR and MAR, is proposed. This assists in determining the
condition of closed eyes or an opening mouth, such as when yawning, and any frames that include
hand motions like nodding or covering an open mouth with the hand as part of a human's intrinsic
ability to manage drowsiness.

5. THEORITICAL ANALYSIS:

In this section, we divide the relevant work into three categories: techniques for detecting driver
sleepiness, face landmark identification algorithms, and visual object tracking algorithms.

HAAR CASCADE ALGORITHM:

The Haar cascade technique can recognize objects in photos regardless of their scale or position.
The algorithm consists of the following steps:

1. obtaining a feed of the driver's face, in this case accomplished by taking photographs of the
driver in real-time using a camera

2. Identification of the driver's face based on the photographs acquired.

3. Determining the locations of key landmarks on the detected face.

4. Using pre-trained algorithms to categorize the ratios of these discovered characteristics as

sleepiness symptoms.

5. Giving an alert when a specific score frame is exceeded.

20
 Figure 2. Flowchart of detection system

The use of HAAR cascade files, which are XML files for HAAR cascade operations, enables the
face detection characteristics. The convolution of these cascades across the whole picture results
in the detection of these characteristics throughout the entire image, the combination of which
constitutes the features we are attempting to identify.

21
The Viola-Jones face detection method uses combinations of simple Haar-Like features to classify
faces. Haar-like features are rectangular digital image features that get their name from their
similarity to Haar wavelets. The value of a two-rectangle feature is the difference between the sum
of the pixels within two rectangular regions. The regions have the same size and shape. A three-
rectangle feature computes the sum within two outside rectangles subtracted from the sum in a
center rectangle. Finally, a four-rectangle feature computes the difference between diagonal pairs
of rectangles.

We take several positive and negative samples for statistical model-based training, and we extract
various properties from these data. The statistical model parameters that are employed as particular
properties to categorize various objects are then condensed from these distinguishing
characteristics. The classification accuracy for these algorithms can be increased by adjusting these
parameters. The basic idea behind the Haar classifier is the use of Haar-like properties to identify
objects in photos. The models are kept in the repository as XML files and may be read using
OpenCV techniques. These consist of face detection models, eye detection models in the facial
areas.

Figure 3.Haar cascade classifier detection images

22
6. EXPERIMENTAL INVESTIGATION

Support Vector Machine

Support Vector Machine is regarded as a classification strategy however, it may be used to solve
both classification and regression issues. It can deal with a variety of categorical and continuous
variables with ease. SVM is a very good classification method. It is a supervised learning method
that is mostly used to categorize data into distinct categories. A collection of label data is used to
train SVM. A supervised learning algorithm examines the training data and then predicts the right
output category for a particular data-set input. To distinguish or categorize any two classes, SVM
creates a decision boundary, which is a hyperplane between them. SVM is also employed in picture
classification and object detection.

Figure 4.SVM hyperplane

SVM, a novel machine learning algorithm, has recently been demonstrated to be a viable tool for
data classification as well as pattern recognition. SVM is a form of a learning system that is built
on the idea of margin maximization.

23
The new data set, called the test set, was created to measure the accuracy of the learning model.
This is used to verify whether the model has a strong performance when predicting unseen data.
The test data set contained a total of 89 samples, with 17 open eyes, 30 blinks, and 42 closed eye.

Figure 5.Importing and installing the required packages

CNN:

In this section, we discuss the principle of evaluating the driver’s fatigue state. With CNN, we use
the angle of the eyes to evaluate the eye state. Moreover, we use the state of the single eye near
the camera to assess the state of the whole eye.

Besides, CNN also measures the state of the mouth to judge if the driver is yawning. After these
assessments, CNN merges these results and evaluates the driver’s degree of drowsiness.

1) eye status recognition

a: Recognition based on cnn: We build a CNN to recognize the eight layers of the eye state.

24
b: Recognition based on angle Owing to the CNN drawbacks (the accuracy of the eye closure
recognition by CNN is poor), we use the angle of the eye to compensate for CNN’s limitations
regarding eye closure recognition. After the CNN validates that the driver’s eye is open, we use
the angle of the eye to validate the result. A blink is the process of the eye closing and opening.
The study employs a CNN model to predict whether or not a person is drowsy depending on
whether their eyes are closed or open. The study's main objective was to maintain the CNN model's
total number of trainable parameters at 250K in order to allow the system to be used on edge or
computationally inefficient devices. The concept directly affects the automotive industry,
improving driving safety and reducing the number of fatalities brought on by drowsy driving.
Neural networks are analogous to layers. Each layer of a neural network has nodes that compute
values depending on attributes or weights. For hidden layers, the activation function while for
output layers, the activation function is either sigmoid or SoftMax.

Figure 6. CNN process

25
CLASSIFICATION MODULE

To categorize the images as alert or sleepy, we develop a Deep Learning Binary Classification
model in this module. A class label is predicted for a specific sample of input data in a classification
task in machine learning. A model will determine the optimal way to map samples of input data to
certain class labels using the training dataset. Such classification jobs with only two class labels are
referred to as binary classification.

Training:
All of the accessible photos are divided into train data (80%) and test data (20%) after data shuffling
and splitting. During training, 20 epochs are run. A portion of the training data is utilised as
validation data while the model is being trained. During validation, we only utilise 10% of the
training data. The validation data is not used to train the model; rather, it is used to assess the
model's performance at the conclusion of each epoch. When a training improvement is noted, we
employ checkpoints in our model to store the model's current state. The model weights are kept in
the checkpoint. When improvements are made, the weights are adjusted, and after the training is
complete, the best model is saved.

Evaluation:
The model's performance should be evaluated after training. Several measures, like as loss,
accuracy, precision, and recall, can be used to assess a classification model. To visualize the graphs
of the various model metrics, we utilise Matplotlib.

Evaluating test results::

By feeding the model new data, testing is done to assess the model's generalizability. Training data
and test data have already been separated from the data. In order to forecast the classes for the test
data, the model is given the 20% of the data that it does not already know. Using the test data, we
determine loss and accuracy to assess the model's performance. We also discover the model's
Precision and Recall values.

26
 accuracy = history.history['accuracy']
val_accuracy = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(len(accuracy))

plt.plot(epochs, accuracy, "b", label="trainning accuracy")

plt.plot(epochs, val_accuracy, "r", label="validation accuracy")
plt.legend()
plt.show()

plt.plot(epochs, loss, "b", label="trainning loss")

plt.plot(epochs, val_loss, "r", label="validation loss")
plt.legend()
plt.show()

Figure 7. Training accuracy

27
 Figure 8.Training loss

CODE

import numpy

from pygame import mixer

import time
import cv2

from tkinter import *

import tkinter.messagebox
# In[4]:
root=Tk()
root.geometry('500x570')
frame = Frame(root, relief=RIDGE, borderwidth=2)
frame.pack(fill=BOTH,expand=1)
root.title('KLU')
frame.config(background='light blue')
label = Label(frame, text="KLEF",bg='light blue',font=('Times 35 bold'))
label.pack(side=TOP)
28
filename = PhotoImage(file="demo.png")
background_label = Label(frame)
background_label.pack(side=TOP)

def hel():
help(cv2)

def Contri():
tkinter.messagebox.showinfo("Contributor","Stuti\n")

def anotherWin():
tkinter.messagebox.showinfo("About",'Driver Cam version v1.0\n Made Using\n-OpenCV\n-
Numpy\n-Tkinter\n In Python 3')

menu = Menu(root)
root.config(menu=menu)

subm1 = Menu(menu)
menu.add_cascade(label="Tools",menu=subm1)
subm1.add_command(label="Open CV Docs",command=hel)

subm2 = Menu(menu)
menu.add_cascade(label="About",menu=subm2)
subm2.add_command(label="Driver Cam",command=anotherWin)
subm2.add_command(label="Contributor",command=Contri)

def exitt():
exit()

29
def web():
capture =cv2.VideoCapture(0)
while True:
ret,frame=capture.read()

gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF ==ord('q'):

break
capture.release()
cv2.destroyAllWindows()

def webrec():

capture =cv2.VideoCapture(0)
fourcc=cv2.VideoWriter_fourcc(*'XVID')
op=cv2.VideoWriter('Sample1.avi',fourcc,11.0,(640,480))
while True:
ret,frame=capture.read()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
cv2.imshow('frame',frame)
op.write(frame)

if cv2.waitKey(1) & 0xFF ==ord('q'):

break

op.release()
capture.release()
cv2.destroyAllWindows()

def webdet():
capture =cv2.VideoCapture(0)
30
 face_cascade = cv2.CascadeClassifier('lbpcascade_frontalface.xml')
eye_glass = cv2.CascadeClassifier('haarcascade_eye_tree_eyeglasses.xml')
while True:
ret, frame = capture.read()

gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

faces = face_cascade.detectMultiScale(gray)
for (x,y,w,h) in faces:
font = cv2.FONT_HERSHEY_COMPLEX

cv2.putText(frame,'Face',(x+w,y+h),font,1,(250,250,250),2,cv2.LINE_AA)
cv2.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)

roi_gray = gray[y:y+h, x:x+w]

roi_color = frame[y:y+h, x:x+w]

eye_g = eye_glass.detectMultiScale(roi_gray)
for (ex,ey,ew,eh) in eye_g:
cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xff == ord('q'):
break
capture.release()
cv2.destroyAllWindows()

def webdetRec():
capture =cv2.VideoCapture(0)
face_cascade = cv2.CascadeClassifier('lbpcascade_frontalface.xml')
eye_glass = cv2.CascadeClassifier('haarcascade_eye_tree_eyeglasses.xml')
fourcc=cv2.VideoWriter_fourcc(*'XVID')
op=cv2.VideoWriter('Sample2.avi',fourcc,9.0,(640,480))

31
 while True:
ret, frame = capture.read()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray)

for (x,y,w,h) in faces:

font = cv2.FONT_HERSHEY_COMPLEX
cv2.putText(frame,'Face',(x+w,y+h),font,1,(250,250,250),2,cv2.LINE_AA)
cv2.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)
roi_gray = gray[y:y+h, x:x+w]
roi_color = frame[y:y+h, x:x+w]

eye_g = eye_glass.detectMultiScale(roi_gray)
for (ex,ey,ew,eh) in eye_g:

cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)
op.write(frame)
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xff == ord('q'):
break
op.release()
capture.release()
cv2.destroyAllWindows()

def alert():
mixer.init()
alert=mixer.Sound('beep-07.wav')
alert.play()
time.sleep(0.1)
32
 alert.play()

def blink():
capture =cv2.VideoCapture(0)
face_cascade = cv2.CascadeClassifier('lbpcascade_frontalface.xml')
eye_cascade = cv2.CascadeClassifier('haarcascade_eye.xml')
blink_cascade = cv2.CascadeClassifier('CustomBlinkCascade.xml')

while True:
ret, frame = capture.read()
gray = cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray)

for (x,y,w,h) in faces:

font = cv2.FONT_HERSHEY_COMPLEX
cv2.putText(frame,'Face',(x+w,y+h),font,1,(250,250,250),2,cv2.LINE_AA)
cv2.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)
roi_gray = gray[y:y+h, x:x+w]
roi_color = frame[y:y+h, x:x+w]

eyes = eye_cascade.detectMultiScale(roi_gray)
for(ex,ey,ew,eh) in eyes:
cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)

blink = blink_cascade.detectMultiScale(roi_gray)
for(eyx,eyy,eyw,eyh) in blink:
cv2.rectangle(roi_color,(eyx,eyy),(eyx+eyw,eyy+eyh),(255,255,0),2)
alert()
33
 cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF ==ord('q'):
break

capture.release()
cv2.destroyAllWindows()

but1=Button(frame,padx=5,pady=5,width=39,bg='white',fg='black',relief=GROOVE,command=we
b,text='Open Cam',font=('helvetica 15 bold'))
but1.place(x=5,y=104)

but2=Button(frame,padx=5,pady=5,width=39,bg='white',fg='black',relief=GROOVE,command=we
brec,text='Open Cam & Record',font=('helvetica 15 bold'))
but2.place(x=5,y=176)

but3=Button(frame,padx=5,pady=5,width=39,bg='white',fg='black',relief=GROOVE,command=we
bdet,text='Open Cam & Detect',font=('helvetica 15 bold'))
but3.place(x=5,y=250)

but4=Button(frame,padx=5,pady=5,width=39,bg='white',fg='black',relief=GROOVE,command=we
bdetRec,text='Detect & Record',font=('helvetica 15 bold'))
but4.place(x=5,y=322)

but5=Button(frame,padx=5,pady=5,width=39,bg='white',fg='black',relief=GROOVE,command=blin
k,text='Detect Eye Blink & Record With Sound',font=('helvetica 15 bold'))
but5.place(x=5,y=400)

34
 but5=Button(frame,padx=5,pady=5,width=5,bg='white',fg='black',relief=GROOVE,text='EXIT',co
mmand=exitt,font=('helvetica 15 bold'))
but5.place(x=210,y=478)
root.mainloop()
# In[ ]:

7. EXPERIMENT RESULTS

Based on the investigation of the experiment analysis the results of the face detection system are

Figure 9.Experiment code

35
 Figure 10. Interface of detection system

Non-Drowsy frames:

If the driver seems to be recognized as not being sleepy, the detected face will be highlighted using a
green-coloured recognized rectangle, and no alarm will be created. In this case, the input will be the
continuous stream of video.

36
Figure 11.Face Detection of the open eyes in facial areas

37
 Drowsy frames:

The input will be a continuous stream of video, and if the driver appears to be tired, the identified eyes
will be highlighted with a blue-colored rectangle, and the system will provide a warning. A sound will
be used as the alert (beep sound). With the sound, the intention is to alert the driver that they are asleep.
The SVM model's Haar cascade classifier is used to identify sleepiness.

Figure 12. Face Detection of closed eyes in facial area

38
 Figure 13. Detection of facial regions

8. DISCUSSION OF RESULTS

In the experimental investigation, we discussed the two methods to test the experiment using SVM
and CNN methods. SVM uses a vector to make the measurement and prediction so that the output
is generated. It takes the input through the classifier and generates the output in real-time. The
main advantage of SVM is that it is very efficient to train.

The haar cascade classifier, an SVM algorithm that yields the most accurate findings, is used to
analyze the experiment's outcomes. The outcomes generated by the suggested approach accurately
identify the movements of the face region.

39
 Figure 14. tracking stages of system

Figure 15. Confusion matrix for an open , closed, yawn and no yawn
categories

9. CONCLUSION:

Our face detection system will detect the driver’s facial areas and decide the person's drowsiness
based on the expression. This approach may be evaluated using machine learning techniques and
datasets. The sleepiness detection method, which uses live camera images to distinguish between
the driver's open and closed eyes in the facial areas, has been categorized in an effective way. With
the use of haar cascade classifier and image processing, the model hopes to offer a solution for
Driver Drowsiness Detection.

40
10. REFERENCES

[1] Ayman altameem , Ankit kumar , Ramesh chandra poonia , Sandeep kumar , and Abdul
khader Jilani saudagar “ Early Identification and Detection of Driver Drowsiness by Hybrid
Machine Learning” (2021) IEEE Access DOI:10.1109/ACCESS.2021.3131601

[2] Wanghua Deng , Ruoxue Wu “Real-Time Driver-Drowsiness Detection System Using Facial
Features” (2019) IEEE Access DOI:10.1109/ACCESS.2019.2936663

[3] Janki Chandiwala , Shrushti Agarwal “Driver’s real-time Drowsiness Detection using
Adaptable Eye Aspect Ratio and Smart Alarm System” (2021) IEEE
DOI:10.1109/ICACCS51430.2021.9441756

[4] Giovanni Salzillo; Ciro Natale; Giovanni B. Fioccola; Enrico Landolfi “Evaluation of Driver
Drowsiness based on Real-Time Face Analysis” (2020) IEEE DOI:
10.1109/SMC42975.2020.9283133

[5] Dey, Sanjay; Chowdhury, Sami Ahbab; Sultana, Subrina; Hossain, Md. Ali; Dey, Monisha;
Das, Sajal K. “Real-Time Driver Fatigue Detection Based on Facial Behaviour along with Machine
Learning Approaches” (2019) IEEE DOI: 10.1109/SPICSCON48833.2019.9065120

[6] Manu B.N “Facial Features Monitoring for Real Time Drowsiness Detection” (2016) IEEE
DOI:10.1109/INNOVATIONS.2016.7880030

[7] Ashish Kumar; Rusha Patra “Driver Drowsiness Monitoring System using Visual Behaviour
and Machine Learning” (2018) IEEE DOI: 10.1109/ISCAIE.2018.8405495

[8] Malaika Vijay∗ , Nandagopal Netrakanti Vinayak∗ , Maanvi Nunna∗ , Subramanyam Natarajan
“Real-Time Driver Drowsiness Detection using Facial Action Units” (2020) IEEE
DOI:10.1109/ICPR48806.2021.9412288

[9] Aditya Nair, Shoaib Mansoori, Rajeev Moghe, Payal Shah and Kiran Talele “Driver Assistant
System using Haar Cascade and Convolution Neural Networks(CNN)” (2019) IEEE
DOI:10.1109/ICOEI.2019.8862779

41
[10] Kartik Dwivedi, Kumar Biswa ranjan , Amit Sethi “Drowsy driver detection using
representation learning” (2014) IEEE DOI:10.1109/IAdCC.2014.6779459

[11] Riad Alharbey, Mohamed M. Dessouky “Fatigue State Detection for Tired Persons in
Presence of Driving Periods” (2022) IEEE DOI: 10.1109/ACCESS.2022.3185251

[12] Varun Chaudhary;Ziyad Dalwai;Vikram Kulkarni “Intelligent Distraction and Drowsiness

Detection System for Automobiles” (2021) IEEE DOI: 10.1109/CONIT51480.2021.9498562

[13] U Gopikrishnan;Renu Jose “DriveCare: A Real-Time Vision Based Driver Drowsiness

Detection Using Multiple Convolutional Neural Networks With Kernelized Correlation Filters
(MCNN-KCF)” (2020) IEEE DOI: 10.1109/ICCCA49541.2020.9250868

[14] V.Uma Maheswari; Rajanikanth Aluvalu; MVV Prasad Kantipudi; Krishna Keerthi

Chennam “Driver Drowsiness Prediction Based on Multiple Aspects Using Image Processing

Techniques” (2022) IEEE DOI: 10.1109/ACCESS.2022.3176451

42