FACE MASK DETECTION

USING TENSORFLOW
A PROJECT REPORT
Submitted by

AVINASH RAI
KIRAN SAVANDE
KEVIN PRAJAPATI
NIMT KHATRI

In Partial fulfillment for the award of

the degree of
BACHELOR OF TECHNOLOGY

in

COMPUTER SCIENCE AND ENGINEERING

Under the Guidance of

Prof. Harsh
Kantawala

Department of Computer Science & Engineering

Parul University, Vadodara

September, 2021
 PARUL UNIVERSITY

CERTIFICATE
This is to Certify that Project-I -Subject code 03105401 of 7th Semester entitled
“Face Mask Detection” of Group No. PUCSE has been successfully completed by

AVINASH RAI - 170303105276

KEVIN PRAJAPATI–
170303105272
KIRAN SAVANDE– 170303105440
NIMIT KHATRI– 1703031050141

under my guidance in partial fulfillment of the Bachelor of Technology (B.TECH)

in Computer Science and Engineering of Parul University in Academic Year

​
Project Guide
Prof. Harsh Kantawala

Head of Department
Prof. Amit Barve
​
ACKNOWLEDGEMENT

Behind any major work undertaken by a team there lies the

contribution of the people who helped to cross all the hurdles to
achieve our goal. It gives us immense pleasure to express our sense
of sincere gratitude towards our respected guide Prof. Harsh
Kantawala, for her persistent, outstanding, invaluable co-operation
and guidance. It is our achievement to be guided under her. She is a
constant source of encouragement and momentum that any intricacy
becomes simple. We gained a lot of invaluable guidance and prompt
suggestions from her during the entire project work. We will be
indebted of her forever and we take pride to work under her.
We also express our deep sense of regards and thanks to Prof. Amit
Barve Head of Computer Science & Engineering Department. We
feel very privileged to have had their precious advice, guidance and
leadership.
Last but not the least, Our humble thanks to the Almighty God.

Date: 17/9/21​

ABSTRACT

The primary objective is to design and manufacture a system to

perform the task of automating face mask detection and to
reduce the wastage of human efforts, time and cost. This system
is integrated with embedded systems for application in malls to
ensure that public safety guidelines are followed. The system will
detect the face of a human being and check if the person is
wearing a mask or not, captured in video frames. Then the
system will send an invoice on the executive manager’s/security
staff device on its own and then appropriate fines will be
collected from those people. Our face mask detection system will
be based on IOT which will reduce the paperwork as well as
chances of any error or ambiguity. This will reduce the load on
security staff as well.
 KEYWORDS: Raspberry Pi, Machine Learning, OpenCV,
Face Detection, Face Recognition, Masked Face,
MobileNetV2 Architecture.
​
CONTENTS

SR Title Page No

Chapter : 1 1.1 Introduction 6

1.1.1 Problem Statement 6

1.1.2 Objectives 6

1.1.3 Scope 7

Chapter : 2 2.1 Literature Review 8

2.2.1 Critical Evaluation of Journal 8-20

Papers

Chapter : 3 3.1 Research Methodology 21

3.1.1 ​Introduction ​to

​Research 21
Methodology

3.1.2 Working 23

3.1.3 Advantages 24

3.1.4 Disadvantages 24

3.1.5 Methodology 24

Chapter : 4 Reference 25

​
LIST OF TABLES
 LIST OF FIGURES
Figure No Figure Description Page No
Figure 3.1.1 Training Face Mask Detector
Figure 3.1.2 Apply Face Mask Detector
Figure 3.1.3 Invoice System
​
CHAPTER 1

1.1 INTRODUCTION

Recently, we are suffering from an untreatable disease COVID-19, so

this approach is very useful to slow down the spread of disease in the
public and also ensure safety of others. Recognition from faces is a
popular and significant technology in recent years. Face alterations and
the presence of different masks make it too much challenging. The
primary concern to this work is about facial masks, and especially to
enhance the recognition accuracy of different masked faces. The
occluded face detection problem has been approached using
MobileNetV2 Convolutional Neural Network Architecture. A feasible
approach has been proposed that consists of detecting the facial regions
with the help of OpenCV. And finally applying a face mask classifier to
each face ROI(Region of Interest) to determine whether that face has a
mask or not.
In this chapter we have mentioned in detail the problem statement,
objectives and scope of our project. Problem
definition defines our inspiration to build this software. Objectives are
the predefined goals that we want to achieve from this system. Scope is
the range or extent of our system that can be achieved in a limited time.
 1.1.1 PROBLEM STATEMENT

Everyday more and more people are infected by this deadly virus called
Corona Virus(Covid-19) and due to that other humans are also affected
even when they take all precautions. Also there’s no stop to the number
of cases increasing gradually. This problem should be fixed. Our system
aims to provide solutions to these problems using Machine Learning,
Deep Learning and Internet of Things.
1.1.2 OBJECTIVES

1) We intend to design a system which will reduce man power and also
ensure safety of people as the application generates invoice and fine is
taken, as result people don't forget to wear a mask so as to avoid a fine
amount.
2) The webcam would sense if any human has a mask on their face by
implementing a

3) face mask classifier in MobileNetV2 architecture in which it needs

to be fine-tuned with pre-trained ImageNet weights.

4) After detecting the faces, with the help of OpenCV , each face ROI is
extracted.

5) At last, a face mask classifier is applied on each face ROI to check

whether there is a face mask or not.

6) If there’s no face mask , then the system will automatically generate an

invoice by

7) extracting facial images and send it to the executive manager/ security

staff’s device at the mall.
​
12
 8) Thus resulting in low energy wastage and time's been saved.

1.1.3 SCOPE

1) Our System will be useful as a helping tool for increasing safety

level and reducing man-power. Also reducing stress for staff
members at the mall.
2) The system will improve as per feedback received from higher
authorities.
3) Implemented on a large scale would help in saving time in large
organizations and Companies
​
12

CHAPTER 2

2.1 LITERATURE REVIEW

In this chapter, we have given our critical evaluation &

summary of all research papers that we read related to
our project.

2.1.1 Critical Evaluation of Journal Papers

Paper 1: A hybrid deep transfer learning model with machine

learning methods for face mask detection in the era of the COVID-
19 pandemic

In the given research paper faculties Mohamed Loey and his team
have introduced a mask face detection model that is based on deep
 transfer learning and classical machine learning classifiers. The
proposed model can be integrated with surveillance cameras to impede
the COVID-19 transmission by allowing the detection of people who are
not wearing face masks. The model is integration between deep transfer
learning and classical machine learning algorithms. They have used
deep transfer learning for feature extractions and combined it with three
classical machine learning algorithms to get highest accuracy and
consume least time in the process of training and detection. A hybrid
model using deep and classical machine learning for face mask detection
is presented in which it consists of two components, i.e feature
extraction using Resnet50 and classification process of face masks using
decision trees, Support Vector Machine (SVM), and ensemble
algorithm. Also three datasets are selected 1) RMFD, 2) SMFD and 3)
LFW. The SVM classifier achieved 99.64% testing accuracy in RMFD.
In SMFD, it achieved 99.49%, while in LFW, it achieved 100% testing
accuracy. The corona virus epidemic has given rise to an extraordinary
degree of worldwide scientific cooperation. As a result AI based on and
ResNet-50, and ResNet-18 are versions of ResNet to get rid of the
problem of vanishing gradients that have their specific residual block.
ResNet-50 with 50- layers are deep, start with a convolution layer, and
end with a fully- connected layer, and in between followed by 16
residual bottleneck blocks each block has three layers of convolution
layer. In classification, the last layer in ResNet-50 was removed and
replaced with three traditional machine learning classifiers Support
vector machine (SVM), decision tree, and ensemble , quickly detect
infections, the AI ’s power is being exploited to address the Covid-
19.Machine learning and Deep Learning can help to fight Covid-19 in
many ways. To better understand infection rates and to trace , they used
Ensemble methods are K-Nearest Neighbors Algorithm (k-NN), Linear
Regression and Logistic Regression . The first component is deep
transfer learning (ResNet50) as a feature extractor. With three traditional
machine learning classifiers (Support vector
​
12
machine (SVM) decision tree, and ensemble) to improve our model
performance. The main contribution of this research is to construct
SVM, decision trees, and ensemble that do not over fit the training
process. We will use such methods and algorithms to provide and
deliver results with maximum accuracy. With the proposed system,
higher accuracy can be achieved. The SVM classifier achieved the
highest accuracy possible with the least time consumed in the training
process. The SVM classifier in RMFD achieved 99.64% testing
accuracy. In SMFD, it gained 99.49%, while in LFW, it reached 100%
testing accuracy.

1) The model is integration between deep transfer learning and classical

machine learning algorithms.

2) They have used deep transfer learning for feature extractions and
combined it with three classical machine learning algorithms
(Decision trees, SVM, and Ensemble).

3) Deep transferring learning ResNet50 is used for the feature

extraction phase while the traditional machine learning model is
used in the training, validation, and testing phase.

4) Also three datasets are used : A dataset of RMFD with real face
masks for (training, and testing phases), will be referred to DS1. A
dataset of SMFD with fake masks for (training, and testing phases),
will be referred to DS2. A combined dataset from DS1, and DS2
for (training, and testing phases), will be referred to DS3. A dataset
of LFW with simulated face masks for (testing), will be referred to
DS4.

Paper 2: Retina Face Mask : A Face Mask Detector

 In this paper, authors Mingjie Jiang, Xinqi Fan, Hong Yan have
proposed a novel face mask detector, Retina Face Mask, which is able to
detect face masks and contribute to public healthcare. To the best of our
knowledge, Retina Face Mask is one of the first dedicated face mask
detectors. In terms of the network architecture, Retina Face Mask uses
multiple feature maps and then utilizes feature pyramid network (FPN)
to fuse the high-level semantic information to achieve better detection,
we propose a context attention detection head and a cross-class object
removal algorithm to enhance the detection ability.
One major protection method for people is to wear masks in public
areas. Retina Face Mask, which is a high- accuracy and efficient face
mask detector to proposed Retina Face Mask is a one-stage detector,
which consists of a feature pyramid network to fuse high-level semantic
information with multiple feature maps, and a novel context attention
module to focus on detecting face masks. A novel cross-class object
removal algorithm to reject predictions with low confidences and the
high intersection of union. To achieve better detection, propose a context
attention detection head and a cross-class object removal algorithm to
enhance the detection ability. Traditional object detection uses a multi-
step process, A well-known detector is the Viola-Joins detector, which is
able to achieve real- time detection. Rather than using handcrafted
features, deep learning based detector demonstrated excellent
performance recently, due to its robustness and high feature extraction
capability. In Retina Face Mask, we adopt ResNet as a standard
backbone, but also include Mobile Net as a backbone for comparison
and for reducing computation and model size in deployment scenarios
with limited computing resources. We will use such methods and
algorithms to provide and deliver results with maximum accuracy. With
​
12

the proposed system, higher accuracy can be achieved. Although

different network components are able to increase the detection
performance, the biggest accuracy improvement is achieved by ResNet
backbone.

Paper 3: FACIAL MASK DETECTION USING SEMANTIC

SEGMENTATION

Authors Toshanlal Meenpal ; Ashutosh Balakrishnan ; Amit Verma

We aim to design a binary face classifier which can detect any face
present in the frame irrespective of its alignment. We present a method
to generate accurate face segmentation masks from any arbitrary size
input image. Beginning from the RGB image of any size, the method
uses Predefined Training Weights of VGG – 16 Architecture for feature
extraction. Proposed model has also shown great results in recognizing
non-frontal faces. Along with this it is also able to detect multiple facial
masks in a single frame. Face Mask Detection should be reliable as well
as accurate so that our desired output can be achieved. Initially
researchers focused on edge and gray value of face image, was based on
pattern recognition model, having a prior information of the face model.
A da boost was a good training classifier. The face detection technology
got a break through with the famous Viola Jones Detector, which greatly
improved real time face detection. We propose this paper with twin
objective of creating a Binary face classifier which can detect faces in
any orientation irrespective of alignment and train it in an appropriate
neural network to get accurate results.

We will use such methods and algorithm to provide and deliver

results with maximum accuracy , reliability &more speedy. We were
able to generate accurate face masks for human objects from RGB
channel images containing localized objects. Proposed network can
detect non frontal faces and multiple faces from single image. The
method can find applications in advanced tasks such as facial part
detection. All the experiments have been performed on Multi Human
Parsing Dataset containing about 5000 images, each with at least two
persons. Out of these, 2500 images were used for training and validation
while the remaining where used for testing the model. The designed
 FCN semantically segments out the facial spatial location with a specific
label. Furthermore, proposed model has also shown great results in
recognizing non-frontal faces. Along with this it is also able to detect
multiple facial masks in a single frame. The post processing provides a
large boost to pixel level accuracy. The mean pixel level accuracy for
facial masks: 93.884%.

Paper 4: DETECTING MASKED FACES IN THE WILD WITH

LLE-CNNs

In the given research paper, authors Shiming Ge ; Jia Li ; Qiting Ye ;

Zhao Luo have mainly focused on the development of a system or we
could say an immediate identification of the person on image or video
stream with the help of MAFA dataset. In particular, the detection of
masked faces,which can be very helpful for many applications and event
analysis. Based on the dataset, we propose LLE-CNNs for masked face
detection which consists of 3 modules. The proposed approach consists
of a proposal module, embedding module and verification module.
Analyze data using locally linear embedding algorithms and dictionaries
to deliver results with maximum accuracy. It ensures that each image
contains at least one face occluded by various types of masks, while the
six main attributes of each masked face, including locations of faces,
eyes and masks, face orientation and mask type
​
12

and are manually annotated and cross-checked by nine subjects.

Efficient low-Resolution face recognition via Bridge Distillation ,it was
having low accuracy. Presenting a dataset of masked faces that can be
used as an additional training source for developing new faces detectors.
Proposing LLE-CNNs for masked face detection, which outperforms 6
state- of-the-art face detectors in detecting masked faces and then
conducting comprehensive analysis on the key challenges in masked
face detection, which may be helpful for developing new face detectors
in the future. We will use such methods , modules and algorithm to
provide and deliver results with maximum accuracy. The proposed
model achieves the best performance in all settings. This may imply that
the data-driven framework may be a feasible solution in finding robust
and effective features for masked face detection. By Using Various
Algorithms and Deep Learning we can easily predict it with the one with
mask and without a mask. This dataset would be analyzed using LLE-
CNNs algorithms to deliver results with maximum accuracy .We
conduct a comprehensive benchmark of our approach and 6 state-of-the-
art face detectors on MAFA. The main objective of this benchmark is to
find out how these face detectors perform in detecting masked faces and
in which cases they will probably succeed and fail. By analyzing the
experimental results of these face detectors, we then conduct a
comprehensive discussion on how to develop face detectors with the
capability of handling faces occluded by various kinds of masks. In the
benchmarking process, we split the MAFA into two subsets, including a
training set and a testing set. The training set consists of 25, 876 images
with 29, 452 masked faces that are randomly selected from MAFA,
while the testing set contains the rest 4, 935 images with 6, 354 masked
faces. In this study, we only report the performances of our approach
and the six face detectors on the testing set, while such results can be
directly used to facilitate their comparisons with new models trained on
the same training set of MAFA in the future.

Paper 5 : Detecting Face with Densely Connected Face proposal

Network.

In This Paper, the author Shifeng Zhang, Zhen lie, Hailin Shi, Stan
Z.li, Xiaobo Wang mainly focused on Face Detection is one of the
fundamental problem in computer vision. This paper mainly shows the
former tends to great efficiency while latter cares more about high
accuracy. To perform well on both speed and accuracy, one natural idea
is to combine the advantages of them. Therefore ,cascade CNN based
 methods are proposed that puts features learned by CNN into cascade
framework so as to boost the performance and keep efficient. In this,
they develop a state-of- the-art face detector with CPU real-time speed.
Core idea is to Shrink the input image and focus on detecting small
faces. Developing a novel face detector, designing a light-weight but
powerful convolutional network for face detection. Here FDDB and
AFW dataset is used to analyze model. For understanding DCFPN ,
each componenent one after another is ablate to examine the affect in
final performance. Efficient low-Resolution face recognition via Bridge
Distillation ,it was having low accuracy. Presenting a dataset of faces
that can be used as an additional training source for developing new
faces detectors. Analyzing model by FDDB dataset. Firstly we replace
fair L1 with Smooth L1 loss. Meantime the target of regression is the
same as RPN. Finally we take the place of DCCL with other 4
convolutional layers ; CNN , RDCL , DCDL , PASCAL. Dataset FDDB
Dataset The proposed model achieves the best performance in all
settings. This may imply that the a
​
12

feasible solution in finding robust and effective features for masked face
detection. In this paper, we propose a novel face detector with real-time
speed on the CPU devices as well as high performance. On the one
hand, our DCFPN has a light weight but-powerful framework that can
well incorporate CNN features from different sizes of receptive field at
multiple levels of abstraction. On the other hand, we use the dense
anchor strategy and propose the fair L1 loss function to handle small
faces well. The state-of-the-art performance on three challenge datasets
shows its ability to detect faces in the uncontrolled environment. The
proposed detector is very fast, achieving 30 FPS to detect faces bigger
than 40 pixels on CPU and can be accelerated to 250 FPS on GPU for
the VGA-resolution images.

Paper 6 : FACE RECOGNITION SYSTEM

 In This Paper, the author Yang Li mainly focused on With the
development of deep learning and the introduction of deep
convolutional neural networks, the accuracy and speed of face
recognition have made great strides, the results from different networks
and models are very different. In this paper, facial features are extracted
by merging and comparing multiple models, and then a deep neural
network is constructed to train and construct the combined features. In
this way, the advantages of multiple 4 models can be combined to
mention the recognition accuracy. After getting a model with high
accuracy, we build a product model. This article compares the pure-
client model with the serverclient model, analyzes the pros and cons of
the two models, and analyzes the various commercial products that are
required for the server-client model. It has high potential to be used in
government, public facilities, security, e- commerce, retailing, education
and many other fields. Deep neural network, face recognition, server-
client model, business model, deep multi-model fusion, convolutional
neural . Deep learning refers to a set of algorithms that solve various
problems such as images and texts by using various machine learning
algorithms in multi-layer neural networks. Deep learning can be
classified as a neural network from the general category, but there are
many changes in the concrete realization. How to accurately and
effectively identify people has always been an interesting topic, both in
research and in industry. With the rapid development of artificial
intelligence in recent years, facial recognition gains more and more
attention. Compared with the traditional card recognition, fingerprint
recognition and iris recognition, face recognition has many advantages,
including but limit to non-contact, high concurrency, and user friendly .
Artificial Neural Network (ANN), Convolutional neural network
(CNN), Representation-based methods : The basic idea is to convert
two-dimensional face input into another space, and then use statistical
methods to analyze face patterns, such as Eigenface, Fisher face, and
SVM. The first time real adaboost applied to object detection, and
 proposed a more mature and practical multi-face detection framework,
the nest structure mentioned on the cascade structure improvements also
have good results. The traditional card recognition, fingerprint
recognition and iris recognition, face recognition has many advantages,
including but limit to non-contact, high concurrency, and user friendly.
We proposed to build a high performance, scalable, agile, and low cost
face recognition system. We divide the proposed approach into several
small sub projects. First, we studied neural network and convolutional
neural network. Based on the theory of deep learning, we built the
Siamese network which will
​
12

train the neural network based on similarities. Then we examine and

compare the available open source data set, we chose ORL dataset and
trained the model with GPU. The model will take a human face image
and extract it into a vector. Then the distance between vectors are
compared to determine if two faces on different picture belongs to the
same person.

Paper 7 : Face Detection: A Survey

In This Paper, the author Erik Hjelm , Boon Kee Low mainly
focused on comprehensive and critical survey of face detection
algorithms. Face detection is a necessary first-step in face recognition
systems, with the purpose of localizing and extracting the face region
from the background. It also has several applications in areas such as
content-based image retrieval, video coding, video conferencing, crowd
surveillance, and intelligent human–computer interfaces. However, it
was not until recently that the face detection problem received
considerable attention among researchers. The human face is a dynamic
object and has a high degree of variability in its apperance, which makes
face detection a difficult problem in computer vision. A wide variety of
techniques have been proposed, ranging from simple edge- based
algorithms to composite high-level approaches utilizing advanced
pattern recognition methods. The human face is a dynamic object and
has a high degree of variability in its apperance, which makes face
detection a difficult problem in computer vision. A wide variety of
techniques have been proposed, ranging from simple edge-based
algorithms to composite high-level approaches utilizing advanced
pattern recognition methods. A high degree of variability in its
apperance, which makes face detection a difficult problem in computer
vision. The face detection problem received considerable attention
among researchers. The human face is a dynamic object and has a high
degree of variability in its apperance, which makes face detection a
difficult problem in computer vision, so a wide variety of techniques
have been proposed, ranging from simple edge-based algorithms to
composite high-level approaches utilizing advanced pattern recognition
methods. A PDBNN is a classification neural network with a
hierarchical modular structure. Training is performed with DBNN
learning rules, which means that the teacher only tells the correctness of
the classification (no exact target values) and LUGS (locally
unsupervised globally supervised) learning is applied. Each subnet is
trained individually with an unsupervised training algorithm K-mean
and vector quantization or the EM algorithm. A wide variety of
techniques have been proposed, ranging from simple edge-based
algorithms to composite high-level approaches utilizing advanced
pattern recognition methods. Face detection is one of the visual tasks
which humans can do effortlessly. However, in computer vision terms,
this task is not easy. The solution to the problem involves segmentation,
extraction, and verification of faces and possibly facial features from an
uncontrolled background. As a visual frontend processor, a face
detection system should also be able to achieve the task regardless of
illumination, orientation, and camera distance. We have presented an
extensive survey of feature-based and image- based algorithms for face
detection, together with a brief presentation of some of the application
areas.Feature-based methods are applicable for real-time systems where
 color and motion is available. Since an exhaustive multiresolution
window scanning is not always preferable, feature-based methods can
provide visual cues to focus attention. Image- based approaches are the
most robust techniques for
​
12

processing gray-scale static images. The most important application for

face detection is still as a preprocessor in face recognition systems. For
offline processing, face detection technology has reached a point where
the detection of a single face in an image with fair resolution (typical
for a face recognition system) is close to being a solved problems.

Paper 8 : Facial Recognition using OpenCV

In This Paper, the author Shervin EMAMI, Valentin Petruț SUCIU

mainly focused on Face detection and recognition has transcended from
an esoteric to a popular area of research in computer vision and one of
the better and successful applications of image analysis and algorithm
based understanding. Because of the intrinsic nature of the problem,
computer vision is not only a computer science area of research, but also
the object of neuro-scientific and psychological studies, mainly because
of the general opinion that advances in computer image processing and
understanding research will provide insights into how our brain work
and vice versa. Application that would allow user access to a particular
machine based on an in-depth analysis of a person’s facial features.
Because of the intrinsic nature of the problem, computer vision is not
only a computer science area of research, but also the object of neuro-
scientific and psychological studies, mainly because of the general
opinion that advances in computer image processing and understanding
research will provide insights into how our brain work and vice versa.
Over the past decade face detection and recognition have transcended
from esoteric to popular areas of research in computer vision and one of
 the better and successful applications of image analysis and algorithm
based understanding. Apply various image pre-processing techniques to
standardize the images that you supply to a face recognition system.
Most face recognition algorithms are extremely sensitive to lighting
conditions, so that if it was trained to recognize a person when they are
in a dark room. OpenCV has the advantage of being a multi-platform
framework; it supports both Windows and Linux, and more recently,
Mac OS X. CXCORE namespace contains basic data type definitions,
linear algebra and statistics methods, the persistence functions and the
error handlers. CVAUX namespace is described in OpenCV's
documentation as containing obsolete and experimental code. ML
namespace contains machine-learning interfaces. HighGUI namespace
contains the basic I/O interfaces and multi-platform windowing
capabilities. CVCAM namespace contains interfaces for video access
through DirectX on 32-bit Windows platforms. Image quality is taken
into consideration, there is a plethora of factors that influence the
system’s accuracy. It is extremely important to apply various image pre-
processing techniques to standardize the images that you supply to a
face recognition system. Most face recognition algorithms are extremely
sensitive to lighting conditions, so that if it was trained to recognize a
person when they are in a dark room, it probably won’t recognize them
in a bright room, etc. Haar Cascade classifier. Face recognition however
is much less reliable than face detection, with an accuracy of 30-70% in
general. Face recognition has been a strong field of research since the
1990s, so when image quality is taken into consideration, there is a
plethora of
factors that influence the system’s accuracy. It is extremely important to
apply various image pre-
​
12

processing techniques to standardize the images that you supply to a

face recognition system with OpenCV. To improve the recognition
performance, there are MANY things that can be improved here, some
of them being fairly easy to implement. For example, you could add
color processing, edge detection, etc. You can usually improve the face
recognition accuracy by using more input images, at least 50 per person,
by taking more photos of each person, particularly from different angles
and lighting conditions. If you can’t take more photos, there are several
simple techniques you could use to obtain more is that at the heart of the
algorithm, it is matching images by basically doing the equivalent of
subtracting the testing image with a training image to see how similar
they are. This would work fairly well if a human performed it, but the
computer just thinks in terms of pixels and numbers. So if you imagine
that it is looking at one pixel in the test image, and subtracting the gray
scale value of that pixel with the value of the pixel in the EXACT same
location of each training image, and the lower the difference then the
better the match.

Paper 9 : A Survey paper for Face Recognition Technologies

In This Paper, the author Kavita , Ms. Manjeet Kaur mainly focused
on The biometric is a study of human behavior and features. Face
recognition is a technique of biometric. Various approaches are used for
it. A survey for all these techniques is in this paper for analyzing various
algorithms and methods. Face recognition is emerging branch of
biometric for security as no faces can be defeated as a security
approach. So, how we can recognize a face with the help of computers
is given in this paper. Face recognition is emerging branch of biometric
for security as no faces can be defeated as a security approach. Early
face Recognition algorithms used simple geometric models, but the
recognition process has now matured into a science of sophisticated
mathematical representations and matching processes. Major
advancements and initiatives in the past ten to fifteen years have
propelled face recognition technology into the spotlight. Face
recognition can be used for both verification and identification. Over the
past few decades now enable similar recognitions automatically. Early
 face recognition algorithms used simple geometric models, but the
recognition process has now matured into a science of sophisticated
mathematical representations and matching processes. There are some
techniques that identify, parameterize and analyze linear subspaces.
Other than linear subspaces there are some statistical face recognition
techniques which are based on non-linear subspaces. Hybrid learning
algorithm to train the RBF Neural Networks: Resting bitch face that
reconigsed the Expression of the person. Single Layer adaptive NN :A
single layer adaptive NN (one for each person) for face recognition,
expression analysis and face verification. Multilayer Perceptron (MLP):
face recognition system with neural networks present results with a
small number of classes the first 50 principal components of the face
images were extracted and reduced to five dimensions using auto
associative neural network. Self-Organizing map (SOM) it provides
dimensionality reduction and invariance to minor changes in the face
image sample. Artificial Neural Network (ANN) is a most successful
tool for pattern recognition problems. Image quality is taken into
consideration, there is a plethora of factors that influence the system’s
accuracy. It is extremely important to apply various image pre-
processing techniques to standardize the images that you supply to a
face recognition system. Most face recognition
​
12

algorithms are extremely sensitive to lighting conditions, so that if it

was trained to recognize a person when they are in a dark room, it
probably won’t recognize them in a bright room, etc. Template based
Methods ; Support Vector Machine (SVM)solve two class pattern
recognition problem. Appearance-based approaches for face recognition
like PCA, LDA, and probabilistic subspace view a 2D face image as a
vector in image space. Face recognition has been an active research area
over last 40 years. The face recognition research has several disciplines
such as image processing, machine learning approach, pattern
recognition, computer vision, and neural networks. Classification is the
main problem. In the process of face recognition it includes, to train the
face images from the known individuals and then to classify the newly
coming test images into one of the classes. The problem of face
recognition is easily solved by Humans where limited memory can be
the main problem. The problems or limitations for a machine learning
face recognition. Facial expression change, Illumination variation ,
Ageing , Pose change , Scaling factor (i.e. size of the image) , Frontal
vs. profile , Presence and absence of spectacles, beard, mustache etc. ,
Occlusion due to scarf, mask or obstacles in front. In this paper, we
presented some major issues on face recognition. These are as follows:
Consequently, face recognition in video possesses more challenges to
the current face recognition systems. Use of three dimensional face
image models has been suggested as a way to compensate for low
resolution, low dimension, poor contrast and non-frontal pose. By the
way of constructing a 3D face model from multiple non-frontal frames
in a video, and then generating a frontal view from the derived 3D
model, and finally using a 2D face recognition algorithm to recognize
the synthesized frontal view, the spatio-temporal information can be
fully employed. Meantime, it will help solve the problem of occlusion,
pose variance and illumination issues caused by video frame's poor
quality.

Paper 10 : Efficient Masked Face Recognition Method during the

COVID-19 Pandemic.

In This Paper, the author Walid Hariri mainly focused on we propose

a reliable method based on discard masked region and deep learning
based features in order to address the problem of masked face
recognition process. The COVID-19 is an unparalleled crisis leading to
huge number of casualties and security problems. In order to reduce the
spread of corona virus, people often wear masks to protect themselves.
This makes the face recognition a very difficult task since certain parts
of the face are hidden. Face Mask Detection should be reliable as well as
 accurate so that our desired output can be achieved. However, wearing
the mask face causes the following problems: i) fraudsters and thieves
take advantage of the mask, stealing and committing crimes without
being identified. ii) community access control and face authentication
are become very difficult tasks when a grand part of the face is hidden
by a mask iii) existing face recognition methods are not efficient when
wearing a mask which cannot provide the whole face image for
description iv) exposing the nose region is very important in the task of
face recognition since it is used for face normalization [18], pose
correction [14], and face matching [9]. Due to these problems, face
masks have signficantly challenged existing face recognition methods.
We start by localizing the mask region. To do so, we apply a cropping in
order to obtain only he informative regions of the masked face (i.e.
forehead and eyes ones). Next, we describe the selected regions using
deep
​
12

learning model. This strategy is more suitable in real-world applications

comparing to restoration approaches. Recently, some works have
applied a supervised learning on the missing region to restore them such
as in [3]. This strategy, however, is a difficult and highly time-
consuming process. We will use such methods and algorithm to provide
and deliver results with maximum accuracy , reliability & more speedy.
Face recognition, COVID-19, Masked face, Deep learning In real-world
scenarios (i.e. unconstrained environments), human faces might be
occluded by other objects such as facial mask. This makes the face
recognition process a very challenging task. Consequently, current face
recognition methods will easily fail to make an efficient recognition.
The proposed method improves the generalization of face recognition
process in the presence of the mask. To accomplish this task, we
proposed a deep learning based method and quantization based
technique to deal with the recognition of the masked faces. The
proposed method can also be extended to richer applications such as
violence video retrieval and video surveillance. The proposed method
achieved a high recognition performance. For the best of our knowledge,
this is the rest work that addresses the problem of masked face
recognition during COVID–19 pandemic. It is worth stating that this
study is not limited to this pandemic period since a lot of people are self-
aware constantly, they take care of their Page 9/13 health and wear
masks to protect themselves against pollution and to reduce other
pathogens transmission. The used faces belong to the Real-World-
Masked-Face-Dataset. This dataset is freely available to industry and
academia. It is available at this link: https://github.com/X-
zhangyang/Real-World-MaskedFace-Dataset.

Paper 11 : FACE MASK DETECTOR USING DEEP LEARNING

(PYTORCH) & COMPUTER VISION (OPEN CV)

In This Paper, the author Walid Hariri mainly focused To identify

the person on image/video stream wearing face mask with the help of
computer vision and deep learning algorithm by using the PyTorch
library. The study analyses a set of video streams/images to identify
people who are compliant with the government rule of wearing medical
masks. This could help the government to take appropriate action
against people who are non-compliant. Face Mask Detection should be
reliable as well as accurate so that our desired
output can be achieved. Based on ImageNet dataset MobileNetV2
outperforms MobileNetV1 and ShuffleNet (1.5) with comparable model
size and computational cost.
1) Train Deep learning model (MobileNetV2).

2) We will use such methods and algorithm to provide and deliver results
with
3) maximum accuracy , reliability &more speedy.
4) Pytorch & Open Vision (OpenCV).
 The current study used OpenCV, Pytorch and CNN to detect whether
people were wearing face masks or not. The models were tested with
images and real-time video streams. Even though the accuracy of the
model is around 60%, the optimization of the model is a continuous
process and we are building a highly accurate solution by tuning the
hyper parameters. MobileNetV2 was used to build the mobile version of
the same. This specific model could be used as a use case for edge
analytics. MobileNetV2 was chosen as an algorithm to build a model
that could be deployed ona mobile device. A customized fully connected
layer which contains four sequential layer​
12

on top of the MobileNetV2 model was developed. The layers are

1) Average Pooling layer with 7×7 weights.

2) Linear layer with ReLu activation function.

3) Dropout Layer.
4)Linear layer with Softmax activation function
with the result of 2 values. The final layer softmax
function gives the result of two probabilities each
one
represents the classification of “mask” or “not mask”. The data set has
been divided into two sets, likely a training and validation set. The
accuracy of image classifier over the training set vs validation.

Paper 12 : Face Detection using Deep Learning: An Improved Faster

RCNN Approach

In This Paper, the author Walid Hariri mainly focused on new face
detection method by extending the state-of- the-art Faster R-CNN
algorithm. Their scheme improves the existing faster RCNN scheme by
combining several important strategies, including feature concatenation,
hard negative mining, and multi-scale training, etc. They conducted a
extensive set of experiments to evaluate the proposed scheme on the
well-known Face Detection Dataset and Benchmark (FDDB), and
achieved the state-of-the-art performance (ranking the best among all
the published approaches. The main results are that they have improved
the state-of-the-art faster RCNN framework by combining a number of
strategies, including feature concatenation, hard negative mining, multi-
scale training, model pretraining, and proper calibration of key
parameters. As a consequence, the proposed scheme obtained the state-
of-the-art face detection performance, making it the best model in terms
of ROC curves among all the published methods on the FDDB
benchmark. Face detection is one of the important key steps towards
many subsequent face-related applications, such as face verification,
face recognition and face clustering, etc. Early research studies in the
literature were mainly focused on extracting different types of hand-
crafted features with domain experts in computer vision, and training
effective classifiers for detection and recognition with traditional
machine learning algorithms. Such approaches are limited in that they
often require computer vision experts in crafting effective features and
each individual component is optimized separately, making the whole
detection pipeline often suboptimal. That's why in this paper they have
came up with new face detection method by extending the state-of-the-
art Faster R-CNN algorithm . To address the first problem, much effort
has been devoted to coming up with more complicated features like
HOG, SIFT, SURF and ACF. Others aimed to speed up the feature
selection in a heuristic way. The well known OpenCV C++ Library took
SVM as the classifier in its face detector. Other approaches, such as
random forest, have also been attempted . In addition, much work has
been done to improve the Faster R-CNN architecture. Also, joint
training conducted on CNN cascade, region proposal network (RPN)
and Faster R-CNN has realized end-to-end optimization. Wan et al
combined Faster R-CNN face detection algorithm with hard negative
mining and ResNet and achieved significant boosts in detection
performance on face detection benchmarks like 1) Feature
Concatenation 2)Hard Negative Mining and 3)Multi-Scale Training.
 Their methodology follows the similar deep learning framework of
Faster RCNN, which has been shown to be a state-of-the-art deep
learning scheme for generic object detection. For
​
12

that, they have trained the CNN model of Faster RCNN using the
WIDER FACE dataset . Then further use the same dataset to test the
pre-trained model so as to generate hard negatives. The hard negatives
are fed into the network as the second step of our training procedure.
The resulting model will be further fine-tuned on the FDDB dataset.
During the final fine-tuning process, they have applied the multi-scale
training process, and adopt a feature concatenation strategy to further
boost the performance of our model. As a final optional step, they have
converted the resulting detection bounding boxes int ellipses as the
regions of human faces are more elliptical than rectangular. The concept
of uploading and training dataset on ImageNet and fine tuning process is
relevant to our methodology. 1) Faster RCNN 2) Pretrained VGG16
selected as backbone CNN network, deep learning framework .. Then,
they conducted an extensive set of experiments on the well-known
FDDB testbed for face detection benchmark, and achieved the state-of-
the-art results which ranked the best among all the published methods.
1) Proposed a new face detection method by extending the state-of-the-
art Faster R-CNN algorithm . 2) Improved the existing faster RCNN
scheme by combining several important strategies, including
feature concatenation, hard negative mining, and multi-scale training.3)
Conducted an extensive set of experiments to evaluate the proposed
scheme on the well-known Face Detection Dataset and Benchmark
(FDDB) and achieved the state-of-the-art performance (ranking the best
among all the published approaches).
​
12
 3.1RESEARCH METHODOLOGY

CHAPTER 3
​

This chapter deals with the research and methodology to do in the

project work. It shows detailed and deep insights into the
experimentation associated with the project. Also, future working of our
project is presented.

3.1.1 Introduction of Research Methodology

We have observed that even after the government and authorities said
to take care of our safety, there are people who roam around like
there’s no virus and few of them don't even wear a mask. Even in
malls, after lockdown if there is any sale all rushes like everything is
normal. So we have come up with a solution which may help in
reducing the spread of the coronavirus.

3.1.2 Working

First we’ll review the dataset to train our custom face mask detector .
We’ll use this Python script to train a face mask detector and review
the results.

After training COVID-19 face mask detector, we’ll proceed to implement

two more additional Python scripts used to:

1) Detect COVID-19 face masks in images.

2) Detect face masks in real-time video streams.

 In order to train a custom face mask detector, we need to break our project
into two distinct phases,
1) Training: Here we’ll focus on loading our face mask detection
dataset from disk, training a model (using Keras/TensorFlow)
on this dataset, and then serializing the face mask detector to
disk

2) Deployment: Once the face mask detector is trained, we can then

move on to loading the mask detector, performing face detection,
and then classifying each face as with_mask or without_mask

To create this dataset:-

I. This dataset consists of no. of images belonging to two classes:

a) with_mask:" no. of images"

b) without_mask:"no. of images images"

1) Taking normal images of faces

2) Then creating a custom computer vision Python script to add face
masks to them, thereby creating an artificial (real-world
applicable) dataset.
​
12

Facial landmarks allow us to automatically infer the location of facial

structures, including:
a).Eyes
b.Nose
c). Eyebrows
d.)
Jawline
Then we’ll create three Python scripts:

a. train_mask_detector.py: Accepts our input dataset and fine-tunes

MobileNetV2 upon it to create our mask detector model.
b. detect_mask_image.py: Performs face mask detection in static images.

c. detect_mask_video.py: Using webcam, this script applies face mask

detection to every frame in the stream.

1) Data augmentation

2) Loading the MobilNetV2 classifier (we will fine-tune this model with
pre-trained Image Net weights)

3) Building a new fully-connected (FC) head

4) Pre-processing

5) Loading image data

After the detection of faces is done, extraction of face region is done and
on each face ROI, face mask classifier is applied and then according to
that a person’s face is detected. If the person is not wearing a mask
system will generate an invoice as a script and send it to the admin or
manager and collect the fine on the spot i.e in mall itself.
 Figure 3.1.1 Training Face Mask Detector

​
12

Figure 1.3.2 Apply Face Mask Detector

 Figure 3.1.3 Invoice System
​
12

3.1.3 Advantages

1) Less amount of time is consumed while capturing and detecting

faces in video frame with reliable, efficient and accurate
manner.
2) No paperwork is required and for backup as on the spot invoice is
given to the person if he/she is not wearing a mask.
3) Device is managed easily and less maintenance work is required.

4) The use of Raspberry pi helps in multitasking since it is a processor

and can be accessed remotely.

3.1.4 Disadvantages

1) Sometimes server slow down its difficult to manage record.

2) It will take multiple picture of person as there are many cameras.
3.1.5 Methodology

1) MobileNetV2: MobileNetV2 is a state of the art for mobile visual

recognition including classification, object detection and semantic
segmentation. This classifier uses Depth wise Separable
Convolution which is introduced to dramatically reduce the
complexity cost and model size of the network, and hence is
suitable to Mobile devices, or devices that have low computational
power. In MobileNetV2, another best module that is introduced is
inverted residual structure. Non-linearity in narrow layers is
deleted. Keeping MobileNetV2 as backbone for feature extraction,
best performances are achieved for object detection and semantic
segmentation.

2) Keras: Subfield of machine learning which takes input to recognize

pattern to output a new set of similar data.

3) ROI: Region of interest (left eye, right eye and lip according to
knowledge based method of face detection.

4) OpenCV: Focus on real time application (interface) for computer

vision machine learning software.

5) Tensor flow: Used for numerical computation and large-scale

machine learning, bundles together a slew of machine learning and
deep learning(neural networking) models and algorithms makes
them useful by common metaphor.
6) Raspberry Pi:

Raspberry Pi is a low cost, credit-card sized computer that plugs

into a computer monitor or TV, and uses a standard keyboard and
mouse. It is a capable little device that enables people of all ages
to explore computing, and to learn how to program in languages
 like Scratch and Python. It’s capable of doing everything you’d
expect a desktop computer to do, from browsing the internet and
playing high-definition video, to making spreadsheets, word-
processing, and playing games. For the admin system we are
going to use invoice system in which ​data of
the ​person without wearing ​mask will be stored.
​
12

References

[1] K. He, X. Zhang, S. Ren and J. Sun, "Deep residual learning for image
recognition", 2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 770-778, 2016.

[2] Roomi Mansoor, Beham and M. Parisa, "A Review Of Face

Recognition Methods", International Journal of Pattern Recognition and
Artificial Intelligence, vol. 27, no. 04, pp. 1356005, 2013.

[3] H. Li, Z. Lin, X. Shen, J. Brandit and G. Hua, "A convolutional neural
network cascade for face detection", IEEE CVPR, pp. 5325-5334, 2015

[4] C.Y. Wen, S.H. Chiu, Y.R. Tseng and C.P. Lu, "The Mask Detection
Technology for Occluded Face Analysis in the Surveillance System",
Journal of forensic sciences, vol. 50, no. 3, pp. 593-601, 2005.1.
RETINAFACEMASK: A FACE MASK DETECTOR (MINGJIE JIANG,
XINQI FAN, HONG YAN).

[5] FACE ATTENTION NETWORK: AN EFFECTIVE FACE

DETECTOR FOR THE OCCLUDED FACES (JIANFENG WANG, YE
YUAN, GANG YU).

[6] DETECTING MASKED FACES IN THE WILD WITH LLE-CNNs

 (SHIMING GE, JIA LE, QITING YE, ZHAAO LUO).

[7] FACE DETECTION AND SEGMENTATION BASED ON

IMPROVED MASK R-CNN (KAIHAN LIN, HUIMIN ZHAO, JUJIAN
LV, CANYAO LI, XIAYONG LIU, RONGJUN CHEN, RUOYAN
ZHAO).

[8] GRID LOSS: DETECTING OCCLUDED FACES (MICHEAL OPTIZ,

GEORG WALTNER, GEORG POIER, HORST POSSEGGER, HORST
BISCHOF).

[9] REAL-TIME FACE MASK IDENTIFICATION USING

FACEMASKNET DEEP LEARNING NETWORK (MADHURA
INAMDAR, NINAD MEHENDALE).

[10] A HYBRID DEEP TRANSFER LEARNING MODEL WITH

MACHINE LEARNING METHODS FOR FACE MASK DETECTION
IN THE ERA OF THE COVID-19 PANDEMIC (MOHAMED LOEY
A, GUNASEKARAN MANOGARAN B, MOHAMED HAMED N.
TAHA D,NOUR ELDEEN M. KHALIFA).
12