FACE RECOGNIZATION USING OPENCV

A Report Submitted
by
RAMAADYUTI BATTABYAL
Department of Computer Science and Engineering.
JIS College Of Engineering, Kalyani.

1
 Face Recognition using OpenCV

The growing interest in computer vision of the past decade. Fueled by the steady
doubling rate of computing power every 13 months, face detection and recognition has
transcended from an esoteric to a popular area of research in computer vision and one
of the better and successful applications of image analysis and algorithm based
understanding. Because of the intrinsic nature of the problem, computer vision is not
only a computer science area of research, but also the object of neuro-scientific and
psychological studies, mainly because of the general opinion that advances in computer
image processing and understanding research will provide insights into how our brain
work and vice versa. Because of general curiosity and interest in the matter, the author
has proposed to create an application that would allow user access to a particular
machine based on an in-depth analysis of a person's facial features. This application will
be developed using Intel's open source computer vision project, OpenCV and
Microsoft's .NET framework.

2
 1. Introduction

What is face recognition? When you look at your friend walking down the street or a
picture of him, you recognize that he is your friend Paulo. Interestingly when you look at
your friend or a picture of him you look at his face first before looking at anything else.
Ever wondered why you do that? This is so that you can recognize him by looking at his
face. Well, this is you doing face recognition.

But the real question is how does face recognition works? It is quite simple and intuitive.
Take a real life example, when you meet someone first time in your life you don't
recognize him, right? While he talks or shakes hands with you, you look at his face,
eyes, nose, mouth, color and overall look. This is your mind learning or training for the
face recognition of that person by gathering face data. Then he tells you that his name
is Paulo. Now your mind is trained and ready to do face recognition on Paulo's face.
Next time when you will see Paulo or his face in a picture you will immediately
recognize him. This is how face recognition work. The more you will meet Paulo, the
more data your mind will collect about Paulo and especially his face and the better you
will become at recognizing him.

Now the next question is how to code face recognition with OpenCV, You might say that
our mind can do these things easily but to actually code them into a computer is difficult.
Thanks to OpenCV, coding face recognition is as easier as it feels. The coding steps for
face recognition are same as we discussed it in real life example above.

 Training Data Gathering: Gather face data (face images in this case) of the
persons you want to recognize
 Training of Recognizer: Feed that face data (and respective names of each
face) to the face recognizer so that it can learn.

3
 Recognition: Feed new faces of the persons and see if the face recognizer
you just trained recognizes them.

OpenCV comes equipped with built in face recognizer, all you have to do is feed it
the face data. It's that simple and this how it will look once we are done coding it.

4
 Over the past decade face detection
and
recognition have transcended from
esoteric to popular areas of
research in
computer vision and one of the
better
and successful applications of
image
analysis and algorithm based
understanding. Because of the
intrinsic
nature of the problem, computer
vision
is not only a computer science area
of
research, but also the object of
neuro-
5
scientific and psychological studies
also,
mainly because of the general
opinion
that advances in computer image
processing and understanding
research
will provide insights into how our
brain
work and vice versa.
A general statement of the face
recognition problem (in computer
vision)
can be formulated as follows: given
still
or video images of a scene,
identify or

6
verify one or more persons in the
scene
using a stored database of faces.
Facial recognition generally involves
two
stages:
Face Detection where a photo is
searched to find a face, then the
image
Over the past decade face detection
and
recognition have transcended from
esoteric to popular areas of
research in
computer vision and one of the
better
and successful applications of
image
7
analysis and algorithm based
understanding. Because of the
intrinsic
nature of the problem, computer
vision
is not only a computer science area
of
research, but also the object of
neuro-
scientific and psychological studies
also,
mainly because of the general
opinion
that advances in computer image
processing and understanding
research
will provide insights into how our
brain
8
work and vice versa.
A general statement of the face
recognition problem (in computer
vision)
can be formulated as follows: given
still
or video images of a scene,
identify or
verify one or more persons in the
scene
using a stored database of faces.
Facial recognition generally involves
two
stages:
Face Detection where a photo is
searched to find a face, then the
image

9
Over the past decade face detection
and
recognition have transcended from
esoteric to popular areas of
research in
computer vision and one of the
better
and successful applications of
image
analysis and algorithm based
understanding. Because of the
intrinsic
nature of the problem, computer
vision
is not only a computer science area
of
research, but also the object of
neuro-
10
scientific and psychological studies
also,
mainly because of the general
opinion
that advances in computer image
processing and understanding
research
will provide insights into how our
brain
work and vice versa.
A general statement of the face
recognition problem (in computer
vision)
can be formulated as follows: given
still
or video images of a scene,
identify or

11
verify one or more persons in the
scene
using a stored database of faces.
Facial recognition generally involves
two
stages:
Face Detection where a photo is
searched to find a face, then the
image
Over the past decade face detection
and
recognition have transcended from
esoteric to popular areas of
research in
computer vision and one of the
better
and successful applications of
image
12
analysis and algorithm based
understanding. Because of the
intrinsic
nature of the problem, computer
vision
is not only a computer science area
of
research, but also the object of
neuro-
scientific and psychological studies
also,
mainly because of the general
opinion
that advances in computer image
processing and understanding
research
will provide insights into how our
brain
13
work and vice versa.
A general statement of the face
recognition problem (in computer
vision)
can be formulated as follows: given
still
or video images of a scene,
identify or
verify one or more persons in the
scene
using a stored database of faces.
Facial recognition generally involves
two
stages:
Face Detection where a photo is
searched to find a face, then the
image

14
Over the past decade face detection
and
recognition have transcended from
esoteric to popular areas of
research in
computer vision and one of the
better
and successful applications of
image
analysis and algorithm based
understanding. Because of the
intrinsic
nature of the problem, computer
vision
is not only a computer science area
of
research, but also the object of
neuro-
15
scientific and psychological studies
also,
mainly because of the general
opinion
that advances in computer image
processing and understanding
research
will provide insights into how our
brain
work and vice versa.
A general statement of the face
recognition problem (in computer
vision)
can be formulated as follows: given
still
or video images of a scene,
identify or

16
verify one or more persons in the
scene
using a stored database of faces.
Facial recognition generally involves
two
stages:
Face Detection where a photo is
searched to find a face, then the
image
Over the past decade face detection
and
recognition have transcended from
esoteric to popular areas of
research in
computer vision and one of the
better
and successful applications of
image
17
analysis and algorithm based
understanding. Because of the
intrinsic
nature of the problem, computer
vision
is not only a computer science area
of
research, but also the object of
neuro-
scientific and psychological studies
also,
mainly because of the general
opinion
that advances in computer image
processing and understanding
research
will provide insights into how our
brain
18
work and vice versa.
A general statement of the face
recognition problem (in computer
vision)
can be formulated as follows: given
still
or video images of a scene,
identify or
verify one or more persons in the
scene
using a stored database of faces.
Facial recognition generally involves
two
stages:
Face Detection where a photo is
searched to find a face, then the
image

19
Over the past decade face detection
and
recognition have transcended from
esoteric to popular areas of
research in
computer vision and one of the
better
and successful applications of
image
analysis and algorithm based
understanding. Because of the
intrinsic
nature of the problem, computer
vision
is not only a computer science area
of
research, but also the object of
neuro-
20
scientific and psychological studies
also,
mainly because of the general
opinion
that advances in computer image
processing and understanding
research
will provide insights into how our
brain
work and vice versa.
A general statement of the face
recognition problem (in computer
vision)
can be formulated as follows: given
still
or video images of a scene,
identify or

21
verify one or more persons in the
scene
using a stored database of faces.
Facial recognition generally involves
two
stages:
Face Detection where a photo is
searched to find a face, then the
image
1.1 Problem Definition:

Over the past decade face detection and recognition have transcended from esoteric to
popular areas of research in computer vision and one of the better and successful
applications of image analysis and algorithm based understanding. Because of the
intrinsic nature of the problem, computer vision is not only a computer science area of
research, but also the object of neuroscientific and psychological studies also, mainly
because of the general opinion that advances in computer image processing and
understanding research will provide insights into how our brain work and vice versa. A
general statement of the face recognition problem (in computer vision) can be
formulated as follows: given still or video images of a scene, identify or verify one or
more persons in the scene using a stored database of faces. Facial recognition
generally involves two stages:

Face Detection where a photo is searched to find a face, then the image is processed
to crop and extract the person’s face for easier recognition.

22
Face Recognition where that detected and processed face is compared to a database
of known faces, to decide who that person is.

1.2 Project Overview/Specifications:

Humans are very good at recognizing faces and complex patterns. Even a passage of
time doesn't affect this capability and therefore it would help if computers become as
robust as humans in face recognition. Face recognition system can help in many ways:
 Checking for criminal records.  Enhancement of security by using surveillance
cameras in conjunction with face recognition system.  Finding lost children's by using
the images received from the cameras fitted at public places.  Knowing in advance if
some VIP is entering the hotel.  Detection of a criminal at public place.  Can be used
in different areas of science for comparing an entity with a set of entities.  Pattern
Recognition. This project is a step towards developing a face recognition system which
can recognize static images. It can be modified to work with dynamic images. In that
case the dynamic images received from the camera can first be converted in to the
static one's and then the same procedure can be applied on them. But then there are
lots of other things that should be considered. Like distance between the camera and
the person, magnification factor, view [top,side, front] etc.

1.3 Hardware Requirements:

 Processor :Pentium processor of 400MHz or higher.

 RAM : Minimum 64MB primary memory.
 Hard disk : Minimum 1GB hard disk space.
 Monitor : Preferably color monitor (16 bit color) and above.
 Webcamera.
 Compact Disk drive.
 A keyboard and a mouse.

23
1.4 Software Requirements:
 Operating System : Windows operating system
 Language : Java
 Front-end tool : Swing
 JDK : JDK 1.5 and above

2. OpenCV Face Recognizers

OpenCV has three built in face recognizers and thanks to OpenCV's clean coding, you
can use any of them by just changing a single line of code. Below are the names of
those face recognizers and their OpenCV calls.

1. EigenFaces Face Recognizer Recognizer -

cv2.face.createEigenFaceRecognizer()
2. FisherFaces Face Recognizer Recognizer -
cv2.face.createFisherFaceRecognizer()
3. Local Binary Patterns Histograms (LBPH) Face Recognizer -
cv2.face.createLBPHFaceRecognizer()

1.EigenFaces Face Recognizer:

 We represent every image in our training as some linear combination of

weights of eigenfaces which is basis for entire training dataset.

24
 We use this weights to represent the image in smaller subspace.(Dimensionality
Reduction — Using k best eigenvectors to represent basis)

 For recognizing we calculate the weights of test image and compare it with
weights of all the images in training dataset,and recognizes the image with least
error between weights of test and training set.

 Training Algorithm

1. First we read all the training images.

2. Convert each [m x n] image matrix into [(m x n) x 1] image vector.(Flatten

image matrix to vector).

3. Find average_face_vector, sum(all image vectors)/number(images).

4. Subtract average_face_vector from every image vector.

5. Stack all (image_vectors — average_face_vector) in matrix forming A = [(m x n) x

i] matrix (where i = number of images).

6. Calculate covariance matrix of above matrix -> C = A*transpose(A).

7. Find eigenvectors and eigenvalues of above covariance matrix.

8. As size of above matrix is huge its very expensive to compute eigenvectors of such a
huge matrix,so we will calculate the eigenvectors of transpose(A)*A as the matrix is
small of i x i dimension.

9. Then we will pre-multiply the eigenvectors of transpose(A)*A with A to get

eigenvectors of A*transpose(A). → (Proof is given in paper)

10. Choose best k eigenvectors such that k << i.

11. Find weights of each image and store it. → Equation given in paper.

25
  Testing Algorithm

1. Normalize the test image -> I = Test_Image_Vector — Average_Face_Vector.

2. Find weights of test image using above training process(use test images which
have not been used in training process to actually see how accurate your algorithm
is recognizing).

3. Calculate error between weights of test image and weights of all images in
training set.

4. Choose the image of training set which has minimum error.

There are many ways to calculate error such euclidean distance,mahalanobis distance
etc.You can use different distances and see the results and conclude which performs
better.

2.FisherFaces Face Recognizer

 Let X be a random vector with samples drawn from c classes:

X = {X1 , X2, . . . , Xc} Xi = {x1 , x2, . . . , xn}

 The scatter matrices SBandSWarecalculatedas :

SB = ∑c i=1 Ni(µi − µ)(µi − µ) T SW = ∑c i=1 ∑ xj∈Xi (xj − µi)(xj − µi) T ,

where µisthetotalmean : µ = 1 N ∑N i=1 xi

And µi isthemeanofclassi ∈ {1, . . . , c} : µi = 1 |Xi | ∑ xj∈Xi xj

 Fisher’s classic algorithm now looks for a projection W, that maximizes the class
separability criterion:

26
  Wopt = arg maxW |W T SBW| |WTSWW|
a solution for this optimization problem is given by solving the General Eigenvalue
Problem: SBvi = λiSwvi S −1 W

SBvi = λivi

 The optimization problem can then be rewritten as:

Wpca = arg maxW |W T STW|

Wfld = arg maxW |WTWT pcaSBWpcaW| |WTWT pcaSWWpcaW|

The transformation matrix W, that projects a sample into the (c-1)-dimensional space is then
given by:

W = WT fldWT pca

You can see that features extracted actually represent faces and these faces are called
fisher faces and hence the name of the algorithm.

3.Local Binary Patterns Histograms (LBPH) Face Recognizer

Now that we know a little more about face recognition and the LBPH, let’s go further and
see the steps of the algorithm:

a.Parameters: the LBPH uses 4 parameters:

 Radius: the radius is used to build the circular local binary pattern and represents
the radius around the central pixel. It is usually set to 1.

 Neighbors: the number of sample points to build the circular local binary pattern.
Keep in mind: the more sample points you include, the higher the computational
cost. It is usually set to 8.

27
  Grid X: the number of cells in the horizontal direction. The more cells, the finer
the grid, the higher the dimensionality of the resulting feature vector. It is usually
set to 8.

 Grid Y: the number of cells in the vertical direction. The more cells, the finer the
grid, the higher the dimensionality of the resulting feature vector. It is usually set
to 8.

b.Training the Algorithm: First, we need to train the algorithm. To do so, we need to
use a dataset with the facial images of the people we want to recognize. We need to also
set an ID (it may be a number or the name of the person) for each image, so the
algorithm will use this information to recognize an input image and give you an output.
Images of the same person must have the same ID. With the training set already
constructed, let’s see the LBPH computational steps.

c.  Applying the LBP operation: The first computational step of the LBPH is to create
an intermediate image that describes the original image in a better way, by highlighting
the facial characteristics. To do so, the algorithm uses a concept of a sliding window,
based on the parameters radius and neighbors.

d.  Extracting the Histograms: Now, using the image generated in the last step, we can
use the Grid X and Grid Y parameters to divide the image into multiple grids, as can be
seen in the following image.

e.  Performing the face recognition: In this step, the algorithm is already trained. Each
histogram created is used to represent each image from the training dataset. So, given
an input image, we perform the steps again for this new image and creates a histogram
which represents the image.

 So to find the image that matches the input image we just need to compare two
histograms and return the image with the closest histogram.

28
 We can use various approaches to compare the histograms (calculate the
distance between two histograms), for example: euclidean distance, chi-
square, absolute value, etc. In this example, we can use the Euclidean distance
(which is quite known) based on the following formula:

 So the algorithm output is the ID from the image with the closest histogram. The
algorithm should also return the calculated distance, which can be used as a
‘confidence’ measurement. Note: don’t be fooled about the ‘confidence’ name, as
lower confidences are better because it means the distance between the two
histograms is closer.

 We can then use a threshold and the ‘confidence’ to automatically estimate if the
algorithm has correctly recognized the image. We can assume that the algorithm
has successfully recognized if the confidence is lower than the threshold defined.

3.1Flow Chart:

29
3.2Data Flow Diagram:

30
3.3 Step by Step Execution:

31
3.4 Coding Face Recognition with OpenCV
The Face Recognition process in this tutorial is divided into three steps.

1. Prepare training data: In this step we will read training images for each
person/subject along with their labels, detect faces from each image and assign
each detected face an integer label of the person it belongs to.
2. Train Face Recognizer: In this step we will train OpenCV's LBPH face
recognizer by feeding it the data we prepared in step 1.
3. Testing: In this step we will pass some test images to face recognizer and
see if it predicts them correctly.

32
To detect faces, I will use the code from my previous article on face detection. So if you
have not read it, I encourage you to do so to understand how face detection works and
its Python coding.

Code Dependencies¶

1. OpenCV 3.2.0.
2. Python v3.5.
3. NumPy Numpy makes computing in Python easy. Amont other things it
contains a powerful implementation of N-dimensional arrays which we will use
for feeding data as input to OpenCV functions.

 Import Required Modules:

1.import cv2

2.import os

3.import numpy as np

 Training Data:
Each folder is named with format sLabel (e.g. s1, s2) where label is actually the
integer label assigned to that person.the mapping for label 0 is empty.
 Prepare training data:

LBP face detector

Preparing data...

Data prepared

Total faces: 23

Total labels: 23

 Train Face Recognizer:

face_recognizer = cv2.face.createLBPHFaceRecognizer()

face_recognizer.train(faces, np.array(labels))

33
  Prediction
def draw_rectangle(img, rect):

def draw_text(img, text, x, y):

def predict(test_img):

print("Predicting images...")
print("Prediction complete")

3.4 Testing Process:

 Unit Testing: A program unit is small enough that the programmer who
developed if can test it in a great detail.
 Integration Testing: Situations like treating the modules when there is no data
entered in the text box is also tested.
 System Testing: This testing is done to check if the system has its requirements
and to find the external behavior of the system.
 Acceptance Testing: This testing is performed finally by user to demonstrate that
the implemented system satisfies its requirements.
 Specification Testing: Specification testing is done to check if the program does
what is should do and how it should behave under various conditions or
combination.
 White Box Testing: White Box Testing is a test case design method that uses the
control structure of the procedural design to derive test cases.
 Black Box Testing: Black Box Testing focuses on the functional requirement of
the software. It enables to derive sets of input conditions that will fully exercise all
functional requirements for a program.
 Boundary Values Analysis: Number of errors usually occurs at the boundaries of
the input domain generally. In this technique a selection of test cases is
exercised using boundary values i.e., around boundaries.

34
4. RESULTS / OUTPUTS:

35
5.CONCLUSIONS & FUTURE SCOPE:
Face Recognition is a fascinating idea to work on and OpenCV has made
it extremely simple and easy for us to code it. It just takes a few lines of
code to have a fully working face recognition application and we can
switch between all three face recognizers with a single line of code
change. It's that simple.

Although EigenFaces, FisherFaces and LBPH face recognizers are good

but there are even better ways to perform face recognition like using
Histogram of Oriented Gradients (HOGs) and Neural Networks. So the
more advanced face recognition algorithms are now a days implemented
using a combination of OpenCV and Machine learning. I have plans to
write some articles on those more advanced methods as well, so stay
tuned!

1. The user will be authenticated not only with the username also with the
image of the user
2. For the processing, some of the lines on the face will be used so that
the image can be identified with the different angles.
3. The image processing process isgood enough to provide security for
the website.
Future Enhancements:
1. The project can be enhanced for processing 3D images.
2. Authentication can be implemented by capturing video clip of a
person.
3. This can also be used to process the signatures of a person for
providing the authentication.
4. We can also use this in real time application.
5. Authentication can be embedded into web application which will be an
added advantage for providing the login for the websites .

36
6.References:
http://www.imageprocessingplace.com/
http://www.graphicsmagick.org/
http://www.imagemagick.org/
http://www.mediacy.com/
http://opencv.willowgarage.com/
http://www.face-rec.org/algorithms/
http://en.wikipedia.org/wiki/Threedimensional_face_recognition /
http://www.cvpapers.com/

Books:
Digtal Image Processing Projects- Rs tech Technology
Image processing by MichealPedilla

37