Proceedings of The Seventh International Conference on Informatics and Applications (ICIA2018), Japan, 2018

Face Detection and Face Recognition in Python Programming Language

Primož Podržaj Boris Kuster
Faculty of Mechanical Engineering, University of Ljubljana
Aškerčeva 6, 1000 Ljubljana, Slovenia
[email protected]

ABSTRACT • a modern language (object oriented, ex-

ception handling, dynamic typing etc.)
Python is becoming increasingly popular program-
ming language. It is a free, high-level language that • concise, easy to read and quick to learn
has a very flat learning curve. It has a wide set of
freely available libraries. In this paper computer vi- • full of freely available libraries, in par-
sion libraries are first discussed. Then Face detec- ticular scientific ones (linear algebra, vi-
tion and Face recognition capabilities of libraries sualization tools, plotting, image analy-
available are analyzed. The basic description of the sis, differential equations solving, sym-
algorithm used in the libraries is given. For each bolic computations, statistics etc.)
major step an example of the resulting image is pro-
• useful in a wider setting: scientific com-
vided. Although just two sample images are given
puting, scripting, web sites, text parsing,
in the paper, the algorithm was analyzed on many
etc.
images. The analysis confirmed that Python is re-
ally the tool of choice for face detection and recog- • widely used in industrial applications
nition tasks.
In comparison with other programming lan-
guages such as C/C++, Java, and Fortran,
KEYWORDS
Python is a higher-level language. The compu-
Python, Image Processing, OpenCV, Face Detec- tation time is therefore typically a little longer,
tion, Face Recognition but it is much easier to program in. In the
case of C and Fortran, wrappers are also avail-
able. PHP and Ruby on the other side are high-
1 INTRODUCTION
level languages as well. Ruby can be compared
Python is a high-level general-purpose pro- to Python but lacks scientific libraries. PHP
gramming language created by Guido van on the other hand is a more web-oriented lan-
Rossum in 1991. It has a design philosophy guage. Python can also be compared to Mat-
that puts emphasis on code readability. It sup- lab, which has a really extensive scientific li-
ports multiple programming paradigms includ- brary. It is however not open source and free.
ing object-oriented, imperative, functional and Scilab and Octave are open source environ-
procedural and has a large standard and com- ments similar to Matlab. Their language fea-
prehensive library. The first release was fol- tures are however inferior to the ones avail-
lowed by Python 2.0 in 2000 and Python 3.0 able in Python. People in general tend to think
in 2008. At the time of writing this paper the that complex problems demand complex pro-
latest version is Python 3.7. Python is a good cesses in order to produce complex solutions.
choice for all the researchers in the scientific Python was developed with exactly the oppo-
community because it is [1]: site philosophy. It has an extremely flat learn-
ing curve and development process for soft-
• free and open source ware engineers [4]. It is used for system ad-
ministration tasks, by NASA both for devel-
• a scripting language, meaning that it is in- opment and as a scripting language in sev-
terpreted eral of its systems, Industrial Light & Magic

ISBN: 978-1-941968-53-6 ©2018 SDIWC 1

 Proceedings of The Seventh International Conference on Informatics and Applications (ICIA2018), Japan, 2018

uses Python in its production of special effects Its a library intended (as the name sug-
for large-budget feature films, Yahoo! uses gests) to be a simplified version of
it (among other things) to manage its discus- OpenCV. It doesnt offer all the possibil-
sion groups and Google has used it to imple- ities of OpenCV, but it is easier to learn
ment many components of its web crawler and and use.
search engine [3]. As Python is also a language
that is easy to learn and both powerful and con- • OpenCV
venient from the start [2], we might soon be It is by far the most capable and most
asking, who is not using it. As already men- commonly used computer vision library.
tioned Python has an extensive set of libraries It is written in C/C++, but Python bind-
which can be imported into a project in order ings are added during the installation. It
to perform specific tasks. The ones that re- also gives emphasis on real time image
ally should be mentioned in any scientific pa- processing. Among the ones, which will
per dealing with mathematics are NumPy and not be presented it might be worth men-
SciPy. NumPy is a library which provides sup- tioning Ilastik. It is a simple, user-friendly
port for large, multi-dimensional arrays. As tool for interactive image classification,
images are in fact large two (greyscale) or three segmentation and analysis.
(color) dimensional matrices, this library is es-
2.1 Python Imaging Library (PIL)
sential in all image processing tasks. It should
also be emphasized that many other libraries Python Imaging Library (PIL) is a library orig-
(not limited to image processing) use NumPy inally written by Fredrik Lundh. As PIL’s last
release dates back to 2009 it is a little outdated
array representation. SciPy is a library built on [5]. It’s successor that also supports Python
the NumPy array object and contains modules 3 is called Pillow [6]. Consequently not both
for signal and image processing, linear alge- of them can be installed at the same time. At
bra, fast Fourier transform, etc. The last library the time of writing the paper, the last version
mentioned in this introductory section is Mat- of Pillow is 5.1.0. The most trivial program
(showing an image) can be written as follows:
plotlib. As the name suggests this library is a
plotting library. Although it is used a lot in all from PIL import Image
im=Image.open(’/path/to/the/image/’)
areas of science, Image processing relies heav-
im.show()
ily on it. In this paper the libraries and their ca-
pabilities in the area of image processing, im- Pillow is able to extract a lot of information
age analysis and computer vision in general are from an image and makes it possible to execute
discussed. The execution of some of the most several standard procedures for image manipu-
common algorithms in the field is also demon- lation, including:
strated together with the resulting images.
• per-pixel manipulations,
2 PYTHON’S IMAGE PROCESSING LI- • masking and transparency handling,
BRARIES
• image filtering, such as blurring, contour-
There are several Python libraries related to
ing, smoothing, or edge finding,
Image processing and Computer vision. The
ones that will be presented in this paper are: • image enhancing, such as sharpening, ad-
justing brightness, contrast or color
• PIL/Pillow
This library is mainly appropriate for sim- Some of them will be demonstrated. Image can
ple image manipulations (rotation, resiz- for example be easily rotated for specific angle
ing, etc.) and very basic image analysis (in our case 45◦ ) and then saved by the follow-
ing code:
(histogram for example)
rotated_image=im.rotate(45)
• SimpleCV rotated_image.save(’rotated.jpg’)

ISBN: 978-1-941968-53-6 ©2018 SDIWC 2

 Proceedings of The Seventh International Conference on Informatics and Applications (ICIA2018), Japan, 2018

A color image can also be split into different import numpy as np

components (red, green and blue). import cv2
img = cv2.imread(’lena-color.jpg’)
r, g, b = im.split() cv2.imshow(’image’,img)
r.show() cv2.waitKey(0)
cv2.destroyAllWindows()
Image can also easily be sharpened or blurred.
In this case it is however important that we As OpenCV is nowadays the most suitable li-
also import the ImageFilter library. The code brary for computer vision we will use it in the
needed is shown below.
remaining part of the paper.
from PIL import ImageFilter
sharp=im.filter(ImageFilter.SHARPEN) 3 FACE DETECTION
blur=im.filter(ImageFilter.BLUR)
OpenCV enables us to do even more complex
Image can for example also be easily cropped tasks relatively easy. There are for example
with the following command routines, which detect face(s) (eyes in an im-
age). The following sequence of commands
cropped_im=im.crop((100,100,400,400)) does just that.
face_cascade=cv2.CascadeClassifier
It was demonstrated that PIL/Pillow is very
(’C:\\Users\\...\\haarcascade
easy if only basic image processing task are _frontalface_default.xml’)
needed. For more detailed analysis and com- eye_cascade=cv2.CascadeClassifier
puter vision SimpleCV and OpenCV are more (’C:\\Users\\...\\haarcascade_eye.xml’)
appropriate. img=cv2.imread(’lena.jpg’)
gray=cv2.cvtColor
(img, cv2.COLOR_BGR2GRAY)
2.2 SimpleCV faces=face_cascade.detectMultiScale
SimpleCV is an interface for Open Source ma- (gray, 1.3, 5)
chine vision libraries in Python. As the name for (x,y,w,h) in faces:
suggests it is a simplified version of OpenCV. cv2.rectangle
It is relatively easy to learn and use, as it is con- (img,(x,y),(x+w,y+h),(255,0,0),2)
cise, has readable interface for cameras, and roi_gray = gray[y:y+h, x:x+w]
makes it possible to execute image manipula- roi_color = img[y:y+h, x:x+w]
tion, feature extraction, and format conversion. eyes = eye_cascade.detectMultiScale
At the time of writing it is however still lim- (roi_gray)
ited to Python2. As now the majority of Python for (ex,ey,ew,eh) in eyes:
users have already transferred to Python3, Sim- cv2.rectangle(roi_color,(ex,ey),
pleCV wont be described in detail. Below a (ex+ew,ey+eh),(0,255,0),2)
simple program can be seen. cv2.imshow(’img’,img)
cv2.waitKey(0)
from SimpleCV import Image cv2.destroyAllWindows()
img = Image(’lena.jpg’)
img.show() The result is shown in Fig. 1. For the image
in Fig. 1 algorithm works perfectly. If how-
As everything included in SimpleCV can be ever a more complex image is used, the result
done in OpenCV, which is still developed, em- (especially for eyes) is not so good. See for
phasis will be put on OpenCV. example Fig. 2. The algorithm itself applies
the so called Haar feature-based cascade clas-
2.3 OpenCV sifiers. It was proposed as an effective object
OpenCV is an open source computer vision li- detection method by Paul Viola and Michael
brary available written in C and C++ which Jones [9]. The number of these features can
runs under Linux, Windows, Mac OS X, iOS, be enormous. But most of them are irrelevant.
and Android. Interfaces are available for
Python, Java, Ruby, Matlab, and other lan- A good feature is for example the fact that the
guages. A very simple program used just to region of the eyes is usually darker than the re-
show an image can be written as follows: gion of the nose and cheeks. A second good

ISBN: 978-1-941968-53-6 ©2018 SDIWC 3

 Proceedings of The Seventh International Conference on Informatics and Applications (ICIA2018), Japan, 2018

ily on various libraries. So modules paths,

face recognition, argparse, picle and os have to
be imported into Python project.
At first, some images of the person we wish
to recognize must be collected. This can be
done either manually, or by the application of
the Microsoft’s Bing API for searching. Ide-
ally, the dataset should contain at least 30 im-
ages of each person. No other persons should
be present in the images used for training. Two
Figure 1. An example of face detection sample images (one for each person) are shown
in Fig. 3. The network architecture used for

Figure 2. An example of face detection Figure 3. Sample images of Bob Kelso and John Dorian

feature could for example be based upon the face recognition is based on ResNet-34 neural
fact that the eyes are usually darker than the network [10]. The Python’s face recognition li-
bridge of the nose. With the increasing num- brary however has fewer layers and the number
ber of such features, we can increase the reli- of filters is reduced by half. The network was
ability of the algorithm. Misclassifications are trained on a dataset of approximately 3 million
of course always a possibility. It should also images (mainly the VGG dataset [11] and the
be noted, that the reliability decreases with the scrubs dataset [12]).
decreasing amount of pixels in the face area. The algorithm is composed of four steps:

1. Finding all the faces

4 FACE RECOGNITION
In the first step we could use the face de-
The field of Face recognition could be defined tection algorithm described in the previ-
as a research area in which images of faces are ous section [9], which is the most com-
grouped into sets belonging to one individuum. monly applied one. The face recognition
Maybe it is easier to understand if we use Face- library however uses a more advanced
book as an example. In the past, Facebook Histogram of Oriented Gradients (HOG)
was capable of detecting faces (see previous method [13]. Color images must first be
section), but then the user had to tag the per- converted to grayscale ones. Then for
son by clicking on the image and specifying its each pixel in the image we look at the
name. Now Facebook has the capability to tag direction in which the image is getting
everyone in the image automatically. This is darker. So we get a matrix of gradients
achieved by Face recognition algorithms. (see Fig. This matrix is to a large ex-
In Python this task can be achieved using tent independent of the brightness varia-
pre-trained convolutional neural networks and tions in the original image. It his however
OpenCV. The whole program relies heav- too big for manipulation. So submatrices

ISBN: 978-1-941968-53-6 ©2018 SDIWC 4

 Proceedings of The Seventh International Conference on Informatics and Applications (ICIA2018), Japan, 2018

3. Encoding Faces
The Deep Convolutional Neural Network
is trained to generate 128 measurements
for each face. The training process uses
three images at a time (the image of a
known person, another image of the same
person and an image of some other per-
son) [15]. This step requires a large
dataset and a lot of computer power. But
it has to be executed only once. Some pre-
Figure 4. A sample of an original image and its his- trained neural networks are also available
togram of gradients on internet.
4. Finding the persons name from the encod-
of 16x16 size are formed. Then the pre- ing
dominant direction for each submatrix is The last step is actually a very basic one.
found. The face being analyzed is compared to
the faces that we have in our database.
2. Posing and Projecting Faces Python’s library uses Support vector ma-
This step deals with the problem that faces chine (SVM) to do that. In principle
in an image might be looking to some any other classification algorithm could
other direction and not straight into the be used.
camera. There are several solutions to this
problem. Python’s library uses the ap- The performance of the algorithm is presented
proach with 68 landmarks that are present on the image shown in Fig. 6. It can be seen
on any face [14]. An example of such an
image is shown in Fig. 5. Then a ma-

Figure 6. An example of the Face detection algorithm

that it works well, despite the fact that both

Bob Kelso and John Dorian are not looking
straight into the camera. In the case of John
Dorian the head is positioned almost perpen-
Figure 5. A face with the landmarks dicular to the axis of the camera. It should also
be noted that the whole image is much darker
chine learning algorithm is trained to lo- then an average one. Despite that the algo-
cate these 68 landmarks on any face. Af- rithm was able to recognize both of them per-
ter that the face is transformed using affine fectly. Another example is shown in Fig. 7). In
transformations so that eyes and mouth this case there is no problem with brightness.
are centered as best as possible. John Dorian is again looking into the direction

ISBN: 978-1-941968-53-6 ©2018 SDIWC 5

 Proceedings of The Seventh International Conference on Informatics and Applications (ICIA2018), Japan, 2018

vastly improve the social aspects of robots or

software packages that can adapt to the user.
It namely gives a very reliable feedback in hu-
man machine interaction.

REFERENCES
[1] C. Fuhrer, J. E. Solem, and O. Verdier, Scientific
Computing with Python 3, Packt Publishing Ltd,
2016. (references)

[2] S. Nagar, Introduction to Python: For Scientists and

Figure 7. Another example of the Face detection algo-
Engineers, Bookmuft, 2016.
rithm

[3] M. L. Hetland, Beginning Python: from novice to

of Bob Kelso. Also his emotion is expressed professional, 3rd Ed., Apress, 2017.
by facial expression. In the case of Bob Kelso
[4] R. V. Hattem, Mastering Python: master the art of
we can see that he is smoking a pipe and also writing beautiful and powerful Python by using all
has some uncommon facial expression. De- of the features that Python 3.5 offers, Packt Publish-
spite that the algorithm could recognize both ing, 2016.
of them correctly.
[5] http://www.pythonware.com/products/pil/

5 CONCLUSION [6] http://python-pillow.org/

The paper is divided into two parts. In the [7] https://en.wikipedia.org/wiki/Python Imaging Library
first one a short overview of the most common
Python’s libraries related to image processing [8] A. Kaehler, and G. Bradski, Learning OpenCV:
and computer vision is given. The second part computer vision in C++ with the OpenCV library,
uses the OpenCV library. In this part the li- 2nd Ed.,” O’Reilly, 2016.
braries for face detection and face recognition
[9] P. Viola, and M. Jones, “Rapid object detection us-
are described and analyzed. Face detection and
ing a boosted cascade of simple features,” Proceed-
face recognition are namely two areas of inten- ings of the 2001 IEEE Computer Society Confer-
sive research, because they enable a better in- ence on Computer Vision and Pattern Recognition,
teraction between computer system or robots vol. 1, pp. I-511-I-518, 2001.
on one side and humans on the other.
[10] K. He, X. Zhang, S. Ren, and J. Sun, Deep residual
Python’s face recognition library turns out to
learning for image recognition,” In Proceedings of
be fast and very reliable tool for face detec- the IEEE conference on computer vision and pattern
tion and face recognition. As Python is a recognition, pp. 770-778, 2016.
high level programming language the library
is well suited to be used just as a face detec- [11] O. M. Parkhi, A. Vedaldi, and A. Zisserman,
tion(recognition) procedure in a wider project “Deep face recognition,” In British Machine Vision
Conference, vol. 1, no. 3, p. 6, September, 2015.
without the need for a detailed knowledge of
the theoretical background of the employed al- [12] H. W. Ng, and S. Winkler, “A data-driven approach
gorithms. So, in our opinion it has a bright fu- to cleaning large face datasets,” IEEE International
ture. Conference on Image Processing (ICIP), pp. 343-
In future it would be very interesting to investi- 347, 2014.
gate the possibilities of Python and its libraries
[13] N. Dalal, and B. Trigs, “Histograms of oriented
for emotion detection. This field is a very hot gradients for human detection,” IEEE Computer So-
topic in human machine interface research. Us- ciety Conference on Computer Vision and Pattern
ing the results of this research it is possible to Recognition (CVPR), vol. 1, pp. 886-893, 2005.

ISBN: 978-1-941968-53-6 ©2018 SDIWC 6

 Proceedings of The Seventh International Conference on Informatics and Applications (ICIA2018), Japan, 2018

[14] V. Kazemi, and J. Sullivan, “One millisecond face

alignment with an ensemble of regression trees,”
Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pp. 1867-1874,
2014.

[15] F. Schroff, D. Kalenichenko, and J. Philbin,

“Facenet: A unified embedding for face recogni-
tion and clustering,” Proceedings of the IEEE con-
ference on computer vision and pattern recognition,
pp. 815-823, 2015.

ISBN: 978-1-941968-53-6 ©2018 SDIWC 7