OVERVIEW OF FACE RECOGNITION TECHNIQUES

Ioan Buciu
Department of Electronics,
University of Oradea, Faculty of Electrical Engineering and Information Technology,
Universitatii, 1, 410087, Oradea, Romania, E-Mail: [email protected]

Abstract – Humans are able to rapidly and accurately subspace approach is given by the Principal Component
recognize familiar faces and identities under widely Analysis (PCA) that extracts features named eigenfaces [1]
varying and difficult viewing conditions, such as representing a face image. PCA represents faces by their
illuminations changing, occlusion, scaling or rotation. projection onto a set of orthogonal axes (also known as
Motivated by its importance in human-to-human principal components, eigenvectors, eigenfaces, or basis
communication and leading to various applications, from images) pointing into the directions of maximal covariance
biometrics to human – computer interaction, the face in the facial image data. By defining the covariance matrix
recognition task is a major issue in the Computer Vision with
field and more. This paper presents a brief survey on
state-of-the art methods used to cope with this
{
C x = E (x − µ x )(x − µ x )
T
}
challenging task. where µ x denotes the mean image. PCA solution is found
by solving the equations system
Keywords: face recognition techniques, technologies,
applications, open problems. C x Z PCA = λZ PCA
with λ as eigenvectors. The basis images corresponding to
I. INTRODUCTION PCA are typically ordered according to the decreasing
amount of variance they represent, i.e., the respective
Among other applications, face recognition is one of the eigenvalues. Here ZPCA comprises the eigenfaces. The
primary biometric tasks, becoming more and more PCA techniques essentially select a subspace that retains
important as advances in technologies such as Internet, most of that variation, and consequently the similarity in
digital cameras require increased security key features. the face space is not necessarily determined by the identity.
Face recognition can operate either on still images or image To overcome this, Belhumeur et al. [2] developed so called
sequences. Also, it can manage either or both tasks: face “Fisherfaces”, an application of Fisher’s linear discriminant
identification (or recognition) and face verification (or (FLD). This technique projects the images onto a subspace
authentication). Face recognition is extreme challenging where the classes are maximally separated by maximizing
task as the recognition performance significantly the between-classes scatter matrix and minimizing the
deteriorates with changes in lighting, pose, scale or within-class scatter matrix at the same time. If we denote
occlusion. The paper is structured as follows. Section II the set of all N = | x | data divided into c classes
presents ones of the most representatives techniques with X ≡ {x1 , x 2 ,..., x c } , then the inter-class scatter
developed for subspace analysis. It also deals with the face matrix Sw is defined as
recognition solutions under varying pose and illumination c
Sw = ∑ ∑ (x − µ i )(x k − µ i )
T
conditions. The Section ends up with the face recognition k
methods for image sequences. Section III describes several i =1 xk ∈xi
face recognition applications. Finally, major limitations of while the between-class scatter matrix Sb is defined as
the existing face recognition systems are mentioned in c
S b = ∑ | xi | (µ i − µ )(µ i − µ )
T
Section IV.
i =1

II. FACE RECOGNITION APPROACHES where µi is the mean image of class xi and µ is the mean
of all data. Here, ZFLD satisfies
A. Face Recognition in Subspaces
ZT S B Z
Subspace analysis methods rely on the hypothesis that the Z FLD = arg max
face images resides in a lower subspace of the input image Z ZT S w Z
space. Thus, many information captured in the input space The solution for finding ZFLD is to solve the generalized
is redundant from the face pattern point of view. The eigeinvalues problems:
features contained in such subspace provide richer
information for face recognition. Probably the most used

173
 S B Z LDA = λS w Z LDA outperformed the parametric eigenspace by a small margin.
A number of 2D model-based algorithms have been
proposed for face tracking through large pose changes.
While PCA minimizes the sample covariance (second-
Separate active appearance models [9] were trained for
order dependence) of the data, independent component
profile, half-profile, and frontal views, with models for
analysis (ICA) [3] minimizes higher-order dependencies as
opposing views created by simple reflection. Using a
well, and the components found by ICA are designed to be
heuristic for switching between models, the system was
non-Gaussian. Bartlett et al. [3] used two ICA
able to track faces through wide angle changes. It has been
configurations to represent faces for recognition. PCA was
shown that linear models are able to deal with considerable
carried out prior to ICA for dimensionality reduction. An
pose variation so long as all the modeled features remained
intermediate step for “whitening” the data has been
visible [10]. A different way of dealing with larger pose
introduced between PCA and ICA processing. The data
variations is then to introduce nonlinearities into the model.
were then decomposed into basis images and de-
Romdhani et al. extended active shape models [11] and
composition coefficients. Their second ICA configuration
active appearance models [12] using a kernel PCA to
(ICA2) yields holistic basis images very similar to those
model shape and texture nonlinearities across views. In
produced by PCA. In that case, ICA is applied to the
both cases models were successfully fit to face images
projection matrix containing the principal components.
across a full 1800 rotation. However, no face recognition
Under this architecture, the linear decomposition
experiments were reported.
coefficients are as independent as possible. A nonlinear
variant of subspace analysis was prposed by Vasilescu and
C. Face Recognition From Image Sequences
Terzopoulos [4] called “Tensorfaces”. Finally, kernel –
based approaches were poposed, inclusing kernel PCA [5],
Recognizing faces in image sequences (video) is a more
kernel LDA [6]. In experiments on two data sets that
difficult task than the one corresponding to recognizing
contained images from 40 and 11 subjects, respectively,
face in still images as it involves simultaneously tracking
with varying pose, scale, and illumination, this algorithm
and recognition. Typically, a video-based face recognition
showed performance clearly superior to that of ICA, PCA,
system operates as follows. The face is first detected and
and KPCA and somewhat better than that of the standard
then tracked over time. Only when a frame satisfying
Fisherfaces.
certain criteria (size, pose) is acquired is recognition
performed using still-to-still recognition technique. For
this, the face part is cropped from the frame and
B. Face Recognition Across Pose and Illumination
transformed or registered using appropriate
transformations. This tracking-then-recognition approach
The most recent evaluation of commercial face recognition
attempts to resolve uncertainties in tracking and
systems shows satisfactory performance for face
recognition sequentially and separately and requires a
verification of the best systems to be on par with
criterion for selecting good frames and estimation of
fingerprint recognizers for frontal, uniformly illuminated
parameters for registration. Probabilistic video analysis has
faces. Recognizing faces across changes in pose and
recently gained significant attention in the computer vision
illumination has proved to be a much more difficult
community since the work of Isard and Blake [13]. The
problem. Although most research has so far focused on
authors introduced a time series state space model
frontal face recognition, there is a sizable body of work on
parameterized by a tracking motion vector (e.g., affine
pose invariant face recognition and illumination invariant
transformation parameters). The CONDENSATION
face recognition. However, face recognition across pose
algorithm was developed to provide a numerical
and illumination has received little attention. One of the
approximation to the posterior distribution of the motion
earliest appearance-based multi-view approaches was the
vector at time t given the observations up to t. The
one developed by Beymer [7]. After a pose estimation step,
CONDENSATION algorithm, also known as the particle
the algorithm geometrically aligns the probe images to
filter, was originally proposed in the signal processing
candidate poses of the gallery subjects using the
literature [14] and has been used to solve many other vision
automatically determined locations of three feature points.
tasks [15, 16], including human face recognition [17].
This alignment is then refined using the optical flow
strategy. Recognition is performed by computing
III. FACE RECOGNITION APPLICATIONS
normalized correlation scores. Good recognition results are
reported on a database of 62 subjects imaged in a number
This Section briefly reviews major face recognition
of poses ranging from -300 to +300 (yaw) and from -200 to
applications.
+200 (pitch). Pentland et al. [8] extended the eigenface
1. Face identification was widely used for
approach to tackle multiple views. The authors compare the
identifying driver licenses, in immigration
performance of a parametric eigenspace (computed using
programs, passports, or welfare registration.
all views from all subjects) with view-based eigenspaces
2. Access control deals with border – crossing,
(separate eigenspaces for each view). In experiments
vehicle access, ATM, computer access, computer
involving 21 people recorded in nine evenly spaced views
network access, online transaction access, online
from minus 900 to +900, view-based eigenspaces

174
 database access. For example, a commercial store data. A microprocessor card, on the other
access control named FaceGate [18] requires the hand, can add, delete, and manipulate information
one wishing to get into a building to enter his in its memory on the card. A microprocessor card
entry code or a card and face a camera on the door also has built-in security features. Contact-less
entry system. By applying a mathematical model smart cards contain a small antenna so the card
to an image of a face, FaceGate generates a unique reader detects the card from a distance. The Smart
biometric “key.” Whenever one wishes to access a Card’s portability and ability to be updated make
building, FaceGate verifies the person’s entry it a technology well suited for securely connecting
code or card, then compares his face with its the virtual and physical worlds. The application of
stored “key.” It registers him as being authorized face recognition technology in smart cards, in
and allows him to enter the building. Access is essence, is a combination of the two. This can be
denied to anyone whose face does not match. seen from the following two examples. Smart
3. Security refers to the terrorist alert issue in cards store the mathematical characteristics of the
airports, secure boarding systems, file encryption, faces during the enrollment stage. The
intranet and internet security, or medical records. characteristics are read out during the verification
Many airports have been adopted face recognition stage for comparison with the live capture of the
technology for improving security. In contrast to person’s face. If granted, the person can have his
face identification issue, face-recognition based stored facial characteristics updated in the card’s
security is a harder task due to many factors, such memory. To mention only one such application,
as different lighting conditions which can not the ZN-Face system [20] combines face
always be controlled, or the large number of faces recognition and smart card technology, is used for
that have to be examined in public places and to protecting secure areas at Berlin airports. Potential
discriminate the interested one from the crowd. threats posed by criminals who often succeed in
For instance, Fresno Yosemite International (FYI) entering high security areas by means of a suitable
airport in California deployed Viisage’s face disguise (e.g., pilot uniforms) are ruled out
recognition technology for airport security effectively. The individual’s face characteristics
purposes. The system is designed to alert FYI’s are stored on a smart card; ZN-Face compares and
airport public safety officers whenever an verifies the card information with the face
individual matching the appearance of a known readings at each access station.
terrorist suspect enters the airport’s security 6. Multimedia management deals with face-based
checkpoint. Anyone recognized by the system searching information, face-based video
would undergo further investigative processes by segmentation and summarization or event
public safety officers. Unfortunately, no detection. Human faces are frequently seen in
satisfactory results (if not quite disappointing) news, sports, films, home video, and other
were obtained using automatic face recognition multimedia content. Indexing this multimedia
systems after several years of real-life testing. content by face detection, face tracking, face
4. Surveillance is another application area where recognition, and face change detection is
face recognition plays a major part, including important to generate segments of coherent video
video surveillance, CCTV control or portal content for video browsing, skimming, and
control. As in face recognition applied in public summarization. Together with speech recognition,
places, surveillance based on face recognition natural language processing, and other image
systems assesses even lower performance. understanding techniques, face processing is a
Unconstrained lighting conditions and large powerful tool for automatic indexing, retrieval,
variability in pose degree make this task and access to the ever-growing digital multimedia
extremely difficult for large scale. It is worth content. One integrated multimedia management
mentioning that, in 1998 Visionics FaceIt system is the “Infomedia” project at Carnegie
technology was deployed for the first time to Mellon University [21]. This project aims to
enhance town center surveillance in Newham create an information digital video library to
Borough of London, which has 300 cameras enhance learning for people of all ages. Thousands
linked to the closed circuit TV (CCTV) control of hours of video content is indexed and archived
room. The city council claims that the technology for search and retrieval by users via desktop
has helped to achieve a 34% drop in crime since computers through computer networks. Face
its installation. Similar systems are in place in databases. Content-based image retrieval tries to
Birmingham, England. In 1999 Visionics was solve the difficulties faced by text-based image
awarded a contract from National Institute of retrieval. Instead of being manually annotated by
Justice to develop smart CCTV technology [19]. text-based keywords, images would be indexed by
5. Smart cards have an embedded microprocessor or their own visual content, such as color and texture.
memory chip that provides the processing power Feature vector is the basis of content-based image
to serve many applications. Memory cards simply retrieval, which captures image properties such as

175
 color and texture. However, these general features [4] M. A. O. Vasilescu and D. Terzopoulos. Multilinear
have their own limitations. Recently, researchers analysis of image ensembles: TensorFaces. In Proceedings
have tried to combine it with other image analysis of European Conference on Computer Vision, pages 447–
technologies, such as face detection and 460, Copenhagen, Denmark, May 2002.
[5] B. Scholkopf, A. Smola, and K.-R. Muller. Nonlinear
recognition, to improve the retrieval accuracy. component analysis as a kernel eigenvalue problem. Neural
7. Low enforcement is closely related to suspect Computation, 10(5):1299–1319, 1998.
tracking and investigation, identifying cheats in [6] M.-H. Yang. Kernel eigenfaces vs. kernel fisherfaces: Face
casinos, criminal face retrieval and recognition. recognition using kernel methods. In Proc. of IEEE
8. Human – computer interaction refers to Internation Conf. on Face and Gesture Recognition, pages
interactive gamming and proactive computing 215–220, Washington, DC, May 2002
9. Other applications include antique photo [7] D. Beymer. Face recognition under varying pose. Technical
verification, very – low bit rate image and video Report 1461, MIT AI Laboratory, Cambridge, MA, 1993.
transmission, etc. [8] A. Pentland, B. Moghaddam, and T. Starner. View-based
and modular eigenspaces for face recognition. In
Proceedings of the IEEE Conference on Computer Vision
IV. LIMITATIONS OF CURRENT FACE and Pattern Recognition, pages 84–91, 1994.
RECOGNITION SYSTEMS [9] T. Cootes, G. Wheeler, K. Walker, and C. Taylor. View-
based active appearance models. Image and Vision
Despite tremendous work performed to build a reliable face Computing, 20:657–664, 2002.
recognition system, the existing face recognition systems [10] A. Lanitis, C. Taylor, and T. Cootes. Automatic
face several limitations. Face recognition technology is still interpretation and coding of face images using flexible
not robust, especially in unconstrained environments, and models. IEEE Transactions on Pattern Analysis and
recognition accuracy is not acceptable, especially for large- Machine Intelligence, 19(7):743–756, 1997.
[11] S. Romdhani, S. Gong, and A. Psarrou. Multi-view
scale applications. Lighting changes, pose changes, and nonlinear active shape model using kernel PCA. In 10th
time differences between the probe image and the gallery British Machine Vision Conference, volume 2, pages 483–
image(s) further degrade the performance. These factors 492, 1999.
have been evaluated in FRVT 2002 using some of the best [12] S. Romdhani, A. Psarrou, and S. Gong. On utilising
commercial systems [22]. For example, in a verification template and feature-based correspondence in multi-view
test with reasonably controlled indoor lighting, when the appearance models. In 6th European Conference on
gallery consisted of 37,437 individuals with one image per Computer Vision, volume 1, pages 799–813, 2000.
person and the probe set consisted of 74,854 probes with [13] M. Isard and A. Blake. Contour tracking by stochastic
two images per person, the best three systems, on average, propagation of conditional density. European Conference on
Computer Vision, pages 343–356, 1996.
achieved a verification rate of 90% at a false alarm rate of [14] N.J. Gordon, D.J. Salmond, and A.F.M. Smith. Novel
1%, 80% at a false alarm rate of 0.1%, and 70% at a false approach to nonlinear/non-Gaussian Bayesian state
alarm rate of 0.01%. This level of accuracy may be (or may estimation. IEE Proceedings on Radar and Signal
not be) suitable for an access control system with a small Processing, 140:107–113, 1993.
database of hundreds of people but not for a security [15] G. Qian and R. Chellappa. Structure from motion using
system at airports where the number of passengers is much sequential Monte Carlo methods. IEEE Internation
larger. The test results in FRVT 2002 can partly explain Conference on Computer Vision, pages 614–621, 2001.
why several systems installed at airports and other public [16] M.J. Black and A.D. Jepson. A probabilistic framework for
places have not received positive feedback based on their matching temporal trajectories. European Conference on
Computer Vision, pages 909–924, 1998.
poor performance. One example is that the crowd [17] B. Li and R. Chellappa. A generic approach to simultaneous
surveillance system tested by Tampa, Florida police tracking and verification in video. IEEE Transactions on
reported 14 instances of a possible criminal match in a 4- Image Processing, 11:530–544, 2002.
day session, but they were all false alarms. The Tampa [18] FaceGate.http://www.premierelect.co.uk/faceaccess.html.
police department has abandoned the system. [19] FaceIt-Hist. http://www.identix.com/company/comp
history.html.
[20] W. Konen and E. Schulze-Kruger. ZN-face: a system for
REFERENCES access control using automated face recognition. In
Proceedings of the International Workshop on Automatic
Face and Gesture Recognition, pages 18–23, 1995.
[1] L. Sirovich and M. Kirby. Low-dimensional procedure for
[21] H. Wactlar, T.K.M. Smith, and S. Stevens. Intelligence
the characterization of human faces. Journal of the Optical
access to digital video: informedia project. IEEE Computer,
Society of America A, 4(3):519–524, 1987.
29-5:46–52, 1996
[2] P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman,
[22] P. Phillips, P. Grother, R. Michaels, D. Blackburn,
“Eigenfaces vs. Fish-erfaces: Recognition Using Class
E.Tabassi, and M. Bone. Face recognition vendor test 2002:
Specific Linear Projection”, IEEE Trans. On Pattern
evalution report. http://www.frvt2002.org.
Analysis and Machine Intelligence, 19(7), pp. 711-720,
1997.
[3] M. S. Bartlett, J. R. Movellan, and T. J. Sejnowski, “Face
recognition by independent component analysis,” IEEE
Trans. Neural Networks, 13(6), pp. 1450-1464, 2002.

176