BIOMETRICS SECURITY BASED ON FACE

RECOGNITION

A THESIS REPORT

Submitted by
ZIAUL HAQUE CHOUDHURY

Under the guidance of

Dr. K.M.MEHATA

In partial fulfillment for the award of the degree of

MASTER OF TECHNOLOGY (BY RESEARCH)

in
INFORMATION TECHNOLOGY

JUNE 2013
ii
iii
iv
 ACKNOWLEDGEMENT

This is one of the most significant and delighting moments in my life; making a
milestone in one of the longest project, the Master of Technology (By Research)
thesis. Finally, I am on the verge of graduating.

I thank my guide Dr. K. M. MEHATA, Dean, School of Computer and

Information Sciences, for giving me the research opportunity in Biometrics
Security. He has always inspired me with all the interesting and challenging
problems. He showed not only which problems we need to solve but how
effectively and how efficiently.

I sincerely express my heartfelt gratitude to Prof. JALEES AHMED KHAN

TAREEN, Vice Chancellor, B.S. Abdur Rahman University, Chennai.

I sincerely thank Dr. V. M. PERIASAMY, Registrar, for furnishing every essential

facility for doing my research project.

I thank Dr. LATHA TAMILSELVAN, Professor and Head, Department of

Information Technology, for her eminent support and encouragement throughout
my research project.

I express my sincere thanks to the research review committee members, from

the Department of Computer Science and Engineering, HOD, Dr. ANGELINA
GEETA, Professor, Dr. ANANTHAKUMAR, from the Madras Institute of
Technology, Chennai.

I would like to thank all the faculty members and staffs of the Department of
Information Technology and Computer Science & Engineering, B.S.Abdur

v
Rahman University for their inspiration, cooperation and provided me all official
and laboratory facilities in various ways for the completion of this thesis.

My obligated respect and thanks to my loving parents and sisters for their love,
sacrifice, inspiration, suggestions and support. They are my first teachers after I
came to this world and have set great examples for me about how to live, study
and work.

(ZIAUL HAQUE CHOUDHURY)

vi
 ABSTRACT

Biometrics is a rapidly developing technology, which has been broadly applied in

forensics such as criminal identification, secured access, and prison security.
The biometric technology is basically a pattern recognition system that
acknowledges a person by finding out the legitimacy of a specific behavioral or
physiological characteristic owned by that person. In this era, face is one of the
commonly acceptable biometrics system which is used by humans in their visual
interaction and authentication purpose. The challenges in the face recognition
system arise from different issues concerned with cosmetic applied faces and of
low quality images.
In this thesis, we propose two novel techniques for extraction of facial
features and recognition of faces when thick cosmetic is applied and of low
quality images. In the face recognition technology, the facial marks identification
method is one of the unique facial identification tasks using soft biometrics. Also
facial marks information can enhance the face matching score to improve the
face recognition performance. When faces are applied by thick cosmetics, some
of the facial marks are invisible or hidden from their faces. In the literature, to
detect the facial marks AAM (Active Appearance Model) and LoG (Laplacian of
Gaussian) techniques are used. However, to the best of our knowledge, the
methods related to the detection of facial marks are poor in performance
especially when thick cosmetic is applied to the faces. A robust method is
proposed to detect the facial marks such as tattoos, scars, freckles and moles
etc. Initially the active appearance model (AAM) is applied for facial feature
detection purpose. In addition to this prior model the Canny edge detector
method is also applied to detect the facial mark edges. Finally SURF is used to
detect the hidden facial marks which are covered by cosmetic items. It has been
shown that the choice of this method gives high accuracy in facial marks
detection of the cosmetic applied faces.

vii
 Besides, another aspect of the face recognition based on low quality
images is also studied. Face recognition indeed plays a major rule in the
biometrics security environment. To provide secure authentication, a robust
methodology for recognizing and authentication of the human face is required.
However, there are numbers of difficulties in recognizing the human face and
authentication of the person perfectly. The difficulty includes low quality of
images due to sparse dark or light disturbances. To overcome such kind of
problems, powerful algorithms are required to filter the images and detect the
face and facial marks. This technique comprises extensively of detecting the
different facial marks from that of low quality images which have salt and pepper
noise in them. Initially (AMF) Adaptive Median Filter is applied to filter the
images. The filtered images are then extracted to detect the primary facial
feature using a powerful algorithm like Active Shape Model (ASM) into Active
Appearance Model (AAM). Finally, the features are extracted using feature
extractor algorithm Gradient Location Orientation Histogram (GLOH).

Experimental results based on the CVL database and CMU PIE database with
1000 images of 1000 subjects and 2000 images of 2000 subjects show that the
use of soft biometrics is able to improve face recognition performance. The
results also showed that 93 percentage of accuracy is achieved. Second
experiment is conducted with an Indian face database with 1000 images and
results showed that 95 percentage of accuracy is achieved.

viii
 TABLE OF CONTENTS

CHAPTER NO TITLE PAGE NO.

ABSTRACT vii

LIST OF FIGURES xiii

LIST OF TABLES xvi

LIST OF ABBREVIATION xvii

1 INTRODUCTION 1

1.1 Networks Security 1

1.2 Networks Security Concepts 1

1.3 Security Issues and challenges 2

1.3.1 Information Security 2

1.3.2 Computer Security 2

1.3.3 Internet Security 4

2 BIOMETRICS SECURITY 5

2.1 History of Biometrics 5

2.2 Biometrics Security 6

2.3 Biometrics System 7

2.4 A brief review of the major biometrics 8

technologies

ix
 2.5 Differences between behavioral & physical 11
biometrics

2.6 The major biometric concept 11

2.7 Applications of biometrics 14

2.8 Criminal justice system 15

2.8.1 Finger printing 15

2.8.2 Face recognition 15

2.9 Challenges on biometrics security: face 15

recognition

2.9.1 Face recognition 17

2.9.2 Face recognition in 2D 17

2.9.3 Face representation 19

3 LITERATURE OVERVIEW 21

3.1 Face authentication using different facial 21

features

3.2 Demographic based authentication 26

3.3 Different face recognition methods 28

3.4 Face annotation method 39

3.5 Facial feature detection algorithms 41

x
 3.6 Edge detection method 42

3.7 Motivation 44

3.8 Thesis Contribution 45

4 ROBUST FACIAL MARKS DETECTION 46

4.1 Introduction 46

4.2 Facial marks finder 50

4.2.1 Facial feature detection 51

4.2.2 Mean shape mapping 51

4.2.3 Face mask construction 53

4.2.4 Facial marks blob detection 54

4.2.5 Facial marks blob classification 59

4.2.6 Facial marks based matching 60

4.3 Experimental results 61

4.4 Summary 63

5 FACIAL MARKS DETECTION FROM LOW 64

QUALITY IMAGES

5.1 Introduction 64

5.2 Proposed method 66

5.3 Facial marks on the face 68

5.4 Facial marks detection method 69

xi
 5.4.1 Noise removal 70

5.4.2 Facial feature detection 73

5.4.3 Designing mean shape & mask construction 73

5.4.4 Gradient location and orientation histogram 75

5.4.5 Blob classification for facial marks 78

5.4.6 Face based matching for facial marks 79

5.5 Experimental results 81

5.6 Summary 83

6 CONCLUSIONS AND FUTURE DIRECTION 84

6.1 Conclusion 84

6.2 Future direction 85

7 BIBILOGRAPHY 86

LIST OF PUBLICATIONS 99

TECHNICAL BIOGRAPHY 100

xii
 LIST OF FIGURES

FIGURE NO. TITLE PAGE NO

2.1 statics of biometrics users in worldwide in 5

the year 2012

2.2 Examples of different types of biometric 8

authentication

2.3 Example applications using face 17

biometric

2.4 Face recognition performance in FRVT 19

2002

4.1 Examples of facial marks Scar, mole, and 46

freckles etc.

4.2 Examples of textual query and a 48

schematic of face region segmentation

4.3 Three different types of example of facial 49

marks results. Full face, partial face, and
off-frontal face

4.4 Schematic of automatic facial mark 50

extraction process

4.5 Schematic of texture mapping process 52

using the triangular Barycentric
coordinate system

4.6 Schematic of the mean face construction 53

xiii
4.7 Mark classification using morphology 57

4.8 Examples of mark detection and 58

classification results

4.9 Mark detection graph 58

4.9.1 Schematic of the mark based indexing 60

scheme

4.9.2 Examples of face image retrieval result. 63

5.1 Example of low quality image with 64

different facial marks

5.2 Example of textual query and a schematic 66

of face region segmentation

5.3 Over all structure of our proposed method 68

5.4 Shows salt and pepper noise images 68

5.5 Different facial marks on the mean face 69

image

5.6 Schematic of Automatic Facial Mark 70

Extraction Process.

5.7 Salt and pepper noise filtered with AMF 71

5.8 shows histogram for noisy color image 72

5.9 shows histogram for noiseless color 72

5.9.1 Schematic of texture mapping process 74

using the triangular Barycentric

xiv
 coordinate system

5.9.2 Schematic of the mean face construction 74

5.9.3 Marks detection performance graph 76

5.9.4 Mark classification using morphology. 78

5.9.5 Examples of facial marks which are 78

detected from face image

5.9.6 Schematic of the mark based indexing 80

scheme

5.9.7 Face matching and retrieval results 83

xv
 LIST OF TABLES

4.9.2 Proposed facial marks matcher & face 62

recognition accuracy

5.9.3.1 Mark Detection Accuracy (MDA) 76

in percentage

xvi
 LIST OF ABBREVIATION

AAM - Active Appearance Model

PCA - Principle Component Analysis

SURF - Speed Up Robust Feature

CCD - Charge Coupled Device

DB - Data Base

ASM - Active Shape Model

SIFT - Scale Invariant Feature Transform

GLOH - Gradient Location Orientation Histogram

3D - 3 Dimensional

AMF - Test Environment as a Service

SMT - Scars, Marks, and Tattoos

xvii
 1. INTRODUCTION

1.1 NETWORK SECURITY

Network security lies in the provisions & policies took over by a network
administrator to protect and keep tabs on unauthorized access such as
modification, misuse, or denial of a computer network and network accessible
resources. Network security necessitates the authorization of access to data in a
network, which is controlled by the network administrator. Users pick out or are
allotted an ID and password or other authenticating information that permits
them access to information and programs. Network security addressing an
assortment of computer networks, both private and public that are applied in
everyday tasks carrying on communications and transactions among individuals,
businesses, and government agencies. Networks can be private, such as inside
a company, and others which might be exposed to public access. Network
security is necessitated in enterprises, organizations, and other parts of the
institutions. It does as its title explicates, it secures the network, as well as
defend and oversee the operations being performed. The most simple and
common way of defending a network resource is by allotting it a corresponding
password and unique name.

1.2 Network security concepts

Network security commences with authenticating the user, through the

‘username’ and a ‘password’, which is referred as one factor authentication.
With two factor authentication, something the user 'has' is also applied (e.g.
'dongle', a security token, an ATM card, or a mobile phone); and with three
factor authentication, another criterion is also applied (e.g. retinal scan, face
recognition or a fingerprint, etc.). Once authenticated, a firewall imposes access
policies such as what services are permitted to be accessed by the network
users.

1
1.3 SECURITY ISSUES & CHALLENGES IN NETWORKS & ENTERPRISES

1.3.1 Information Security

“Information security is the practice of defending information from unauthorized

access, use, modification, disclosure, perusal, inspection, Denial of use of
knowledge, misuse, Facts, recording or destruction.” From the above definition,
Information Security does guarantees protection. The confidentiality, integrity
and availability (CIA) triad is one of the core principles of information security. In
the year 1992 and subsequently revised in 2002, the OECD's guidelines for the
Security of Information Systems and Networks proposed the nine generally
accepted principles: a) Awareness, b) Responsibility, c) Response, d) Ethics, e)
Democracy, f) Security Management, g) Risk Assessment, h) Reassessment,
and i) Security Design and Implementation. Building upon those, in the year
2004 the NIST's Engineering Principles for Information Technology
Security (EPITS) proposed 33 principles. In the year 2002, Donn
Parker suggested an alternative model for the classic CIA triad that he named
the six atomic elements of information. The elements are i) confidentiality, ii)
possession, iii) integrity, iv) authenticity, v) availability, and vi) utility. The merits
of the Parkerian hexad are subjects of debate amongst security professionals.

1.3.2 Computer security

With the introduction of the computer, the requirement for automated tools for
defending files and other information stored on the computer became apparent.
This is particularly the case for a shared system, and the requirement is even
more discriminating for systems that can be got at over a public telephone
network, the Internet, or data network. The aggregation of tools planned to
protect data and to thwart hackers is what constitutes the computer security.

2
1.3.2.1 Security by design

Some of the techniques that are commonly used to implement computer security
include:

The principle of least perquisite, where each part of the system has only the
perquisite that are required for its function. That way even if a assailer benefits
access to that part, they have only bounded access to the whole system.

Automated theorem proving to demonstrate the correctness of crucial

software subsystems.

Unit testing and code reviews to make modules more secure where formal
rightness proofs are not possible.

Defense in depth, where the design is such that more than one sub-system
requires to be breached to compromise the unity of the system and the
information it holds.

Default secures settings and design to ‘fail secure’ rather than ‘fail insecure’.
Ideally, a secure system should need a deliberate, knowledgeable, conscious,
and free decision on the part of legitimate authorities in order that make it
insecure.

Audit trails tracking system activity, so that when a security breach goes on,
the mechanism and extent of the breach can be found out. Putting in audit trails
remotely, where they can only be appended, it can keep intruders from covering
their tracks.

Full revelation to ensure that when bugs are found the ‘window of vulnerability’
is kept as short as potential.

3
1.3.3 Internet security

Security is influenced by the introduction of spread out systems and the use of
networks and communications for carrying data between the end user and the
computer and between the computer and computer. Network security evaluation
is needed to protect data during their transmission. In fact, the term network
security is somewhat misleading because virtually all government, business, and
academic organizations interlink their data processing equipment with a
collection of interlinked networks. Such a collection is frequently referred to as
an internet and the term internet security is applied.

4
 2. BIOMETRICS SECURITY

2.1 HISTORY OF BIOMETRICS

The term “Biometrics” is derived from the Greek words “bio” means “life” and
“metrics” means “to measure”. Automatized biometric systems have become
obtainable only over the last few decades due to important advances in the field
of computer science. Lots of these new automated techniques, nevertheless,
are based on thoughts that were originally imagined hundreds or even
thousands of years ago. One of the most former and most basic examples of a
characteristic that is utilized for recognition is the human face. Since the
commencing of civilization, humans have applied faces to identify known
‘familiar’ and unknown ‘unfamiliar’ individuals. Figure 2.1, the statics of
biometrics users in worldwide in the year of 2012.

Figure 2.1, statics of biometrics users in worldwide in the year 2012

5
This simple task became more and more challenging as the human population
increased and as more commodious methods of travel brought in many new
individuals into small once communities. The conception of human to human
acknowledgement is also seen in behavioral very common biometrics such as
gait and speaker recognition. Individual use these features, somewhat without
awareness to recognize known individuals on a day by day basis.

2.2 BIOMETRICS SECURITY

A biometric is a mensuration of biological features such as fingerprint, iris

pattern, retina image and face or hand geometry or behavioral features such as
gait, voice, or signature. Biometric technology utilized these characteristics to
recognize individuals automatically. Most preferably the characteristic should be
universally present and unique to the individual, easily measurable and stable
over time. No biometric features have been officially demonstrated to be unique,
even though they are usually sufficiently distinct for practical applies. Dissimilar
biometrics will be more desirable for dissimilar applications depending. For
example, on whether the objective is to discover someone with their cooperation
or from a distance without their cognition. As illustrated below, biometrics can be
applied to answer two principal questions:-
Are you who you claim to be?
Who are you?

 Are you who you claim to be?

Confirming that someone is who they claim to be normally relied on security

pass, or something that they know, that is password. Absolute confidence
cannot be provided by either of them. For instance, security passes can be
skipped and passwords are sometimes unknowingly written down. Biometric
technology offers an extra level of assurance, even if the person’s

6
characteristics or a password are easily known and can therefore be replicated.
To affirm an individual’s identity, their biometrics is scanned and converted into
electronic form and kept either on a card that remains in their possession or in a
database. On requesting access to an IT system or a building, the biometric is
scanned again and equated with the record to confirm their individuality. Where
the objective is simply to recognize an individual as someone with permission to
use the system, there is no actual need to link the biometric data with any other
personal information.

 Who are you?

If a database of known individuals has been developed it is possible to answer

the question ‘who are you?’ Against the database in a ‘one-to-many’ search, the
biometric of the unknown person is compared. Their identity can be determined
if their biometric has been entered into the database on a previous occasion;
this is much quicker than a manual system. High quality data are needed if the
database searches are to give accurate results.

2.3 BIOMETIRC SYSTEM

A biometric recognition or simply a biometric system is a pattern recognition

system that authenticates an individual based on his biometric traits. The
biometric recognition technology consists of four main modules: (a) sensor that
captures samples of a biometric trait, (b) feature extraction module that extracts
certain salient features of the biometric sample captured by the sensor, (c)
system database that stores the features extracted by the feature extraction
module, and (d) matcher module that equalizes the features extracted from the
biometric samples with the features stored in the system database. See Figure
2.2, for an illustration of a typical biometric recognition technology.

7
 Figure 2.2, Examples of different types of biometric authentication

2.4 A BRIEF REVIEW OF THE MAJOR BIOMETRIC TECHNOLOGIES

There are a total of seven major biometric technologies available today. They
are:

Fingerprint recognition;
Hand geometry recognition;
Facial recognition;
Iris and retina recognition;
Voice recognition;
Keystroke recognition;
Signature recognition;

8
Of these technologies, hand geometry recognition, fingerprint recognition and
iris recognition are most predominant. For DNA recognition (examining the
unique strands found in DNA samples), earlobe recognition (examining the
geometry of the earlobe) and gait recognition (the way and manner in which
somebody walks), considerable time and effort is being invested in biometric
technologies of the future. About the biometric technology a brief description is
given below.

Fingerprint recognition

The location and determination of the unique characteristics of the fingerprint

are involved in fingerprint recognition. Various ‘valleys’ and ‘ridges’, which form
the basis for the loops, arches and swirls on your fingertip is included in the
fingerprint. Different kinds of breaks and discontinuities are contained in the
ridges and valleys. These are called as ‘minutiae’. The unique features are
located and determined from these minutiae. Two types of minutiae are there:
bifurcations (the location where a single ridge splits into two ridges), ridge
endings (the location where the ridge actually ends).

Hand geometry recognition

Hand geometry identification requires looking for unique features in the structure
of the hand. These features contain the length, thickness and width of the finger,
the distances between the hand’s overall bone structure and finger joints. Of
these unique characteristics, a 3-dimensional image is taken. As other
identifiers, the hand does not contain as many unique characteristics.

9
Facial recognition

Facial recognition contains taking many images or pictures of the face and
extracting the unique facial features as well as distances from or between the
nose, ears, mouth, eyes and cheeks.

Iris and retinal recognition

Examining the unique features of the iris is entailed in iris recognition. The
colored section between the white region of the eye (also known as the sclera)
and the pupil is the iris. To control the size of the pupil (the part of the eye that
allows light to pass through) is its primary purpose. The trabecular mesh-work
(the tissue that gives the iris its ‘radial’ impression) is included in the unique
features of the iris and as well as other physiological properties that is freckles,
furrows, rings, and the corona. Examining the pattern of blood vessels in the
retina that is located at the back of the eye is involved in retinal recognition. At
the juncture of the optic nerve (the area where the nerve leaves the brain and
enters the eye) is focused on the examination.

Voice recognition

With voice recognition, it is the unique patterns of an individual’s voice, as

produced by the vocal tract, which is examined. A text phrase is usually recited
in order to capture the voice inflections. The vocal tract consists of the oral
pharynx, nasal pharynx, laryngeal pharynx, oral cavity, and the nasal cavity.

Keystroke recognition

Keystroke identification works by analyzing the unique way in which an

individual character on a computer keyboard. Variables contain typing speed,

10
the distance of time that keys are held down, and the time taken between c
successive keystrokes.

Signature recognition

Signature recognition analyzes the way and manner in which we sign our name.
Unique features include alteration in timing, speed and pressure during the
signing process. It is significant to understand that it is not the signature itself
that is examined.

2.5 DIFFERENCES BETWEEN BEHAVIORAL AND PHYSICAL BIOMETIRCS

The given biometric technologies are divided in two categories: behavioral

biometrics and physical biometrics. In general, As captured by a biometric
system, generally behavioral biometrics can be defined as the non-biological or
non- physiological features (or unique identifiers). This category includes
signature as well as keystroke recognition as behavioral biometrics also covers
any mannerisms or behavior displayed by an individual. As captured by a
biometric system, physical biometrics may be defined as the biological and
physiological features (or unique identifiers). Hand geometry recognition,
fingerprint recognition, facial recognition, iris and retinal recognition, and voice
recognition are included in this category.

2.6 THE MAJOR BIOMETIRC CONCEPTS

It is important that one has a basic understanding of the key concepts that are
associated with biometrics, when examining the various biometric technologies
and systems. Each of these concepts is discussed below:

11
 Verification and identification
Biometric templates
The processes of enrolment, verification and authorization;
Biometric performance standards;

Verification and identification

The aim of the verification process is to establish someone’s claimed identity. It

assigns you a number when you first enroll in a biometric system, which is linked
to your biometric template. On the basis of this number the database containing
your template is searched. You will be extended a given service or privilege, if a
positive match is established. A one-to-one (or 1:1) relationship is said to exist
as the number is linked to the template. The identification process looks to
establish someone’s identity as its name suggests. The entire database needs
to be searched in order for the system to ‘recognize’ you in the absence of a
unique number. A one-to-many (1: n) relationship is said to exist as the template
can belong to anyone in the database. To identify and track known criminals, a
good example of a 1: n database is AFIS (Automated Fingerprint Identification
Services), which is used by many law enforcement agencies.

Biometric templates

It takes numerous images/recordings of your biological and non-biological data

(i.e. a voice recording or keystroke pattern) when you first enroll in a biometric
system. These raw images and recordings are subsequently consolidated into
one main image, called as the ‘biometric sample’. The unique features
(discussed above) are captured and extracted from this sample. Next, for the
purposes of verification and identification, they are converted to a ‘biometric
template’. The biometric system does not contain actual images of the biological
or non biological data, it only stores the mathematical files. Depending on the

12
biometric system in use, the type of mathematical file is created in which binary
mathematical files are created by fingerprint recognition and hand geometry
recognition systems. Hexadecimal files are generated by iris recognition
systems.

The processes of enrollment, verification and authorization

When you try to enter a high security area, your finger print need to be
registered and added to a database. This process is known as enrollment.
Various images of the fingerprint are taken, that are combined in a single
biometric sample. To extract the unique features from the given sample, a
mathematical formula or extraction algorithm is used next. These features are
subsequently stored in a mathematical file known as an enrollment template that
is used for the verification purposes. The next step includes verification. You are
required to identify yourself by placing your finger on a scanner of fingerprint in
order to enter a high-security area. ‘An image’ of your print will be captured by
the scanner’s sensor. The unique features are extracted from the image
subsequently by the extraction algorithm and store these in a file known as the
verification template. In order to determine the extent to which they match, the
next stage includes comparing the verification template to the enrollment
template, which are obtained with the help of a matching algorithm. Based on
the amount of overlap between both templates, the latter gives a score. If an
agreed value (or threshold) is lesser than this score, you are authorized to enter
the area. If the threshold value is higher than the score, you are denied access
(and the verification process may be repeated). Although much happens during
the processes of enrollment, verification and authorization, only a few seconds is
taken to complete. It is important to remember that an enrollment template is
never completely (100%) similar to the verification template at this point.

13
Biometric performance standards

Based on the performance of several standards, biometric systems are rated.

The most important are the False Acceptance Rate (FAR), the False Rejection
Rate (FRR) and the Equal Error Rate (EER) as shown in figure 1. It can be
defined as the probability of a registered user, as it is wrongly rejected by the
biometric system, the FRR, which are also known as a type 1 error. As the
probability of an impostor being wrongly authorized by the biometric system, the
FAR (also known as type 2 errors) can be defined. The probability of the FAR
and the FRR is reflected by the EER (or crossover rate) which is almost the
same. There are also other standards of biometric. To Enroll rate (FTE), for
example, the failure, which defines the statistical probability that a person is
simply unable to enroll in a biometric system. For the system to capture, this
may be attributable to the fact that the person in question does not have enough
unique features. The Ability to Verify rate (ATV) indicates the overall percentage
of users that can be verified by a system of biometric. The ATV can also be the
combination of the FTE and the FRR. Mathematically, this relationship can be
represented as follows:
ATV = [(1-FTE) * (1-FRR)]

2.7 APPLICATIONS OF BIOMETRICS

For use in access/I.D. cards and for small-scale security of buildings and IT
systems, there is growing interest in the use of biometrics. At a national level, on
current and potential large-scale applications this brief is focused. These contain
the immigration and asylum, Criminal Justice System and port and border
security. The law has regulated the collection of fingerprints by the police and
immigration service. General legislation regulates other biometric technology.

14
2.8 CRIMINAL JUSTICE SYSTEM

2.8.1 Fingerprinting

Automated fingerprinting is the only biometric in general use (see box on NAFIS
opposite) at a national level in the UK. Prisons already take ink fingerprints from
prisoners who are convicted. As proof, these can be compared against the
police database in which the intended person is being held. A system which is
automatic would give rapid confirmation of a person’s identity and allow
information about individuals to be shared quickly and easily.

2.8.2 Face Recognition

Everyone is charged or convicted of an offence by the police through

photographs. Digital databases of static photos are already held by some police
forces. To set up a national face recognition database might be an option,
comparable with NAFIS in that it could carry out searches based on static
images taken in police stations. Face recognition technology could also be used
to complement the estimated 1 million CCTV cameras already in operation in
the UK, by recognizing individuals covertly at a distance. This is a greater
technical challenge than the use of static images collected with the knowledge
and co-operation of the individual. Potentially, covert recognition could be used
in real time, allowing the instant identification of individuals, or offline, enabling
hours of CCTV recordings to be searched quickly and without human fatigue.

2.9 CHALLENGES ON BIOMETIRC SECURITY: FACE RECOGNITION

To establish a subject's identity based on facial characteristics is the ability of

face recognition. Various techniques from different research fields are required
in automated face recognition such as (i) computer vision, (ii) machine learning,

15
(iii) pattern recognition, and (iv) image processing. From a number of subjects,
face images are enrolled into the system as gallery data in a typical face
recognition system, and using a one-to-one or one-to- many scheme, the test
subject’s face image (probe image) is matched to the gallery data. The one-to-
many and one-to-one matching are called identification and verification
respectively. To interact with each other, face recognition is one of the
fundamental methods used by human beings. Using a pair of photographs dates
back to 1871 in a British court [28], attempts are made to match faces. For the
purpose of automatic person recognition, with still and video images, techniques
for automatic face recognition have been developed over the past three
decades. A wide range of applications, including enforcement of law,
applications of civil, and systems of surveillance are there in face recognition. To
smart home systems face recognition applications have also been extended. For
better interactive communications between human and machines [29], the
recognition of the human face and expression is used. Using the face Figure 2.3
shows some applications of biometrics.

(c) (d)

16
Figure 2.3, Example applications using face biometric: (a) ID cards, (b) face
matching and retrieval (c) access control and (d) Dyna Vox Eye Max system
(controlled by eye gaze and blinking.)

The face has several advantages that make it one of the most preferred
biometric traits. The face biometric is easy to capture even at a very far
distance. The internal feelings (emotion) of the subject (e.g., happiness or
sadness) and the person's age are also conveyed besides the identity of the
face. This makes an important topic in human computer interaction as well as
person recognition of face recognition. There has been a significant
improvement in performance of face recognition for the past one decade for
many applications [29] [30] but it is still below acceptable levels. To overcome
the performance bottleneck in 2D still face recognition, video input, and different
features (e.g., skin texture), recent efforts have focused on using 3D models.
This chapter starts with a survey of recognition of face in 2D and presents the
challenges in problems of face recognition.

2.9.1 Face Recognition

Using a one-to-one or one-to-many scheme, images of faces from a number of

subjects are enrolled into the system as gallery data and the face image of a test
subject (probe image) is matched to gallery data in a typical scenario of face
recognition. Three distinct modalities are there that are used in recognition of
face applications: 2D, 3D, and video. The face recognition problems will be
reviewed in these domains in the following sections.

2.9.2 Face Recognition in 2D

Using 2D still images, face recognition has been well studied for over a decade
[42] [43] [44]. Face recognition systems based on 2D still image, a snapshot of a

17
user is acquired and compared with a gallery of snapshots to establish an
identity of a person. The user is expected to be cooperative and provide a frontal
face under uniform lighting conditions with a simple background to enable the
capture and segmentation of a high quality image of the face in this procedure.
Although, it is now well known that the performance of the single-shot 2D image
based face recognition systems [29] can be drastically degraded by small
variations in pose and lighting. 2D face recognition is usually categorized
according to the number of images used in matching Some of the well-known
algorithms for 2D face recognition are based on Principle Component Analysis
(PCA) [42] [43], Linear Discriminant Analysis (LDA) [45], Elastic Graph Bunch
Model (EGBM) [46], and correlation based matching [47]. Continuously 2D face
recognition technology is evolving. Identification accuracy of around 70% was
achieved given near frontal pose and normal lighting conditions on a large
database (121,589 images from 37,437 subjects, Figure 2.5) in the Face
Recognition Vendor Test (FRVT) 2002 [48]. The FRVT 2006 evaluations were
performed in a verification scenario and the best performing system showed a
0.01 False Reject Rate (FRR) at a False Accept Rate (FAR) of 0.001 given the
high resolution (400 pixels between eyes1) 2D images (by Neven Vision [49]) or
3D images (by Viisage2 [50]). In closed set or open set scenarios, face
recognition can be performed. The open set recognition considers the possibility
that the probe subject is not enrolled in the gallery but the closed set recognition
tests always have the probe subject enrolled in the gallery data. Therefore in
retrieving candidate matches in open set recognition tests [51], a threshold value
(on match score) is typically used.

18
Figure 2.4, Face recognition performance in FRVT 2002 (Snapshot of computer
screen) [90].

2.9.3 Face Representation

Most face recognition techniques use one of two representation approaches: i)

local feature based [52] [53] [54] [55] [56] [57] or ii) holistic based [43] [42] [58]
[59]. Local feature based approach to identify local features (e.g., eyes, nose,
mouth, or skin irregularities) in a face and generate a representation based on
their configuration of geometry. The approach based on holistic localize the face
and use the entire face region in the sensed image to generate a representation.
For this purpose, a dimensionality reduction technique (e.g., PCA) is used. The
information of discrimination present in micro facial features (for instance,
speckles or moles) is usually ignored and considered as noise. Applying further
transformations on the holistic representation is also a common technique (e.g.,
LDA). A combination of local and holistic representations has also been studied
19
[60] [61]. Local feature based approaches can be further categorized as: i)
component based [62] [63], ii) modular [64] [67] [66] [68] [65], and iii) skin detail
based [69] [70]. Component based approaches try to identify local facial
primitives (e.g., eyes, nose, and mouth) and either use all or a subset of them to
generate features for matching. Modular approaches subdivide the face region,
irrespective of the facial primitives, to generate a representation. Skin detail
based approaches have recently gained attention due to the availability of high
resolution (more than 400 pixels between the eyes) face images. Facial
irregularities (e.g., freckles, moles, scars) can be explicitly or implicitly captured
and used for matching in high/low resolution face images.

20
 3. LITERATURE OVERVIEW

3.1 Face authentication using different facial features

To authenticate a person's face in the literature, the authors J. E. Lee, A. K.

Jain, and R. Jin et al. (2008) used Scars, marks and tattoos (SMT) are being
increasingly used for suspect and victim identification in forensics and agencies
of law enforcement. In particular, tattoos are getting serious attention because of
their visual and demographic characteristics as well as their increasing
prevalence. Scars, marks, and tattoos (SMT) are imprints of skin that have been
shown to be useful by law enforcement agencies for identification of a non-
skeletalized body of a victim or a suspect using a false identity. The author
illustrated the method of detecting SMT provide more discriminative information
than the traditional demographic indicators such as age, height, gender, race,
and gender to identify a person. To detect the SMT they used Scale Invariant
Feature Transform (SIFT) extracts repeatable characteristic feature points from
an image and generates descriptors representing the texture around the feature
points. They contributed their method based on a CBIR system for tattoo
images, with the growing use of tattoos for victim and suspect identification in
forensics and law enforcement agencies such a system will be of great value in
apprehending suspects and identifying victims. They introduced image similarity
computation based on SIFT features [2]. Demographic information such as
gender, ethnicity and facial marks as the soft biometrics traits. They leveled the
gender of each face images into three categories – male, female, and unknown
as well as ethnicity into three categories – Caucasian, African-American, and
unknown. The author currently leveled the gender and ethnicity as a manual.
They proposed a fully automated facial marks extraction system using global
and local analysis method. First they applied Active Appearance Model to detect
and remove the primary facial feature from face image using PCA algorithm
such as eye, eyebrows, nose and mouth. The local irregularities are detected
using Laplacian of Gaussian (LoG) or Difference of Gaussian (DoG). The
21
detected facial marks are used to calculate by their morphology and color along
with the location. They developed a soft biometric trait-based face matching
system. It uses gender and ethnicity information and facial marks. This soft
biometric matcher can be combined with any facial matcher to improve the
recognition accuracy or used by itself when a face matcher fails because of face
occlusion. The author also showed that facial marks can help in discriminating
identical twins. With the proposed soft biometric matcher, users can issue
semantic queries to retrieve images of interest from a large database [1]. Facial
skin exhibits various small scale structures in the surface (wrinkles, scars) and
the texture (nevi – a general term for pigment lesions like birthmarks and moles)
that stands out from the normal skin appearance and represent potentially
valuable references for individual distinction. Among such skin irregularities
moles are especially suited for identification. Their predictable appearance, also
under changing illumination, facilitates detection. And their numerous
appearance in conjunction with unique distribution patterns scales well with
extensive galleries. Furthermore moles require no abstract encoding, in contrast
to most other facial features. The goal of their paper is to present techniques for
detection and validation of moles that are prominent enough to be used for
identification. Relying on such small scale variations is an unusual approach in
face recognition. Conventional recognition algorithms are designed to work on
low resolution images. J. S. Pierrard and T. Vetter et al. (2007) introduces a
novel framework capable of detecting prominent moles. They attach importance
to the fact that, in contrast to existing methods are not bound to a certain
recognition scenario. That means in particular that they worked independent of
pose and illumination and that they are able to deduce personally identifying
features from one gallery image per individual. Their systems are divided into
three main steps corresponding to the three properties that characterize a local
region as a birthmark namely appearance, location and context. From distance a
mole appears simply as small dark region of circular shape surrounded by a
brighter area, i.e. a so called ’blob’. This description also holds under varying

22
viewing conditions (pose/illumination). A very sensitive multi scale detection
scheme to identify even the most subtle mole is also discussed. Due to its
sensitivity, the detector also responds to typical facial features such as nostrils,
corners of eyes eyebrows and mouth as well as to unexpected deviations like
strands of hair. Across individuals, these points are not discriminate and it is
crucial for their scheme that they are rejected. With prior knowledge derived
from the Morphable Model, they compute a binary segmentation of the face in
order to rule out points in those non-skin areas. In contrast to most other skin a
detection/segmentation scheme our approach is texture based and therefore
requires no color input or training. Finally the notion of saliency was introduced
which allows to assess the importance of each birthmark candidate for
recognition. This procedure takes the relationship between a point’s size and
contrast and the texture of its neighborhood into account. In essence it
represents a combined measure of confidence and uniqueness. Below a certain
saliency threshold, points are immediately discarded. Their framework
incorporates face-class specific knowledge from the 3D Morphable Model
developed by Blanz and Vetter. The model is derived from 3D laser scans,
registered to a common reference in which shape and texture are
parameterized. A statistical analysis of this data is performed by Principal
Component Analysis. The resulting sets of orthogonal eigenvectors represent
facial prototypes which can be linearly combined to form new faces. For a set of
model parameters (linear coefficients) and additional parameters for pose and
illumination, a realistic face image can be synthesized. Given a photograph of a
face, an analysis-by-synthesis approach is used to fit these parameters such
that the rendered face model matches the input image. The result is a complete
3D reconstruction of the face where shape and texture in occluded regions (due
to pose) have been automatically replenished from the model. They detect
moles by means of normalized cross correlation (NCC) matching. As a template,
a Laplacian-of-Gaussian filter mask is used, since it is very much similar to the
blob-like appearance of moles. The NCC has not scaled invariant and the object

23
size is not known a priori. This requires computation of the matching for several
resolutions, using templates of varying scale. With a growing number of
resolutions a straightforward implementation becomes very inefficient. Their
paper presented a novel approach to exploit local skin irregularities as features
for face identification. They focused on the methodology of detection and
evaluation of such regions and showed that it is possible to determine a
person’s identity based on only a few well-chosen pixels [5]. D. Lin and X. Tang
et al. (2006) introduced a multi layer framework for high resolution face
recognition exploiting features in various scales. Each face image is factorized
into four layers: facial organs, the appearance of global, skins, and details of
irregularities. They employed Multilevel PCA followed by the Regularized LDA to
model global appearance and organs of the face. But, the description of texture
of the skin and details of irregularities, for which representation of conventional
vector are not appropriate. For this problem, Discriminative Multiscale Texton
Features and SIFT- Activated Pictorial Structure are proposed to describe the
skin and details of subtle respectively. In order to combine the information
effectively conveyed by all layers, they further designed a metric fusion
algorithm adaptively placing emphasis onto the highly confident layers. They
established a novel and effective paradigm for high resolution face recognition,
incorporating new representations for modeling skin texture and irregular details
and a flexible fusion scheme [6]. We call a face recognition method as Digital
Image Skin Correlation, DISC, which is based on a feature tracking method.
Unlike other feature tracking methods that require a set of sparse markers to be
attached to the actor’s face or have a sparse set of points, the natural texture of
the skin for tracking the motions of the face is used by DISC. The distribution of
pores of skin on the face provides a highly dense texture, which they exploit for
capturing fine-scale facial movements. By tracking an individual’s skin pores
during a facial expression, such as a subtle smile, they can generate a vector
field that describes the individual’s facial movement. They refer to these vector
fields as facial deformation fields. Since facial movements are comprised of a

24
complex sequence of muscle actions, which are unique to an individual, it is
almost impossible to imitate an individual’s facial expression. This suggests that
they can use the facial deformation fields for face recognition. S. Pamudurthy, E.
Guan, K. Mueller, and M. Rafailovich et al. (2005) said their method leverages
the strengths of both still-image and video-based methods, using high-resolution
images to extract temporal information. An important feature of their method is
that it is completely based on affordable mainstream technologies. The only
hardware they require is a high-resolution digital camera. In their experiments,
they had observed that a digital camera with resolution greater than 4 mega
pixels is sufficient to capture images that enable us to accurately track the skin
pores on an individual’s face. Digital Image Skin Correlation (DISC) is based on
a material science technique called the Digital Image Speckle Correlation. DISC
analyzes two images taken before and after the skin has deformed and yields a
vector field corresponding to the skin deformation. Displacement vectors are
derived by the DISC by tracking pores of the skin in the digital images before
and after deformation. In order to recognize an individual, they capture two high-
resolution images of the subject’s face – one with a neutral expression and the
other with a subtle smile. They used DISC to compute the deformation field that
represents the way the subject’s deforms of face. Then, they generate two
scalar fields from each vector field, instead of directly using the field of
deformation as the feature for recognition, which are then used in the
recognition process. They call these scalar fields’ projection images since they
are obtained by projecting each vector in the deformation field onto the X- and
Y-axes, respectively. Alternatively, they also separated the deformation field into
magnitude and orientation fields, and possibly used their gradients as features.
In order to validate the effectiveness of their framework and test its robustness,
they compared the ability of their approach with that of still-image-based
approaches in the task of recognizing people with heavy makeup. For this
purpose, two small databases have been created since the standard recognition
of the face databases does not contain the kind of images and tracking data

25
they require. To compare their approach to still-image-based recognition, they
used the PCA+LDA algorithm from the CSU Face Identification Evaluation
System (CSU FaceId Eval) method as their baseline algorithm. The author
contributed a novel method for face recognition that is based on tracking the
dense texture of skin pores naturally present on the face instead of placing
cumbersome and less dense markers on the person’s face. Their approach
combines the strengths of both the still-image methods and video-based
methods [7].

3.2 Demographic based authentication

To verify the faces they used two methods first method namely; “attribute”
classifiers – uses binary classifiers trained to recognize the presence or absence
of describable aspects of visual appearance (e.g., gender, race, and age). And
second method – “simile” classifiers – removes the manual labeling required for
attribute classification and instead learns the faces similarity, or faces regions to
particular reference people. In this paper they attempt to advance the state-of-
the art for face verification in uncontrolled settings with no cooperative subjects.
To this end, they present two novel and complementary methods for verification
of face. The idea of extracting and comparing “high-level” traits or visual
features, of an image of the face that are not sensitive to pose, expression,
illumination, and other imaging conditions which are common to both methods.
Their approach for face verification does not use expensive computation to align
pairs of faces. The relatively short (65–3000 dimensional) vector of outputs from
the trait classifiers (attribute and simile) are computed on each face
independently. Comparing two faces is simply a matter of comparing these trait
vectors. They introduced Attribute classifier, Smile classifiers, Pub Fig data set
and Human Performance Evaluation. Also they used to perform face verification
on a pair of images; they compare the score of both images. In their method
number of steps are involved: Extract low level features, Compute visual traits,
and Perform Verification. The author presented and evaluated two approaches

26
for face verification using traits computed on face images – based on
describable attributes and novel simile classifiers. N. Kumar, A. C. Berg, P. N.
Belhumeur, and S. K. Nayar et al. (2009) contributed this was the first time such
attributes have been applied to face verification [4]. Utilization of soft biometrics
traits like height , age ,gender, weight, and ethnicity to complement the identity
information provided by the primary biometric identifiers. The biometric
recognition system is divided into two subsystems. One subsystem is called the
primary biometric system and it is based on traditional biometric identifiers like
face, hand-geometry and fingerprint. The second subsystem which is also
known as the secondary system of biometric is based on the traits of soft
biometric like age, gender, and height. The author contributed utilization of
ancillary user information like gender, height, and performance of the traditional
biometric systems like fingerprint can be improved by the ethnicity. Although
these soft biometric characteristics are not as permanent and reliable as the
traditional biometric identifiers like a fingerprint, they provide some information
about the identity of the user that leads to higher accuracy in establishing the
user identity [3]. To improve the Forensic Examination, the FBI also preferred
Forensic Audio, Video and Image Analysis Unit (FAVIAU) perform forensic
examinations such as human identification (from image and video) including
comparison of faces, comparison of ears, comparison of hands, and
determination of height. The author conducted two types of examination to
involve biometrics performed by image examiners within FAVIAU- photographic
comparisons and photogrammetry. In both of these examinations, images
depicting the subject, i.e. the 'questioned individual', are submitted to the FBI for
analyses. The photographic comparisons of peoples are generally one to one
comparisons of person questioned and unknown. The goal of photographic
comparisons is to determine if the questioned individual is the suspect. If the
questioned individual and the unknown are identical, the process is known as
individualization or identification in forensic community. The other types of
photogrammetry are the indirect determination of special measurements of the

27
object using photographic image. The main objective of photogrammetry is to
determine the height of a questioned individual or to determine the length of a
weapon used in the commission of a crime. This also involves two methods
reserve projection or analytical photogrammetry. Height determination is
commonly done to eliminate the suspect or to link multiple crimes, because
human height follows a Gaussian distribution and thus can rarely used for
identification of a person [10].

3.3 Different face recognition methods

A local feature based method for matching facial sketch images to photographs
of faces was used for performing such matching and this is the first known
feature based method. Starting with a sketch training set of correspondences of
the photo such as a set of sketch and photo images of the same subjects, B.
Klare and A. K. Jain et al. (2010) demonstrated the ability to match the sketch
photos directly using SIFT feature descriptors. Their method measures the
similarity between a sketch and a photo by their distance from the training set of
sketch or photo pairs. They first computed using SIFT feature representation for
each sketch query, gallery photograph and each sketch or photo
correspondence to the dictionary. The author contributed that their proposed
sketch or photo matching method is an effective than others. Their method uses
local image feature to describe both sketch and photo images in a common
representation framework [8]. The facial biometrics community is currently
focused on developing new algorithms or attempting to fuse biometrics. The
simple solution to enhance the accuracy of automated systems right now is to
utilize human subject matter experts in the procedure. Every person contrasting
facial images will be trained to the highest possible level of knowledge and
practice in their field. The human face is effectively an elastic sculpture built
upon the foundation of bones and muscles, wrapped in skin, and subject to
change with expression, age, and damage. The specifics that need to be
included in a training program are the properties of the face, the alteration of the

28
face, and the methods of comparing faces. The author elaborated that all facial
images should be trained in the structure of the bones and muscles of the head.
Training should include a link between external expressions and the internal
muscle groups involved in making such expressions, in order for the examiner to
compare images depicting differing facial expressions. Furthermore, because
wrinkles are typically formed perpendicular to the direction of muscle extension,
knowledge of the muscles of the head is important to understanding the aging of
the face. The outermost layer of the face, the skin, bears the texture that is most
critical in individualizations. Commonly it is the co-location of facial blemishes,
marks, and patterns thereof that allow a facial image examiner to identify a
person to the exclusion of all others. Also they pay attention to the
characteristics of the ear. N. A. Spaun et al. (2009) also mentioned the
drawbacks to facial biometrics since the face is not stable over the life and aging
of a face is the most common alteration depicted [9]. For recognition of human
faces from single images out of a large database containing one image for each
person, faces are represented by labeled graphs based on Gabor wavelet
transform. Image graphs of new faces are extracted by an elastic graph
matching process and can be compared by a simple similarity function. The
author made three major extensions to this system; firstly they used the phase
of the complex Gabor wavelet coefficient to achieve more accurate location of
the nodes and to disambiguate patterns which would be similar in their
coefficient magnitudes. Secondly, they employed object adapted graphs, where
the nodes refer to specific facial landmarks called fiducial points. Across large
viewpoint changes, the correct correspondences between two faces can then be
found. Thirdly, they had introduced a new data structure, called the bunch
graph, which serves as a generalized representation of faces by combining jets
of a small set of individual faces. This allows the system to find the fiducial
points in one matching process, which eliminates the need for matching each
model graph individually. This reduces computational effort significantly. A first
set of graphs is generated manually. Nodes are located at fiducial points and

29
edges between the nodes as well as correspondences between nodes of
different poses are defined. Once the system has an FBG (possibly consisting of
only one manually defined model), graphs for new images can be generated
automatically by elastic bunch graph matching. Initially, when the FBG contains
only some faces, it is required to correct and review the resulting matches, but
once the FBG is rich enough (approximately 70 graphs) one can rely on the
matching and generate large galleries of model graphs automatically. Matching
a FBG on a new image is done by maximizing a graph similarity between an
image graph and the FBG of identical pose. It depends on the jet similarities and
a topography term, which takes into account the distortion of the image grid
relative to the FBG grid. The author had applied it to face recognition. They also
tested the system with respect to rotation in depth and differences in facial
expression. They did not investigate robustness to other variations, such as
illumination changes or structured background [11]. A new approach for face
recognition one that is insensitive to large variations in lighting and facial
expressions. P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman et al. (1997)
noted that lighting variability contains intensity and also the number and
direction of light sources. Their approach to face recognition exploits two
observations: a) All of the images of a surface of Lambertian, which are taken
from a fixed viewpoint that are under varying illumination, lie in a linear
subspace of 3D of the high-dimensional image space. b) Because of regions of
specularities, shadowing and expressions of the faces, the above observation
does not exactly right. In practice, certain facial regions may have varied from
image to image that often deviates significantly from the linear subspace and
consequently are less reliable for face recognition. They made use of these
observations by finding a linear projection of the faces from the high-dimensional
image space to a significantly lower dimensional feature space which is
insensitive both to variation in lighting direction and facial expression. They
considered projection directions that are nearly orthogonal to the within-class
scatter, projecting away variations in lighting and facial expression while

30
maintaining discriminability. Their method ‘fisherfaces’ which is a derivative of
Fisher’s Linear Discriminant (FLD) maximizes the ratio of between-class scatter
to that of within-class scatter. The experiments suggest a number of
conclusions: a) all methods perform well if presented with an image in the test
set which is similar to an image in the training set. b) The ‘fisherface’ method
appears to be the best at extrapolating and interpolating over lighting variation,
even though the Linear Subspace method is a close second. c) Removing the
largest three principal components does improve the performance of the
Eigenface method in the presence of lighting variation, but does not achieve
error rates as low as some of the other methods described here. d) In the limit,
as more principal components are used in the Eigenface method, performance
approaches that of correlation method. Similarly performance improves as the
dimensionality of the feature space is increased when the first three principal
components have been taken out. However, that performance seems to level off
at about 45 components of the principal. A similar point of diminishing returns is
found out by Sirovitch and Kirby when using Eigenfaces to represent face
images. e) The Fisherface method appears to be the best at simultaneously
handling variation in lighting and expression. The Linear Subspace method
suffers when confronted with a variation in facial expression as expected [12].
Another way appearance-based face recognition method is the Laplacianface
approach [13]. The images of face are mapped into a subspace of face for
analysis by using Locality Preserving Projections (LPP). Unlike from Principal
Component Analysis i.e. PCA and Linear Discriminant Analysis i.e. LDA, where
the Euclidean structure of the face space is seen effectively, an embedding is
found out by LPP where local information is preserved, and a face subspace is
obtained and the essential face manifold structure is detected at best. The faces
of the Laplacian are the minimal linear approximations to the Eigen functions of
the Laplace Beltrami operator on the manifold of face. They showed how PCA,
LDA, and LPP arise from the same principle applied to different choices of this
graph structure. Author illustrated that while the Eigenfaces method aims to

31
preserve the global structure of the space of images, and the Fisherfaces
method aims to preserve the discriminating information; the Laplacianfaces
method aims to preserve the local structure of the space of the image. The local
manifold structure is more significant than the global in many real-world
classification problems, especially when nearest-neighbor like classifiers,
Euclidean structure is used for classification. Although it is unsupervised, LPP
seems to have power of the discrimination. An efficient subspace learning
algorithm for face recognition should be able to discover the nonlinear manifold
structure of the face space. Their Laplacianfaces method which is proposed to
explicitly also consider the manifold structure which is modeled by an adjacency
graph. Moreover, by finding the optimal linear approximations to the Eigen
functions of the Laplace Beltrami operator, the Laplacian faces are obtained on
the face manifold. By finding the optimal linear approximations to the
eigenfunctions of the Laplace Beltrami operator on the manifold, LPP is
obtained. LLE is nonlinear, while LPP is linear. Moreover, LLE is defined only on
the training data points while LPP is defined everywhere, and it is not clear how
to evaluate the maps for recent test points. In contrast, to locate it in the reduced
representation space, LPP may be simply applied to any new data point. The
manifold ways of face analysis (representation and recognition) are introduced
in that paper in order to detect the underlying nonlinear manifold structure in the
manner of linear subspace learning [13]. P. S. Penev and J. J. Atick et al. (1996)
constructs, from the global PCA modes, a local topographic representation of
objects in terms of local features. The procedure, which they call local feature
analysis i.e. LFA, a dense set of local feed forward is derived in a receptive
field, defined at each point of the grid of receptor and different from each other,
that is optimally matched to the input ensemble, and whose outputs are as
decorrelated as possible. Since the objects from the input ensemble span only a
very-low-dimensional subspace, the dense set of outputs necessarily contains
residual correlations. They used these residual correlations to define lateral
inhibition which acts to specify the output. Thus, the final representation is a

32
local sparse-distributed representation in which only a small number of outputs
are active for any given input. The number of active units is of the order of the
dimensionality of the PCA subspace, but their subset changes from one input
example to another, providing valuable additional information about the locations
of the features in the currently represented example. They gave a practical
implementation for finding approximations to the stable states of this network
dynamics that is computationally very efficient on a serial machine. They
illustrate LFA by using it to derive local features in three different object
ensembles that are similar to ensembles for which global representations have
been derived previously. The first ensemble 2D images of faces without
background serves to illustrate the ability of the method to derive local features
intrinsic to objects; it yields receptive fields for noses, mouths, eyes, cheeks, etc.
The second ensemble 3D surfaces of human heads while producing the same
conceptual features, has the added advantage of being much more controlled
since it contains no variability due to scale, light or albedo in the data. In this
case we derive high-quality 3D filters that are matched to the 3D features. The
third ensemble comprises of 2D images of faces on a background. Besides the
regular ‘face’ features, the most prominent additional ones that LFA derives, are
those signaling transitions from the background to the head. The positions of
activation provide an accurate outline of the head that could subsequently be
used for segmentation. In their paper they had shown how to overcome the main
limitation of the PCA representation by developing LFA. A method for deriving a
dense set of local topographic feed forward receptive fields that are defined at
each point of the receptor grid, different from each other, optimally matched to
the second-order statistics of the input ensemble, and having outputs as
decorrelated as possible. Then they used the residual correlations to define
lateral inhibition which specifies the output, recovering the low dimensionality,
and further decorrelating the representation. The resulting receptive fields are
matched to the structures that are expected at their respective centers, thus
giving a representation of the objects in terms of their local features [14]. A new

33
unsupervised nonlinear feature extraction method is called spectral feature
analysis i.e. SFA. The advantages of SFA over traditional feature extraction
methods are: (1) SFA does not suffer from the small-sample-size problem; (2)
SFA can extract discriminatory information from the data and that linear
discriminant analysis which can be subsumed under the SFA framework is
shown; (3) SFA can effectively discover the nonlinear structure that are hidden
in the data. For face recognition tasks, these appealing properties make SFA
very suitable. Their work here is motivated by the recent works on spectral
clustering, which is a class of clustering methods based on the spectral graph
theory. These clustering methods usually take two steps. First, the patterns are
embedded in a lower-dimensional space such that the clusters are more
“obvious” or separate. Then, a classical clustering algorithm (such as the K-
means clustering algorithm) is performed in the embedded space. Spectral
clustering can yield impressively good results when traditional clustering
approaches fail, and it has been shown to be very powerful in fields such as
computer vision and VLSI design. They presented in their paper a related
unsupervised feature extraction method called spectral feature analysis (SFA).
The embedded data resulting from the first step of spectral clustering will be
referred to as spectral features. Since spectral clustering can only compute
spectral features for the training set, a natural question is how to obtain spectral
features for the unseen (i.e., testing) data. To address this problem, they first
proposed a weighted extension of the standard kernel principal component
analysis (KPCA) called weighted kernel principal component analysis (WKPCA).
Then, they showed that the spectral clustering is a special case of WKPCA and,
consequently, spectral features of the test set can be obtained by WKPCA. Their
method is particularly suitable for face recognition because of the following
properties: SFA does not suffer from the small-sample-size problem. In both
LDA and LPP, the resultant computational problems involve generalized
eigenvalue problems, which in turn require matrix inversions. The sizes of the
matrices involved, depending on the dimensionality of the image, which are

34
often very large compared to the number of training images. These matrices
may thus be singular and the small sample-size problem occurs. On the other
hand, although the resultant computational problem in SFA is still a generalized
Eigenvalue problem, the size of the matrix that has to be inverted is only
dependent on the number of training patterns. As spectral clustering can
effectively group patterns into clusters, SFA can also effectively extract
discriminative information from the data. Furthermore, they will prove that LDA
can be subsumed under the SFA framework. SFA can effectively discover the
intrinsic nonlinear manifold structure hidden in the data. Indeed, they will prove
that kernel LPP is SFA. In other words, their method can exploit and preserve
the data’s nonlinear manifold structure [15]. Also describes image features that
have many properties that make them suitable for a scene or matching differing
images of an object. To image scaling and rotation, the features are invariant
and partially invariant to change in illumination and 3D camera viewpoint. In both
the spatial and frequency domains, they are well localized, reducing the
probability of disruption by occlusion, clutter, or noise. With efficient algorithms,
large numbers of features can be extracted from typical images. Additionally, the
features are highly different, that allows a single feature to be correctly matched
with high probability against a large feature of the database, giving a basis for
object and recognition of scenes. The extracting cost of these features is
minimized by taking a cascade filtering approach, in which the more expensive
operations are applied only at locations that pass an initial test. There are the
major stages of computation used to generate the set of features of image: 1.
Space of scale detection of extrema: The first stage of computation searches
over all scales and locations of image. To identify potential interest points that
are invariant to scale and orientation, it is implemented efficiently by using a
difference-of-Gaussian function. 2. Localization of keypoint: A detailed model is
fit to determine the location and scale at each location of the candidate. Based
on measures of their stability, key points are selected. 3. Assignment of
orientation: One or more orientations are assigned to each keypoint location

35
based on local directions of image gradient. In image data, all operations of the
future are performed which has been transformed relative to the assigned
orientation, location and scale for each feature. As a result, invariance is
provided to these transformations. 4. Descriptor of Keypoint: At the selected
scale in the region around each keypoint, the local image gradients are
measured in which it is transformed into a representation which allows for
significant levels of local shape distortion and change in illumination. This
method is named as the Scale Invariant Feature Transform i.e.SIFT, as it
converts image data into scale-invariant coordinates relative to features of local.
From a set of reference images, SIFT features are first extracted and stored in a
database for image matching and recognition. By comparing individually each
feature from the new image to this previous database and finding candidate
matching features based on Euclidean distance of their feature vectors, a new
image is matched. Due to their distinctiveness, the SIFT keypoints described in
their paper are particularly useful which enables to be selected from a large
database of other keypoints and gives the correct match for a keypoint. By
assembling a high-dimensional vector representing the image gradients within a
local region of the image, this distinctiveness is achieved. The keypoints have
been shown to be invariant to rotation of the image and scale and robust affine
distortion across a substantial range, noise addition, and illumination change.
From typical images, large numbers of keypoints can be extracted, which result
to the robustness among the clutter in extracting small objects. The keypoints
are detected over a complete range of scales and small local features are
available for matching objects which are occluded small and highly, while for
images subject to noise and blur, large keypoints performed well. Their paper
has also presented methods for using the keypoints for object recognition [16].
To extract the feature most powerful algorithm SURF (Speed-Up Robust
Features) was introduced in the year 2006. The author claimed that his
algorithm is more powerful than SIFT algorithm. They used SURF as feature
extraction corresponding points between two images of the same scene or

36
object of many computer vision applications. The search between two
corresponding points can be divided into three steps. First, ‘interest points’ are
selected at distinctive locations in the image, for instance corners, T-junctions
and blobs. Repeatability is the most valuable property of an interest point
detector. The reliability of a detector is expressed by the repeatability for finding
the same physical interest points under different viewing conditions. Secondly, a
feature vector represents the neighborhood of every interest point. This
descriptor must be robust to noise, displacements of detection and geometric
and deformations of photometric and at the same time, it has to be distinctive.
Finally between different images, the descriptor vectors are matched. Based on
a distance between the vectors, the matching is done. Also they focused on the
scale and in plane rotation invariant detectors and descriptors. Common issue
between feature complexity and robustness to commonly occurring photometric
deformations offered by it was studied. Perspective effects, anisotropic scaling
and skew are assumed to be second order effects that are covered to some
degree by the overall robustness of the descriptor. The descriptor can be
extended towards affine invariant regions using affine normalization of the
ellipse. Their work involved mainly on the interest point detection scheme and
new descriptor. They used Hessian matrix detection scheme because of its
good performance in computation time and accuracy. Their proposed SURF
had more fast and performance interest point detection-description scheme
which outperforms the current state of the art, both in speed and accuracy
claimed by the author [26]. Another survey conducted recently on Speed up
robust feature detection explores the reduction of the scale space “Octaves” as
a key speed improvement. Rather than simply discarding the higher octaves,
which may reduce accuracy, their paper developed a more sophisticated
approach, which is termed as best octaves, that examines all octaves and then
chooses two octaves that provide the best matching performance according to
their argument. The author demonstrated that the best octaves outperform
SURF variants obtained using the convolutional approach. They claimed that

37
this was the first systematic approach to SURF octave section and the time
complexity is reduced [27]. The use of multiple sensors addresses the problem
of noisy sensor data, but all other potential problems associated with unimodal
biometric systems remain. A recognition system that works on multiple instances
of the same biometric can ensure the presence of a live user by asking the user
to provide a random subset of biometric measurements. A multimodal biometric
system based on different traits is expected to be more prone to noise, the
problem of non-universality is addressed, the matching accuracy is improved,
and reasonable protection is provided against spoof attacks. Therefore, based
on multiple biometric traits the development of biometric systems has received
considerable attention from researchers. In a multimodal biometric system that
uses different traits of biometric, various fusion levels are possible: fusion at the
feature level of extraction, matching the level of score or level of decision. It is
difficult to consolidate information at the feature level because the feature sets
used by different biometric modalities may either be inaccessible or not
compatible. At the decision level, fusion is too rigid since only a limited amount
of information is available at this level. Therefore, integration at the matching
score level is generally preferred to the ease in accessing and combining
matching scores. Given the scores of different modalities, they used the
combination approach to fusion and address some of the issues involved in
computing a single matching score. Normalization is required to transform these
scores into a common domain before combining them as the matching scores
generated by the different modalities are heterogeneous. There has been no
detailed study of these techniques while several normalization techniques have
been proposed. Based on the face, fingerprint and hand –geometry modalities,
in this work, they have systematically studied the effects of different
normalization schemes on the performance of a multimodal biometric system.
The author examined the effect of different score normalization techniques on
the performance of a multi model biometric system and demonstrated that there
is a considerable improvement in the recognition performance of a multi model

38
biometric system that uses the face, fingerprint and hand geometry traits for
user authentication [22].

3.4 Face annotation methods

Investigations are also growing on new user scenario for face annotation in
which users are allowed to multi select a group of photographs and assign
names to that photograph. Then the system will attempt to propagate names
from photograph level to infer the correspondence between faces and images
from photographs. The authors measured the facial similarity using thumbnails
to see who is in each photograph uses his/her mouse to multi select all of the
photographs with a particular individual that he/she wants to level. This interface
further reduces the users labeling efforts because the function of browsing
thumbnails becomes common in most photograph management system, and the
user works on photograph level, instead of face level. Thus users can level the
group of photographs in a batch way. Also the author derived the objective
function as the sum of similarities between each pair of face of the same
individual in different photographs. Maximization of the objective function leads
to the optimal solution for name propagation. The author further takes
advantages from user’s interactive operation between annotation and
propagation to make the system more effective in annotation and propagation.
This means that the name propagation accuracy can be improved if some faces
have annotated before, and similarly, the annotation accuracy can be improved
if some photographs have been associated with names [17]. A novel face
annotation method that systematically and automatically combines (or
leverages) context information about time and space (or background) of
personal photos with conventional FR technologies. In their proposed method,
traditional FR techniques can take benefits to improve annotation accuracy by
taking into account context information. In addition, their method requires no
training data labeled by hand from photos in contrast to previous research in this
field. Practically, it is highly desirable in most cases with a shortage of labeled

39
data. Their proposed method largely consists of three steps: situation clustering
based on contextual information, person clustering with an efficient fusion of
person identity features, and face recognition for the purpose of annotating
faces on personal photos. Using context-based situation and person clustering,
they obtained a useful group, called ‘person cluster’, of multiple facial images
that belong to the same person. In the practical FR applications, they may be
varied significantly due to practical problems like changes in illumination and
pose, although facial traits are extracted from the same person. To overcome
those limitations by combining their multiple evidences, multiple facial images of
the same person are used for FR methods. Multiple facial images included in
each person cluster are to be annotated by conventional FR methods in
conjunction with effective information fusion strategies. Three representative
subspace FR methods are adopted as a FR framework in the following: Principal
Component Analysis i.e. PCA or “eigenfaces”, Fisher-Linear Discriminant
Analysis i.e. FLDA or “fisher-faces, and Bayesian i.e. “Probabilistic eigenspace”.
To efficiently take advantages of multiple facial features per person, feature and
measurement level fusion strategies are used. To demonstrate the propose of
face annotation approach, they conducted extensive experiments with personal
photos used in the standard MPEG-7 VCE-3. The authors concluded that their
method provides new insights for efficient integration of contextual information of
multimedia contents with computer vision technologies [18]. Face detection and
recognition have numerous multimedia applications of wide interest, in which
one is automatic face annotation. A real-time face annotation system
implemented on a cellular-phone-based embedded platform. They developed
the face detector based on the OpenCV face detector. This detection method,
originally proposed by Viola and Jones, (2001) is one of the most widely used
algorithms and can achieve a great detection rate with fast computation. This
method applies a cascade classifier structure that consists of a series of stages.
Each stage consists of several weak classifiers that can quickly reject non-face
regions so more computational resources are reserved to examine more

40
promising, face-like regions. For the face recognition purpose the author used
ARENA algorithm. Also they used ARENA for comparing all pixel values of the
detected face to those of images in the training set [19].

3.5 Facial feature detection algorithms

To detect a facial feature used Active appearance model (AAM). An AAM

contains a statistical model of the shape and the gray level appearance of the
object of interest. The approach follows the method described in Edwards et al.
(1996) but include extra normalization and weighting steps. The models were
generated by combining a model of shape variation with a model of the
appearance variation in a shape normalized frame. The collected training set of
images which are labeled, where on each example object, key landmark points
are marked. For instance to build a face model we need face images marked
with points at key position to outline the main feature. Given such a set, they
generated a statistical model of shape variation. The labeled points on a single
object describe the shape of that object. Then the set is aligned into a common
coordinate frame which is represented by a vector x, The Principal Component
Analysis (PCA) is applied to the data. To build the statistical model of the grey
level appearance they warp each example image so that its control points match
the mean shape using triangulation algorithm. They used training set of 400
images of faces, each labeled 122 points around the main features. From that
they generated a shape model with 23 parameters. To new images, they
demonstrated an iterative scheme for fitting an Active Appearance Model (AAM).
This method makes use of learned correlation between model displacement and
the resulting difference image [20]. For various image analysis and synthesis
problems, Active Appearance Model (AAM) framework is a very useful method
that can fit the shape and appearance model to the input image. However, it
often fails to accurately converge to the landmark points of the input image since
the goal of the AAM fitting algorithm is to minimize the residual error between
the model appearance and the input image. To alleviate this weakness, They

41
combined Active Shape Models i.e. ASM into AAMs, by using the local profile
model, to find correct landmark points. They derived a gradient based iterative
method by modifying the objective function of the ASM search since the original
objective function of the ASM search is not appropriate for combining these
methods. They proposed a new fitting method that combines the objective
functions of both ASM and AAM into a single objective function in a gradient
based optimization framework. The proposed AAM + ASM algorithm pre-
computes the linear shape, appearance, and profile model independently from a
set of the landmark training images and uses these models simultaneously to
find the optimal model parameters for an input image. By integrating the AAM
and ASM error terms and optimizing them simultaneously, we could obtain more
accurate solutions than using only the AAM error term. This improvement is due
to the fact that the ASM error term enforces to move the shape points to nearby
edge-like points. If we only take the AAM error term, the AAM fitting often fails to
converge to the ground truth points because there are no distinctive texture
patterns in the cheek area of the face. One thing to note here is that the purpose
of the proposed algorithm is not obtaining an illumination robustness fitting but
obtaining a more accurate fitting. Experimental results show that the proposed
fitting method successfully improves the fitting results in terms of root mean
square (RMS) positional errors when compared to ASM, AAM, and TC-ASM
[21].

3.6 Edge detection method

John Canny has developed a method called Canny edge detection in 1986 to
detect edges. Their paper presented a method to develop a mathematical model
for these two criteria which can be used to design detectors for arbitrary edges.
They also discovered that the first two criteria are not "tight" enough, and that it
is necessary to add a third criterion to circumvent the possibility of multiple
responses to a single edge. To derive detectors for several common image
features, including step edges, they used the criteria in numerical optimization.

42
They find that there is a natural uncertainty principle between detection and
localization performance on specializing the analysis to step edges, in which two
main goals are there. They derive a single operator shape with this principle,
which is minimal at any scale. A simple approximate implementation is there in
the optimal detector in which edges are marked at maxima in gradient
magnitude of a Gaussian-smoothed image. They extend this simple detector
using the operators of several widths to cope with different signal-to-noise ratios
in the image. For the fine-to-coarse integration of information from operators at
different scales, they present a common method, called feature synthesis. They
finally proved that step edge detector performance improves considerably as the
operator point spread function is extended along the edge [23]. Similarly,
optimal filter for edge detection has been studied by Didier Demigny et al (1997).
They defined the criteria to compare the performance of different filters in their
application the discrete domain. The validity of this approach is confirmed by
some comparisons with experimental results. They also defined an optimal
continuous filter and optimal discrete filter. They compared the performances of
the sampled version of the continuous filter to those of the optimal discrete filter.
They highlighted the existence of two classes of derivative filters, according to
whether the impulse response in the continuous domain is continuous or not at x
= 0. For a filter with a discontinuous CIR, They had computed an upper bound
for the criterion of low responses multiplicity. On the other hand, criteria for
better detection and low-responses multiplicity can be used with the smoothing
filters. Finally, these criteria can be used to compute an optimal filter directly in
the discrete domain [24]. Recently one published paper studied a region
growing method, which is largely used in medical image segmentation. To
improve the region growing method, they used three kinds of region growing
method to segment the origin medical image namely confidence connected
threshold, connected threshold and connected threshold with neighborhood. To
evaluate the performance of edges locating and decide which is the most
suitable, they used canny edge detection. The Canny edge detection algorithm

43
is used to obtain the small edges, and then applied the edge information to
compare different region growing methods to choose the suitable one. Further,
they implemented image filtering and edge detection and they claimed that their
method improves the small edges’ locating and promotes the last segmentation
result [25].

3.7 MOTIVATION

The face is one of the most acceptable objects for biometrics study and it has
also been the most common method of recognition that human use in their
visual interactions. The problem with authentication systems based on
fingerprint, voice, iris and the most recent gene structure (DNA fingerprint) has
been the problem of data acquisition. The method of acquiring face images is
non intrusive and thus face can be used as a biometric trait for covert (where
user is unaware that he is being subjected) system. Now the issue is whether
faces covered with thick layer of cosmetic items to cover all scars, marks on the
faces could be recognized with considerable accuracy. This type of study
enhances the accuracy of face recognitions under different noises and thereby
increases the level of biometric security. With this motivation a study has been
done on these issues and two methods have been proposed to address several
issues that arise due to layer of marks and images with low quality. The
relevance of facial marks or micro features ( e.g. scars, birthmarks ) to face
recognition is also presented. The main subject of this research work
concentrates on two aspects (i) face recognition methods for cosmetic applied
face images (ii) low quality face images.
In this thesis the main focus is to develop the face recognition system
based on cosmetic applied face and to provide secure authentication in
biometrics system. The proposed method is to detect the facial marks such as
tattoos, scars, freckles and moles covered by cosmetic material from the face
image. This method proposed gives high accuracy in facial marks detection of
the cosmetic applied faces. In fact the use of the robust marks detection method
44
with soft biometrics traits provided the best facial recognition technology.
However, there are a number of difficulties in recognizing the human face and
authentication of the person perfectly. The difficulties include low quality images
due to sparse dark or light disturbances. To overcome such kind of problems,
suitable algorithms are required to filter the images and detect the face and
facial marks. This technique comprises extensively of detecting the different
facial marks from that of low quality images which have salt and pepper noise in
them.

3.8 THESIS CONTRIBUTIONS

We have developed methods to improve face recognition performance in two

ways: utilizing secondary features (e.g., facial marks), i) Facial marks detection
from the cosmetic applied faces, ii) Facial marks detection from the low quality
image. The contributions of the thesis are summarized below.
 An automatic facial mark detection method is developed that can be used
in face retrieval and matching. We have applied an Active Appearance
Model (AAM) [20] to localize the face boundary and mask primary facial
features such as e.g., eyes, eyebrows, nose, and mouth. Canny edge
detection [23] and Speed Up Robust Feature (SURF)[26] is then applied
on the rest of the face area to detect facial marks.
 Facial marks detection method is developed that can be used in face
retrieval and matching. We have applied an Active Appearance Model
(AAM) and Active Shape Model (ASM) [21] to localize the face boundary
and mask primary facial features such as e.g., eyes, eyebrows, nose, and
mouth. Gradient Location Oriented Histogram (GLOH) [78] is then applied
on the rest of the face area to detect facial marks. A fusion of mark based
matching and a commercial matcher shows that the recognition
performance can be improved. The methods used in this thesis are
described below.

45
4. ROBUST FACIAL MARKS DETECTION METHOD USING AAM AND SURF

4.1 INTRODUCTION

The facial marks detection provides a significant role in face recognition

and matching in biometrics environment. Face recognition still remains a
challenging problem to recognize the face. A number of studies are carried out
to improve the face recognition performance by developing feature
representation schemes. These features include the salient skin regions, which
appear on the human face such as (eg. scars, moles, freckles, wrinkles, etc.)
[10]. Due to improvement in image processing and computer vision algorithms,
the use of facial feature has played a significant role. Local feature has a unique
capability to investigate face images in the forensic application by enhancing
image accuracy.

Figure 4.1, Examples of facial marks Scar, mole, and freckles etc.

46
 The above information is very essential to provide identification of the
suspected folks for forensic experts to provide testimony [9]. Formal face
recognition system encode the face images by applying either local or global
texture features. Local techniques initially detect the individual components of
the human face such as mouth, chin, nose, eyes etc. and prior to encoding the
textural content of each of these components (e.g., EBGM and LFA) [11] [14]. In
global techniques, during the face recognition process, the entire face is
considered as a single entity (e.g., PCA, LDA, Laplacianfaces etc.) [12].
Example of facial marks are given in figure 4.1. Therefore, the given method has
small classifiable marks on the human faces which are considered as noisy.
Hence it is not used in the matching process. However, it’s shown in the Face
Recognition Vendor Test (FRVT 2006) [71], the face skin details such as facial
marks are highly significant to attain the face recognition accuracy. Hence, facial
marks can be used to filter the database to speed up the retrieval process or
differentiate identical twins. Facial marks belong to a more general term, Scars,
Marks, and Tattoo (SMT), which is gaining increasing attention for its utility of
subject identification under non-ideal conditions for forensic investigation [72].
There have been only a few studies utilizing facial marks for the face recognition
purpose [5][73]. Previous studies illustrate that facial marks are primarily
focused on facial recognition performance using standard face image data set.
Park and Jain [5] expressed that facial marks are very essential for identifying
twins using the semi automatic concept. Also Spaun [9][10] explained that facial
examination process carried out in the low enforcement agencies. One of the
major examination steps involves identifying “class” and “individual”
characteristics. The ‘class’ characteristics involve overall facial shape, presence
of hair, hair color, shape of the nose, presence of marks etc. Similarly ‘individual’
characteristics include the number and location of scars, tattoos, location
wrinkles etc in a faces. Lin at el [6] first utilized the SIFT operator [16] to extract
facial irregularities and fused them with global face matcher. Facial irregularities
and skin texture were used as an additional means of distinctiveness to achieve

47
performance improvement. However, the individual types of facial marks not
defined. Therefore the method is not suitable for face database indexing.
Pierrard et al. [5] proposed a method to extract moles using normalized cross
correlation method and a morphable model, they claimed that their method is
pose and lighting invariant since it uses a 3D morphable model. Therefore,
besides moles they did not consider other types of facial marks. Lee et al. [2]
introduced “scars, marks, and tattoos (SMT)” in their tattoo image retrieval
system. While tattoos can exist on any body part and are more descriptive, we
are interested in marks appearing exclusively on the face which typically show
simple morphologies. Pamudurthy et al. [7] used distribution of fine-scale skin
marks pattern. The correlation coefficients calculated between registered skin
patches are aggregated to obtain a matching score.

We completely focused on the detection of the facial marks from the face
image and the hidden facial marks that are covered by cosmetic items. We use
different powerful algorithms to find the facial marks from the cosmetic applied
faces. Our method shows better performance contrast with the existing study.
Figure 4.2, shows the mark based matcher helps in indexing each face image
based on facial marks, these indices will enable fast retrieval as well as for
textual or key word based query. Figure 4.3, shows the detected facial marks in
different types of faces.

Figure 4.2, Examples of textual query and a schematic of face region

segmentation.
48
Figure 4.3, Three different types of example of facial marks results. Full face,
partial face, and off-frontal face.

We initially apply the (AAM) Active Appearance Model using PCA to

detect the facial features. Some facial features such as eyebrows, eyes, nose,
and mouth are subtracted from the detected face image. We create a mean
shape to detect the face automatically and also construct a mask for the face
image. Finally, we apply the canny algorithm to identify local irregularities by
detecting the edges in the image and Speed Up Robust Feature (SURF) to
extract the facial features. Therefore the detected facial marks were combined to
enhance the face matching accuracy. Our technique completely differs
significantly from the previous studies in the following aspects: (1) initially we
extract all the facial marks that are locally salient and covered by cosmetic
items. (2) We concentrate on finding semantically meaningful facial marks
instead of extracting texture patterns that are implicitly based on facial marks.
The proposed facial marks determination concept will be helpful to forensics
and law enforcement agencies because it will supplement existing facial
matchers to improve the identification accuracy.

49
4.2 FACIAL MARK FINDER

The proposed facial mark detection method is based on the appearance

of salient localized regions on the face image which are covered by cosmetic
items. Therefore, a feature detector SURF [26] has been applied to detect the
hidden facial marks. Hence, to detect the direct facial feature on the face image
it will increase the number of false positives facial marks due to presence of
facial features such as eyes, eyebrows, nose, and mouth. Therefore, we initially
detected the facial features of the face image and subtracted the unwanted
regions to find the accurate marks present in the face, after that we extracted
different facial marks present in the face image. The complete process involves
different steps; the complete facial mark detection process is illustrated in Figure
4.4. We will elaborate with the following steps 1) facial feature detection 2) mean
shape mapping 3) face mask construction 4) facial marks blob detection and 5)
facial marks blob classification.

Figure 4.4, Schematic of automatic facial mark extraction process.

50
4.2.1. Facial Feature Detection

We applied Active Appearance Model (AAM) using PCA [20], [74] to

detect automatically facial features with 120 landmarks that delineate the facial
features such as eyes, eyebrows, nose, mouth, and face boundary (Fig. 9).
Therefore the facial features detection process will be disregarded in the
sequence of facial mark detection process which provides easy identification of
the marks. Active Appearance Models both the shape and texture of face
images using the Principal Component Analysis (PCA). We labeled manually a
set of landmark points in the training data:

1.1, A set of shape information X={𝓍1,….𝓍𝒩} and corresponding textures G =

{ℊ1,…… ℊ𝒩} are obtained. Where {𝓍1,….𝓍𝒩} is the training set of aligned shapes
and {ℊ1,…… ℊ𝒩} is the training set of shape free normalized image patches.

1.2, Applying to get eigen value using PCA both on X and G, the principal
components of shape and texture, Px and PG are obtained. Where Px , PG is the
matrix of eigenvectors.

1.3, The image shape and texture, 𝒳new and 𝒢new , of a new face can be
expressed as 𝒳new = 𝒳µ + Pxbx and 𝒢new=gµ + PGbG, where 𝒳µ and gµ ( bX and bG)
are means (weight vectors) of X and G, respectively. The 120 landmark points
were labeled on a subset of database images for training.

4.2.2. Mean Shape Mapping

Active Appearance Model has been applied to detect the landmarks.

Therefore to simplify the mark detection, we map each face image to the mean
shape. Consider Si, i=1,…,N represent the shape of each of the N face images
in the database (gallery) based on the 120 landmarks. Hence the mean shape is
calculated using the equation Sµ= iN1 Si .

51
 r1 r’1

p P’

r2 r3 r’2 r’3

Figure 4.5, Schematic of the texture mapping process using the triangular
Barycentric coordinate system.

Each face image Si is mapped to the mean shape Sµ by using the

Barycentric coordinate-based [75] with the texture mapping process. Initially,
both Si and Sµ are subdivided into a set of triangles. Given a triangle T in S i, its
corresponding triangle T' is found in Sµ . Let r1,r2 , and r3( r'1 ,r'2 and r'3 ) be the
three vertices of T(T' ). Then, any point p inside T is expressed as p=αr1 + r2 +
r3, and the corresponding point p' in T' is similarly expressed as p'=αr' 1 + r'2 +
r'3, where α + + =1. In this way, the pixel value is mapped and shown in
Figure 4.5, using the schematic of the Barycentric mapping process. This
mapping process has repeated for all the points which is present inside all the
triangle, the texture in Si is mapped to Sµ . Hence, after this mapping process, all
face images are normalized in terms of scale and rotation and it facilitated us to
represent each facial mark in a face-centered common coordinate system.
Figure 4.6, shows the schematic of mean face construction.

52
 Figure 4.6, Schematic of the mean face construction.

4.2.3. Face Mask Construction

From the mean shape, we constructed a mask known as Sµ to suppress

false positive values, after applying the facial feature detection for blob
detection. Applying the blob detection operator to the face image mapped into
the mean shape and a mask constructed from Sµ is used to suppress blob
detection on the facial features detection process. Consider the mask
constructed from the mean shape to be denoted as Mg, namely, a generic mask.
However, the generic mask does not cover the user-specific facial features, eg.
beard or small wrinkles around the eyes or mouth, that are likely to increase the
false positives. Therefore we also build Ms known as a user-specific mask using
the edge image. Hence the conventional Canny edge detector [23][24] is applied
to obtain the edge from the face image. The user specific mask Ms, constructed
as a sum of Mg and edges that are connected to Mg is used as an aid to
removing most of the false positives present around the beard or small wrinkles
around eyes or mouth.

53
4.2.4. Facial Marks Blob Detection

Our facial marks mostly appear and present in isolated blobs. To detect
the facial marks which are covered such as cosmetic items, we use a familiar
canny edge detector algorithm [23][24][25] and SURF [26] is applied to extract
the facial marks appeared on the face. The algorithms are illustrated given
below:

4.2.4.1, Canny edge detector: It was developed by John F Canny in 1986. It

allows to detect the blob. Motivated by this algorithm, we detect the hidden facial
marks which are covered with cosmetic items. It uses the Gaussian filter, and it
introduces a two-dimensional Gaussian function, and it uses first order
derivation to filter the image. The two-dimensional Gaussian function is given:

where G is the gradient value, σ is known as variance of Gaussian function to

control the length of Gaussian windows and x,y are the distance away from the
target pixel. The original image denoted as T(x, y) is filtered by Gaussian filter
and the result is given below:

Where F(x,y) is the Gradiant function.

For the G(x, y, σ) having two variables, the partial derivation along x and y
direction are given :

54
Where Gx is the first derivative in the horizontal direction and Gy is the vertical
direction. The filtered image can be described as convolution of T(x,y) and
partial derivation of G(x, y, σ) along x, y:

Where Fx(x,y) and Fy(x,y) are the derivatives of the image in the x and y
directions.

One of the pixels (x, y)’s gradient magnitude and direction in image is given by
F(x,y) and ɵ,

Canny edge detection can detect all the edges which is present in the skin in
silent region.

55
4.2.4.2, SURF: We use SURF (Speed Up Robust Feature) [26] as a feature
detector to extract the facial marks. We also applied Hessian matrix base
detector because of its good performance in computation time and accuracy.
Therefore, we apply Hessian-Laplace detector [76] rather than using a different
measure for selecting the location and the scale and we rely on the determinant
of the Hessian for both. The Hessian matrix H(x, σ) in x at scale σ is defined as
follows:

where Lxx (x, σ) is the convolution of the Gaussian second order

derivative with the image I in point x, and similarly for Lxy (x, σ) and

Lyy (x, σ). To complete the procedure of facial mark detection process is
enumerated below:

a) Landmark detection from the face using AAM.

b) Mapping to mean shape Sµ.
c) Mask construction.
d) Apply Canny edge detector to detect the edges.
e) SURF is applied to extract the features.
f) Represent the marks with bounding box.

(1) Linear (2) Circular

56
 (3) Irregular

Figure 4.7, Mark classification using morphology.

(1) (2)

(3) (4)

57
 (5) (6)

Figure 4.8, Examples of mark detection and classification results. (1),(2), (3), (4),
(5) and (6) Symbols are used to denote six different classes of marks.

Figure 4.9, Mark detection graph

LOG is the existing mark detection performance.

SURF is the our proposed mark detection performance.

58
 Mark detection is evaluated in terms of mark detection accuracy and
image brightness, shown in figure 4.9. The mark detection accuracy of mark
detection with the range of brightness contrast varies from ( 30% , 60%) to
(60%, 10%) in existing work and our proposed work varies from ( 35%, 64%) to (
60%, 10%) from the face image. These results showed the automatic mark
detection accuracy and performance. Our mark detection performance graph
Fig. 4.9, shows that marks detection accuracy is better than the existing marks
detection accuracy. We evaluated the automatic mark detection process which
helps us to improve the face recognition accuracy.

4.2.5. Facial Marks Blob Classification

Each detected blob such as Marks, Moles etc. are assigned a bounding
box tightly enclosing the blob. Then we classify a mark in a hierarchical fashion:
linear versus all, followed by circular point versus irregular. Linearity
classification of a blob used two eigenvalues such as λ1 and λ2 are obtained
from the eigen decomposition on x and y coordinates of blob pixels. When λ 1 is
larger than λ2, the mark is considered as a linear blob. We calculate the second
moment of the blob pixels M2 for the circularity detection. A circle RM2 with
radius M2 will enclose most of the blob pixels if they are circularly distributed.
Therefore, a decision can be made based on the ratio of the number of pixels
within and outside of RM2. The color of the blob can be decided based on the
ratio of the mean intensity of the pixels inside and outside of the blob (e in/eout ).

The classification of blobs is shown in the figures 4.7 and 4.8 shows example
images of facial marks detection results with ground truth and automatically
extracted facial marks using our new method which is covered by cosmetic
items. We demonstrated that our proposed method detects the facial marks
robustly detected.

59
4.2.6 FACIAL MARK BASED MATCHING

We encoded the detected facial marks into a 48 bins histogram

representing the morphology, color, and location of facial marks. To encode the
facial marks, the face image is subdivided into eight different regions in the
mean shape space given in Figure 4. Each mark is encoded by six digit binary
number representing its morphology and color. If facial marks are more than one
found in the same region, a bit by bit summation is applied. The six bin values
are concatenated for the eight different regions in the order as shown in Figure
4.9.1, to generate the 48 bin histogram. If a mark is obtained on the borderline of
the face segments, it is considered in both regions also considering the
variations of the segments across multiple face images of the same subject. The
soft biometric traits based matcher is used to retrieve the data from the
database. Soft biometrics can be combined with any facial matcher to improve
the overall accuracy [22]. The weights are chosen to obtain the best face
recognition accuracy.

Figure 4.9.1, Schematic of the mark based indexing scheme.

60
4.3 EXPERIMENTAL RESULTS

4.3.1. Database

We used two different kinds of databases, CVL database [83] and CMU
PIE database [84][85] to design two separate databases, DB1 and DB2 are
used to evaluate the proposed mark-based matcher. DB1 consists of images
from CVL and CMU PIE databases with 1000 images, one image per subject,
for probe and 2000 images, one per subject for the gallery. DB2 consists of
CMU PIE face images with 1000 images from 1000 subjects for probe and 2000
images from 2000 subjects for the gallery. The image size varies from 384×480
to 640 × 480 (width height) for the CMU PIE and 1000 ×2000 for CVL databases
both at 96 dpi resolution. Therefore, we manually labeled the fifteen facial mark
types for DB2 and eight facial mark types (e.g., based on color and morphology)
for the CVL images in DB1 to create the ground truth. This allows us to evaluate
the proposed facial mark extraction method. We used the images from the CVL
database in DB1.

4.3.2. Face image matching and retrieval

We measure the automatic mark detection method in terms of recall and

precision value. Recall and precision are commonly utilized as performance
evaluation in statistical classification and information retrieval tasks [86]. These
measures are defined in terms of false positive, true positive and false negative.
False positives are the number of automatically detected marks that are not in
the ground truth, true positives are the number of automatically detected marks
that match the ground truth, and false negatives are the number of ground truth
marks that were not detected by the automatic detector.
Facial marks play a critical role to recognize the given face. We focused
on detecting the facial marks which are permanently on the face. Therefore

61
some of the facial marks are not stabilizing in human face for example pimples,
acne or zits. To enhance the matching accuracy, we fixed up the permanent
marks such as moles, scar, birth marks etc. on the face image. It will improve
performance of the recognition or identification of the human face. Also we
focused those face images that includes facial marks but that are covered by
cosmetic items. Our method detects the all kind hidden facial marks on the face
image. We simply filter out the unnecessary facial edges and fix the particular
importance of facial marks. It improves the matching accuracy and enhanced
the results. We collected a small amount of facial images which contains 1000
images. We retrieved the images and it shows 93% of accuracy from the face
database. Figure 4.9.2, shows the detected facial marks that are retrieved from
the database. Our database contains numerous detected face images. We have
two databases DB1 and DB2; DB1 data base contains the facial marks detected
images and DB2 database contains the face matching data. Our proposed mark
detection method is implemented using Matlab 7.10.0(R2010a).

Table 4.9.2, Proposed facial marks matcher & face recognition accuracy

Automatic marks detection Existing method Proposed method

( Cosmetic applied
accuracy face)

Based on facial marks

92.10% 93%

62
 Figure 4.9.2. Examples of face image retrieval result.

4.4 SUMMARY

Facial marks from the cosmetic applied faces such as e.g., freckles, moles, and
scars are salient localized regions appearing on the face which is to be useful in
face recognition and provide more security in biometric authentication. An
automatic facial mark extraction methodology has been developed that shows
good performance in terms of matching and retrieval. The fusion of facial marks
with a state of the art face matcher improves the face recognition performance.
This shows that micro-level features such as facial marks do offer some
discriminating information to enhance the security. Most of the facial marks
detected are semantically meaningful so the users can issue queries to retrieve
images of interest from a large database.

63
5. BIOMETRICS SECURITY: FACIAL MARKS DETECTION FROM THE LOW
QUALITY IMAGES

5.1 INTRODUCTION
A number of studies were examined to improve the face recognition by
developing features representation schemes. These features include the salient
skin regions, which appear on the human face such as scars, moles, freckles,
wrinkles, etc. [10]. Previous studies illustrate that facial marks are primarily
focused upon in evaluating facial recognition performance using standard face
image data set. Park and Jain [73] expressed that facial marks are very
essential in identifying twins using the semi automatic concept. They also
labelled the demographic information such as ethnicity and gender. To detect
the facial features the authors applied AAM manually and facial marks are
detected using LoG with morphological operator. This method though was not
enough to detect the facial marks from the low quality images [73]. But, facial
marks have been used to speed up the retrieval process in order to differentiate
the human faces [77].

Figure 5.1, Example of low quality image with different facial marks.

64
Figure 5.1, shows the example of different kinds of facial marks on the human
face. An existing study found that for the face recognition purpose facial marks
have been utilized very rarely [73][70]. Also Spaun [9][10] explained that facial
examination has to provide identification of “class” and “individual”
characteristics. The ‘class’ involves the overall facial shape, presence of hair,
hair color, shape of nose, presence of marks etc. Similarly ‘individual’
characteristics involve the number & location of scars, tattoos, location wrinkles
etc. on a face. Lin at el. [6] first utilized the SIFT operator [16] to extract facial
irregularities and fused them with global face matcher. However, the individual
types of facial marks are not defined. Therefore their method is not suitable for
face database indexing. Krystian Mikolajczyk and Cordelia Schmid et al [78]
introduced GLOH is a descriptor which extends SIFT by changing the location
grid and using PCA to reduce the size. It is also designed to increase its
robustness and distinctiveness. Lee et al. [2] introduced “scars, marks, and
tattoos (SMT)” in their tattoo based image retrieval system. While tattoos can be
drawn or made at any point of the body, the study of face recognition with
tattoos assumes greater significance. Pierrard et al. [70] proposed a method to
extract moles using normalized cross correlation method and a morphable
model. They claimed that their method is pose and lighting invariant since it
uses a 3D morphable model. They did not consider other types of facial marks
besides moles. Eigen faces [79] was one of the first successful face recognition
methods. It can handle variations and blur in the images quite well, but it
requires a large training set of images. Figure 5.2, shows how marks based
matcher helps in indexing each face image based on facial marks. These
indices enable fast retrieval and also for textual or key word base query.

65
Figure 5.2, Example of textual query and a schematic of face region
segmentation.

5.2 PROPOSED METHOD

Park and Jain et al.[73] used AAM (Active Appearance Model) using PCA
(Principle Component Analysis) to detect the facial landmark from the face
image. The local irregularities that are facial marks (moles, scars etc.) are
extracted using LoG or DoG and morphological operators. Their methods are
applicable only for normal with high quality images. In the literature we found
that their methods are poor to detect the facial marks from the low quality
images. In view of this we considered the low quality images due to sparse dark
or light disturbances known as salt and pepper noise type. Generally these
types of marks affect only a small number of image pixels. Typical sources of
noise include flecks of dust inside the camera as well as from faulty CCD
(Charge Coupled Device) elements. To overcome these inherent problems, we
have proposed a facial mark detection method that can be applied on low quality
images. Our proposed method differs significantly from the existing study in
determining the facial marks from the low quality images. Figure 5.4, shows

66
images which contain salt and pepper noise, (i.e. sparse dark or light
disturbances). Initially, we filter out those face images containing salt and
pepper noise using the Adaptive Median Filter (AMF). Therefore to detect the
landmarks from the face image, we applied powerful techniques ASM (Active
Shape Model) into AAM (Active Appearance Model) using PCA (Principle
Component Analysis). We also subtract the unwanted facial features such as
eyebrows, eyes, nose, and mouth from the face image. Finally, the local
irregularities such as scars, moles, freckles etc. are extracted using GLOH
(Gradient Location Orientation Histogram). Our proposed method showed the
best result contrast with existing study. The complete architecture of the
proposed method with necessary steps involved in our research work is given in
fig 5.3. Our proposed method differs from the previous works in the following
aspects; (i) we concentrate on filtering the face images which contain salt and
pepper type. (ii) All the facial marks that are locally salient due to low quality are
extracted. (iii) We concentrate on finding semantically meaningful facial marks
instead of extracting texture patterns that implicitly include facial marks. These
proposed facial marks determination concepts can be helpful in forensics and
law enforcement agencies as it can supplement existing facial matchers to
improve the identification accuracy.

Input image with noise

To filter the images used AMF To detect the landmarks we applied

( Adaptive Median Filter ) ASM + AAM using PCA

Removed unwanted features eye,

nose, mouth and eye brows

Designing the mean shape to detect

the face automatically

Blob classification for facial marks Constructed the generic mask using
Canny algorithm
67
 Facial marks detection using GLOH
( Gradient Location Oriented Histogram )

Figure 5.3, Overall structure of our proposed method.

Figure 5.4, shows salt and pepper noise images

5.3 FACIAL MARKS ON THE FACE

Facial marks are located in different regions of the face. To know the different
categories of facial marks present in face, we need to analyse the categories of
marks. Different kinds of facial marks are freckle, mole, scar, pockmark, acne,
whitening, dark skin etc. Freckle is a single or a set of dark spots present in the
face. Wherever there was a dense set of spots we labelled them in a single
bounding box. A mole typically appears large in size and darker in color
compared to the other spots. Scar represents the discolored region in the skin

68
due to a cut or injury. Acne is a red region caused due to a pimple and is stable
for a few days to several months. Whitening represents a skin region that
appears brighter in contrast with the surrounding region. We took into
consideration the wrinkles that are larger and omitted the smaller wrinkles near
the eyes and mouth. We also ignored the beard and facial hair in constructing
the ground truth. All other kinds of marks which are not mentioned above are
labelled under the “others” group. Figure 5.5, shows the different facial marks in
a facial image. The average number of marks observed in our database is 9 per
subject. All the images in the database showed at least 2 marks per subject.

Figure 5.5, Different facial marks on the mean face image

5.4 FACIAL MARKS DETECTION METHOD

The proposed mark detection method is based on the Gradient Location

Orientation Histogram (GLOH) [78] that detects local silent points of a low
quality input image or a noisy image. Therefore, to detect the facial feature, it
will increase the local extrema of facial features such as eyes, eyebrows, nose,

69
and mouth. To avoid detecting unwanted local facial features we subtracted
eyes, eyebrows, nose, and mouth from the face image using ASM into AAM
using PCA. To detect the facial landmark we applied ASM into AAM using PCA
and we provide masking process using [21]. The complete facial mark detection
process is illustrated in Figure 5.6, with the following steps 4.1) Noise removal
4.2) Facial feature detection 4.3) Designing the mean shape and mask
construction 4.4) Gradient Location Orientation Histogram and 4.5) Blob
classification for facial marks.

Figure 5.6, Schematic of Automatic Facial Mark Extraction Process.

5.4.1 Noise Removal

Adaptive Median Filter (AMF) [80] is applied to remove the noises (salt and
pepper noise) from the face image and performs spatial processing to find out

70
the pixels that have been affected by impulse noise in an image. Moreover, it
groups pixels as noise by comparing each pixel in the image in its neighbouring
pixels. A pixel which is different from a majority of its neighbours, as well as
being not structurally aligned with those pixels to which it is similar, is defined as
impulse noise. The median pixel value of the pixels in the neighbourhood that
have passed the noise labelling test then replace these noise pixels. The filtered
image from salt and pepper noise type using AMF is shown in Figure 5.7, and
Figure 5.8, 5.9, shows the histogram of an image. The purpose of applying the
AMF is

1). Removing impulse noise from the image.

2). Smoothing of other noises from the image.

3). Reducing distortion, like excessive thinning or thickening of object

boundaries.

Noise with salt & pepper Filtered with AMF

Figure 5.7, Salt and pepper noise filtered with AMF

71
 Figure 5.8, shows histogram for noisy color image

Figure 5.9, shows histogram for noiseless color image

A histogram of an image is produced first by discretization of the colors in the

image into a number of bins, and counting the number of image pixels in each
bin. The x axis of the graph represents the tonal variation, while the y axis
represents the number of pixels in that particular tone.

72
5.4.2 Facial Feature Detection

The integrated Active Shape Model (ASM) into Active Appearance Model (AAM)
[21] is applied to detect automatically 80 landmarks that delineate the facial
features such as the mouth, nose, eyes, eyebrows and face boundary. For
reliability, we first detect the landmarks of two eyes, nose and mouth. These
facial features will be disregarded in the subsequent facial mark detection
process. ASM into AAM identifies both the shape and texture of face images
using the Principle Component Analysis (PCA). ASM finds the shape
parameters so that the profile of each model is similar to the pre-learned profile.
It is another way of keeping the shape parameter within the learned range in the
parameter space. Similarly, AAM finds the shape and appearance model
parameters such that the model instance is most similar to the input image.
Models that are obtained from the current model parameter are as similar as the
input image. ASM and AAM are hence combined to reduce the error rate and
detect the face with perfect landmark points.

5.4.3 Designing Mean Shape and mask construction

(5.4.3.1) Active Shape Model in Active Appearance Model has been applied to
detect the landmarks, thereby to simplify the mark detection. We then map each
face image to the mean shape. Consider that Si where i=1, 2…, N represents
the shape of each of the N face images in the database (gallery) based on the


N
80 landmarks. The mean shape is calculated using the equation Sµ= i 1
Si .

r1 r’1

p P’

r2 r3 r’2 r’3
73
Figure 5.9.1, Schematic of the texture mapping process using the triangular
Barycentric coordinate system.

The schematic of a mean face construction system is shown in Figure 5.9.1.

Each face image Si is mapped to the mean shape Sµ by using the Barycentric
coordinate-based [75] texture mapping process. Initially, both Si and Sµ are
subdivided into a set of triangles in such a way that given a triangle T in S i, its
corresponding triangle T' is found in Sµ . Let r1,r2 , and r3( r'1 ,r'2 and r'3 ) be the
three vertices of T(T' ). Then, any point p inside T is expressed as p=αr1 + r2 +
r3, and the corresponding point p' in T' is similarly expressed as p'=αr' 1 + r'2 +
r'3, where α + + =1 . In this way, the pixel value is mapped and shown in
Figure 5.9.2, using the schematic of the Barycentric mapping process.

Figure 5.9.2, Schematic of the mean face construction.

This mapping process is repeated for all the points which are present inside the
triangle and the texture in Si is mapped to Sµ. Hence, after the mapping process,
all face images are normalized in terms of scale and rotation and hence this
facilitates us to represent each facial mark in a face-centered common
74
coordinate system. Fig. 5.9.2, shows the schematic of mean face construction.
(5.4.3.2) We construct a generic mask and derive a user specific mask to
suppress false positive value detection around the facial feature. The user
specific mask covers small unusual landmarks around the facial feature. We
suppress the false positives of small wrinkles or beard that are connected to the
facial feature. We then build a user specific mask from the edge image obtained
using the canny edge detector [23].

5.4.4 Gradient location and orientation histogram (GLOH)

Gradient location and orientation histogram (GLOH) [78] is a descriptor which

extends SIFT by changing the location grid and using PCA to reduce the size. It
is also designed to increase its robustness and distinctiveness. The SIFT
descriptor is computed for a log-polar location grid with 3 bins in radial direction
i.e. the radius set to 6, 11 and 15 and 8 in angular direction, which results 17
location bins. It is to be mentioned that the central bin is not divided in angular
directions and the gradient orientations are quantized in 16 bins. It provides a
272 bin histogram and the size of this descriptor is reduced using PCA. The 128
largest eigenvectors are used for description. Shape context is similar to the
SIFT descriptor, but it is based on edges. Shape context is a 3D histogram of
edge point locations and orientations. Edges are extracted by using the Canny
[22][23] detector. Location is quantized into 9 bins of a log-polar coordinate
system. The radius set to 6, 11 and 15 and orientation quantized into 4 bins
(horizontal, vertical and two diagonals). This leads to a 36 dimensional
descriptor which is allowed to detect the features present on the face image and
it has shown to give better results than using the same weight for all edge
points, as proposed in [81]. Therefore the original shape context was computed
only for edge point locations and not for orientations.

75
 Figure 5.9.3, Marks detection performance graph

LOG, SIFT -- the existing mark detection performance.

GLOH -- the proposed mark detection performance.

Image 10 15 20 25 30 35 40 45 50 55 60 65
Brightness
(%)

LOG(MDA) 69 66 62.5 59.3 54 48.5 42 36.9 34 31.3 28 24

SIFT(MDA) 80 76 72.7 69.5 67 64.9 59 55.3 49 44 40 37

GLOH(MDA) 83 79 75.4 71.9 69 66.1 62 58.1 51 48 44 43

Table 5.9.3.1, Mark Detection Accuracy (MDA) in percentage

76
Mark detection performance is evaluated in terms of mark detection accuracy
and image brightness, as shown in figure 5.9.3. The Y axis denotes the mark
detection accuracy in percentage and X axis denotes the image brightness lavel
in percentage. The mark detection accuracy of mark detection with the range of
brightness contrast varies from ( 26% , 65%) to (69%, 10%) and ( 37%, 65% )
to ( 80%, 10% ) in existing work and our proposed work varies from ( 43%, 65%)
to ( 83%, 10%) from the face image. The mark detection accuracy is provided in
the above table 5.9.3.1. The mark detection performance graph in fig. 5.9.3,
shows that the marks detection accuracy by our method is better than existing
marks detection accuracy. This helps to improve the face recognition accuracy.
The mark detection accuracy is based on an Indian face database [82].

(1) Linear

(2) Circular

77
 (3) Irregular

Figure 5.9.4, Mark classification using morphology.

Figure 5.9.5, Examples of facial marks which are detected from face image.

5.4.5 Blob Classification for Facial Marks

The classification of blobs is shown in the figure 5.9.4. Each detected blob such
as Marks, Moles, freckles etc. are assigned a bounding box. Pixels in the
bounding box are binarized with a threshold value selected from the mean value
78
of the surrounding pixels. Local extrema is brighter or darker than its
surrounding region, so the average value of the surrounding area can serve
effectively for fixing the bounding box. Then a mark is classified in a hierarchical
fashion: linear versus all, followed by circular point versus irregular. In the
linearity classification of a blob, λ1 and λ2 are the two eigenvalue that we
obtained from the eigen decomposition on the x and y coordinates of blob pixels.
When λ1 is larger than λ2, the mark is considered as a linear blob. Then the
second moment of the blob pixels M2 for the circularity detection is calculated. A
circle RM2 with radius M2 will enclose most of the blob pixels. In view of this, a
decision can be made based on the ratio of the number of pixels within and
outside of RM2.

Figure 5.9.5, elaborates facial marks detection results using our new
method.

5.4.6 FACE BASED MATCHING FOR FACIAL MARKS

We then encoded the detected facial marks into a 48 bins histogram

representing the morphology, color, and location of facial marks. For encoding,
the face image is subdivided into eight different regions in the mean shape
space. Each mark is encoded by six digits binary numbers representing its
morphology and color. If more than one facial mark is found in the same region,
a bit by bit summation is applied. The six bin values are concatenated for the
eight different regions in the order as shown in Figure 5.9.6, to generate the 48
bin histogram. If a mark is obtained on the borderline of the face segments, it is
included in both regions considering the variation of the segments across
multiple facial images of the same subject. Given the indices obtained from face
images, the histogram intersection method is used to calculate the matching
scores or filter the candidate face image.

79
 Right Left Right Left Right Left Right Left

Forehead Forehead Eye Eye Check Check Chin Chin

1 1 1 1 1 1 1 1

1 1 1 1 1 1

Point Point Linear Linear Irregular Irregular

Dark Bright Dark Bright Dark Bright

Figure 5.9.6, Schematic of the mark based indexing scheme.

80
5.5 EXPERIMENTAL RESULTS

5.5.1 Database

The face images data sets which contain low quality images are named as DB1
and DB2 which are collected from an Indian face database [82]. DB1 and DB2
are used to evaluate the proposed mark-based matcher. We gathered 1000
images to demonstrate the proposed facial mark detection method. The image
size in our database is 640*480 (width * height) with 96 dpi resolution. We
manually labeled different types of facial marks as defined in section III as
ground truth. This process allows us to evaluate the proposed facial marks
extraction method. DB1 is used to evaluate the effectiveness of the indexing
scheme to improve the facial individuality. The soft biometric traits based
matcher is used to retrieve the data from the database. It can be combined with
any facial matcher to improve the overall accuracy [22]. Weights are chosen to
obtain the best face recognition accuracy.

5.5.2 Image matching and retrieval for security

We measure the automatic mark detection method in terms of recall and

precision value. Recall and precision are commonly utilized as performance
evaluation in statistical classification and information retrieval tasks [86]. These
measures are defined in terms of false positive, true positive and false negative.
False positives are the number of automatically detected marks that are not in
the ground truth, true positives are the number of automatically detected marks
that match the ground truth, and false negatives are the number of ground truth
marks that were not detected by the automatic detector.

We have used DB1 and DB2 for the soft biometrics retrieval and
matching experiments. We applied our marks based matcher on facial images to
identify the human face. The soft biometric matcher successfully retrieves the
correct images to probe the face images. The facial feature points are manually
81
labeled with 80 landmarks for the partial face and automatically labeled for the
database images. We retrieved the images and it shows 95% of accuracy from
the face database. Figure 5.9.7, shows the face matching and retrieval results in
which all the marks are detected automatically from the database. As some of
the facial marks are not stabilized in our face for example pimples, acne or zits,
problems occurred during the matching. To enhance the matching accuracy we
fixed up the permanent marks such as moles, scar, birth marks etc. on the face
image. It improves the recognition or identification of the particular person. Our
method detects majority of the facial marks from a low quality face image. We
simply filtered out the unnecessary facial edges. It also improved the matching
accuracy and enhances the results. The proposed mark detection method is
implemented using Matlab 7.10.0(R2010a).

82
 Figure 5.9.7, face matching and retrieval results

5.6 SUMMARY

Facial marks from the low quality images which is salt and pepper noise type
due to sparse dark light disturbance due faulty CCD elements and extracting the
features such as e.g., freckles, moles, and scars are salient localized regions
appearing on the face which is to be useful in face recognition and provide more
secure in biometric authentication. An automatic facial mark extraction
methodology has been developed that shows good performance in terms of
matching and retrieval. The fusion of facial marks with a state of the art face
matcher improves the face recognition performance. This shows that micro-level
features such as facial marks do offer some discriminating information to
enhance the security. Most of the facial marks detected are semantically
meaningful so the users can issue queries to retrieve images of interest from a
large database.

83
 6. CONCLUSION AND FUTURE DIRECTION

6.1. Conclusion

We have proposed a new facial mark detection process as well as face

matching which are covered by the cosmetic items on each face. We have
shown that the proposed facial based indexing scheme helps us in matching the
facial marks and to enhance the security based on biometrics. For marks
detection and enhancing purpose, we used powerful canny edge detection
method and SURF algorithm for feature extraction purpose. In contrast to the
previous studies where the facial marks are presented implicitly and with poor
accuracy, we presented the most effective method to detect the facial marks.
We are currently working for further improvement with robust mark detection
methods and we are also studying image resolution requirements for reliable
mark extraction.
We have also proposed a new facial mark detection process from the low
quality images. The images are filtered using the Adaptive Median Filter to
improve the quality of images. We have shown that the proposed method is
also useful for face matching and retrieval to enhance the security and in
biometrics system. GLOH algorithm is used for feature extraction. In contrast to
the previous studies that employed the facial marks implicitly or with poor
accuracy, here we present the most effective method to detect the marks. We
are currently working for further improvement in the different quality of face
images to detect the facial marks. We are also studying image resolution
requirements for reliable marks extraction.

84
6.2 Future Direction

Based on the contributions of this thesis, the following research directions

appear promising. Additional features such as morphology or color of the facial
mark based matching should be considered. This will improve the matching
accuracy with facial marks and enable more reliable face image retrieval. The
face image retrieval system can be combined with other robust face matchers
for faster search. Since each facial mark is locally defined, marks can be easily
used in matching and retrieval given partial faces. The proposed facial mark
detection, and matching system should be evaluated in an image based
recognition systems. The mark based matching techniques can be combined to
build a unified system for video based face recognition.

85
Bibliography

86
REFERENCES

[1] A. K. Jain and U. Park, “Facial marks: Soft biometric for face recognition,”
in Proc. IEEE ICIP, 2009, pp. 1–4.

[2] J.E. Lee, A. K. Jain, and R. Jin, “Scars, marks and tattoos (SMT): Soft
biometric for suspect and victim identification,” in Proc. Biometric
Symposium, Biometric Consortium Conf., 2008, pp. 1–8.

[3] A. K. Jain, S. C. Dass, and K. Nandakumar, “Soft biometric traits for

personal recognition systems,” in Proc. ICBA, 2004, vol. 3072, LNCS, pp.
731–738.

[4] N. Kumar, A. C. Berg, P. N. Belhumeur, and S. K. Nayar, “Attribute and

simile classifiers for face verification,” in IEEE Int. Conf. Computer Vision
(ICCV), 2009, pp. 1–8.

[5] J. S. Pierrard and T. Vetter, “Skin detail analysis for face recognition,” in
Proc. CVPR, 2007, pp. 1–8.

[6] D. Lin and X. Tang. “From macrocosm to microcosm.” In Proc. CVPR,

pages 1355–1362, 2006.

[7] S. Pamudurthy, E. Guan, K. Mueller, and M. Rafailovich, “Dynamic

approach for face recognition using digital image skin correlation,” in
Proc. Audio- and Video-based Biometric Person Authentication (AVBPA),
2005, pp. 1010–1017.

[8] B. Klare and A. K. Jain, “Sketch to photo matching: A feature-based

approach,” in Proc. SPIE, Biometric Technology for Human Identification
VII, 2010, pp. 1–10.
87
[9] N. A. Spaun, “Facial comparisons by subject matter experts: Their role in
biometrics and their training,” in Proc. ICB, 2009, pp. 161–168.

[10] N. A. Spaun, “Forensic biometrics from images and video at the Federal
Bureau of Investigation,” in Proc. BTAS, 2007, pp. 1–3.

[11] L. Wiskott, J.-M. Fellous, N. Kruger, and C. von der Malsburg, “Face
recognition by elastic bunch graph matching,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. 19, no. 7, pp. 775–779, Jul. 1997.

[12] P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman, “Eigenfaces vs.

fisherfaces: Recognition using class specific linear projection,” IEEE
Trans. Pattern Anal. Mach. Intell., vol. 19, no. 7, pp. 711–720, Jul. 1997.

[13] X. He, S. Yan, Y. Hu, P. Niyogi, and H.-J. Zhang, “Face recognition using
laplacianfaces,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 3, pp.
328–340, Mar. 2005.

[14] P. S. Penev and J. J. Atick, “Local feature analysis: A general statistical

theory for object representation,” Comput. Neural Syst., vol. 7, pp. 477–
500, 1996.

[15] F. Wang, J. Wang, C. Zhang, and J. T. Kwok, “Face recognition using

spectral features,” Pattern Recognition., vol. 40, no. 10, pp. 2786–2797,
2007.

[16] D. G. Lowe, “Distinctive image features from scale invariant keypoints,”

Int. J. Comput. Vision (IJCV), vol. 60, no. 2, pp. 91–110, 2004.

88
[17] L. Zhang, Y. Hu, M. Li, W. Ma, and H. Zhang, “Efficient propagation for
face annotation in family albums,” in Proc. ACM Int. Conf. Multimedia,
2004, pp. 716–723.

[18] J. Y. Choi, S. Yang, Y. M. Ro, and K. N. Plataniotis, “Face annotation for

personal photos using context-assisted face recognition,” in Proc.ACM
Int. Conf. Multimedia Information Retrieval, 2008, pp. 44–51.

[19] S.-W. Chu, M.-C. Yeh, and K.-T. Cheng, “A real-time, embedded face
annotation system,” in Proc. ACM Int. Conf. Multimedia, 2008, pp. 989–
990.

[20] T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active appearance

models,” in Proc. ECCV, 1998, vol. 2, pp. 484–498.

[21] Jaewon Sung, Takeo Kande, Daijin Kim, “A Unified Gradient-Based

Approach for Combining ASM into AAM” International Journal of
Computer Vision 75(2), 297–309, 2007.

[22] A. K. Jain, K. Nandakumar, and A. Ross, “Score normalization in

multimodal biometric systems,” Pattern Recognition, vol. 38, no. 12, pp.
2270–2285, 2005.

[23] John Canny, “A computational approach to edge detection”, IEEE

Transactions on Pattern Analysis and Machine Intellgence, vol. PAMI-8,
No.6, November 1986.

[24] Didier Demigny and Tawfik Kamlé, “A Discrete Expression of Canny’s

Criteria for Step Edge Detector Performances Evaluation” IEEE
Transactions on pattern analysis and machine intelligence, vol. 19, no.
11, November 1997.
89
[25] Hong-rui Wang, Jian-li Yang, Hai-jun Sun, Dong Chen, Xiu-ling Liu, “An
improved Region Growing Method for Medical Image Selection and
Evaluation Based on Canny Edge Detection” 978-1-4244-6581-
1/11/$26.00 ©2011 IEEE

[26] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool, “SURF: Speeded Up
Robust Features” Computer Vision and Image Understanding 110 (2008)
346–359

[27] Shoaib Ehsan, Nadia Kanwal, Adrian F. Clark, Klaus D. McDonald-Maier,

“An Algorithm for the Contextual Adaption of SURF Octave Selection
With Good Matching Performance: Best Octaves” IEEE Transactions on
image processing, vol. 21, no. 1, January 2012.

[28] G. Portera and G. Doran. An anatomical and photographic technique for

forensic facial identification. Forensic Science International, 114:97-105,
2000.

[29] S. Z. Li and A. K. Jain (eds.). Handbook of Face Recognition. Springer-

Verlag,Secaucus, NJ, 2005.

[30] P. J. Phillips, P. Grother, R. J. Micheals, D. M. Blackburn, E. Tabassi, and

M. Bone. Face Recognition Vendor Test 2002: Evaluation Report, Tech.
Report NISTIR 6965, NIST, 2003.

[31] G. Yang and T. S. Huang. Human face detection in a scene. In Proc.

IEEE Conference on Computer Vision and Pattern Recognition, pages
453-458, 1993.

90
[32] M.H. Yang, D. Kriegman, and N. Ahuja. Detecting faces in images: A
survey. IEEE Transactions on Pattern Analysis and Machine Intelligence,
24(1):34-58,2002.

[33] R. Chellappa, C. L. Wilson, and S. Sirohey. Human and machine

recognition of faces: A survey. Proc. IEEE, 83(5):705-740, 1995.

[34] R.L. Hsu, Mohamed Abdel-Mottaleb, and A. K. Jain. Face detection in

color images. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 24(5):696-706, 2002.

[35] I. Craw, D. Tock, and A. Bennett. Finding face features. In Proc.

European Conference on Computer Vision, pages 92-96, 1992.

[36] H. A. Rowley, S. Baluja, and T. Kanade. Neural network-based face

detection. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 20(1):23-38,1998.

[37] P. A. Viola and M. J. Jones. Robust real-time face detection. International

Journal of Computer Vision, 57(2):137-154, 2004.

[38] K. Levi and Y. Weiss. Learning object detection from a small number of
examples: the importance of good features. In Proc. IEEE Conference on
Computer Vision and Pattern Recognition, volume 2, pages 53-60, 2002.

[39] B. Heisele, T. Serre, M. Pontil, and T. Poggio. Component-based face

detection. In Proc. IEEE Conference on Computer Vision and Pattern
Recognition,volume 1, pages 657-662, 2001.

91
[40] J. Wu, S. C. Brubaker, M. D. Mullin, and J. M. Rehg. Fast asymmetric
learning for cascade face detection. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 30(3):369-382, 2008.

[41] Open Computer Vision Library,

http://sourceforge.net/projects/opencvlibrary.

[42] M. Kirby and L. Sirovich. Application of the Karhunen-Loeve procedure

for the characterization of human faces. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 12(1):103-108, 1990.

[43] M. Turk and A. Pentland. Eigenfaces for recognition. Cognitive

Neuroscience, 3:72-86, 1991.

[44] W. Zhao, R. Chellappa, P. J. Phillips, and A. Rosenfeld. Face recognition:

A literature survey. ACM Computing Surveys (CSUR), 35(4):399-458,
2003.

[45] R. A. Fisher. The statistical utilization of multiple measurements. Annals

of Eugenics, 8:376-386, 1938.

[46] L. Wiskott, J.-M. Fellous, N. Kruger, and C. von der Malsburg. Face
recognition by elastic bunch graph matching. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 19(7):775-779, 1997.

[47] J. P. Lewis. Fast normalized cross-correlation. Vision Interface, pages

120-123, 1995.

92
[48] P. J. Phillips, P. Grother, R. J. Micheals, D. M. Blackburn, E. Tabassi, and
M. Bone. Face Recognition Vendor Test 2002: Evaluation Report, Tech.
Report NISTIR 6965, NIST, 2003.

[49] Neven Vision, FR SDK,

http://neven-vision-s-fr-dk.software.informer.com/.

[50] L-1 Identity Solutions, http://www.l1id.com.

[51] J. Stallkamp, H. K. Ekenel, and R. Stiefelhagen. Adaptive background

mixture models for real-time tracking. In PProc. IEEE International
Conference on Computer Vision, pages 1-8, 2007.

[52] A. Hadid T. Ahonen and M. Pietikainen. Face recognition with local binary
patterns. In Proc. European Conference on Computer Vision, pages 469-
481, 2004.

[53] W. Zhang, S. Shan, W. Gao, X. Chen, and H. Zhang. Local gabor binary
pattern histogram sequence (LGBPHS): A novel non-statistical model for
face representation and recognition. In Proc. IEEE International
Conference on Computer Vision, pages 786-791, 2005.

[54] L. Wiskott, J.-M. Fellous, N. Kruger, and C. von der Malsburg. Face
recognition by elastic bunch graph matching. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 19(7):775-779, 1997.

[55] P. S. Penev and J. J. Atick. Local feature analysis: a general statistical

theory for object representation. Network: Computation in Neural
Systems, 7:477-500, 1996.

93
[56] B. S. Manjunath and R. Chellappa. A feature based approach to face
recognition. In Proc. IEEE Conference on Computer Vision and Pattern
Recognition, pages 373-378, 1992.

[57] S. Arca, P. Campadelli, and R. Lanzarotti. A face recognition system

based on local feature analysis. In Proc. Audio- and Video-Based
Biometric Person Authentication, pages 182-189, 2003.

[58] J. P. Hespanha P. N. Belhumeur and D. J. Kriegman. Eigenfaces vs.

fsherfaces: recognition using class speci¯ c linear projection. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 19(7):711-
720, 1997.

[59] N. Vaswani and R. Chellappa. Principal components null space analysis

for image and video classification. IEEE Transactions on Image
Processing, 15(7):1816-1830, 2006.

[60] M. Grudin. On internal representation in face recognition systems. Pattern

Recognition, 33(7):1161-1177, 2000.

[61] K. M. Lam and H. Yan. An analytic-to-holistic approach for face

recognition based on a single frontal view. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 20(7):673-686, 1998.

[62] B. Heisele, P. Ho, J. Wu, and T. Poggio. Face recognition: component-

based versus global approaches. Computer Vision and Image
Understanding, 91(1):6- 21, 2003.

94
[63] T.K. Kim, H. Kim, W. Hwang, and J. Kittler. Component-based LDA face
description for image retrieval and MPEG-7 standardisation. Image and
Vision Computing, 23(7):631-642, 2005.

[64] R. Gottumukkal and V. K. Asari. An improved face recognition technique

based on modular pca approach. Pattern Recognition Letters, 25(4):429-
436, 2004.

[65] X. Geng and Z.H. Zhou. Image region selection and ensemble for face
recognition. Journal of Computer Science Technology, 21(1):116-125,
2006.

[66] A. M. Martinez. Recognizing imprecisely localized, partially occluded, and

expression variant faces from a single sample per class. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 24(6):748-
763, 2002.

[67] A. Pentland, B. Moghaddam, and T. Starner. View-based and modular

eigenspace for face recognition. In Proc. IEEE Conference on Computer
Vision and Pattern Recognition, pages 84-91, 1994.

[68] K. Tan and S. Chen. Adaptively weighted sub-pattern PCA for face
recognition. Neurocomputing, 64:505-511, 2005.

[69] D. Lin and X. Tang. From macrocosm to microcosm. In Proc. IEEE

Conference on Computer Vision and Pattern Recognition, pages 1355-
1362, 2006.

95
[70] J. S. Pierrard and T. Vetter. “Skin detail analysis for face recognition.” In
Proc. IEEE Conference on Computer Vision and Pattern Recognition,
pages 1-8, 2007.

[71] P. J. Phillips, W. T. Scruggs, A. J. O’Toole, P. J. Flynn, K.W. Bowyer, C.

L. Schott, and M. Sharpe. Face Recognition Vendor Test 2006: FRVT
2006 and ICE 2006 Large-Scale Results Tech. Report NISTIR 7408,
NIST, 2007.

[72] FBI Next Generation Identification,

http://www.fbi.gov/aboutus/cjis/fingerprintsbiometrics/ngi/ngi2.

[73] U. Park and A. K. Jain. Face matching and retrieval using soft biometrics.
IEEE Transactions on Information Forensics and Security, 5(3):406–415,
2010.

[74] M. B. Stegmann, The AAM-API: An open source active appearance

model implementation, in Proc. MICCAI, 2003, pp. 951–952.

[75] C. J. Bradley. The Algebra of Geometry: Cartesian, Areal and Projective

Coordinates. Bath: High perception, 2007.

[76] Mikolajczyk, K., Schmid, C. “Indexing based on scale invariant interest

points.” In: ICCV. Volume 1. (2001) 525 – 531.

[77] O. Déniz, G. Bueno, J. Salido, F. De la Torre, “Face recognition using

Histograms of Oriented Gradients” Pattern Recognition Letters 32 (2011)
1598–1603.

96
[78] Krystian Mikolajczyk and Cordelia Schmid, “A performance evaluation of
local descriptors” IEEE Transactions on Pattern Analysis and Machine
Intelligence, Volume 27 Issue 10, Page 1615-1630, October 2005.

[79] M.A. Turk and A.P. Pentland, “Face Recognition Using Eigenfaces,” In
Proceedings, IEEE Conference on Computer Vision and Pattern
Recognition, pp. 586-591, June 1991.

[80] Samuel Morillas, Valent´ın Gregori and Almanzor Sapena, “Adaptive

Marginal Median Filter for Colour Images” Sensors 2011, 11, 3205-3213;
doi:10.3390/s110303205.

[81] S. Belongie, J. Malik, and J. Puzicha. “Shape matching and object

recognition using shape contexts.” IEEE Transactions on Pattern Analysis
and Machine Intelligence, 24(4):509-522, 2002.

[82] Jain, V., Mukherjee, A.: The Indian Face Database (2002),
http://vis- www.cs.umass.edu/~vidit/IndianFaceDatabase/

[83] Computer Vision laboratory, Faculty of Computer and Information

Science, University of Ljubljana, Slovenia,
http://www.lrv.fri.unilj.si/facedb.html.

[84] Sim T, Baker S, Bsat M. The CMU pose, “illumination, and expression
database.” IEEE Transaction on Pattern Analysis and Machine
Intelligence. 2003 December;25(12):1615–1618.

[85] T. Sim, S. Baker, and M. Bsat. “The CMU pose, illumination, and
expression (PIE) database.” In Proceedings of the 5th IEEE International
Conference on Face and Gesture Recognition, 2002.

97
[86] C. V. van Rijsbergen. “Information Retrieval (2nd ed.).” London:
Butterworths, 1979.

98
 LIST OF PUBLICATIONS

[1]. Ziaul Haque Choudhury, K.M. Mehata, “Robust Facial Marks Detection
Methods Using AAM and SURF” International Journal of Engineering
Research and Applications (IJERA), ISSN: 2248-9622, Vol. 2, Issue 6,
November- December 2012, pp.708-715.

[2]. Ziaul Haque Choudhury, K.M. Mehata, “Biometrics Security: Facial Marks
Detection from the Low Quality Images” International Journal of
Computer Applications (0975 – 8887) Volume 66 – No.8, March 2013.

99
 TECHNICAL BIOGRAPHY

Mr. Ziaul Haque Choudhury (RRN. 1067201) was born on 31th Dec 1980, in
Durgapur, Silchar, Cachar, Assam. He did his schooling in Govt. Boys H.S. School,
Silchar. He received B.Sc. degree in Computer Science from Bharathidasan University,
Trichy in the year 2007. He did M.Sc.Engineering degree in Information Technology,
Department of Computer Science & Engineering, Annamalai Univerisity,
Chidamabaram in the year 2009. He has got one year industrial experience. He was
employed (Network Engineer) with Allsec Technologies Ltd., Chennai. He is currently
pursuing his M.Tech.(By Research) Degree in Network Security (Biometrics Security) in
the Department of Information Technology, B.S. Abdur Rahman University, Chennai.
His area of interests includes Network Security, Biometrics Security, Pattern
Recognition, Image Processing etc. The e-mail ID is: [email protected] and the
contact number is: 9003481822.

100
101