2014 IEEE 22nd Signal Processing and Communications Applications Conference (SIU 2014)

Gabor Wavelet Transform Based Facial Expression

Recognition Using PCA and LBP
Muzammil Abdulrahman, Tajuddeen R. Gwadabe, Fahad J. Abdu, Alaa Eleyan
Electrical & Electronics Engineering Department, Mevlana University, Turkey
[email protected], {tgwadabe,fjibrin}@yahoo.com, [email protected]

ABSTRACT was investigated. The approach is based on a localized

representation of FE features and on the fusion of classifier
This paper proposed a facial expression recognition outputs. Abboud and Davoine [6] suggested a bilinear
approach based on Gabor wavelet transform. Gabor wavelet factorization expression classifier for facial recognition.
filter is first used as pre-processing stage for extraction of Ekman and Friesen [7], proposed well known FE
the feature vector representation. Dimensionality of the representation and coding system, called facial action
feature vector is reduced using Principal Component coding system (FACS). Their goal was to describe all the
Analysis (PCA) and Local binary pattern (LBP) algorithms. visually distinct local facial muscle a movement using 46
Experiments were carried out of using Japanese female actions units (AUs), a FE corresponds to the combination of
facial expression (JAFFE) database. In all experiments a set of AUs. Lyons et al [8] suggested a technique for
conducted using JAFFE database, results obtained reveal classifying facial images automatically based on labelled
that GW+LBP has outperformed other approaches in this elastic graph matching and 2D Gabor Wavelet (GW)
paper with an average recognition rate of 90% under the representation. Researches in [9, 10, 11] demonstrated that
same experimental setting. GW representation is a good feature extraction technique
because of its biological relevance and computational
Index Terms— Facial Expression Recognition, Gabor properties. W.K. Kong et al [12] used GW as a feature
Wavelet Transform, Principal Component Analysis, Local extraction technique for palm print recognition, A.B.
Binary Patterns. Sharizai et al [13] enhances finger print images by the use of
Gabor Wavelet Transform (GWT).
I. INTRODUCTION The rest of the paper is organized as follows. Section 2 and
3 briefly give an overview of Gabor wavelet filter and
The increase need in researches in the area of digital Dimensionality reduction (i.e. LBP, PCA) respectively.
processing and pattern recognition give rise to the Section 4 describes the proposed method and section 5
development of algorithms for different applications such as discusses the simulation results. Finally the paper is
automated access control, human-computer interaction, and concluded in section 6.
surveillance [1]. For automated access control, the
commonly used method is face recognition but the faces II. GABOR WAVELETS
might exhibit different Facial Expressions (FE). Face
recognition and FE analysis has attracted the interest of A GW filter is an essential tool used to extract local features
researchers in the field of computer vision. both in spatial and frequency domain which can be applied
Although human being can recognize different FE exhibited on images to extract features aligned at particular angles
by face very easily [2], it is not so for computers. In a (orientations). The GWs filter captures significant visual
survey on automatic Facial Expression Recognition (FER) features such as spatial localization, orientation selectivity,
[3], Scientists conducted experiments in order to recognize frequency selectivity, and quadrature phase relationship
the FE from un-occluded facial images taken under [14]. The GW kernel in spatial domain is a complex
controlled laboratory conditions. But at times, the human exponential modulated by a Gaussian function [15].
subject may be talking, thus changing his facial features or The GWs kernel can be defined by the following equation:
his face may be partially occluded. Rosenblum et al. [4] −(
x' 2 + y ' 2
1 )
ψ ( x, y , ϖ , θ ) =
'
applied a technique of networks where the complexity of
2
e 2σ 2 eiϖ x (1)
recognizing FE was grouped into three layers of 2πσ
decomposition. In [5], FER in the presence of occlusion

2265
 2014 IEEE 22nd Signal Processing and Communications Applications Conference (SIU 2014)

where (x,y) denote the pixel position in the spatial domain, computed over a region is used as a texture descriptor. Fig.
ϖ is the central frequency of a sinusoidal plane wave, θ is 2 gives an example of the basic LBP operator.
the orientation of the Gabor filter and σ is the standard
deviation along x and y directions. The parameters x' and
y ' can be defined by the following equations: 32 18 25 1 0 0
threshold
12 27 50 0 1
' '
x = x cos θ + y sin θ , y = − y sin θ + y cos θ (2) 81 78 42 1 1 1
binary sequence: 10011110
Gabor filter bank of various frequencies and orientations has Figure 2 Example of LBP operators
been frequently used to extract features of face image.
However in previous work review [14-17] mainly a forty The limitation of the basic LBP operator is that its small 3×3
Gabor filter bank at five scales, v={0,1,…,4} and eight neighborhood cannot capture the dominant features with
orientations, µ={0,1,…,7} with σ=2π , ϖ = 2 were used. large scale structures. As a result, to deal with the texture at
Fig.1 shows the magnitude of the Gabor at four scales and different scales, the operator was later extended to use
the Real Gabor filter bank with 4 different scales and 6 neighborhoods of different sizes [19]. One of such is the
different orientations. uniform patterns: an LBP is ‘uniform’ if it contains at most
one 0-1 and one 1-0 transition when viewed as a circular bit
string. For instance, 00000000, 001110000 and 11100001
are uniform patterns. It is observed that uniform patterns
account for nearly 90% of all patterns in the (8, 1)
neighborhood and for about 70% in the (16, 2)
neighborhood in texture images, where the notation ( P, R )
denotes a neighborhood of P equally spaced sampling
points on a circle of radius of R that form a circularly
symmetric neighbor set [4].
(a) (b)
B. Principal Component Analysis
Figure 1 (a) Magnitude of the Gabor filter bank at four scales (b) PCA [20] can be defined as an orthogonal linear
Real part of the Gabor kernels at four scales and six orientations. transformation that transforms data to a new coordinate
system such that the highest variance by any projection of
The GW feature representation ϕm,n ( x, y ) is obtained by the data comes to sit on the first coordinate (called the first
principal component), the second highest variance on the
convolving the Gabor filter bank ψ ( x, y,ϖ , θ ) with input second coordinate, and so on. Thus PCA finds orthogonal
image. basis for data, sorts dimensions in order of importance and
ϕm,n ( x, y ) = I ( x, y ) ∗ψ ( x, y,ϖ ,θ ) (3) discard low significance dimensions. The main purpose of
PCA is to reduce the dimensionality of the raw data while
retaining as much as possible the variation present in the
In practice usually the magnitude of the convolution output data set [17].
Gabor feature vector representation is used for FER, PCA can also be thought as a way of recognizing patterns in
because they do vary slowly with the displacement while the data, and expressing the data in such a way as to emphasize
phases are very sensitive under small displacement [18]. their similarities and differences. Since pattern in data may
be difficult to find especially in data of high dimension.
III. DIMENTIONALITY REDUCTION PCA is used in many pattern recognition applications such
FER [21] and face recognition [22].
A. Local Binary Patterns
LBP operator takes a local neighborhood around each pixel,
IV. PROPOSED METHOD
thresholds the pixels of the neighborhood at the value of the
central pixel and uses the resulting binary valued image In this section we explain the method used in this paper. It
patch as a local image descriptor. It was originally defined can be summarized using the diagram shown in Fig. 3. The
for 3×3 neighborhoods, giving 8 bit codes based on the 8 FE database used is the Japanese Female Facial Expression
pixels around the central one, the operator labels the pixels (JAFFE) Database, the images where categorized into
of an image by thresholding a 3×3 neighborhood of each training and testing. All expressions in the database where
pixel with the center value and considering the results as a first pre-processed using GWT with 5 scales and 8
binary number, and the 256-bin histogram of the LBP labels orientations and no down sampling were done. After the

2266
 2014 IEEE 22nd Signal Processing and Communications Applications Conference (SIU 2014)

pre-processing each image within the database has a Gabor+LBP has better performance with averaged
corresponding 40 images and each image is 256×256. Due recognition rate 90% and promise to be the best result
to this large number of images, dimensionality reduction is compared to the rest, though all the remaining algorithms
used. Two methods; PCA and LBP are used here. A uniform appear to demonstrate a good performance as well when
LBP is used in which each image is divided into a region of compared to previous work reviewed.
64×64, a radius of 1 used with 8 samples within each radius.
K-nearest neighbor is used for similarity measure and in this VI. CONCLUSION
case we used Euclidean distance (L2).
Gabor wavelets were used as a pre-processing stage
followed by dimensionality reducing using PCA/LBP for
Preprocessing (Gabor Dimensionality
Facial Expression
Database
Wavelet Transform) Reduction (PCA/LBP) facial expression recognition in this paper. Experimental
evaluations of the proposed approach were conducted on
JAFFE database. The results obtained showed that pre-
processing with Gabor wavelets improves the performance
of directly applying both PCA and LBP.
Decision Similarity Measure (L2)

Table 1 Confusion Matrix for FER using PCA

Figure 3 block diagram of the proposed method

Surprise
Disgust
Neutral

Happy
Anger

Fear

Sad
PCA
The database contains 213 images of 10 subjects with 7
facial expressions namely; neutral, anger, disgust, fear,
happy, sad and surprise. Fig. 4 shows example of the images Neutral 90 0 0 0 0 0 10
from the database. 137 images containing all the expressions Anger 9 64 18 9 0 0 0
were used for training while 76 images were used for Disgust 0 8 83 0 0 8 0
testing. Fear 8 0 8 67 0 8 8
Happy 0 0 0 10 80 10 0
Sad 18 0 0 18 18 45 0
Surprise 0 0 0 0 0 10 90

Table 2 Confusion Matrix for FER using LBP

Surprise
Disgust
Neutral

Happy
Anger

Fear

Sad
LBP

Figure 4 Examples of images from the JAFFE database.

Neutral 100 0 0 0 0 0 0
V. SIMULATION RESULTS Anger 0 90 10 0 0 0 0
Disgust 0 18 81 0 0 0 0
The simulation of the proposed work was implemented in Fear 8 0 0 83 0 0 8
MATLAB using JAFFE database. The JAFFE database Happy 8 0 0 8 50 25 8
contains 213 images posed by 10 female with 7 facial Sad 0 9 0 0 18 73 0
expressions. They were taken from Japanese female model Surprise 0 0 0 0 0 0 100
each having a resolution of 256×256 pixels. The mages in
the database are greyscale images in tiff format. We Table 3 Confusion Matrix for FER using Gabor
categorized the images into 137(64%) images containing all
the expressions used for training data set while 76(36%)
Surprise
Disgust
Neutral

Happy
Anger

Fear

Sad

used as a testing data set. In order to demonstrate and Gabor

compare the performance of the proposed algorithms, we
first obtain the PCA, LBP, Gabor, Gabor+PCA, Gabor+ Neutral 100 0 0 0 0 0 0
LBP features and used the K-nearest neighbour with Anger 9 91 0 0 0 0 0
Euclidean distance (L2) for similarities measure to classify Disgust 0 8 83 0 8 0 0
the face images. The results of the confusion matrix are Fear 0 0 0 75 17 8 0
given in tables, 1, 2, 3, 4 and 5 for PCA, LBP, Gabor,
Happy 0 0 0 10 90 0 0
Gabor+PCA, and Gabor+LBP, respectively. From the result
Sad 0 0 0 0 9 91 0
shown below, it reveals that the performance of
Surprise 0 0 10 0 0 0 90

2267
 2014 IEEE 22nd Signal Processing and Communications Applications Conference (SIU 2014)

[7] P. Ekman, and W. Friesen, “Facial Action Coding System: A

Table 4 Confusion Matrix for FER using Gabor+PCA Technique for the Measurement of Facial Movements”,
Consulting Psychologists Press, California, 1978.
[8] M. J. Lyons, J. Budynek, and S. Akamatsu, “Automatic
Gabor+PCA

Surprise
Disgust
Neutral Classification of Single Facial Images,” IEEE Transactions on

Happy
Anger

Fear

Sad
Pattern Analysis and Machine Intelligence, Vol. 21, No. 12,
pp. 1357–1362, 1999.
[9] S. Bashyal, and G. Venayagamoorthy, “Recognition of Facial
Neutral 100 0 0 10 0 0 0 Expressions using Gabor Wavelets and Learning Vector
Anger 18 82 0 0 0 0 0 Quantization,” Engineering Applications of Artificial
Disgust 0 0 67 0 0 25 8 Intelligence, Vol. 21, No. 7, pp. 1056-1064, 2008.
Fear 0 0 0 58 17 25 0 [10] W. Gu, C. Xiang, Y. Venkatesh, D. Huang, and H. Lin, “Facial
Expression Recognition using Radial Encoding of Local Gabor
Happy 0 0 0 10 70 10 10
Features and Classifier Synthesis,” Pattern Recognition, Vol.
Sad 0 9 18 0 9 63 0 45, No. 1, pp. 80-91, 2012.
Surprise 0 0 0 10 0 0 90 [11] A. Eleyan, and H. Demirel, “Performance Comparison among
Complex Wavelet Transforms Based Face Recognition
Systems,” Image Processing and Communication Conference,
Table 5 Confusion Matrix for FER using Gabor+LBP AISC84, pp. 201-209, August 2010.
[12] W. K. Kong, D. Zhang and W. Li, “Palm Print Feature
Extraction using 2-D Gabor Filters”, Pattern Recognition
Gabor+LBP

Surprise
Disgust
Neutral

Happy

Society, pp. 2339-2347, 2003.

Anger

Fear

Sad

[13] N. Karimimehr, A. A. B. Sharazi and B. M. Kashavar, “Finger

Print Image Enhancement using Gabor Wavelet Transform”,
18th Iranian Conference on Electrical Engineering Proceeding,
Neutral 100 0 0 0 0 0 0 pp. 316, 2010.
Anger 0 91 0 0 9 0 0 [14] S. Fazli, R. Afrouzian, and H seyedarabi, “High- Performance
Disgust 0 0 92 0 8 0 0 FE using Gabor Filter and Probabilistic Neural Network”,
0 0 0 8 17 0 IEEE International Conference on Intelligent Computing and
Fear 75
Intelligent System, Vol. 4, pp. 93-96, 2009.
Happy 0 0 0 0 100 0 0 [15] T. S. Lee, “Image Representation Using 2D Gabor Wavelets”,
Sad 0 0 0 18 0 82 0 IEEE Transactions on Pattern Analysis and Machine
Surprise 0 0 10 0 0 0 90 Intelligence, Vol. 18, pp. 959-971, 1996.
[16] S. Zhang, L. Li, and Z. Zhao, “Facial Expression Recognition
Based on Gabor Wavelet and Sparse Representation”, IEEE
REFERENCES International Conference on Signal Processing, Vol.2, pp. 816-
819, 2012.
[17] C. Liu, H. Wechsler, “Independent Component Analysis of
[1] Z. Niu and X. Qiu, “Facial Expression Recognition Based on
Gabor Features for Face Recognition”, IEEE Transactions on
Weighted Principal Component Analysis and Support Vector
Neural Networks, Vol. 14, pp. 919-928, 2003.
Machines”, IEEE 3rd International Conference on Advanced
[18] G. Donato, M. Bartlett, J. Hager, P. Ekman, and T. Sejnowski,
Computer Theory and Engineering, pp. 174-178, 2010.
“Classifying Facial Actions”, IEEE Transactions on Pattern
[2] N. Vretos, and I. Pitas, “A Model-Based Facial Expression
Analysis and Machine Intelligence, Vol. 21, No. 10, pp. 974-
Recognition Algorithm using Principal Components Analysis”,
989, 1999.
IEEE 16th International Conference on Image Processing, pp.
[19] T. Ojala, M. Pietikainen, and T. Maenpaa, “Multi-resolution
3301-3304, 2009.
Gray Scale and Rotation Invariant Texture Analysis with Local
[3] M. Pantic, and L. Rothkrantz, “Automatic Analysis of Facial
Binary Patterns”, IEEE Transactions on Pattern Analysis and
Expressions: The State of the Art”, IEEE Transactions on
Machine Intelligence, Vol. 24, No. 7, pp. 971-987, 2010.
Pattern Analysis and Machine Intelligence, Vol. 22, No. 12,
[20] M. Turk, and A. Pentland, “Eigenfaces for Recognition”,
pp. 1424–1440, 2010.
Journal of Cognitive Neuroscience, Vol. 3, No. 1, pp.71-86,
[4] M. Rosenblum, Y. Yacoob, and L. S. Davis, “Human
1991.
Expression Recognition from Motion using a Radial Basis
[21] A. Garg and V. Choudhary. “Facial Expression Recognition
Function Network Architecture,” IEEE Transactions on Neural
Using Principal Component Analysis’’, International Journal
Networks, Vol.7, No.5, pp.1121-1138, 1996.
of scientific Research Engineering and Technology, Vol.1, No.
[5] F. Bourel, C. Chibelushi, and A. Low, “Recognition of Facial
4, pp. 039-042, 2012.
Expressions in the Presence of Occlusion”, Proceedings of the
12th British Machine Vision Conference, Vol. 1, pp. 213–222, [22] A. Eleyan, H. Demirel, “Face Recognition System Based on
2001. PCA and Feed-forward Neural Network”, International Work-
[6] B. Abboud and F. Davoine, “Appearance Factorization for Conference on Artificial Neural Networks Proceedings, LNCS,
Facial Expression Recognition and Synthesis,” Proceedings of pp 935-942, June 2005.
International Conference on Pattern Recognition, pp.163–166,
2004.

2268