Facial Emotion Recognition Using NLPCA and SVM

There have been considerable studies on expression classification of facial images; however, results have been unsatisfactory under varied illumination, pose variation, and occlusions. Deng et al. [10] used a Gabor filter for extracting features and implemented PCA and LDA for feature reduction. Experiments performed on the JAFFE database yielded 97% accuracy.

Praseeda Lekshmi et al. [11] implemented an RBF neural network to classify four facial expressions using the JAFFE database. Kishore et al. [12] proposed a method which is constructed from the fusion of facial features. Gabor wavelets and DCT were implemented to retrieve the features and PCA was implemented to reduce dimensionality. The Hybrid Emotional Neural Network was used as a classifier to classify emotions. Experiments performed on the Cohn-Kanade (CK) dataset yielded 98.5% of Recognition rate.

For recognizing the facial expressions and classification a feature learning-based pixel difference representation was proposed by Sun et al. [13]. A DFM called as discriminative feature matrix was first generated from original images and a discriminative feature dictionary was constructed. DFD served as the input to vertical 2D linear discriminate analysis in the direction (V-2DLDA). To classify the images, a nearest neighbor classifier was used. The proposed method was evaluated on a CK+ dataset and yielded an accuracy of 91.87 %.

Uddin et al. [9] stated a depth camera-based novel approach. Local directional rank histogram pattern (LDRHP) and local directional pattern (LDP) was used to retrieve the features. KPCA was used for dimensionality reduction. Finally, the convolutional neural network was used for classification.

RamiReddy et al. [8] proposed a method which is constructed from the combination of local and global features. DLBP and DCT were used to get global and local features from non-dynamic face expression images respectively. For classification RBF neural network was used. Evaluation was performed using the Cohn-Kanade dataset, which yielded an accuracy of 97 %.

Kumar et al. [14] proposed a framework depending on getting the instructive regions of a face image. In the feature extraction stage, LBP features were extracted and then Procrustes analysis was used to model the reference image. Features were finally extracted only from selected regions. An SVM was used as a classifier to classify the expressions. Experiments performed on the MUG database yielded an accuracy of 98 %.

To recognize and classify facial expressions in video sequences, Kamarol et al. [15] proposed an approach based on appearance features. At first step, Viola-Jones face detector was used for detecting faces from frames. The Spatio temporal texture map (STTM) algorithm was then used to model the facial features. SVM classifier was implemented to classify the features. The accuracy achieved was approximately 95.37 on the CK+ database.

Siddiqi et al. [16] stated an approach based on localized features. SWLDA was implemented to find the features from images which contain expressions of a face. SWLDA used partial F-test values to choose localized features from the frames. An HCRF model was used for recognition. Four datasets were used for performance evaluation of the proposed approach and yielded an accuracy of 96.37 %.

Qayyum et al. [17] Proposed an approach based on a stationary wavelet transform. Features were extracted using the stationary wavelet transform. The horizontal and vertical sub-bands of a stationary wavelet transform contain muscle movement information of facial expressions. DCT was applied on these sub-bands for feature dimensionality. A Feed Forward Neural Network (FFNN) with back propagation algorithm was implemented to recognize and classify the expressions. Average accuracies of 96.61% and 98.83% were achieved on extended Cohn-Kanade and JAFFE datasets, respectively.

Ding et al. [18] stated a method for recognizing and classifying facial expressions in videos. A peak expression frame from the video was detected using DLBP. Logarithm-Laplace (LL) domain was used to obtain accurate facial features. The Taylor expansion theorem was applied to extract the features and a Taylor feature map was created, which was given as an input to Taylor Feature Pattern (TFP) for extracting accurate features. CK and JAFFE datasets were used for testing.

4.1 Haar wavelet

The Haar wavelet is a transform that includes multi-layer decomposition. The Haar wavelet transform is implemented for the extraction of global attribute from a face image by decomposing image of face into wavelet coefficients. Decomposed wavelet coefficients consist of four portions, each with a quarter of the authentic image area as presented in Figure 2. LA (All Components), LH (Horizontal Component), LV (Vertical Components) and LD (Diagonal Component). We have used LA coefficients to obtain features from facial expression images.

2.png

Figure 2. Haar wavelet decomposition

4.2 Gabor wavelet

The extraction of local features like eyes, mouth, cheeks, and nose, involves finding attributes in a localized manner. Local attributes can be pulling out using Gabor wavelets [7] in frequency and spatial domain. The Gabor wavelet transformation is represented by

$\begin{align} & {{G}_{o,s}}(x,y)=({{o}^{2}}+{{s}^{2}})\div (2{{\sigma }^{2}})*{{\ell }^{-(({{o}^{2}}+{{s}^{2}})*({{x}^{2}}+{{y}^{2}}))\div (2{{\sigma }^{2}})}} \\ & *[{{\ell }^{(i*(ox+sy)}}-{{\ell }^{(-{{\sigma }^{2}}\div 2)}}] \\ \end{align}$ (1)

where o is the orientation and s is the scale, which are obtained by the following equations.

$o={{k}_{m}}\div {{f}^{n}}*\cos (pi*m\div 8)$ (2)

$o={{k}_{m}}\div {{f}^{n}}*\cos (pi*m\div 8)$ (3)

${{k}_{m}}=pi\div 2,\text{ }f=√2$ (4)

Here, f= √2 is the spacing factor between the kernels and km is the frequency domain.

The Figure 3 shows the real part of the Gabor kernels at 4 scales and 5 orientations and their magnitude.

3.png

Figure 3. Real part of the Gabor kernels

In the proposed method, Gabor wavelets with five orientations and four scales are used. Gabor kernels are convolved with the face image to extract facial features using the below relationship.

$Om,n=I(x,y)*{{G}_{u,v}}(x,y)$ (5)

I(x,y) is a gray-level face image, and G_u,v(x,y) is a Gabor kernel.

The Figure 4 shows the Gabor wavelet representation of an image.

4.png

Figure 4. Gabor wavelet representation of an image

By concatenating all features using orientation selectivity, spatial frequencies, and spatial localities, a feature vector is generated. Because Gabor kernels are complex representations, a high-dimensionality feature vector is generated.

6.1 Concatenated fusion

In this phase, the features extracted from both methods are concatenated for increasing the accuracy of the model. Normalization is implemented so that features share a common scale.

$S=({{S}_{\text{1}}}+\text{ }{{S}_{\text{2}}})$ (6)

6.2 Weighted fusion

The weighted summation method is applied to combine or fuse normalized features. Weights are set based on feature ranking. Feature ranking is computed by determining the variance between a feature vector and the mean of the feature set. Based on this view, weights are assigned to the features. The weighted summation is given as

$S=\underset{n=0}{\overset{N}{\mathop \sum }}\,{{\text{w}}_{\text{n}}}{{s}_{n}}$ (7)

In this study, four methods are proposed, namely PCA_SVM with concatenated fusion (Proposed method-1 with CF), PCA_SVM with weighted fusion (Proposed method-1 with WF), NLPCA-SVM with concatenated fusion (Proposed method-2 with CF), and NLPCA-SVM with weighted fusion (Proposed method-2 with WF). In all the methods, Gabor and Haar wavelets are used to extract global features as well as local features from facial expression images and an SVM is used as a classifier to recognize six emotions.

In the PCA_SVM with a concatenated fusion method, PCA was implemented to decrease the dimensionality of features. A concatenated fusion technique was implemented for fusing global and local features.

In the PCA_SVM with a weighted fusion method, PCA was implemented to decrease the dimensionality of features. A weighted fusion technique was implemented for fusing global and local features.

In the NLPCA_SVM with a concatenated fusion method, NLPCA was implemented to decrease the dimensionality of features. A concatenated fusion technique was implemented for fusing global and local features.

In the NLPCA_SVM with a weighted fusion method, NLPCA was implemented to decrease the dimensionality of features. A weighted fusion technique was implemented for fusing global and local features.

1. Training samples: <classes, expressions> I(x,y)=I _{<no of classes, no of samples, 6 expressions>}(x, y)

2. (i) Let Gabor wavelet transform is G_u,v(x, y)

    (ii) O_m,n= I(x, y)*G_{u ,v}(x, y)

    (iii) Feature set: X_i = O_m,n

3. Case I: Feature extraction using PCA

    (i) $\bar{x}=\frac{1}{M}\sum\limits_{i=1}^{M}{\mathop{x}_{i}}$

    (ii) ${{\omega }_{i}}={{x}_{i}}-\bar{x}$ 

    (iii) $c=\frac{1}{M}\sum\limits_{n=1}^{M}{\mathop{\omega }_{n}}\mathop{\omega }_{n}^{T}=B{{B}^{T}}$

    (iv) $C: u_1, u_2, ... u_N$

    (v)Feature vector 

Case II: Feature extraction using NLPCA

    (i) (pc, net, network) = nlpca(X_i ,n_c)

    (ii)  pc_new = nlpca_get_components(net,V1)

   (iii) V1 = nlpca_get_data (net, pc_new)

Here n_c is no of nonlinear components from dataset

4. Let X_i is a feature set

where H_n is Haar Wavelet transform

5. Repeat step 3 on the feature set which is generated in step 4 and form feature vector V2.

6. Case I:

    (i) Train set=V1+V2

    (ii) Model= svmtrain (Train set, Group);

    (iii) Repeat step 1 to step5 on test sample: T(x, y)

    (iv) svmclassify(model, T(x, y))

Case II:

    (i) Train set= $∑_{n=0}^N w_n s_n$

    (ii) Model=svmtrain(Train set, Group);

    (iii) Repeat step 1 to step5 on test sample: T(x, y)