Learning Bases of Activity for Facial

Expression Recognition
 Domain Introduction:
• Image processing is a method to perform some operations on an image, in order to get an
enhanced image or to extract some useful information from it.
• It is a type of signal processing in which input is an image and output may be image or
characteristics/features associated with that image.
• Nowadays, image processing is among rapidly growing technologies. It forms core research area
within engineering and computer science disciplines too.
• Image processing basically includes the following three steps:
• Importing the image via image acquisition tools;
• Analyzing and manipulating the image;
• Output in which result can be altered image or report that is based on image analysis.
 Objective:
• The main objective of this process is to identify the facial expression of the person to classify the
mental state of the person, on the other hand it is typically said by the application like, identifying
the customer feedbacks by the facial expressions.

• The main goal of the process is to improve the facial expression recognition rate of the person.
 Abstract:
• In this approach, we propose a novel data-driven feature extraction framework that represents
facial expression variations as a linear combination of localized basis functions, whose coefficients
are proportional to movement intensity.

• We show that the linear basis functions of the proposed framework can be obtained by training a
sparse linear model with Gabor phase shifts computed from facial videos.

• The proposed framework process addresses generalization issues that are not tackled by existing
learnt representations, and achieves, with the same learning parameters, state-of-the-art results in
recognizing both posed expressions and spontaneous micro-expressions.
 Introduction:
• Facial expression recognition has potential applications in different aspects of day-to-day life not

yet realized due to absence of effective expression recognition techniques.

• This facial expression approach discusses the application of Gabor filter based feature extraction

in combination feed forward neural networks (classifier) for recognition of seven different facial

expressions from still pictures of the human face.

• The study presented here gives simple method in facial expression recognition. Facial expression

recognition has potential applications in different aspects of day-to-day life not yet realized due to

absence of effective expression recognition techniques.

 Introduction:
• This discusses the application of Gabor filter based feature extraction in combination feed forward
neural networks (classifier) for recognition of seven different facial expressions from still pictures
of the human face.

• The study presented here gives simple method in facial expression recognition to detect the micro
expressions in the facial image.
 Existing System:
 In existing system, several methods are used to extract image face features vector, which presents
small inter-person variation.

 This feature vector is feed to a multilayer perceptron to carry out the face recognition or identity
verification tasks.

 Proposed system consists in a combination of Gabor and Eigenfaces to obtain the feature vector.

 Evaluation results show that proposed system provides robustness against changes in illumination,
wardrobe, facial expressions, scale, and position inside the captured image, as well as inclination,
noise contamination and filtering.
 Disadvantages:
• One problem that can be associated with the use of these methods is the fact that they are very
sensitive to variations in pose, illumination, occlusion, aging, and rotation changes of the face.

• These techniques do not work well in case of a small sample size.

• LBP does not provide the directional information of the facial frame.

• The performance of this approach also degrades with variation in illumination.

 Proposed system:
• In most cases of facial expression classification, the process of feature extraction yields a
definitively large number of features and subsequently a smaller sub-set of features needs to be
selected according to some optimality criteria.

• Gabor filters have been proved to be effective for expression recognition because of its superior
capability of multi-scale representation.

• Gabor wavelet can use very better description of biological visual neuron about receptive field,.
According to the needs of special vision, it can adjust the spatial and frequency properties to face
expression characteristic wanted, so Gabor filter wavelet is suitable for people face analysis and
treatment of expression.
 Advantages:
• It has good predictive ability, and computationally.

• It is less expensive than other existing methods.

• The advantage of this method is that it is very efficient for seeking localized features.

• It also provided good recognition rates when used across multiple datasets.

• The proposed HCRF model also showed significant improvement over existing work in terms of
recognition accuracy.
Block Diagram:

Input Face image

Input Face image

Face Detection and

Face Detection and
extraction
extraction

Feature Extraction
Feature Extraction

Classification
Classification

Performance
Performance
Estimation
Estimation
Architecture Diagram:
 Flow diagram
Dataset
Dataset
Input Face image
Input Face image

Face Detection and

Face Detection and polynomial-kernel
extraction polynomial-kernel
extraction Support vector
Support vector
machine
Facial Feature machine
Peculiarity
Facial Feature Peculiarity
Points using Gabor Extraction
Points using Gabor Extraction

Classification
Classification
Accuracy | Sensitivity |
Accuracy | Sensitivity | Performance
Specificity |PPV | NPV Performance
Specificity |PPV | NPV Estimation
Estimation
 Modules:
• Input Face image

• Face Detection and extraction

• Feature Extraction

• Classification

• Performance Estimation
Literature Survey
 Title Year Author Methodology Advantages Disadvantages

Building High- 2012 Quoc V. Le Control experiments The input data the parameters are
level Features Marc’Aurelio show that this feature detector are 200x200 not shared
Using Large Scale Ranzato Rajat is robust not images, much across different
Unsupervised Monga only to translation but also to larger than locations in the
Learning scaling and typical 32x32 image.
out-of-plane rotation. We also images used in
find that the deep learning
same network is sensitive to and
other high-level unsupervised
concepts such as cat faces and feature learning
human bod-ies.
 Title Year Author Methodology Advantages Disadvantages

Spatio-Temporal 2012 Yimeng In this paper, we propose The It do not capture

Phrases for Zhang an approach that efficiently co-occurring ST higher
Activity y? identifies both local and long- phrases are order and long range
Recognition , Xiaoming range motion further used to dependencies
Liu interactions; taking the \push" compute a kernel
z activity as an example, our for discrimina-
, Ming-Ching approach can tive learning
Chang capture the combination of the with SVM
z hand movement of one person
, Weina Ge and the
z foot response of another
, and Tsuhan person, the local features of
Chen which are both
spatially and temporally far
away from each other.
 Title Year Author Methodology Advantages Disadvantages

Joint 2012 Minh Hoai In this paper we propose a The system can each original video
Segmentation and Zhen-Zhong novel method that avoids use multi-class samples was used
Classification of Lan Fernando the limitations of the above SVM to train a only
Human Actions in De la Torre approaches by jointly model for once
Video performing video temporal actions
segmentation and action
recognition. Unlike standard
approaches based on
extensions of dynamic
Bayesian
networks, our method is based
on a discriminative temporal
extension of the spatial bag-
of-words model that has
been very popular in object
recognition.
 Title Year Author Methodology Advantages Disadvantages

Learning 2011 William We represent videos by The model is significant resources

Spatiotemporal Brendel and spatiotemporal graphs, used for parsing could be wasted
Graphs of Human Sinisa where nodes correspond to new videos in on learning hand-
Activities Todorovic multiscale video segments, terms picked parts and
and edges capture their of detecting and relations which may
hierarchical, temporal, and localizing not
spatial rela-tionships. Access relevant activity be the most relevant
to video segments is provided parts. for representing and
by our new, multiscale recognizing the ac-
segmenter. Given a set of tivity.
training spatiotemporal
graphs, we learn their
archetype graph, and pdf ’s
associated with model nodes
and edges.
 Title Year Author Methodology Advantages Disadvantages

Efficient Activity 2012 Chao-Yeh We propose an efficient This same that limits detections
Detection with Chen and approach that unifies activity property of to
Max-Subgraph Kristen categorization with space- linear SVMs is cases where the
Search Grauman time localization. The main used in subject of the
idea is to pose activity to enable activity does not
detection as a maximum- efficient change its
weight connected subwindow spatial position much
subgraph problem over a search for object over time.
learned space-time graph detection
constructed on the test
sequence.
 System Requirements
SOFTWARE REQUIREMENTS:

OS : Windows

Software : Mat lab

HARDWARE REQUIREMENTS:

Processor : Intel Pentium.

RAM : 2GB
 References:
• [1] Y.-L. Tian, T. Kanade, and J. F. Cohn, “Recognizing action units for facial expression
analysis,” IEEE Trans. Pattern Anal. Mach. Intell. vol. 23, no. 2, pp. 97–115, Feb. 2001.

• [2] Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, “A survey of affect recognition methods:
Audio, visual, and spontaneous expressions,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no.
1, pp. 39–58, Jan. 2009.

• [3] M. Pantic, “Machine analysis of facial behaviour: Naturalistic and dynamic behaviour,” Philos.
Trans. Roy. Soc. B, Biol. Sci., vol. 364, no. 1535, pp. 3505–3513, 2009.

• [4] H. Gunes and B. Schuller, “Categorical and dimensional affect analysis in continuous input:
Current trends and future directions,” Image Vis. Comput., vol. 31, no. 2, pp. 120–136, 2013.
 References:
• [5] E. Sariyanidi, H. Gunes, and A. Cavallaro, “Automatic analysis of facial affect: A survey of registration,
representation, and recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 6, pp. 1113–1133, Jun.
2015.

• [6] Z. Ambadar, J. W. Schooler, and J. F. Cohn, “Deciphering the enigmatic face: The importance of facial
dynamics in interpreting subtle facial expressions,” Psychol. Sci., vol. 16, no. 5, pp. 403–410, 2005.

• [7] T. Wu, M. S. Bartlett, and J. R. Movellan, “Facial expression recognition using Gabor motion energy
filters,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, Jun. 2010, pp. 42 47.

• [8] G. Zhao and M. Pietikainen, “Dynamic texture recognition using local binary patterns with an application
to facial expressions,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 6, pp. 915–928, Jun. 2007.
 References:
• [9] B. Jiang, M. Valstar, B. Martinez, and M. Pantic, “A dynamic appearance descriptor approach
to facial actions temporal modeling,” IEEE Trans. Cybern., vol. 44, no. 2, pp. 161–174, Feb. 2014.

• [10] H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, “Joint fine-tuning in deep neural networks for
facial expression recognition,” in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2015, pp. 2983–2991.

• [11] S. Elaiwat, M. Bennamoun, and F. Boussaid, “A spatio-temporal RBMbased model for facial
expression recognition,” Pattern Recognit., vol. 49, pp. 152–161, Jan. 2016.

• [12] M. Liu, S. Shan, R. Wang, and X. Chen, “Learning expressionlets on spatio-temporal

manifold for dynamic facial expression recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., Jun. 2014, pp. 1749–1756.