2017 IEEE 17th International Conference on Advanced Learning Technologies

Building a face expression recognizer and a face expression database for an

intelligent tutoring system

Ramon Zatarain-Cabada, Maria Lucia Barron-Estrada, Francisco Gonzalez-Hernandez, Hector Rodriguez-Rangel

Posgrado en Ciencias de la Computación
Instituto Tecnológico de Culiacán
Culiacán, Sinaloa, México
[email protected]

Abstract—this work presents the building and validating of a a sporadic and in a short time. All therefore the above
face expression database and a face expression recognizer. The indicates the real importance of understanding those
face expression recognizer uses a geometric-based technique emotions [4]. It is significantly important to understand and
that measures distances between the central point on the face
and other 68 facial landmark points. These measures are
detect emotion in the area of ITS.
transformed into features to train a support vector machine.
II. RELATED WORKS
The database was built inside an educational context while
students program in Java code. The tests validate the accuracy A. Face expression databases
of the recognizer applying a ten-fold cross-validation.
A face expression database is a dataset of images
Keywords—Intelligent-tutoring-system; artificial-intelligence; containing people shows expressions which represent human
face-expression-recognition; face-expression-database emotions. The most important databases are explained below.
Cohn-Kanade (CK) [5] is a dataset of 486 sequences from
I. INTRODUCTION 97 positioned persons. The database seeks to represent the
Nowadays, no one can deny the relevance and the image sequence inside the Facial Action Coding System
importance of artificial intelligence. We can find the use of (FACS). In this way, each expression receive an emotion
artificial intelligence methods in subjects such as social label. The CK includes basic emotions as well as Action
networks, smart devices, games, industrial management Units (AU) annotations. Cohn-Kanade Plus (CK+) [6] is an
control processes, and education. About the educational extension of original CK (version 2). This version includes
issue, the artificial intelligence has been used to individualize posed expressions, non-posed (spontaneous) expressions
the students learning process, e.g. identifying learning styles recording from 84 novel subjects and more metadata
or recognizing personal features on students [1]. information.
Intelligent tutoring systems (ITS) are the answer of Radboud Faces Database (RaFD) [7] contains an image set
researchers to integrate the computational tutors with of 49 models in two subsets: 39 Caucasian Dutch adults and
artificial intelligence to get an improved personalization 10 Caucasian Dutch children. All model expressed eight
using artificial intelligence [2]. The ITSs are systems facial expressions (anger, happiness, fear, sadness, contempt,
designed to handle the learning processes of students. An ITS surprise, disgust, and neutral) with three gaze directions and
can help students to know which topics have not been studied five camera angles. They obtained a total of 120 images per
yet. Also, an ITS can recognize affective states of students to model which had to comply various requisites like wearing a
design and perform the best pedagogical strategies for type of shirt or no hair on the face.
students.
B. Face Expression Recognition
ITSs usually use the emotional Ekman theory, which
describes how human faces move different face parts to Expression recognition deals about how computers get the
express an emotion, either moving a mouth corner or raising ability to understand and process human expressions. Next,
an eyebrow. These emotions are known as basic emotions. an analysis of the main techniques to recognize expressions
is presented below.
Basic emotions have as characteristic being spontaneous and
Geometric-based techniques [8] are techniques that
expressive [3]. However, there are other emotions which
represent faces as a facial points representation. These
emerge when students are working in their learning tasks.
representation is describe by a concatenation of X and Y
These emotions are known as learning-centered emotions. coordinates of fiducial points. The points are used as a feature
Learning-centered emotions relate to educational context and input in a machine-learning and predicting an expression
they emerge during deep learning activities such as thought, system, to represent a face in these techniques there are two
reflection or problem-solving. Some research found a relation types of representations of models, free model, and model-
to emotions as frustration, boredom or engagement during based.
complex cognitive processes. Those emotions emerge during Local Binary Pattern (LBP) [9] is a technique wherein the
a prolonged time unlike basic emotions which appear during original purpose was to identify textures on different surfaces

2161-377X/17 $31.00 © 2017 IEEE 391

DOI 10.1109/ICALT.2017.141
using grayscale images. The technique takes the pixel value bigger than geometric-based techniques. Next, how the
of the image center as the threshold. Each pixel value is recognizer performs its task is described.
compared against the threshold; if the threshold is bigger than The recognizer starts extracting features from faces. To do
the pixel value then the resulted is 0, otherwise, it is 1. The that, we use Dlib library [12]. Dlib locates a set of 68
last step creates a sequence of binary numbers which is landmark points on the face. These points are located using a
converted into its decimal representation. template previously trained by dlib. The coordinate values
Gabor [10] is a representation of convolving images may change depending on where the face is located in the
generated from a set of Gabor filters using various scales and photo. To solve that problem, the mean of both axes (X and
orientations. Gabor filters encode componential information, Y) are calculated, which results in the point coordinates of
and depending on the registration scheme, the overall the sort of a gravity center of all face landmarks. The left
representation may implicitly convey configurable image in Figure 1 shows the location of the central point. The
information. This technique can be used with simple distances from the center to every landmark point are
dimensionality reduction techniques such min, max and mean obtained after the locating the center point is obtained. The
grouping. right image in Figure 1 shows the drawn lines from the central
III. BUILDING A FACE EXPRESSION DATABASE point.
To recognize learning-centered emotions, we decided to
build our own facial database. Four emotions were picked
because they have relation to educational contexts, those
emotions are frustration, boredom, engagement, and
excitement. To label every photo in the database with such
emotions we used a mobile electroencephalography (EEG)
technology called Emotiv Epoc [11]. To relation the photo
content with a programming context, emotions were captured
while students coded solutions for problems in the Figure 1. The left image has the highlighted central point, the right
programming language Java. Next, we explain the process to image has the drawn distances from the central point
capture and label every photo.
Faces may be tilted because students move their neck
A. Building and Filtering of Database during exercise, which might confuse the classifier. The
It was developed a methodology in order to inducing rotations are corrected offsetting all calculated angles by the
situations in which students express emotions related to angle of the nose bridge. This rotates the entire vector array
education to build the database. The methodology searches so that tilted faces become similar to non-tilted faces with the
capturing the moment when a student expresses an emotion same expression. A relative angle is calculated with
during the developing a Java code. The methodology arctangent depending on if the nose bridge is perpendicular
proposed consisting in the following steps: to the horizontal plane for adding or subtracting a
1. The user codes a Java program; meanwhile, the compensation value (90 degrees). Equation 1 shows how to
Emotiv EPOC device captures brain activity and the calculate the relative angle, atan is the function for the inverse
webcam takes a photograph every 5 seconds.
of the tangent, x26 and x29 represent specific coordinates,
2. Every user photograph is labeled with the user
and distance measures the distance between the points x26,
emotion obtained at that moment from the Emotiv
x29 and the central point.
EPOC device.
3. The photo previously labeled is saved into the Face ƒ–ƒሺ݀݅‫݁ܿ݊ܽݐݏ‬ሺ‫ʹݔ‬͸ǡ ‫ͻʹݔ‬ሻሻ ‫ͳ כ‬ͺͲ
݈ܽ݊݃݁ ൌ
expression Database. ߨ
Equation 1. Calculation of relative angle
Finally, we obtained a database containing a total of 7,019
photographs. However, many photographs and their
The final value for the relative angle is established by
corresponding emotions had not a matching consistency. We
Equation 2.
proceed to filter the database by eliminating those incorrect ‫ ݔ‬൏ Ͳǡ ‫ ݔ‬൅ ͻͲ
registers (face-emotion mismatch) using the human judge ݈ܽ݊݃݁ ൌ ቄ
‫ ݔ‬൒ Ͳǡ ‫ ݔ‬െ ͻͲ
obtaining a final database of 730 photographs. In order to test Equation 2. Compensation of angles
our face expression database, we performed a verification
using a face expression recognizer which is defined in the Coordinates, distances, and relative angles are
following section. concatenated as input for a support vector machine. The
support vector machine (SVM) is configured according to the
IV. BUILDING A FACE EXPRESSION RECOGNIZER
scikit-learn implementation [13] using default parameters of
The geometric-based technique was selected for this work the implementation.
because appearance-based techniques need a set of features

392
 V. EXPERIMENTS, RESULTS, AND DISCUSSIONS [3] P. Ekman, "An argument for basic emotions,"
The validation test of our database using our recognizer Cognition & emotion, vol. 6, pp. 169-200, 1992.
consists in training the classifier using 90% of the database [4] S. D’Mello and A. Graesser, "Dynamics of affective
and 10% remaining for testing. The test is a ten-fold states during complex learning," Learning and
validation cross. The recognizer processed a total of 655 Instruction, vol. 22, pp. 145-157, 2012.
photographs in training time. Table 1 shows the distribution [5] T. Kanade, J. F. Cohn, and Y. Tian,
of classes, the accuracy obtained for this work, and the "Comprehensive database for facial expression
accuracy for another work explained below. analysis," in Automatic Face and Gesture
Recognition, 2000. Proceedings. Fourth IEEE
TABLE 1. DISTRIBUTION OF CLASSES International Conference on, 2000, pp. 46-53.
Emotion Number of Accuracy of this Accuracy of [6] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z.
classes work work [19] Ambadar, and I. Matthews, "The extended cohn-
Boredom 17 64% 76% kanade dataset (ck+): A complete dataset for action
Engagement 430 64% 91%
Excitement 114 83% 68%
unit and emotion-specified expression," in 2010
Frustration 94 62% 55% IEEE Computer Society Conference on Computer
Vision and Pattern Recognition-Workshops, 2010,
The number of classes presented in Table 1 are the classes pp. 94-101.
for this work and not match with classes used in the work. [7] O. Langner, R. Dotsch, G. Bijlstra, D. H.
The recognizer got a good result in engagement and boredom Wigboldus, S. T. Hawk, and A. van Knippenberg,
emotions. It is evident that excitement and frustration did not "Presentation and validation of the Radboud Faces
get such high scores. To find out if our results were low we Database," Cognition and Emotion, vol. 24, pp.
needed to compare it against other similar works. We decided 1377-1388, 2010.
to compare the work against a single research wherein [8] K.-C. Huang, S.-Y. Huang, and Y.-H. Kuo,
authors used the same emotions of this work [14]. The only "Emotion recognition based on a novel triangular
difference found in the comparison work is the excitement is facial feature extraction method," in Neural
named as delighted, but its description is same. After a Networks (IJCNN), The 2010 International Joint
review, we concluded that results were satisfactory until now, Conference on, 2010, pp. 1-6.
two emotions obtained a result higher than the compared [9] S. Happy and A. Routray, "Automatic facial
research. expression recognition using features of salient
facial patches," IEEE transactions on Affective
VI. CONCLUSIONS Computing, vol. 6, pp. 1-12, 2015.
Detecting emotions is a complicated task even for human [10] Y.-I. Tian, J. F. Cohn, and T. Kanade, "Evaluation
beings, using the face only to detect emotions is a challenge of Gabor-wavelet-based facial action unit
which can become insufficient. It is a research in continuous recognition in image sequences of increasing
development. This work presents the building of a face complexity," presented at the Automatic Face and
expression database focused on learning-centered emotions, Gesture Recognition, 2002. Fifth IEEE International
the database has relation to educational context. Also, the Conference on, 2002.
work presents the building of a geometric-based recognizer [11] (2016). Emotiv EPOC. Available:
that calculates coordinates, distances, and angles in different http://emotiv.com/
faces. The tests validate that this work has successful results [12] D. E. King, "Dlib-ml: A machine learning toolkit,"
in two emotions considering that this work is a beginning to Journal of Machine Learning Research, vol. 10, pp.
recognize centered-learning emotions. This paper can be used 1755-1758, 2009.
as a method to build a face expression database with an
[13] F. Pedregosa, G. Varoquaux, A. Gramfort, V.
educational approach. The database can be found through the
Michel, B. Thirion, O. Grisel, et al., "Scikit-learn:
next URL: ftp://javasensei.ddns.net/corpus.rar
Machine learning in Python," Journal of Machine
REFERENCES Learning Research, vol. 12, pp. 2825-2830, 2011.
[14] N. Bosch, S. D'Mello, R. S. Baker, J. Ocumpaugh,
[1] B. M. McLaren, O. Scheuer, and J. Mikšátko,
V. Shute, M. Ventura, et al., "Automatic Detection
"Supporting collaborative learning and e-
of Learning-Centered Affective States in the Wild,"
discussions using artificial intelligence techniques,"
in IUI '15 Proceedings of the 20th International
International Journal of Artificial Intelligence in
Conference on Intelligent User Interfaces, 2015, pp.
Education, vol. 20, pp. 1-46, 2010.
379-388.
[2] B. P. Woolf, Building Intelligent Interactive Tutors.
Amherst, Massachusetts, Estados Unidos: Morgan
Kaufmann Publishers, 2009.

393