Real-time Convolutional Neural Networks for

Emotion and Gender Classification

Octavio Arriaga Paul G. Plöger Matias Valdenegro
Hochschule Bonn-Rhein-Sieg Hochschule Bonn-Rhein-Sieg Heriot-Watt University
Sankt Augustin Germany Sankt Augustin Germany Edinburgh, UK
Email: [email protected] Email: [email protected] Email: [email protected]

Abstract—In this paper we propose an implement a general

convolutional neural network (CNN) building framework for
designing real-time CNNs. We validate our models by creat-
ing a real-time vision system which accomplishes the tasks of
face detection, gender classification and emotion classification
simultaneously in one blended step using our proposed CNN
architecture. After presenting the details of the training pro-
cedure setup we proceed to evaluate on standard benchmark
sets. We report accuracies of 96% in the IMDB gender dataset
and 66% in the FER-2013 emotion dataset. Along with this we
also introduced the very recent real-time enabled guided back-
propagation visualization technique. Guided back-propagation
uncovers the dynamics of the weight changes and evaluates
the learned features. We argue that the careful implementation
of modern CNN architectures, the use of the current regu-
larization methods and the visualization of previously hidden
features are necessary in order to reduce the gap between slow
performances and real-time architectures. Our system has been
validated by its deployment on a Care-O-bot 3 robot used during Fig. 1: Samples of the FER-2013 emotion dataset [4].
RoboCup@Home competitions. All our code, demos and pre-
trained architectures have been released under an open-source
license in our public repository.

I. I NTRODUCTION
The success of service robotics decisively depends on a
smooth robot to user interaction. Thus, a robot should be
able to extract information just from the face of its user,
e.g. identify the emotional state or deduce gender. Interpret-
ing correctly any of these elements using machine learning
(ML) techniques has proven to be complicated due the high
variability of the samples within each task [4]. This leads to
models with millions of parameters trained under thousands of
samples [3]. Furthermore, the human accuracy for classifying
an image of a face in one of 7 different emotions is 65% ±
5% [4]. One can observe the difficulty of this task by trying
to manually classify the FER-2013 dataset images in Figure
1 within the following classes {“angry”, “disgust”, “fear”, Fig. 2: Samples of the IMDB dataset [9].
“happy”, “sad”, “surprise”, “neutral”}.
In spite of these difficulties, robot platforms oriented to
attend and solve household tasks require facial expressions becomes unfeasible. In this paper we propose an implement
systems that are robust and computationally efficient. More- a general CNN building framework for designing real-time
over, the state-of-the-art methods in image-related tasks such CNNs. The implementations have been validated in a real-time
as image classification [1] and object detection are all based on facial expression system that provides face-detection, gender
Convolutional Neural Networks (CNNs). These tasks require classification and that achieves human-level performance when
CNN architectures with millions of parameters; therefore, classifying emotions. This system has been deployed in a
their deployment in robot platforms and real-time systems care-O-bot 3 robot, and has been extended for general robot
platforms and the RoboCup@Home competition challenges.
Furthermore, CNNs are used as black-boxes and often their
learned features remain hidden, making it complicated to
establish a balance between their classification accuracy and
unnecessary parameters. Therefore, we implemented a real-
time visualization of the guided-gradient back-propagation
proposed by Springenberg [11] in order to validate the features
learned by the CNN.
II. R ELATED W ORK
Commonly used CNNs for feature extraction include a
set of fully connected layers at the end. Fully connected
layers tend to contain most of the parameters in a CNN.
Specifically, VGG16 [10] contains approximately 90% of all
its parameters in their last fully connected layers. Recent
architectures such as Inception V3 [12], reduced the amount
of parameters in their last layers by including a Global
Average Pooling operation. Global Average Pooling reduces
each feature map into a scalar value by taking the average over
all elements in the feature map. The average operation forces
the network to extract global features from the input image.
Modern CNN architectures such as Xception [1] leverage from
the combination of two of the most successful experimental
assumptions in CNNs: the use of residual modules [6] and
depth-wise separable convolutions [2]. Depth-wise separable
convolutions reduce further the amount of parameters by
separating the processes of feature extraction and combination
within a convolutional layer.
Furthermore, the state-of-the-art model for the FER2-2013
dataset is based on CNN trained with square hinged loss Fig. 3: Our proposed model for real-time classification.
[13]. This model achieved an accuracy of 71% [4] using
approximately 5 million parameters. In this architecture 98%
of all parameters are located in the last fully connected layers. to each reduced feature map. Our initial proposed architecture
The second-best methods presented in [4] achieved an is a standard fully-convolutional neural network composed of
accuracy of 66% using an ensemble of CNNs. 9 convolution layers, ReLUs [5], Batch Normalization [7]
III. M ODEL and Global Average Pooling. This model contains approx-
imately 600,000 parameters. It was trained on the IMDB
We propose two models which we evaluated in accordance gender dataset, which contains 460,723 RGB images where
to their test accuracy and number of parameters. Both models each image belongs to the class “woman” or “man”, and it
were designed with the idea of creating the best accuracy achieved an accuracy of 96% in this dataset. We also validated
over number of parameters ratio. Reducing the number of this model in the FER-2013 dataset. This dataset contains
parameters help us overcoming two important problems. First, 35,887 grayscale images where each image belongs to one
the use of small CNNs alleviate us from slow performances of the following classes {“angry”, “disgust”, “fear”, “happy”,
in hardware-constrained systems such robot platforms. And “sad”, “surprise”, “neutral”}. Our initial model achieved an
second, the reduction of parameters provides a better gener- accuracy of 66% in this dataset. We will refer to this model
alization under an Occam’s razor framework. Our first model as “sequential fully-CNN”.
relies on the idea of eliminating completely the fully connected Our second model is inspired by the Xception [1] archi-
layers. The second architecture combines the deletion of the tecture. This architecture combines the use of residual mod-
fully connected layer and the inclusion of the combined ules [6] and depth-wise separable convolutions [2]. Residual
depth-wise separable convolutions and residual modules. Both modules modify the desired mapping between two subsequent
architectures were trained with the ADAM optimizer [8]. layers, so that the learned features become the difference of the
Following the previous architecture schemas, our initial ar- original feature map and the desired features. Consequently,
chitecture used Global Average Pooling to completely remove the desired features H(x) are modified in order to solve an
any fully connected layers. This was achieved by having in the easier learning problem F (X) such that:
last convolutional layer the same number of feature maps as
number of classes, and applying a softmax activation function H(x) = F (x) + x (1)
 to a reduction of one percent with respect to our initial
implementation. Furthermore, we tested this architecture in
the FER-2013 dataset and we obtained the same accuracy of
66% for the emotion classification task. Our final architecture
weights can be stored in an 855 kilobytes file. By reducing our
architectures computational cost we are now able to join both
models and use them consecutively in the same image without
any serious time reduction. Our complete pipeline including
(a)
the openCV face detection module, the gender classification
and the emotion classification takes 0.22 ± 0.0003 ms on a
i5-4210M CPU. This corresponds to a speedup of 1.5× when
compared to the original architecture of Tang.
We also added to our implementation a real-time guided
back-propagation visualization to observe which pixels in the
image activate an element of a higher-level feature map.
Given a CNN with only ReLUs as activation functions for
the intermediate layers, guided-back propagation takes the
derivative of every element (x, y) of the input image I with
respect to an element (i, j) of the feature map f L in layer L.
The reconstructed image R filters all the negative gradients;
(b) consequently, the remaining gradients are chosen such that
Fig. 4: [2] Difference between (a) standard convolutions and they only increase the value of the chosen element of the
(b) depth-wise separable convolutions. feature map. Following [11], a fully ReLU CNN reconstructed
image in layer l is given by:
l l+1 l+1
Since our initial proposed architecture deleted the last fully Ri,j = (Ri,j > 0) ∗ Ri,j (2)
connected layer, we reduced further the amount of parame-
IV. R ESULTS
ters by eliminating them now from the convolutional layers.
This was done trough the use of depth-wise separable con- Results of the real-time emotion classification task in un-
volutions. Depth-wise separable convolutions are composed seen faces can be observed in Figure 5. Our complete real-
of two different layers: depth-wise convolutions and point- time pipeline including: face detection, emotion and gender
wise convolutions. The main purpose of these layers is to classification have been fully integrated in our Care-O-bot 3
separate the spatial cross-correlations from the channel cross- robot.
correlations [1]. They do this by first applying a D × D filter An example of our complete pipeline can be seen in Figure
on every M input channels and then applying N 1 × 1 × M 6 in which we provide emotion and gender classification.
convolution filters to combine the M input channels into N In Figure 7 we provide the confusion matrix results of our
output channels. Applying 1 × 1 × M convolutions combines emotion classification mini-Xception model. We can observe
each value in the feature map without considering their spatial several common misclassifications such as predicting “sad”
relation within the channel. instead of “fear” and predicting “angry” instead “disgust”.
Depth-wise separable convolutions reduces the computation A comparison of the learned features between several emo-
with respect to the standard convolutions by a factor of N1 + tions and both of our proposed models can be observed in
1
D 2 [2]. A visualization of the difference between a normal
Figure 8. The white areas in figure 8b correspond to the pixel
Convolution layer and a depth-wise separable convolution can values that activate a selected neuron in our last convolution
be observed in Figure 4. layer. The selected neuron was always selected in accordance
Our final architecture is a fully-convolutional neural net- to the highest activation. We can observe that the CNN learned
work that contains 4 residual depth-wise separable convolu- to get activated by considering features such as the frown, the
tions where each convolution is followed by a batch nor- teeth, the eyebrows and the widening of one’s eyes, and that
malization operation and a ReLU activation function. The each feature remains constant within the same class. These
last layer applies a global average pooling and a soft-max results reassure that the CNN learned to interpret understand-
activation function to produce a prediction. This architecture able human-like features, that provide generalizable elements.
has approximately 60, 000 parameters; which corresponds to These interpretable results have helped us understand several
a reduction of 10× when compared to our initial naive common misclassification such as persons with glasses being
implementation, and 80× when compared to the original CNN. classified as “angry”. This happens since the label “angry”
Figure 3 displays our complete final architecture which we is highly activated when it believes a person is frowning
refer to as mini-Xception. This architectures obtains an accu- and frowning features get confused with darker glass frames.
racy of 95% in gender classification task. Which corresponds Moreover, we can also observe that the features learned in
 Fig. 7: Normalized confusion matrix of our mini-Xception
Fig. 5: Results of the provided real-time emotion classification network.
provided in our public repository

our mini-Xception model are more interpretable than the ones

learned from our sequential fully-CNN. Consequently the use
of more parameters in our naive implementations leads to less
robust features.
V. F UTURE WORK
Machine learning models are biased in accordance to their
training data. In our specific application we have empirically
found that our trained CNNs for gender classification are
biased towards western facial features and facial accessories.
We hypothesize that this misclassfications occurs since our
training dataset consist of mostly western: actors, writers and
cinematographers as observed in Figure 2.
Furthermore, as discussed previously, the use of glasses
might affect the emotion classification by interfering with
the features learned. However, the use of glasses can also
interfere with the gender classification. This might be a result
from the training data having most of the images of persons
wearing glasses assigned with the label “man”. We believe
that uncovering such behaviours is of extreme importance
when creating robust classifiers, and that the use of the
visualization techniques such as guided back-propagation will
become invaluable when uncovering model biases.

Fig. 6: Results of the provided combined gender and emotion

inferences demo. The color blue represents the assigned class
woman and red the class man
 VI. C ONCLUSIONS
We have proposed and tested a general building designs
for creating real-time CNNs. Our proposed architectures have
been systematically built in order to reduce the amount of
parameters. We began by eliminating completely the fully
connected layers and by reducing the amount of parameters
in the remaining convolutional layers via depth-wise separable
convolutions. We have shown that our proposed models can
be stacked for multi-class classifications while maintaining
real-time inferences. Specifically, we have developed a vision
system that performs face detection, gender classification
and emotion classification in a single integrated module. We
have achieved human-level performance in our classifications
tasks using a single CNN that leverages modern architecture
constructs. Our architecture reduces the amount of parameters
(a) 80× while obtaining favorable results. Our complete pipeline
has been successfully integrated in a Care-O-bot 3 robot.
Finally we presented a visualization of the learned features
in the CNN using the guided back-propagation visualization.
This visualization technique is able to show us the high-level
features learned by our models and discuss their interpretabil-
ity.
ACKNOWLEDGMENTS
We gratefully acknowledge the continued support by the b-it
Bonn-Aachen International Center for Information Technology
and the Hochschule Bonn-Rhein-Sieg.
R EFERENCES
[1] François Chollet. Xception: Deep learning with depthwise separable
convolutions. CoRR, abs/1610.02357, 2016.
[2] Andrew G. Howard et al. Mobilenets: Efficient convolutional neural
networks for mobile vision applications. CoRR, abs/1704.04861, 2017.
[3] Dario Amodei et al. Deep speech 2: End-to-end speech recognition in
english and mandarin. CoRR, abs/1512.02595, 2015.
(b) [4] Ian Goodfellow et al. Challenges in Representation Learning: A report
on three machine learning contests, 2013.
[5] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse
rectifier neural networks. In Proceedings of the Fourteenth International
Conference on Artificial Intelligence and Statistics, pages 315–323,
2011.
[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep
residual learning for image recognition. In Proceedings of the IEEE
conference on computer vision and pattern recognition, pages 770–778,
2016.
[7] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift. In Interna-
tional Conference on Machine Learning, pages 448–456, 2015.
[8] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic
optimization. arXiv preprint arXiv:1412.6980, 2014.
[9] Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation
of real and apparent age from a single image without facial landmarks.
International Journal of Computer Vision (IJCV), July 2016.
[10] Karen Simonyan and Andrew Zisserman. Very deep convolu-
tional networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556, 2014.
[11] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin
Riedmiller. Striving for simplicity: The all convolutional net. arXiv
(c)
preprint arXiv:1412.6806, 2014.
Fig. 8: All sub-figures contain the same images in the same [12] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and
Zbigniew Wojna. Rethinking the inception architecture for computer
order. Every row starting from the top corresponds respec- vision. In Proceedings of the IEEE Conference on Computer Vision and
tively to the emotions {“angry”, “happy”, “sad”, “surprise”} Pattern Recognition, pages 2818–2826, 2016.
(a) Samples from the FER-2013 dataset (b) Guided back- [13] Yichuan Tang. Deep learning using linear support vector machines.
arXiv preprint arXiv:1306.0239, 2013.
propagation visualization of our mini-Xception model (c)
Guided back-propagation visualization of our sequential fully-
CNN.