1.

Software and tools used for implementation :

Python3.7
 OpenCv
 Tensorflows
 Keras
 Numpy
 Sklearn
 Matplotlib
2. Database :
For a best performance, we should train the network with a lot of samples of
images.
This would increase the accuracy and improve the performance of the model.
Unfortunately, the large amount of data in datasets do not exist publicly, but we
have access to two public databases(FER2013 and CK+).
For our system, we will use FER2013 database.
3. Implementation
The system of Facial Emotion Recognition consists of two modules :
The first one is for face detection and the second one is for emotion recognition.
In the following, we will detail each modul.
3.1 face detection :
to detect faces, we used to choose the method proposed by paul Viola and
michael Jones.
In our system, we opted fior Haarcascade_eye.xml from the OpenCv library
which provides the Haar cascade method.
3.2 Recognition method :
As we , we will use the convolutional neural network to recognize the facial
emotion of the detected faces.
The first thing to be considered is the architecture chosen and the number of
layers implemented in this architecture.
In our project, we will use two differents architectures and we will compare the
performance of these two architectures.
The first architecture chosen is VGG16, a simple architecture easy to
implement and handle on python.
 The second one was inspired from VGG16, but we reduced the number of
layers.
Our system contains several modules, in the following we will specify each modules
their role and pupose.
3.3 librairies :
In this module, we imported all the librairies we needed during the phase of
training.
from __future__ import print_function
import keras
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow. keras.models import Sequential
from tensorflow.keras.layers import
Dense,Dropout,Activation,Flatten,BatchNormalization
from tensorflow.keras.layers import Conv2D,MaxPooling2D
from tensorflow.keras.models import load_model
import os
import numpy as np
import matplotlib.pyplot as plt
from livelossplot.tf_keras import PlotLossesCallback

from sklearn.utils import shuffle

from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report,confusion_matrix
import itertools

3.4 Loading dataset and data augmentation:

we applied a data augmentation for our data because we need add more data to our
training set.

num_classes = 5
img_rows,img_cols = 48,48
batch_size = 8

train_data_dir = 'C:/Users/HP/Desktop/projectEssai/train'
validation_data_dir = 'C:/Users/HP/Desktop/projectEssai/validation'

train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=30,
shear_range=0.3,
zoom_range=0.3,
width_shift_range=0.4,
height_shift_range=0.4,
 horizontal_flip=True,
fill_mode='nearest')

validation_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
train_data_dir,
color_mode='grayscale',
target_size=(img_rows,img_cols),
batch_size=batch_size,
class_mode='categorical',
shuffle=True)
validation_generator = validation_datagen.flow_from_directory(
validation_data_dir,
color_mode='grayscale',
target_size=(img_rows,img_cols),
batch_size=batch_size,
class_mode='categorical',
shuffle=True)

Found 24282 images belonging to 5 classes.

Found 5937 images belonging to 5 classes.
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_8 (Conv2D) (None, 48, 48, 32) 320
_________________________________________________________________
activation_11 (Activation) (None, 48, 48, 32) 0
_________________________________________________________________
batch_normalization_10 (Batc (None, 48, 48, 32) 128
_________________________________________________________________
conv2d_9 (Conv2D) (None, 48, 48, 32) 9248
_________________________________________________________________
activation_12 (Activation) (None, 48, 48, 32) 0
_________________________________________________________________
batch_normalization_11 (Batc (None, 48, 48, 32) 128
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 24, 24, 32) 0
_________________________________________________________________
dropout_6 (Dropout) (None, 24, 24, 32) 0
 3.5 CNN architecture in KERAS

Vgg16 architecture :
model = Sequential()
# Block-1
model.add(Conv2D(32,
(3,3),padding='same',kernel_initializer='he_normal',input_shape=(img_rows,i
mg_cols,1)))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Conv2D(32,
(3,3),padding='same',kernel_initializer='he_normal',input_shape=(img_rows,i
mg_cols,1)))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
# Block-2
model.add(Conv2D(64,(3,3),padding='same',kernel_initializer='he_normal'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Conv2D(64,(3,3),padding='same',kernel_initializer='he_normal'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
# Block-3
model.add(Conv2D(128,(3,3),padding='same',kernel_initializer='he_normal'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Conv2D(128,(3,3),padding='same',kernel_initializer='he_normal'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
# Block-4
model.add(Conv2D(256,(3,3),padding='same',kernel_initializer='he_normal'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Conv2D(256,(3,3),padding='same',kernel_initializer='he_normal'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
# Block-5
model.add(Flatten())
model.add(Dense(64,kernel_initializer='he_normal'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Dropout(0.5))
# Block-6
model.add(Dense(64,kernel_initializer='he_normal'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Dropout(0.5))
# Block-7
model.add(Dense(num_classes,kernel_initializer='he_normal'))
model.add(Activation('softmax'))

print(model.summary())

this architectures consists of several layers, eleven in all (8 convolutional layers and

3 fully connected).

If we increase the number of layers the accuracy will decrease.

We applied a batchNormalisation after every convolutional layer and RELU(rectified

linear unit) to eliminate the negative values. At the end, we added a layer of average

Global Pooling,

We applied then the Softmax function to calculate the rate of the 5 classes of

expressions.

Simple CNN architecture:

# Initialising the CNN
model = Sequential()
# 1 - Convolution
model.add(Conv2D(64,(3,3), padding='same', input_shape=(48, 48,1)))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# 2nd Convolution layer
model.add(Conv2D(128,(5,5), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# 3rd Convolution layer
model.add(Conv2D(512,(3,3), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# 4th Convolution layer
model.add(Conv2D(512,(3,3), padding='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# Flattening
model.add(Flatten())
# Fully connected layer 1st layer
model.add(Dense(256))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.25))
# Fully connected layer 2nd layer
model.add(Dense(512))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.25))

model.add(Dense(5, activation='softmax'))

3.6 Optimization :

when we train this model, we are basically trying to solve an optimization problem.
We are trying to optimize weigts given arbitrarly.
Our task is to find the weigths that most accurately map our input data to correct the
output class.
During the training this weights updated and saved in the file.h5.
The weights are optimised using an optimization algorithm.
In our algorithm we used the optimize SGD( stochastic gradient descent).
from tensorflow.keras.optimizers import RMSprop,SGD,Adam
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping,
ReduceLROnPlateau

checkpoint = ModelCheckpoint('Emotion_little_vgg.h5',
monitor='val_loss',
mode='min',
save_best_only=True,
verbose=1)
#earlystop = EarlyStopping(monitor='val_loss',
#min_delta=0,
#patience=3,
#verbose=1,
#restore_best_weights=True
#)
reduce_lr = ReduceLROnPlateau(monitor='val_loss',
factor=0.2,
patience=3,
 verbose=1,
min_delta=0.0001)
callbacks = [PlotLossesCallback(),checkpoint,reduce_lr]

model.compile(loss='categorical_crossentropy',
optimizer = Adam(lr=0.001),
metrics=['accuracy'])

3.7 Learning :
The next step is to learn the given data. For learning the network we use the
« fit() » function.
nb_train_samples = 24176
nb_validation_samples = 3006
epochs=15
history=model.fit_generator(
train_generator,
steps_per_epoch=nb_train_samples//batch_size,
epochs=epochs,
callbacks=callbacks,
validation_data=validation_generator,
validation_steps=nb_validation_samples//batch_size)

Epoch 00015: saving model to

C:\Users\HP\Desktop\Project\model_weights.h5
448/448 [==============================] - 1775s 4s/step - loss: 0.9309
- accuracy: 0.6492 - val_loss: 1.0208 - val_accuracy: 0.6223

An epoch refers to a single pass of the entire data set to the network during the
training.

4. Results :
5. Mesures of performance :
 Fig1. Learning and validating accuracy and loss in VGG16
accuracy
training (min: 0.255, max: 0.464, cur: 0.464)
validation (min: 0.294, max: 0.586, cur: 0.586)

Loss
training (min: 1.295, max: 1.725, cur: 1.295)
validation (min: 1.043, max: 1.551, cur: 1.043)

Epoch 00015: val_loss improved from 1.05397 to 1.04265, saving model to

Emotion_little_vgg.h5
3022/3022 [==============================] - 1431s 473ms/step - loss:
1.2950 - accuracy: 0.4640 - val_loss: 1.0426 - val_accuracy: 0.5860

Accuracy=58%

 If we increase the epoch up to 25, the value of accuracy

will increase up to 65%.
 Fig2. Fig1. Learning and validating accuracy and loss in CNN

accuracy
training (min: 0.377, max: 0.671, cur: 0.671)
validation (min: 0.410, max: 0.690, cur: 0.690)

Loss
training (min: 0.855, max: 1.499, cur: 0.855)
validation (min: 0.829, max: 1.384, cur: 0.829)

Epoch 00015: val_loss improved from 0.78075 to 0.72863, saving model to

model_weights.h5
3035/3035 [==============================] - 1215s 400ms/step - loss:
0.6999 - accuracy: 0.7360 - val_loss: 0.7286 - val_accuracy: 0.7279
Accuracy=72 %
Confusion matrix in KERAS

test_dir=validation_data_dir

import sklearn
from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report,confusion_matrix
import matplotlib.pyplot as plt
%matplotlib inline
test_datagen = ImageDataGenerator(rescale=1./255)
test_generator = test_datagen.flow_from_directory(
test_dir,
target_size=(img_rows, img_cols),
color_mode='grayscale',
batch_size=batch_size,
class_mode='categorical',
shuffle=True)
FLOW1_model = load_model('C:/Users/HP/Desktop/projectEssai/Emotion_little_vgg(3).h5')
#cnfusion Matrix and Classification Report
Y_pred = FLOW1_model.predict_generator(test_generator, test_generator.samples //
test_generator.batch_size)
y_pred = np.argmax(Y_pred, axis=1)
y_pred = y_pred.reshape(-1)
print('Confusion Matrix')
print(confusion_matrix(test_generator.classes, y_pred))
print('Classification Report')
target_names = ['happy','sad','neutral','angry', 'surprise']
print(classification_report(test_generator.classes, y_pred, target_names=target_names))
#Evaluating using Keras model_evaluate:
x, y = zip(*(test_generator[i] for i in range(len(test_generator))))
x_test, y_test = np.vstack(x), np.vstack(y)
loss, acc = FLOW1_model.evaluate(x_test, y_test, batch_size=batch_size)
print("Accuracy: " ,acc)
print("Loss: ", loss)

Confusion Matrix
[[ 17 346 253 168 176]
[ 26 684 496 291 327]
[ 15 468 321 204 208]
[ 15 441 300 194 189]
[ 9 312 213 125 138]]
Classification Report
precision recall f1-score support

happy 0.21 0.02 0.03 960

sad 0.30 0.38 0.34 1824
neutral 0.20 0.26 0.23 1216
angry 0.20 0.17 0.18 1139
surprise 0.13 0.17 0.15 797

accuracy 0.23 5936

macro avg 0.21 0.20 0.19 5936
weighted avg 0.22 0.23 0.21 5936
742/742 [==============================] - 60s 81ms/step - loss: 1.0494 -
accuracy: 0.5864
Accuracy: 0.5864218473434448
Loss: 1.0493624210357666

1. Discussions

Total params Trainable Non trainable accuracy

params params

VGG16 1.328.037 1.325.861 2176 58%

CNN 4.477.701 4.473.733 3.968 72%

The performance of the VGG16 model obtained on FER2013 dataset is relatively

low(58%) compared to the performance of the second architecture 72% due to
the number of parameters trained in the second architecture.