Facial Emotion
Recognition
using
Convolutional
Neural
Networks
Under the guidance of Dr. Annapurna Jonnalagadda
Group Members
Sudesha Basu Majumder Parichay Singh Sreyan Ghosh
(19BEE0228) (19BEE0229) (19BEE0232)
Basic Coding and Coding and creating Coding and creating
Documentation the Web App the Web App
Background
With the advent of intelligent conversational agents who can understand human
speech, we have stepped into a new era in computational intelligence. But this is
still very much in its initial stages where only about 60%-70% of human speech is
recognised correctly. This is mainly due to one primary reason. A reason that has
been on the minds of the poets and storytellers alike for centuries. “What a man
says is not what he means”. Words used in a sentence have multiple connotations
and meanings. For a mathematical algorithm to understand this is impossible
without visual cues. Elements of speech like irony, sarcasm and rhetorics can only
be understood by these artificially intelligent systems only if visual cues are
recognised. In this project, we have built and trained a convolutional neural
network (CNN) from scratch to recognize facial expressions. This will enable us to
understand what people actually “mean” when they “say” something. The data
consists of 48x48 pixel grayscale images of faces. The objective is to classify
each face based on the emotion shown in the facial expression into one of seven
categories (0=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral).
We have used OpenCV to automatically detect faces in images and draw
bounding boxes around them. Once we have trained, saved, and exported the
CNN, we will directly serve the trained model to a web interface and perform real-
time facial expression recognition on video and image data.
The aim of our project is to create a web app that can take
in a live video feed through the camera of the device,
identify if a face is present in the frame, and also identify
the emotion being portrayed by the face. The utility of this
model can be found in intelligent conversational agents
where the mood of the person interacting plays a crucial
role in understanding the context in which the words are
being spoken.
Objectives
What we want To accomplish our task we would be using a customized
neural network consisting of 4 convolutional layers, and 2
to achieve fully connected layers to predict 7 types of facial emotions.
The dataset we would be using is an open-source facial
data dataset from Kaggle graciously provided by Pierre-Luc
Carrier and Aaron Courville. More details on the dataset can
be found in the next section.
Project Timeline
Phase 04
Phase 02 Creating a
Creating the Flask app
Convolutional to serve
Neural predictions
Network
.
(CNN) model.
01 03 04 05
Phase 01
02 Phase 03 Phase 05
Generating Training Using the
training, and model to
validation Evaluating recognise
and testing the model. facial
batches. expression
in videos.
Our Tech Stack
Colab was our primary code editor
We will use Flask to create our web-app
We will be using OpenCV to detect faces We plan to use Tensorflow as our primary library
in the video feed we will process. for bringing the model to life
Expected
Results
SWOT of our Idea
Strength
Although trained on an image based dataset, it works on video as well as live camera
feed.
Applications include autonomous driving assistance and autonomous intensive
healthcare and monitoring.
Weakness
A person might not show his actual emotions by means of facial expressions. This
gap cannot be bridged by our model.
Opportunities
The global emotion detection and recognition market size is projected to grow from
USD 21.6 billion in 2019 to USD 56.0 billion by 2024, at a Compound Annual Growth
Rate (CAGR) of 21.0% during the forecast period.
Factors such as the rising need for socially intelligent artificial agents, increasing
demand for speech-based biometric systems to enable multifactor authentication,
technological advancements across the globe, and growing need for high operational
excellence are expected to work in favor of the market in the near future
Threats
Due to our model's accuracy being less than 70%, it cannot be used in applications
where there can be serious tradeoffs.
Overview of Research Methods
arXiv:1603.04467v2 TensorFlow: Large-Scale Machine Learning on
Heterogeneous Distributed Systems
arXiv:1803.04108v4 Style Aggregated Network for Facial Landmark
Detection
arXiv:1409.1556v6 Very Deep Convolutional Networks for Large-Scale
Image Recognition
IJEAT ISSN:2249-8958 Emotion Recognition from Facial Expression using
Deep Learning
I.J. Image, Graphics and Signal Processing, 2012, 8, 50-56 Human
Emotion Recognition System
IJRTE ISSN:2277-3878 Effective Facial Emotion Recognition using
Convolutional Neural Network Algorithm
arXiv:1910.11113 Facial Emotion Recognition Using Deep Learning: Ching-
Da Wu , Li-Heng Chen
Imavis.2008.08.005 Facial expression recognition based on Local Binary
Points: A comprehensive study
DINWC.2018.8356995. Research on face recognition based on deep
learning: X. Han and Q. Du
Conference on Computer Vision and Pattern Recognition, 2001 Rapid
Object Detection using a Boosted Cascade of Simple Features: Paul Viola,
Michael Jones
DOI:10.1109/TAFFC.2020.2981446 Deep Facial Expression Recognition: A
survey
DOI:10.1109/TIP.2006.884954 Facial Expression Recognition in Image
Sequences Using Geometric Deformation Features and Support Vector
Machines
ICEMI, Yangzhou, 2017 Human face detection algorithm via Haar cascade
classifier combined with three additional classifiers: L. Cuimei, Q. Zhiliang,
J. Nan and W. Jianhua
IEEE CVPR 2018/2168-2177 Facial Expression Recognition by De-
Expression Residue Learning
