GST - Bengaluru Campus

Department of Computer Science and Engineering

Human Emotion Detection

Project Phase-I Presentation

Under the guidance of Presented by

Dr. A VADIVEL G.SHREYESH 322010320058
T.HARSHTIH 322010321009
G.MOKSHITHA 322010325005
16/05/2024 V.CHANDRA VIGNESH REDDY 322010320023
1
 Outlines
• Research Objectives
• Abstract
• Existing work
• Problem Statement
• Introduction
• Literature Work
• Proposed Models
• Architecture Diagram
• Evaluation
• Conclusion
• References

05/16/2024 2
 Research Objectives:

 The following are the goals of a project on human emotion

detection that makes use of machine learning and facial recognition
technology:to create a system that automatically detects emotions
using facial expression recognition machine learning algorithms.

 To scan facial expressions, extract data from them, and deliver

precise findings in real-time

 to better customer service by evaluating personality attributes in

interviews, product testing, and client feedback
 enabling the detection, extraction, and evaluation of facial
expressions to enable the automatic recognition of human emotion
in photographs.

 Deep network synchronization of voice signals and visual sequences

for emotion recognition

 To create a reliable system that can use live stream to distinguish

and detect human emotion
 ABSTRACT:

 Detection of Emotion in the Audio and Video plays a major role in

many technological sectors. Mainly used in market research and
testing. Emotion detection technology can be developed through
various methods.
 This technology can be developed by using deep learning with
convolutional neural networks (CNN) and uses various techniques and
algorithms.
 The main objective here is to classify each face and audio based on the
emotions shown into seven categories which include Anger, Disgust,
Fear, Happiness, Sadness, Surprise.
 For face detection using the Hear cascades is a machine learning
based algorithm where a cascade function will be trained with a set
of input data

 OpenCV contains many pre-trained classifiers for face, eyes, smile

etc. The deep learning is a subset of machine learning

 Deep learning is used by Google to translate the information form

one language to another using deep learning approach.

 The main objective here in this project is, to read the emotion of the
people and displaying them.
 Existing work:

 Text-Based Emotion Detection: - *Sentiment Analysis: This is a

common approach that involves determining the sentiment (positive,
negative, neutral) expressed in a text.

 Emotion Lexicons: Using pre-built emotion lexicons that associate

words with specific emotions and aggregating these associations to
determine the predominant emotion.

 Deep Learning Models: Utilizing recurrent neural networks (RNNs),

long short-term memory networks (LSTMs), or transformers like BERT
to capture emotional context in text.

 Hybrid Approaches: Combining linguistic features with deep learning

models to improve emotion detection accuracy.
 Speech-Based Emotion Detection: - *Acoustic Features: Extracting
features like pitch, intensity, and voice quality to detect emotions
from speech signals. - *Prosodic Patterns: Analyzing patterns in
speech rhythm, intonation, and tempo to infer emotions. - *Deep
Learning on Audio: Using convolutional neural networks (CNNs) and
recurrent networks to process raw audio for emotion classification.
 Image-Based Emotion Detection: - *Facial Expression Recognition:
Detecting emotions from facial expressions using deep learning
models, including convolutional neural networks (CNNs) and
convolutional/recurrent hybrid models. - *Visual Features:
Extracting features from images and using machine learning
algorithms to classify emotions. - *Multimodal Approaches:
Combining facial expression analysis with other modalities like text
or speech to improve accuracy.
 PROBLEM STATEMENT:

 Humans are well trained in reading the emotions of others, in fact, at

just 14 months old, babies can already tell the difference between
happy and sad. But can computers do a better job than us in
accessing emotional states?

 A deep learning neural network that gives machines the ability to

make inferences about our emotional states. In other words, we give
them eyes to see what we can see.
 INTRODUCTION:

 The emotion detection technology has many uses like providing

security whereas the emotion detection technology using audio and
video will be useful for detection of truthfulness or potential
deception.
 By using this technology, it gets easy to know a person's way of
thinking and his interest in various aspects which will be very
advantageous for various companies for improving their sales.
 Emotion Detection through audio analyzing is a technology that
improves product and service performance by monitoring customer
behaviour for evaluation.

 Facial expression detection extracts and analyses the information

taken from the images or videos, it will be able to deliver the exact or
the unbiased emotional responses as data.
 We achieve facial detection by detecting the faces and by analysing
the movement of our eyes, nose, lips etc. and analysing changes in
the appearance of the facial features and classifying various
expressions.

 In this project, deep learning with the convolutional neural networks

(CNN) approach is used. Neural networks need to be trained with
large amounts of data and have a higher computational power
 Literature Work

 Being that emotion is a subjective concept, utilizing the knowledge

and science underlying labeled data and separating the elements
that make it up has long been a difficult topic in the industry. Since
the development of deep learning in computer vision, the issue of
emotion recognition has attracted a lot of attention from
researchers. For this specific purpose, we provide two independent
methods in this paper.
 This project employs convolutional neural networks (CNN) and deep
learning. Large volumes of data must be used to train neural
networks, which also require more processing power. The findings
demonstrate that our CNN model can outperform cutting-edge
techniques for emotion recognition with more depth and fine-
tuning.
 One of the most popular uses of computer vision is face expression
recognition. It is assiduously utilized in medical applications like
monitoring elderly health, counseling, and figuring out a client's
health, as well as figuring out a patient's mental readiness for
treatment.
 Emotions of the students are studied in the context of e-learning,
and the teaching and learning processes are modified as a result.
However, a precise and straight orientation of the face is required in
order to detect any face and its components.
 If the face is rotated or titled, the algorithms for detection will not
function. In their article, they present an algorithm that can only
recognise faces at specific angles and in any orientation.
 Proposed Solutions:

Sketch of the Methodology/Methodologies you would

adopt to Solve the Problem –

Figure 1: Methodology:
Figure 2: Flow Diagram
 The process for examining the straightforward facial expressions in
videos is described in Figure 1. The following is a list of the simple
emotion analysis steps.

 Real-time data from visual input and a dataset of emotionsData

cleaning, segmentation, and region of interest (ROI) for image
processing and EEG signal processingRecognize the human face in
the video frames using face recognition software.Research: Feature
Extraction-SelectionML/DL methods for First Stage (Image
Processing) and Second Stage (EEG Signal-based) Emotion Detection
 ARCHITECTURE:

Flowchart of Face Expression Detection

Feature Extraction
 Our project's major goal is to develop a model that can be applied
in a variety of fields, like feedback response, medical, etc., with the
aid of both audio and video components.

 For the small project, we employed CNN; however, for a larger

project, we wish to apply a more sophisticated methodology.

 The objective is to perfect the expression to its nearly optimal

performance and make it as accurate as feasible.

 Different emotions are discerned in speech emotion detection by

the pitches of the choice and variations in the graph's aptitude.
 EVALUATION:

 Annotation and Ground Truth: To evaluate the system, a labeled

dataset is required where human annotators have categorized the
emotions. This dataset serves as the ground truth for evaluating
system

 performance.Metrics: Common metrics include accuracy, precision,

recall, F1-score, and confusion matrix. These metrics help quantify
the system's performance in emotion classification.
 Cross-Validation: Splitting the dataset into training and testing sets
helps assess how well the system generalizes to new, unseen data.
 ED process for a face region and facial landmarks detection input
image.

 Facial landmarks are visually noticeable places on the face, such as

the tip of the nose, the corners of the mouth, and the ends of the
brows, as depicted in Figure.

 The local texture of a landmark or the pairwise positions of two

landmark points
 CONCLUSION:

 Detecting the facial expressions, using fusion-based approach by

Image processing and EEG signal processing will yield greater
accuracy.

 Detecting the facial expressions and emotions in real time would

accompany to decode the psychological problems.

 Businesses can process images, and videos in real-time for

monitoring video feeds or automating video analytics (feedback /
reviews – Eg: Movies / Product), thus saving costs and making life
better for the users.
 REFERENCES:

 Ekman, P., Friesen, W.. Facial Action Coding System: A

Technique for the Measurement of Facial Movement. Palo
Alto: Consulting Psychologists Press; 1978

 Darwin C., The Expression of the Emotions in Man and

Animals, Murray J., London, 1872.

 J. M. Saragih, S. Lucey and J. F. Cohn, "Real-time avatar

animation from a single image," Face and Gesture 2011, Santa
Barbara, CA, USA, 2011
 Ko, K.E., Sim, K.B.. Emotion recognition in facial image sequences
using a combination of aam with facs and dbn. In: Proceedings of the
Third International Conference on Intelligent Robotics and
Applications - Volume Part I; ICIRA’10. Berlin, Heidelberg: Springer-
Verlag. ISBN 3-642-16583-4,978-3-642-16583-2; 2010, p. 702–712.
URL: http://dl.acm.org/citation.cfm?id=1939443.1939518.

 Dalal, N., Triggs, B.. Histograms of oriented gradients for human

detection. In: Computer Vision and Pattern Recognition, 2005. CVPR
2005. IEEE Computer Society Conference on; vol. 1. 2005, p. 886–893
vol. 1. doi:10.1109/CVPR.2005.177.