Recognition of face emotion in Real-time 2023-24

Abstract

This project presents a real-time face emotion recognition system with an accuracy of 62%,
utilizing OpenCV, a widely used open-source computer vision library. The primary goal of
the system is to detect faces in live video streams and classify their emotions into predefined
categories such as happiness, sadness, anger, and surprise. The system processes facial
features using machine learning techniques to predict these emotions.

To achieve real-time performance, the system captures video frames, detects faces, and
extracts relevant facial features. These features are then analyzed to determine the most likely
emotion being expressed. The system's moderate accuracy of 62% indicates room for
improvement, but it provides a solid foundation for practical applications.

Potential applications of this technology include human-computer interaction, where

understanding user emotions can enhance user experience, mental health monitoring, where
detecting emotional states can provide insights into mental well-being, and the entertainment
industry, where emotion recognition can be used for interactive content. Future work will
focus on improving the system's accuracy, expanding the range of detectable emotions, and
optimizing performance for various real-world scenarios.

Dept. of CSE, NCE, Hassan Page 1

Recognition of face emotion in Real-time 2023-24

Objectives

1.Real-Time Detection: Implement a system capable of detecting and analyzing facial

emotions in live video streams, providing immediate feedback on emotional states.

2.Emotion Classification: Classify detected facial expressions into predefined categories

such as happiness, sadness, anger, surprise, fear, and disgust.

3.Feature Extraction: Utilize OpenCV to effectively detect faces and extract relevant facial
features necessary for accurate emotion classification.

4.Machine Learning Integration: Apply machine learning techniques to improve the

accuracy of emotion recognition and handle variations in facial expressions.

5.Performance Optimization: Ensure the system operates efficiently with minimal latency
to support real-time applications.

6.User Interaction: Enhance human-computer interaction by enabling systems to respond

appropriately to users' emotional states, improving user experience and engagement.

7.Mental Health Monitoring: Provide a tool for monitoring emotional well-being,

potentially aiding in mental health assessments and interventions.

8.Applicability: Develop a flexible and scalable system that can be integrated into various
applications, such as entertainment, security, customer service, and therapeutic tools.

9.Accuracy Improvement: Continuously improve the accuracy of emotion recognition

through ongoing research, data collection, and algorithm refinement.

10.Expanding Emotion Range: Increase the range of detectable emotions to provide a more
comprehensive understanding of human emotional states.

Dept. of CSE, NCE, Hassan Page 2

Recognition of face emotion in Real-time 2023-24

Scope

1.Real-Time Emotion Detection:

 Implement a system that captures live video streams, detects faces, and identifies
emotions in real-time using OpenCV.

2.Emotion Classification:

 Focus on classifying basic emotions such as happiness, sadness, anger, surprise, fear,
and disgust.

3.Feature Extraction and Analysis:

 Utilize OpenCV to detect faces and extract key facial features (e.g., eyes, mouth,
eyebrows) necessary for emotion classification.

4.Machine Learning Integration:

 Integrate machine learning models to analyse facial features and predict emotions
based on trained data.

5.Accuracy and Performance:

 Achieve an initial accuracy rate of 62%, with plans for future improvement through
data refinement and algorithm optimization.
 Ensure the system operates efficiently with minimal latency for real-time applications.

5.Applications
Develop a versatile system that can be applied in various domains, including:

 Human-computer interaction: Improve user interfaces and experiences.

 Mental health monitoring: Provide tools for assessing and tracking emotional well-
being.
 Entertainment: Create interactive content that responds to user emotions.
 Security: Monitor emotional states in security and surveillance systems.
 Customer service: Enhance customer interactions by understanding and responding to
emotions.

Dept. of CSE, NCE, Hassan Page 3

Recognition of face emotion in Real-time 2023-24

Methodology

1.Data Collection and Preprocessing:

 Collect a dataset of facial images labeled with emotions (e.g., happiness, sadness,
anger).
 Preprocess the images by resizing, normalizing pixel values, and augmenting the
dataset for training.

2.Model Development:

 Design a convolutional neural network (CNN) using TensorFlow and Keras to extract
features from facial images.
 Compile the model with appropriate loss function and optimizer for multi-class
classification.

3.Training and Evaluation:

 Split the dataset into training and validation sets.

 Train the CNN model on the training data, monitoring validation accuracy and loss.
 Evaluate the model's performance on the validation set using metrics like accuracy,
precision, recall, and F1-score.

4.Real-Time Emotion Detection:

 Integrate the trained model into a real-time system using OpenCV to capture video
frames.
 Process each frame to detect faces using opencv-contrib-python and extract facial
regions for emotion prediction.
 Display the predicted emotions in real-time on the video stream.

5.Deployment and Optimization:

 Deploy the real-time emotion recognition system to a target environment.

 Optimize the system for performance, ensuring minimal latency and efficient resource
utilization.

Dept. of CSE, NCE, Hassan Page 4

Recognition of face emotion in Real-time 2023-24

Implications

1.Data Collection and Preprocessing:

 Use Pandas and NumPy to load and preprocess the facial image dataset.
 Augment the dataset using techniques like rotation, scaling, and flipping to increase
data diversity.

2.Model Development:

 Create a CNN model architecture using TensorFlow and Keras, incorporating

convolutional, pooling, and fully connected layers.
 Compile the model with appropriate loss function (e.g., categorical cross-entropy) and
optimizer (e.g., Adam).

3.Training and Evaluation:

 Train the model using the augmented dataset, monitoring training progress using tqdm
in a Jupyter Notebook environment.
 Evaluate the model's performance on a separate validation set, analyzing accuracy and
other metrics using scikit-learn.

4.Real-Time Emotion Detection:

 Use OpenCV to capture live video frames and process them for face detection.
 Extract facial regions from detected faces and feed them into the trained model for
emotion prediction.
 Display the predicted emotions on the video stream in real-time.

5Deployment and Optimization:

 Deploy the real-time emotion recognition system using VS Code for coding and
debugging.
 Optimize the system's performance by fine-tuning model parameters, optimizing code
for efficiency, and leveraging hardware acceleration if available.

Dept. of CSE, NCE, Hassan Page 5