AI-Driven Emotion Recognition and Adaptive

Communication for Autism Education

 Department of Artificial Intelligence and Data Science

Project Co-Ordinator : Arun Kumar R AI&DS

Team Leader : Tejasvni K - 727821TUAD056

Team Member 1 : Lokesh T - 727821TUAD029

Team Member 2 : Rajee R - 72782TUAD605

 Table of Contents
 Objective
 Abstract
 Introduction
 Literature Survey
 Problem Definition
 Innovation and Methodology
 Conclusion
 Base Paper Details
 References
Objective :

The objective of this project is to develop a real-time emotion recognition and adaptive
communication system to assist autistic children in improving their social interaction and emotional
recognition capabilities. By leveraging AI technologies such as deep learning, machine learning,
and prompt engineering, the system aims to accurately detect and classify emotions from facial
expressions and provide tailored communication methods and educational content. This approach
seeks to create an accommodating and supportive environment that enhances the quality of life and
learning outcomes for autistic children through real-time assistance and personalized educational
tools.
Abstract :
Autism Spectrum Disorder (ASD) presents significant challenges in social
interaction and communication, particularly in recognizing and responding to emotions. This
project proposes an innovative solution that leverages AI technologies, including Machine Learning
(ML), Deep Convolutional Neural Networks (DCNN), and prompt engineering, to assist autistic
children in real-time emotion recognition and adaptive communication. By utilizing a multi-layered
approach integrating DCNN models, autoencoders, and IoT frameworks, the system will identify
and interpret emotions accurately. The subsequent phase involves the deployment of tailored
communication methods and educational curriculums, designed based on the emotional state
detected, ensuring an accommodating and supportive interaction environment. This holistic
approach aims to enhance the quality of life and learning outcomes for autistic children by
providing them with real-time assistance and personalized educational tools.

Keywords: Autism Spectrum Disorder, emotion recognition, deep learning, autoencoders, IoT,
real-time detection, adaptive communication, educational technology, machine learning.
Introduction :

• Challenges in Autism Care: Addresses difficulties in emotion recognition and social interaction
for children with Autism Spectrum Disorder (ASD).
• AI-Powered Emotion Recognition: Utilizes deep learning models (DCNN) to accurately detect
and classify emotions from facial expressions.
• Adaptive Communication Techniques: Implements prompt engineering and NLP to provide
tailored, empathetic communication based on detected emotions.
• Integration of IoT and Fog Computing: Ensures low-latency, real-time processing using IoT
devices and fog computing for efficient data handling.
• Personalized Educational Tools: Adjusts educational content dynamically to create a supportive
learning environment, enhancing outcomes for autistic children.
Literature Survey :

S. Title Authors Year Inference Limitations

No
1. Real-time facial Fatma M. 2024 This paper introduces a The study notes
emotion Talaat, real-time emotion misclassification issues
recognition Zainab recognition system for with overlapping emotions
model based on H. Ali, children with Autism and a small dataset,
kernel Reham Spectrum Disorder stressing the need for
autoencoder and R. (ASD) using a DCNN, model refinement and
convolutional Mostafa, achieving 95.23% dataset expansion for
neural network Nora El- accuracy. However, better generalizability.
for autism Rashidy challenges with emotion
children overlap and a small
dataset limit its
generalizability.
2. Autism Inge Kamp- 2024 This article The ICD-11's broad ASD
spectrum Becker critiques the ICD- criteria may lead to
disorder in 11’s broader ASD overdiagnosis and false
ICD-11—a criteria, arguing that positives, complicating
critical it may increase clinical practice and
reflection of diagnostic research by reducing
its possible variability, leading diagnostic specificity and
impact on to more false increasing sample
clinical positives and heterogeneity.
practice and complicating access
research to services and
research efforts.
3. Deep learning Vidhya S, 2023 The paper reviews deep The paper highlights challenges
with image- Rajesh R, learning methods for in deep learning for ASD, such
based autism Rohith P, ASD detection, as dataset variability, the need
spectrum Miruthyunj classification, and for larger datasets, and real-
disorder Sanjay S, rehabilitation using world application limitations,
analysis: A Pradeep R, image and video data, and notes its coverage is
systematic Jeevanantham discussing their limited to studies up until June
review S benefits and challenges 2023.
for advancing diagnosis
and treatment.
Problem Definition:
1. Real-Time Emotion Recognition System:

Focus: Development and implementation of a deep learning-based emotion recognition

model using DCNNs for accurate facial expression classification.
Components: Data collection, preprocessing, model training, integration of IoT and fog
computing for real-time processing.

2. Adaptive Communication and Educational Customization:

Focus: Design and deployment of adaptive communication methods and personalized

educational content based on detected emotional states.
Components: Prompt engineering, NLP for empathetic responses, dynamic curriculum
adjustment, and integration with educational software.
3. Model Development:

Details: Build and train DCNN models for emotion classification.

4. Real-Time Processing:

Details: Integrate IoT and fog computing for low-latency data handling.
5. Communication Adaptation:

Details: Create NLP-based communication prompts tailored to emotional states.

6. Educational Content Customization:

Details: Adjust educational material dynamically based on detected emotions.

7. System Integration:

Details: Combine emotion recognition, communication, and educational tools into

a unified platform.
Innovation and Methodology :

AI-Driven Emotion Detection:

• The project introduces an advanced emotion detection system that leverages Deep
Convolutional Neural Networks (DCNN) to achieve highly accurate real-time recognition of
facial expressions. By utilizing sophisticated pre-trained models such as Xception and
MobileNet, the system surpasses traditional emotion recognition methods in terms of
precision and reliability.

Data Collection and Preparation:

• The project begins with the collection of a diverse dataset of facial expressions that cover a
wide range of emotions. This dataset is then carefully preprocessed to ensure high quality and
robustness, preparing it for model training.
Adaptive Communication Framework:
• Another significant innovation is the adaptive communication framework, which employs
prompt engineering and Natural Language Processing (NLP) to generate personalized,
empathetic responses tailored to the child's emotional state. This ensures that interactions are
contextually appropriate and supportive, enhancing the overall communication experience.

Model Development and Training:

• Deep learning models are developed using advanced architectures like Xception and MobileNet.
These models are trained and fine-tuned on the preprocessed dataset using Keras and
TensorFlow, focusing on achieving high accuracy in emotion recognition.
Real-Time Integration:
• The trained models are integrated into an Internet of Things (IoT) framework, with fog
computing employed to manage and process data locally. This setup ensures real-time data
handling and minimal latency, providing immediate feedback.

Communication and Curriculum Adaptation:

• Adaptive communication methods are designed using NLP to create contextually relevant
prompts based on real-time emotional data. Additionally, the educational content is dynamically
adjusted to align with the child's emotional state, enhancing the learning experience.
Conclusion :

In conclusion, this project introduces a comprehensive solution for supporting autistic children by
integrating advanced AI-driven emotion recognition, adaptive communication techniques, and
personalized educational tools. Through the use of Deep Convolutional Neural Networks (DCNN),
IoT devices, and fog computing, the system offers real-time, responsive assistance tailored to the
emotional and educational needs of children with Autism Spectrum Disorder (ASD). This innovative
approach not only enhances emotional understanding and communication but also creates a more
supportive and effective learning environment, ultimately aiming to improve the quality of life and
educational outcomes for autistic children.
Base Paper Details :
• Title : A biometrics-generated private/public key cryptography for a blockchain-based e-voting

system

• Authors : Jide Kehinde Adeniyi , Sunday Adeola Ajagbe , Emmanuel Abidemi Adeniyi ,
Pragasen Mudali , Matthew Olusegun Adigun , Tunde Taiwo Adeniyi , Ojo Ajibola

• Year : 2024

• Journal Name : Egyptian Informatics Journal

• Cite Score : 11.1

• Impact Factor : 5.0

• Journal Details : Voting aims to provide the best decision or select the most selected option
for the largest group of voters. Malicious parties gaining access, and otherwise tampering
with election results, or the votes make this effort counterproductive. To alleviate this, this
study examined the introduction of blockchain. The transparent and immutable nature of the
blockchain makes this data impossible to alter and allows the election results to be
transparent. To further increase the transparency of the system while keeping voters
anonymous, a biometric based cryptography was introduced. The biometric was introduced as
the source for the private key for each voter while a public was generated to act as the identity
of the voter. The biometric trait of each individual is unique and cannot be forged, hence the
identity of the voter is secured. The public key available cannot be traced by to the private
key, hence, identity of the voter is anonymous. The system showed an encouraging
performance after testing.
• System Design: This AI system will assist caregivers by providing real-time emotion
recognition and generating communication or action suggestions based on the detected
emotions of autistic children. It will utilize multi-modal data (visual and behavioural),
generate personalized emotion profiles, and leverage explainable AI to ensure transparency
of its predictions. Additionally, the system will be designed for global usability through
multi-language support.
Key Features
1. Multi-Modal Emotion Recognition:
• Inputs: Visual data (facial expressions via camera) and behavioural data (questionnaire)
• Processing: Deep learning models (CNNs for facial recognition and
• RNNs/LSTMs for emotion analysis) process the multi-modal data to
• identify the child’s emotional state.
• Emotion Output: The system will map facial expressions and
• behaviours to predefined emotional states (e.g., happy, frustrated, sad,agitated).
2. Multi-Modal Emotion Recognition:
• The system will build individual emotion profiles for each child by learning their unique
behavioral patterns over time.
• Emotion models will evolve with the child’s emotional expressions, providing more
accurate predictions as the system gains more data.
3. Generative AI for Communication Assistance:
• Based on the identified emotions, the system will generate actionable suggestions for
caregivers (e.g., calming techniques if frustration is detected).
• It will also provide language-based suggestions to help the child communicate more
effectively, especially when they are struggling to express emotions verbally.
4. Explainable AI:
• The model will provide detailed explanations for its emotion predictions. For example, it
will show which features (e.g., raised eyebrows or body language) led to the detection of
a certain emotion.
• Caregivers and medical professionals can review these explanations to better understand
the system’s reasoning. It will also provide language-based suggestions to help the child
communicate more effectively, especially when they are struggling to express emotions
verbally.
5. Multi-language Support:
• The system will integrate with multi-language LLMs to support caregivers in different
languages, making it usable globally.
• Communication suggestions and action plans will be generated in the caregiver’s native
language.
Instructions for the System:
1. Data Collection:
• Collect and preprocess multi-modal emotion datasets (Autistic Children Facial Dataset :
https://www.kaggle.com/datasets/imrankhan77/autistic- children-facial-data-set).
• Annotate the data with emotional states and create diverse training samples.
2. Model Training:
• Emotion Recognition Models:
 Use CNNs (Convolutional Neural Networks) for visual emotion
recognition.
 Use LSTMs (Long Short-Term Memory) or RNNs (Recurrent Neural
Networks) for behavioral emotion recognition.
• Personalization:
 Train the system to update the child’s emotion profile over time using
reinforcement learning techniques.
• Generative AI:
 Integrate hugging face LLMs to generate communication and action
suggestions based on the identified emotions.
3. Interface Design:
• Create a mobile or web-based UI for caregivers that displays real-time emotion detection
results, actionable insights, and explanations for predictions.
• The UI will include features like child profiles, emotion logs, and a notification system
for critical emotions (e.g., frustration or distress).
4. Explainability:
• Incorporate an explainable AI module that uses SHAP or LIME (Local Interpretable
Model-agnostic Explanations) to explain why a certain emotion was detected.
5. Multi-language Support:
• Use OpenAI’s language models to translate communication suggestions into multiple
languages, making the system accessible globally.
Expected Output:
1. Step-by-Step Guide:
• Provide clear documentation on how to implement the emotion recognition models,
including dataset preprocessing, model architecture, and training techniques.
• Include guidance on integrating Generative AI for communication assistance,
personalization techniques for emotion profiles, and explainability tools.
2. Code Implementation:
• Python code for integrating multi-modal inputs (camera and microphone data) for real-
time emotion detection.
• Code demonstrating how to use a pre-trained GPT model to generate communication
suggestions.
• Code for implementing explainable AI using SHAP or LIME to interpret the model’s
predictions.
3. Emotion Model Personalization:
• Explain how the system updates its emotion recognition capabilities over time by
collecting more data from each child and retraining the model periodically to adapt to
the child’s emotional behavior.
4. Optimization for Edge Devices:
• Suggestions for optimizing the system to run efficiently on edge devices (such as tablets
or mobile phones) by using model compression techniques like quantization and pruning
to reduce computational load while maintaining performance.
References :
[1] Adeniyi, Jide Kehinde, et al. "A biometrics-generated private/public key cryptography for a
blockchain-based e-voting system." Egyptian Informatics Journal 25 (2024): 100447.

[2] Ossai, V.C., Okoro, I.C., Alagbu, E.O., Agbonghae, A.O. and Okafor, I.N., Enhancing E-voting
systems by Leveraging Biometric Key Generation (Bkg). American journal of Engineering research
(AJER), 2, pp.180-190.

[3] Ahmad, M., Rehman, A. U., Ayub, N., Alshehri, M. D., Khan, M. A., Hameed, A., & Yetgin, H.
(2020). Security, usability, and biometric authentication scheme for electronic voting using multiple
keys. International Journal of Distributed Sensor Networks, 16(7), 1550147720944025.

[4] Ajish S, AnilKumar KS. Secure mobile internet voting system using biometric authentication and
wavelet based AES. Journal of Information Security and Applications. 2021 Sep 1;61:102908.
[5] Ologunde E. Cryptographic Protocols for Electronic Voting System. Available at SSRN 4823470.
2023 Dec 29.

[6] Babenko, L., Pisarev, I., & Popova, E. (2019, September). Cryptographic protocols
implementation security verification of the electronic voting system based on blind intermediaries. In
Proceedings of the 12th International Conference on Security of Information and Networks (pp. 1-5).

[7] Cetinkaya, Orhan. "Analysis of security requirements for cryptographic voting protocols." In 2008
Third International Conference on Availability, Reliability and Security, pp. 1451-1456. IEEE, 2008.

[8] Damgård, I., Groth, J. and Salomonsen, G., 2003. The theory and implementation of an electronic
voting system. Secure Electronic Voting, pp.77-99.
[9] Patidar, K. and Jain, S., 2019, July. Decentralized e-voting portal using blockchain. In 2019 10th
International Conference on Computing, Communication and Networking Technologies (ICCCNT) (pp. 1-
4). IEEE.

[10] Lyu, J., Jiang, Z.L., Wang, X., Nong, Z., Au, M.H. and Fang, J., 2019, August. A secure decentralized
trustless E-voting system based on smart contract. In 2019 18th IEEE International Conference On Trust,
Security And Privacy In Computing And Communications/13th IEEE International Conference On Big
Data Science And Engineering (TrustCom/BigDataSE) (pp. 570-577). IEEE.

[11] Hardwick, F.S., Gioulis, A., Akram, R.N. and Markantonakis, K., 2018, July. E-voting with
blockchain: An e-voting protocol with decentralisation and voter privacy. In 2018 IEEE International
Conference on Internet of Things (iThings) and IEEE Green Computing and Communications
(GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data
(SmartData) (pp. 1561-1567). IEEE.
[12] Hjálmarsson, F.Þ., Hreiðarsson, G.K., Hamdaqa, M. and Hjálmtýsson, G., 2018, July.
Blockchain-based e-voting system. In 2018 IEEE 11th international conference on cloud computing
(CLOUD) (pp. 983-986). IEEE.

[13] Alvi, Syada Tasmia, Mohammed Nasir Uddin, and Linta Islam. "Digital voting: A blockchain-
based e-voting system using biohash and smart contract." In 2020 third international conference on
smart systems and inventive technology (ICSSIT), pp. 228-233. IEEE, 2020.

[14] Alvi, S.T., Uddin, M.N., Islam, L. and Ahamed, S., 2022. DVTChain: A blockchain-based
decentralized mechanism to ensure the security of digital voting system voting system. Journal of
King Saud University-Computer and Information Sciences, 34(9), pp.6855-6871.

[15] Jagjivan, M.P., Shrikant, J.P., Vijay, J.N., Pradeep, K.R. and Suhas, P.A., 2021, October. Secure
Digital Voting system based on Aadhaar Authentication by using Blockchain Technology. In 2021
IEEE Mysore Sub Section International Conference (MysuruCon) (pp. 861-870). IEEE.