Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

1. INTRODUCTION
Emotions play a vital role in human communication, influencing decision-making, behavior, and
interpersonal interactions. While traditional emotion recognition methods rely on facial
expressions, speech, and physiological signals, handwriting analysis offers a unique and less
intrusive approach to understanding emotions. Handwriting carries distinct characteristics—such
as stroke pressure, slant, spacing, and thickness—that can reflect a writer's emotional state.

This project focuses on Handwriting-Based Emotion Detection Using Deep Learning and
Image Processing, leveraging computer vision and machine learning techniques to classify
emotions from handwritten text. The system extracts key handwriting features and combines them
with deep learning models to improve accuracy in emotion classification.

The approach consists of two primary components:

1. Image-Based Feature Extraction – Utilizing MobileNetV2, a pre-trained convolutional

neural network (CNN), to extract high-level features from handwriting images.

2. Handwriting-Specific Feature Analysis – Extracting and analyzing features such as

stroke thickness, slant angle, spacing, character size, pressure variation, and baseline
alignment using OpenCV.

To enhance model performance, data augmentation techniques—including rotation, shearing,

scaling, and noise addition—are applied to improve dataset diversity. The extracted features are
fed into a hybrid deep learning model, which consists of a CNN for image analysis and a fully
connected neural network for feature-based classification.

The project aims to provide a robust system for emotion detection through handwriting
analysis, which has potential applications in psychology, forensics, education, and AI-driven
sentiment analysis. Future improvements may include expanding the dataset, optimizing
hyperparameters, and real-time deployment for practical use.

Evolution of Translation Systems

• Traditional Graphology-Based Analysis – Relied on human interpretation of

handwriting traits to assess emotions, lacking scientific validation.
• Statistical and Rule-Based Methods – Used predefined handwriting features like stroke
pressure and letter spacing for basic emotion classification but struggled with
handwriting variability.
• Machine Learning-Based Analysis – Employed classifiers like Naïve Bayes and SVMs
to improve accuracy, but required extensive feature engineering.

Dept of AIML 2024-25 1|Page

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

• Deep Learning and Neural Networks – CNNs (e.g., MobileNetV2) enabled automatic
feature extraction from handwriting images, significantly improving classification
accuracy.
• Real-Time AI-Powered Emotion Detection – Modern AI models integrate deep
learning and computer vision for real-time, automated emotion recognition, with
applications in mental health and forensic analysis.

1.1 Existing System

Current handwriting-based emotion detection systems primarily rely on traditional graphology

techniques or basic machine learning models. These approaches have certain limitations in
accuracy, scalability, and real-time applicability.

1. Graphology-Based Methods

• Handwriting analysis is done manually by experts, focusing on factors like stroke

pressure, slant, and spacing.

• Highly subjective and lacks scientific reliability.

2. Feature-Based Machine Learning Models

• Uses algorithms like Naïve Bayes, Support Vector Machines (SVMs), and
Decision Trees for classification.

• Requires manual feature extraction (e.g., stroke thickness, baseline alignment).

• Struggles with handwriting variations and lacks adaptability to unseen data.

3. Basic Image Processing for Handwriting Recognition

• Involves thresholding, contour detection, and edge detection for extracting

handwriting features.

• Provides useful insights but lacks deep contextual understanding of emotional

patterns.

Limitations of the Existing System

• Lack of Automation – Most systems require manual feature selection or graphology

expertise.

• Limited Accuracy – Struggles with different handwriting styles, ink pressure, and
writing speed.

• No Deep Learning Integration – Existing models do not leverage advanced CNN

architectures for feature extraction.
Dept of AIML 2024-25 2|Page
Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

1.2 Proposed System

The proposed system aims to develop an automated, deep learning-based handwriting

emotion detection model that combines computer vision and handwriting feature analysis
to improve accuracy and scalability. This system overcomes the limitations of traditional
methods by integrating CNN-based image processing with handwriting-specific feature
extraction for precise emotion classification.

Key Features of the Proposed System:

1. Hybrid Model for Emotion Detection

• Utilizes MobileNetV2, a pre-trained CNN, to extract deep image features from

handwriting samples.

• Extracts handwriting-specific features such as stroke thickness, slant angle,

spacing, character size, and writing pressure using OpenCV.

• Combines both feature sets in a multi-input neural network for enhanced

classification.

2. Automated Feature Extraction & Deep Learning Integration

• Eliminates the need for manual feature selection used in traditional graphology
methods.

• Deep learning automates feature learning, making the system adaptable to

different handwriting styles.

3. Data Augmentation for Improved Performance

• Applies rotation, shearing, scaling, noise addition, and brightness adjustments to

increase dataset diversity.

• Ensures robust training and prevents overfitting.

4. Real-Time Emotion Detection

• Capable of processing handwriting samples in real-time, making it suitable for

applications in psychology, forensic analysis, and human-computer interaction.

5. Higher Accuracy and Scalability

• Combining image-based and feature-based analysis improves model accuracy.

• Can be extended to large-scale datasets for broader applications.

Dept of AIML 2024-25 3|Page

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

2.Literature Survey
Handwriting-based emotion detection has been an evolving research area, integrating graphology,
machine learning, and deep learning techniques to improve accuracy and reliability. Various
studies have explored different approaches to analysing handwriting for emotional state detection.

1. Traditional Graphology-Based Approaches

Study: Graphology and Personality Traits Analysis (1990s–2000s)

• Early studies focused on manual handwriting analysis to assess emotions and personality
traits.

• Features such as stroke slant, letter spacing, and baseline alignment were interpreted
subjectively.

• Limitations: Lack of scientific validation, subjective results, and dependency on expert

analysis.

2. Feature-Based Machine Learning Methods

Study: Emotion Recognition from Handwriting Using SVM and Naïve Bayes (2015)

• Introduced feature-based classification using handwriting metrics (e.g., stroke pressure,

spacing, slant).

• Applied Support Vector Machines (SVM) and Naïve Bayes classifiers for emotion
detection.

• Limitations: Manually extracted features led to limited adaptability to different

handwriting styles.

3. Computer Vision-Based Handwriting Analysis

Study: Handwriting Recognition Using Image Processing and Deep Learning (2017)

• Used OpenCV and edge detection algorithms to extract structural handwriting features.

• Combined image processing with statistical handwriting attributes to classify emotions.

• Limitations: Unable to capture deeper semantic representations of handwriting patterns.

Dept of AIML 2024-25 4|Page

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

4. Deep Learning-Based Approaches

Study: Handwriting Emotion Detection Using CNNs and Transfer Learning (2020–Present)

• CNNs (Convolutional Neural Networks) and transfer learning models (e.g., MobileNetV2,
ResNet) were applied to handwriting images.

• Eliminated the need for manual feature selection by automatically extracting deep features
from handwriting patterns.

• Combined CNNs with handwriting-specific feature analysis (e.g., stroke density, pressure
variation).

• Achieved higher accuracy and robustness compared to previous approaches.

Key Findings from Literature

• Traditional graphology is highly subjective and lacks scientific consistency.

• Machine learning models improve accuracy but require extensive manual feature
extraction.
• Deep learning with CNNs provides automated feature learning, making emotion
detection more scalable and accurate.

• Hybrid models combining CNNs with handwriting-specific features yield the best results
for emotion classification.

Dept of AIML 2024-25 5|Page

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

3.Software and Hardware Specification

3.1 Problem Statement

Emotions significantly influence human behaviour, decision-making, and communication. While

most emotion recognition systems focus on facial expressions, speech, or physiological signals,
handwriting remains an overlooked but valuable indicator of emotional states. Various handwriting
characteristics, such as stroke pressure, slant, spacing, and thickness, can reflect the psychological
and emotional conditions of an individual. However, traditional handwriting analysis techniques,
primarily based on graphology, are highly subjective, inconsistent, and lack scientific validation.

Existing handwriting-based emotion detection methods often rely on machine learning models that
require manual feature extraction and struggle with handwriting variations across different
individuals. These approaches lack automation, scalability, and the ability to generalize effectively
across diverse handwriting styles. Additionally, previous research in this area has failed to integrate
deep learning techniques, limiting the accuracy and robustness of existing systems.

To overcome these challenges, this project proposes a hybrid deep learning-based handwriting
emotion detection system. The model integrates CNN-based image processing (MobileNetV2) and
handwriting-specific feature extraction using OpenCV to automatically analyse handwriting
patterns. It also leverages data augmentation techniques (such as rotation, shearing, noise addition,
and scaling) to improve model performance. The system is designed to enable real-time,
automated, and scalable emotion detection with potential applications in mental health assessment,
forensic analysis, and AI-driven sentiment analysis.

5.2 Software Requirements:

1. Operating System

• Windows 10/11, macOS, or Linux

2. Programming Language

• Python 3.7 or later

3. Libraries & Dependencies

• Computer Vision & Image Processing:

• OpenCV (pip install opencv-python) – For handwriting image processing

• PIL (Pillow) (pip install pillow) – For handling images

Dept of AIML 2024-25 6|Page

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

• Machine Learning & Deep Learning:

• TensorFlow/Keras (pip install tensorflow) – For deep learning model training

• Scikit-learn (pip install scikit-learn) – For data preprocessing and model evaluation

• Data Handling & Processing:

• NumPy (pip install numpy) – For numerical computations

• Pandas (pip install pandas) – For handling dataset operations

• Feature Extraction & Augmentation:

• SciPy (pip install scipy) – For statistical computations

• Albumentations (pip install albumentations) – For advanced data augmentation

4. IDE/Text Editor

• VS Code, PyCharm, Jupyter Notebook, or Google Colab

5. Optional Tools

• Virtual Environment Manager (venv or conda) for dependency isolation

• GPU Support: NVIDIA CUDA & cuDNN (for TensorFlow GPU acceleration)

3.3 Hardware Requirements

System Requirements Overview

• The system extracts handwriting features (stroke thickness, slant, pressure, etc.) and
combines them with image-based analysis for emotion classification.

• Uses deep learning (MobileNetV2) and traditional handwriting feature analysis for
emotion detection.

• Outputs emotion prediction (Happy, Angry, Calm, Stressed).

Dept of AIML 2024-25 7|Page

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

3.4 Importance

Handwriting-based emotion detection is a cutting-edge application of deep learning and

computer vision that provides valuable insights into an individual's emotional state based on their
handwriting style. Unlike traditional emotion recognition methods, which rely on facial
expressions or speech, handwriting analysis offers a non-intrusive approach that can be used in
various scenarios, such as psychological assessments, educational settings, and forensic
investigations. By leveraging deep learning techniques, this project enables accurate and
automated emotion detection, reducing the need for manual interpretation and increasing
efficiency.

In education, this technology can help teachers and counsellors assess students' emotional well-
being by analysing their handwritten assignments or notes. Early identification of stress or
anxiety through handwriting can facilitate timely intervention, improving mental health support
in academic institutions. Additionally, in forensic and behavioural studies, handwriting-based
emotion detection can assist in analysing suspect behaviour, identifying distress signals, and
enhancing criminal investigations. Such applications demonstrate the project's potential in both
academic and professional fields.

From a technological perspective, this project integrates computer vision, machine learning, and
feature extraction to develop a robust emotion classification model. By combining image-based
handwriting analysis with statistical feature extraction, it improves accuracy compared to
standalone methods. The implementation of this system can pave the way for future research in
emotion recognition, human-computer interaction, and AI-driven mental health diagnostics. Its
significance extends beyond academia, with applications in healthcare, security, and personal
well-being, making it a valuable contribution to the field of artificial intelligence and behavioural
analysis.

Dept of AIML 2024-25 8|Page

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

4.SYSTEM DESIGN

4.1 System Architecture

4.1.1 User Authentication

• Users log in through a secure login interface.

• Authentication ensures authorized access to the application.

4.1.2 Main Menu

• After login, users are presented with a main menu with the following options:

• Emotion Detection: Upload a handwriting image for analysis.

• View History: Access past emotion analysis reports.

• Emotion Insights: Get forensic insights and tips based on detected emotions.

4.1.3 Handwriting Feature Extraction Module

• Users upload a handwriting image for emotion detection.

• The system processes the input using:

• Image preprocessing (grayscale conversion, noise removal, resizing).

• Feature extraction techniques (stroke thickness, slant angle, pressure, spacing, etc.).

• Extracted handwriting features are stored and used for classification.

4.1.4 Emotion Classification Module

• The extracted handwriting features are processed using a deep learning model (CNN
integrated with feature-based classification).

• The system classifies emotions into categories such as:

• Happy

• Calm

• Angry

• Stressed

• The detected emotion is displayed to the user along with forensic insights.

Dept of AIML 2024-25 9|Page

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

4.1.5 Graphical User Interface (GUI)

• Built using HTML, CSS, and JavaScript for a user-friendly web interface.

• Features include:

• File upload option for handwriting images.

• Real-time emotion detection results.

• Forensic insights and emotion handling tips.

• User-friendly navigation for accessing history and insights.

4.2 Use case Diagram

Fig 4.1 Flowchart of model

Dept of AIML 2024-25 10 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

4.3 File Structure and Explanation

project_folder/

├── static/

│ ├── index.html

│ ├── dashboard.html

│ └── emotion/

│ ├── emotion_detection.html

│ ├── history.html

├── app.py

├── model/

│ ├── emotion_model.h5

│ ├── class_indices.json

├── dataset/

│ ├── handwriting_samples

│ ├── augmented_images

└── requirements.txt

1. Static/ Folder:

• Contains all the HTML files used for the front-end of the project.

• The emotion/ sub-folder contains pages for emotion detection and history viewing.

2. HTML Files:

• index.html:

• The login page where users authenticate.

• Redirects to dashboard.html upon successful login.

• dashboard.html:

• Serves as the main menu of the project.

• Provides navigation to emotion_detection.html and history.html.

Dept of AIML 2024-25 11 | P a g e
Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

• emotion/emotion_detection.html:

• Page where users upload handwriting images for emotion detection.

• Displays the detected emotion along with forensic insights.

• emotion/history.html:

• Displays previously analyzed handwriting images with their detected emotions.

3. app.py:

• The Flask application file that serves as the backend for the project.

• Handles routing between index.html, dashboard.html, emotion_detection.html, and

history.html.

• Includes logic for processing uploaded handwriting images, extracting features, and
interacting with the trained deep learning model.

4. model/

• emotion_model.h5: The trained deep learning model for emotion detection.

• class_indices.json: A JSON file storing class labels mapped to emotion categories (e.g.,
Happy, Angry, Calm, Stressed).

5. dataset/

• handwriting_samples/: Stores raw handwriting images collected for training.

• augmented_images/: Contains augmented handwriting images to improve model

generalization.

Dept of AIML 2024-25 12 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

Flow of the System:

1. Login:

• User logs in via index.html.

• On successful authentication, redirected to dashboard.html.

2. Main Menu:

• In dashboard.html, the user chooses between:

• Emotion Detection (emotion_detection.html)

• View History (history.html)

3. Functional Pages:

• emotion_detection.html:

• Accepts handwriting image uploads.

• Processes the image and displays the detected emotion with forensic insights.

• history.html:

• Shows a list of past emotion detection results.

4. Backend:

• app.py processes user inputs from the HTML forms.

• The uploaded handwriting image is preprocessed and analyzed using the deep learning
model (emotion_model.h5).

• The detected emotion is sent back and displayed on the respective page.

Dept of AIML 2024-25 13 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

5.Implementation

5.1 Frontend development

Fig 5.1 Frontend code

Fig5.2 Frontend Code

Dept of AIML 2024-25 14 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

Fig5.3 Frontend Code

Fig 5.4 Frontend Code

Dept of AIML 2024-25 15 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

5.2 Emotion detection model

import os

import random

import cv2

import pandas as pd

import numpy as np

import tensorflow as tf

from tensorflow.keras.layers import Input, Dense, Flatten, Dropout, Concatenate

from tensorflow.keras.models import Model

from tensorflow.keras.preprocessing.image import ImageDataGenerator

from tensorflow.keras.applications import MobileNetV2

from PIL import Image

from sklearn.model_selection import train_test_split

from tensorflow.keras.optimizers import Adam

def extract_handwriting_features(image_path):

if not os.path.exists(image_path):

raise FileNotFoundError(f"Image file not found: {image_path}")

img = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)

if img is None:

raise ValueError(f"Invalid image file: {image_path}")

img = cv2.resize(img, (224, 224))

Dept of AIML 2024-25 16 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

_, binary = cv2.threshold(img, 127, 255, cv2.THRESH_BINARY_INV)

kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3))

dilated = cv2.dilate(binary, kernel, iterations=1)

thickness = np.sum(dilated - binary) / (np.sum(binary) + 1e-5)

edges = cv2.Canny(binary, 50, 150, apertureSize=3)

lines = cv2.HoughLines(edges, 1, np.pi / 180, 200)

angles = []

if lines is not None:

for rho, theta in lines[:, 0]:

angles.append(theta * 180 / np.pi)

slant_angle = np.mean(angles) if angles else 0

contours, _ = cv2.findContours(binary, cv2.RETR_EXTERNAL,

cv2.CHAIN_APPROX_SIMPLE)

bounding_boxes = [cv2.boundingRect(c) for c in contours]

bounding_boxes = [box for box in bounding_boxes if box[2] > 5]

bounding_boxes = sorted(bounding_boxes, key=lambda x: x[0])

if len(bounding_boxes) > 1:

spacing = np.mean([

bounding_boxes[i + 1][0] - (bounding_boxes[i][0] + bounding_boxes[i][2])

for i in range(len(bounding_boxes) - 1)

])

Dept of AIML 2024-25 17 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

else:

spacing = np.nan

char_sizes = [box[3] for box in bounding_boxes]

avg_char_size = np.mean(char_sizes) if char_sizes else 0

pressure_values = img[binary > 0]

pressure = np.mean(pressure_values) if len(pressure_values) > 0 else 0

pressure_variation = np.std(pressure_values) if len(pressure_values) > 0 else 0

stroke_count = len(contours)

stroke_density = stroke_count / (img.shape[0] * img.shape[1] + 1e-5)

baseline_alignment = np.mean([box[1] for box in bounding_boxes]) if bounding_boxes else 0

feature_vector = np.array([

thickness, slant_angle, spacing, avg_char_size,

pressure, stroke_density, pressure_variation, baseline_alignment

])

feature_vector = (feature_vector - np.min(feature_vector)) / (np.max(feature_vector) -

np.min(feature_vector) + 1e-5)

return feature_vector.tolist()

df = pd.read_csv('C:\\Users\\Aditya\\OneDrive\\Documents\\Mini_Project\\Handwriting.csv')

image_paths = df['Image Path'].tolist() if 'Image Path' in df.columns else []

Dept of AIML 2024-25 18 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

emotions = df['Emotion'].tolist() if 'Emotion' in df.columns else []

sentence=df['Sentence'].tolist() if 'Sentence' in df.columns else []

def data_augmentation(image):

angel=random.uniform(-5,5)

h,w=image.shape[:2]

M=cv2.getRotationMatrix2D((w//2,h//2),angel,1)

rotated=cv2.warpAffine(image,M,(w,h))

shearing=random.uniform(-0.1,0.1)

M_shear=np.float32([[1,shearing,0],[0,1,0]])

shear=cv2.warpAffine(rotated,M_shear,(w,h))

scale_factor=random.uniform(0.9,1.1)

scaled=cv2.resize(shear,None,fx=scale_factor,fy=scale_factor,interpolation=cv2.INTER_CUBIC
)

noise=np.random.normal(0,0.5,scaled.shape).astype(np.uint8)

noise=cv2.add(scaled,noise)

alpha=random.uniform(0.8,1)

beta=random.randint(-5,5)

adjusted=cv2.convertScaleAbs(noise,alpha=alpha,beta=beta)

return adjusted

output_folder='C:\\Users\\Aditya\\OneDrive\\Documents\\Mini_Project\\aug_images'
Dept of AIML 2024-25 19 | P a g e
Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

os.makedirs(output_folder,exist_ok=True)

augmented_data=[]

for index,row in df.iterrows():

image_path=row['Image Path']

emotion = row['Emotion']

image=cv2.imread(image_path)

filename=os.path.basename(image_path).split('.')[0]

emotion_folder = os.path.join(output_folder, emotion)

os.makedirs(emotion_folder, exist_ok=True)

for i in range(6):

augmented_image=data_augmentation(image)

output_path=os.path.join(emotion_folder,f"{filename}aug{i}.png")

cv2.imwrite(output_path,augmented_image)

augmented_data.append([output_path,emotion,sentence])

augmented_df=pd.DataFrame(augmented_data,columns=['Image Path','Emotion','Sentence'])

all_image_path=df['Image Path'].tolist()+augmented_df['Image Path'].tolist()

all_emotions=df['Emotion'].tolist()+augmented_df['Emotion'].tolist()

augmented_df=pd.concat([df,augmented_df],ignore_index=True)

features=[]

for image_path in all_image_path:

features.append(extract_handwriting_features(image_path))

features_df = pd.DataFrame(features, columns=[ 'Thickness', 'Slant_Angle', 'Spacing',

'Avg_char_size',

'Pressure', 'Stroke_density', 'Pressure_variation', 'Baseline_alignment'])

Dept of AIML 2024-25 20 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

augmented_df=pd.concat([augmented_df,features_df],axis=1)

augmented_df.to_csv('C:\\Users\\Aditya\\OneDrive\\Documents\\Mini_Project\\Augmented_Han
dwriting.csv')

original_images_folder = 'C:\\Users\\Aditya\\OneDrive\\Documents\\Mini_Project\\Images'

augmented_images_folder =
'C:\\Users\\Aditya\\OneDrive\\Documents\\Mini_Project\\aug_images'

X = np.array(features)

y = pd.get_dummies(augmented_df['Emotion']).values

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

combined_datagen = ImageDataGenerator(rescale=1.0/255, rotation_range=20,

width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2, zoom_range=0.2,
horizontal_flip=True, fill_mode='nearest')

train_datagen = combined_datagen.flow_from_directory(

original_images_folder,

target_size=(224, 224),

batch_size=64,

class_mode='categorical'

val_datagen = ImageDataGenerator(rescale=1.0/255)

val_generator = val_datagen.flow_from_directory(

augmented_images_folder,

target_size=(224, 224),

batch_size=64,

Dept of AIML 2024-25 21 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

class_mode='categorical'

def combined_data_generator(image_generator, feature_data, labels, batch_size=64):

while True:

img_batch, label_batch = next(image_generator)

feature_batch = feature_data[:img_batch.shape[0]]

yield (np.array(img_batch), np.array(feature_batch)), np.array(label_batch)

image_input = Input(shape=(224, 224, 3))

mobilenet = MobileNetV2(weights='imagenet', include_top=False, input_tensor=image_input)

x = Flatten()(mobilenet.output)

x = Dropout(0.5)(x)

image_model = Model(inputs=image_input, outputs=x)

features_input = Input(shape=(X.shape[1],))

y = Dense(64, activation='relu')(features_input)

y = Dropout(0.5)(y)

features_model = Model(inputs=features_input, outputs=y)

combined = Concatenate()([image_model.output, features_model.output])

z = Dense(128, activation='relu')(combined)

z = Dropout(0.5)(z)

z = Dense(64, activation='relu')(z)

output = Dense(y_train.shape[1], activation='softmax')(z)

Dept of AIML 2024-25 22 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

model = Model(inputs=[image_input, features_input], outputs=output)

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

history = model.fit(

combined_data_generator(train_datagen, X_train, y_train, batch_size=64),

epochs=1,

validation_data=combined_data_generator(val_generator, X_val, y_val, batch_size=64)

model.save('C:\\Users\\Aditya\\OneDrive\\Documents\\Mini_Project\\emotion_detection_model.
h5')

import json

class_indices = {idx: emotion for idx, emotion in enumerate(augmented_df['Emotion'].unique())}

with open('C:\\Users\\Aditya\\OneDrive\\Documents\\Mini_Project\\class_indices.json', 'w') as f:

json.dump(class_indices, f)

Dept of AIML 2024-25 23 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

Explanation

1. Importing Libraries

• cv2: OpenCV library for image processing tasks.

• pandas: For reading and manipulating the dataset in CSV format.

• numpy: For numerical operations, e.g., creating and manipulating arrays.

• tensorflow: For building the deep learning model (Keras backend).

• sklearn.model_selection: For splitting the data into training and validation sets.

• PIL: For image manipulation.

• random: For data augmentation (random transformations).

2. Extract Handwriting Features

• extract_handwriting_features(image_path): This function processes a grayscale image

and extracts several features from the handwriting, such as:

• Stroke Thickness: Measures the difference in thickness between the dilated and
original images.

• Slant Angle: Measures the angle of the lines in the handwriting using Hough Line
Transform.

• Spacing: Measures the average space between characters.

• Character Size: Measures the average character height.

• Writing Pressure: Analyzes pixel intensity to estimate pressure.

• Stroke Density: The density of strokes in the image.

• Baseline Alignment: Measures the vertical alignment of the strokes.

• After extracting these features, the function returns a normalized feature vector for each
image.

3. Data Augmentation

• data_augmentation(image): This function applies several random transformations to

images to generate augmented data, including:

Dept of AIML 2024-25 24 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

• Rotation

• Shearing

• Scaling

• Adding noise

• Adjusting brightness and contrast

• These augmented images help in improving the model's generalization ability by training
on a more diverse dataset.

4. Loading and Preprocessing Data

• df: Loads the CSV file containing information about images, their paths, emotion labels,
and associated sentences.

• Augmenting Data: The code loops over the dataset, generates augmented versions of each
image, and stores them in an output folder, updating the dataset accordingly.

• Feature Extraction: For each image, handwriting features are extracted and stored in a
new dataframe (features_df).

• augmented_df: This dataframe now contains both original and augmented images with
associated emotion labels, sentences, and extracted handwriting features.

• The augmented dataset is saved to a new CSV file (Augmented_Handwriting.csv).

5. Preparing Data for Model

• X: A NumPy array containing the extracted features of all images.

• y: One-hot encoded emotion labels (Emotion column from the dataset).

• train_test_split: Splits the data into training and validation sets (80% for training and 20%
for validation).

6. Image Data Augmentation with ImageDataGenerator

• combined_datagen: An image data generator used for augmenting training images by

applying random transformations like rotation, width and height shifts, shear, zoom, etc.

• train_datagen: Applies the combined augmentations to the images in the original folder
(original_images_folder).

• val_generator: This is used for validation images, without additional augmentations, but
rescaling the pixel values.

Dept of AIML 2024-25 25 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

7. Custom Data Generator

• combined_data_generator(image_generator, feature_data, labels, batch_size): This

generator combines both image data (from train_datagen) and the extracted features
(from X_train). It yields batches of image data along with corresponding handwritten
feature vectors and labels.

8. Building the Model

• Image Model:

• The model uses MobileNetV2 as a feature extractor. It takes in an image of size

(224, 224, 3) and processes it through MobileNetV2 (pretrained on ImageNet).

• The last layer of MobileNetV2 is flattened and passed through a Dropout layer for
regularization.

• Feature Model:

• This model takes in the handwriting features and processes them through a Dense
layer (64 units) followed by a Dropout layer.

• Combining Models: The outputs from both the image and feature models are
concatenated and passed through additional dense layers to generate a final
output.

• Final Output Layer: A softmax layer that outputs the predicted emotion for each
input image.

9. Compiling and Training the Model

• The model is compiled with the Adam optimizer and categorical cross-entropy loss (since
it’s a multi-class classification problem).

• The model is trained for 1 epoch using the combined data generator, which feeds both
image data and handwriting features during training.

• The model is saved to disk (emotion_detection_model.h5).

10. Saving Class Indices

• class_indices: Maps emotion labels (e.g., happy, sad, angry) to numerical indices.

• This dictionary is saved to a JSON file (class_indices.json) so that the model can later
map predictions to human-readable emotion labels.

Dept of AIML 2024-25 26 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

Output

Fig 5.5 Emotion detection output

Fig 5.6 Dataset images

Dept of AIML 2024-25 27 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

6.Results

Website view

Fig 6.1 Website view

Fig 6.2 Prediction section

Dept of AIML 2024-25 28 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

Fig 6.3 Features section

Fig 6.4 Contact section

Dept of AIML 2024-25 29 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

9.Applications
1. Psychological & Mental Health Assessment

This system can be used by psychologists and therapists to assess an individual's emotional state
based on handwriting. By detecting stress, anxiety, or depression through handwriting patterns,
mental health professionals can provide early intervention and tailored therapy for individuals
struggling with emotional distress.

2. Education & Student Monitoring

In academic settings, teachers and counsellors can use this technology to monitor students'
emotional well-being through their handwritten assignments. It helps identify students
experiencing stress, anxiety, or disengagement, allowing timely intervention to improve their
learning experience and mental health.

3. Forensic Investigations & Criminal Profiling

Law enforcement agencies can leverage handwriting-based emotion detection to analyze suspect
notes, threat letters, or criminal documents. This can provide insights into a person's emotional
state at the time of writing, helping forensic experts in criminal profiling and psychological
assessments of suspects.

4. Employee Well-Being & Workplace Productivity

Organizations can use this system to assess employees’ emotional well-being through their
handwritten documents, notes, or reports. By identifying workplace stress or burnout early, HR
departments can implement wellness programs to improve employee productivity and mental
health.

5. Personalized Human-Computer Interaction (HCI)

In AI-driven applications, this technology can enhance human-computer interaction by adapting

responses based on a user’s emotional state. For instance, AI-powered writing assistants or e-
learning platforms can modify their tone or suggestions based on the detected emotion in
handwriting.

6. AI-Powered Sentiment Analysis in Market Research

Companies can use handwriting emotion detection to analyse customer feedback or handwritten
surveys. By understanding customer sentiments from physical responses, businesses can refine
their products, services, and marketing strategies based on consumer emotions.

Dept of AIML 2024-25 30 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

Conclusion
The handwriting-based emotion detection system leverages deep learning and computer vision to
classify emotions such as happiness, anger, calmness, and stress by analysing handwriting features
like stroke thickness, slant, pressure, and spacing. This non-intrusive approach has valuable
applications in mental health assessment, education, forensic investigations, workplace well-
being, and human-computer interaction. By automating emotion recognition, the system enhances
accuracy and efficiency, reducing the need for manual interpretation. Its integration with AI-driven
applications opens new possibilities in personalized user experiences, sentiment analysis, and
fraud detection. Future advancements, such as expanding datasets, incorporating multilingual
handwriting, and real-time analysis, can further improve its accuracy and applicability, making it
a powerful tool for behavioural analysis, security, and mental health support.

The future scope of the Emotion Detection System includes expanding the dataset with diverse
handwriting samples for improved accuracy, integrating real-time handwriting analysis using
digital tablets, and leveraging advanced deep learning models like transformers for better feature
extraction. A multimodal approach combining handwriting, facial expression, and voice tone
analysis will enhance emotion recognition. Cloud and mobile deployment will increase
accessibility, while applications in security, such as fraud detection and signature verification,
will broaden its use. Additionally, AI-powered adaptive learning will enable continuous
improvements through user feedback for more precise and personalized emotion detection

Dept of AIML 2024-25 31 | P a g e

Handwriting-Based Emotion Detection Using Deep Learning and Image Processing

9.References
1. Deep Learning for Handwriting-Based Emotion Detection – Research papers and
studies on deep learning applications for handwriting analysis and emotion recognition.

Example: Handwriting Analysis for Emotion Recognition Using CNNs and RNNs (IEEE,
2022).

2. Computer Vision and Feature Extraction Techniques – Studies on stroke analysis,

slant angle, and pressure variation in handwriting for psychological assessments.

Example: Feature Extraction Methods for Handwritten Text Recognition (Springer,

2021).

3. Handwriting and Psychology – Research on the correlation between handwriting

features and emotional states in forensic and mental health studies.

Example: Graphology and Emotional State Detection (Journal of Behavioural Science,

2020).

4. Machine Learning for Sentiment and Emotion Analysis – Research on using AI for
emotion recognition from textual and handwritten data.

Example: AI-Driven Sentiment Analysis for Handwritten Texts (ACM, 2023).

5. Neural Networks and Transformer Models in Handwriting Recognition – Studies on

MobileNet, CNNs, and transformer-based architectures for improving handwriting-based
classification.

Example: Transformer-Based Handwriting Recognition for Multimodal Emotion Analysis

(Nature AI, 2022).

Dept of AIML 2024-25 32 | P a g e