A Surνey οf Μachine Learning Τechniques in Math Ρroblem Solving and

Impact Analysis

Dr. Lakshmi Β N
Assistant Ρrοfessοr
Deρt. Cοmρuter Science and Εngineering
ΒΜS Institute οf Τechnοlοgy and Μanagement
Βangalοre, Κarnataka- 560094
Pranjal Tyagi
Raghvendra Sharma
Computer Science & Engineering
Computer Science & Engineering
BMS Institute Of Technology & Management
BMS Institute Of Technology & Management
Bangalore,India
Bangalore,India

Abstract—This paper presents a dual-model system aimed

at enhancing communication and interactive problem-solving
Communication poses a difficult barrier for the individuals with
for individuals with hearing impairments and those needing
hearing impairments, whether it has been by sign language that
intuitive math support. The first model uses AI to interpret
they have come to depend on most on their efforts to
and solve mathematical expressions input through hand
gestures, while the second model translates sign language into
communicate with people. But because most of the general

text using MediaPipe and Streamlit. The integration of these population does not know sign language, they can not
models creates a unified platform that bridges communication communicate effectively.Also, traditional ways to interact with
gaps and promotes accessible learning through innovative to gather that information are not as simple.icant barriers in
technology.
communication, often relying on sign language as their primary

Keywords — AI, Gesture Recognition, Sign Language mode of interaction. However, the limited understanding of
Detection, MediaPipe, Streamlit, Educational Technology sign language among the general population restricts their
ability to communicate effectively.
This makes hand gestures a less than ideal vehicle for
I. INTRODUCTION mathematical content, and thus underscores the desire for
simpler, more accessible interfaces. In this context, the most
Revolutioning due to the rapid advance in artificial important issues are high accuracy for gesture and sign
intelligence (AI) and machine learning (ML), several fields language detection, real-time processing to have immediate
have come up with new solutions to improve human–computer feedback, and to make the best user interfacing. These
interaction. Assistive solutions for the disabled are one challenges present the need for making effective assistive
important application of these technologies. The potential of technologies that meet the hearing impaired users' needs and
AI- driven systems, particularly those in sign language solution seekers in interactive math support.
recognition and gesture based interactions in bridging the The main purpose of the project is to create two new advanced
communication gap for deaf and hard of hear community has AI models and combine them into a unified platform in order to
been demonstrated. In addition to allowing communication, improve communication and problem solving abilities. We start
these technologies also enable intuitive interfaces for with constructing a gesture based math problem solving model
educational and problem solving applications. that can accurately predict and solve mathematical problems by

Yet, though progress has been made, current solutions suffer using hand gestures.
from failure to achieve accuracy, real time performance, or A second, is we’re going to work on a sign language detection
ease of use. AI models that can interpret sign language and system where using MediaPipe and Streamlit, it will read sign
mathematical gestures held promise to settle these issues and language signs and make it readable in real time! By integrating
deliver better and more accessible assistive technologies. The these models together into a single platform, users can interact
purpose of this project was to create an integrated system built with mathematics while they communicate using sign language.
around gesture based math problem solving and sign language
detection to improve communication and learning for users.
The objective of this project is to make communication easier robust way to perform real time gesture recognition
for the deaf and hard of hearing community, and to do it which is useful for the project of sign language
through an interactive and an optimal way of learning and detection.
solving mathematical problems.
This project contributes in very important ways to the field of [2] Kumar, Thankachan, and Dominic (2023) aim to
AI driven assistive technologies. First, we develop two develop a system to ease communication for those

innovative AI models: It is then applied to two problems one with hearing and vocal disabilities. The system

for gesture based math problem solving, and another for sign recognizes static and dynamic gestures of American

language detection. Second, we make seamless integration Sign Language (ASL), specifically the letters a-z. It

between these models within the same platform possible, with uses skin color segmentation and Viola-Jones face

'ease of transition between functionalities. Thirdly, we ensure detection to focus on hands and face, eliminating

real time processing capabilities for both models so that we facial regions for static gestures. Zernike moments

can get user experience and immediate feedback. Fourth, we are used for static gesture recognition, and curve

create a user friendly interface that accommodates people at feature vectors for dynamic gestures, classified with

different levels of technical expertise, thus making the system SVM classifiers. The Sphinx module maps spoken

more accessible to a wider audience. Finally, we implement alphabet to text, and subsequently to a picture or

the communications and learning accessibility for the hearing video. The system performs in dynamic

impaired as well as for those who need interactive math backgrounds with minimal clutter, contributing to

support, using innovative technology to enhance educational real-time hand detection and gesture recognition.
and communicative accessibility.
[3] M. J, Krishna, S. N. S and S. K use machinelearning
II. RELATED WORK
to address communication barriers for deaf and
dumb individuals.. For acquiring image they utilize
Previous research has shown the potential of using AI in
OpenCV, for training they use CNN and the output
gesture recognition and sign language detection. Studies like
is the text. Unlike other studies of partial sign
[1] have demonstrated the effectiveness of convolutional
language identification, this study focuses on full
neural networks (CNNs) in recognizing hand gestures, while
recognition of American Sign Language (ASL)
[2] have explored the use of MediaPipe for real-time hand
using 26 letters and 10 numbers. Finger and hand
tracking. However, integrating these technologies into a
motions are extracted to differentiate static and
single platform that addresses both mathematical problem-
dynamic gestures. It also emphasizes the necessity
solving and sign language translation remains relatively
of accessible technology, and not depend on
unexplored. This paper builds on existing work by
external sensors.
combining these approaches to create a unified system that
[4] Which, in the paper, explores how this might be
enhances accessibility and learning experiences.
used in video games by using gestures captured
through a user’s video camera to performactions in
[1] Khubchandani and Karthick (2023) build a sign
the game. Hand gestures on Hill Climb Racing and
language recognition system using Convolutional
body gestures in Subway Surfers are used to
Neural Networks (CNNs) on the Microsoft Kinect
investigate. The practicability of this approach is
platform led to GPU acceleration.. They then
indicated through promising real time results using
preprocess the hand gesture data before passing
a webcam, and the approach is shown to be useful
video samples to the neural network to achieve
as a learning and rehabilitation tool using gesture
95% recognition accuracy. Most errors came from
based controls.
poor lighting an image quality. Suggested future
research directions to improve the system efficacy [4] Focuses on Human Computer Interaction and
and reduce errors were presented. This represents a Gesture Recognition through the use of vision
 based, sound wave based and Radio Frequency them into text or speech.
(RF) methods with a focus on vision based.. The  Integration: We integrate the Streamlit interface
paper proposes the use of deep neural network to with the sign language detection model to allow
improve the gesture recognition since this relates for a seamless user experience.
to interference from factors like palm size, skin
color and occlusion, although it is susceptible to
such interference. The study then uses MediaPipe
to detect 21 hand landmarks and get their
coordinates and trains RNN, CNN, hybrid (CNN
and RNN), and Transformer Encoder models.
With a 99.28% accuracy, the optimized RNN
model was achieved. Practical contactless
interaction application is envisaged by this
research.

III. METHODOLOGY

A. System Architecture

The system is composed of two primary components: A

gesture based math problem solving module and a sign
language detection module. The components are united in a Figure 1. Overview of the system
single platform to make Educational and Communicative
Accessibility possible. IV. IMPLEMENTATION

A. Programming Languages and Tools

Gesture-Based Math Problem Solving: Languages Used
 Objective: This interprets and resolves hand  Python: For its large libraries and frameworks to
gestures input of mathematical expressions through support machine learning, image processing, and
the use of AI. web development, the application development is
 Architecture: We train a convolutional neural done through the main programming language.
network (CNN) to learn to recognize hand gestures Libraries and Frameworks
for mathematical symbols. Then, the symbols are  MediaPipe: For real-time hand tracking and gesture
recognized so that the math problems associated recognition that is necessary for sign language
with the given image are solved. detection and gesture based math problem solving.
 Integration: Streamlit interface has been developed  Streamlit: It offers a convenient way to allow users to
using Streamlit, while integrating it with the gesture interact with the system through a web interface
recognition model to give real time feedback and creating ease of accessibility and instant feedback.
input.  OpenCV: It’s a library for performing image
Sign Language Detection: processing tasks like capturing image frames from
 Objective: Translate sign language into readable the webcam and doing various forms of image
text or speech by using MediaPipe and transformations.
Streamlit.  NumPy: It used to handle data of numerical
 Architecture: Hand Landmark detection is operations and image data as array, for processing
performed by hand tracking module in the and manipulating image data.
MediaPipe. We process these landmarks to find  Pillow: PIL: a Python Imaging Library used to
particular sign language symbols and interpret manipulate and convert images for preprocessing of
dataset.
 B. Algorithms language recognition.

Convolutional Neural Network (CNN): They are to be able  Python: Is the main language when we are

to recognise hand gestures and interpret mathematical implementing the algorithms and also integrating

symbols. Image recognition tasks are performed in a CNN other components.

model. A dataset of hand gesture images is used to train the AI Models:

model to classily differentiate between different hand  Convolutional Neural Networks (CNNs): Used for

gestures. PatchNet for recognition is multiple layers, processing images, e.g. feature extraction from hand

convolutional layers, pooling and fully connected layers, to gestures, sign language and facial images.

extract features and predict on input images.  Long Short-Term Memory (LSTM) Networks: Used
for sequence prediction, a critical task for reading
C. Hand Detection Landmark
continuous sign language gestures in time.
In order to improve hand detection and gesture recognition
Experimental Procedures:
accuracy as well as decrease false positives. It comes down
 Gesture Recognition: Users make predefined
to processing hand landmarks detected by MediaPipe with
mathematical gestures in front of the webcam. We
advanced techniques. Using machine learning models
capture these gestures, these process them with the
trained on gesture data, the algorithm filters and refines
CNN and the LSTM models, and provide the
these landmarks to identify correctly gestures.
equation that gives the mathematical solution.

Figure 3. 21 landmarks detection

D. Experimental Setup

For this study, we used advanced AI technologies to develop

 Sign Language Detection: The system has the
an experimental setup that integrates sign language detection
ability for the user to perform a sign language
and math problem solving systems. The key components of
gesture and these gestures are captured and
interpreted to show the appropriate text or speech
output.

V. RESULTS

A. Results for Math Problem Solving

Our gesture based math problem solving system was

quite impressive, yielding very high reliability and accuracies
the setup include:
solving complex mathematical expressions typed through
Hardware Configuration:
hand gestures. But it interpreted and solved problems of all
 Webcam: It is used to capture real time video input
kinds including combination problems,
of hand gestures and Sign Language.
minimal errors for multi step ones like ((3+5)*2). Most of
 Computer: Running AI models fast on the video
all, you could perform the calculation and provide real time
data with a high performance GPU.
feedback in the same system.
Software and Tools:
 Streamlit: For generating the user interface to under a variety of environmental conditions. It kept high
facilitate the system’s interactivity by the user. accuracy across hands of different shapes and sizes using a
 MediaPipe: Used for real time tracking of hand, Convolutional Neural Network (CNN). The user interface
face and body for accurate gesture and sign based on Streamlit can be responsive and user friendly. The
 system was successful both educationally, as it provided an solution improves educational accessibility. The system
engaging means of teaching mathematics to students, and it received excellent user reaction: people seem to like the
was a high satisfaction system that users enjoyed using. system, mainly for being easy to use, for having an intuitive
Additionally, it had the potential to be used as assistive design and being stable. The Gesture Learn platform is an
technology, to enable individuals with disabilities to engage invaluable educational tool through interactive learning and
with mathematical content in a new way. Future work can communication.
expand the range of expressions that it can handle, as well
as increase robustness. The system's success overall I. FUTUTRE SCOPE
exemplifies the potential for AI-based applications to
There are many ways for the future improvements of the
revolutionize the entire mathematical education experience
Gesture Learn system. As we expand the gesture library and
and promote easier accessibility to individuals from less
enhance real time processing, accuracy increases and latency
privileged backgrounds.
decreases. Multilingual support can get extended and with
B. Results for Sign Language Detection the connections to learning management systems
accessibility and the implementation of the curriculum will
also improve. Key is collaborating with educational
institutions to get input from the real world and then
longitudinal studies to measure long term impact. In addition,
Gesture Learn not only can support other disciplines besides
mathematics, but this will help Gesture Learn to act as a
more diverse educational tool. In a system based on
inclusiveness with a focus on an improved educational
environment, these innovations seek to increase system
The model successfully showed sign language gesture
capabilities.
to be performed using MediaPipe for hand tracking as well
as gesture recognition. This was all a system which even II ACKNOWLEDGEMENT

translated sign language into readable text, making it very We would also like to thank the CSE department of BMS
easy to communicate for the deaf and mute community. Institute of Technology for the facilities and support
Hand tracking module of MediaPipe captured robust hand provided to us to carry out this work.
landmarks which helped the system decode a variety of
REFERENCES
sign language gestures. Evaluation of the system included a
series of tests on users practicing different sign language
[1] H. Khubchandani and K. T, "Sign Language Recognition," 2023 9th
gestures. The system translated gestures into text without
International Conference on Information Technology Trends (ITT),
error, quite reliably. Real time interaction was enabled
Dubai, United Arab Emirates, 2023, pp. 199-203, doi:
through integration with a user friendly interface giving 10.1109/ITT59889.2023.10184252.
immediate visual feedback for users as they gestured. In
communication and educational contexts, particularly, [2] A. Kumar, K. Thankachan and M. M. Dominic, "Sign language

having such a feature was very suitable. Finally, we recognition," 2016 3rd International Conference on Recent Advances
in Information Technology (RAIT), Dhanbad, India, 2016, pp. 422-
ensured that the sign language detection system is a useful
428, doi: 10.1109/RAIT.2016.7507939.
tool for increasing communication accessibility and
inclusivity for the deaf and mute community.
[3] M. J, B. V. Krishna, S. N. S and S. K, "Sign Language Recognition
VI. CONCLUSION using Machine Learning," 2022 International Conference on
With the Gesture Learn system, previously unheard of promise Innovative Computing, Intelligent Communication and Smart
Electrical Systems (ICSES), Chennai, India, 2022, pp. 1-5, doi:
has been demonstrated in the capacity of math instruction and
10.1109/ICSES55317.2022.9914155.
communication for the deaf and mute community. By using AI
driven gesture recognition, sign language detection, the
[4] U. Patel, S. Rupani, V. Saini and X. Tan, "Gesture Recognition
Using MediaPipe for Online Realtime Gameplay," 2022
IEEE/WIC/ACM International Joint Conference on Web
Intelligence and Intelligent Agent Technology (WI-IAT), Niagara
Falls, ON, Canada, 2022, pp. 223-229, doi: 10.1109/WI-
IAT55865.2022.00039.

[5] J. Zhao, X. H. Li, J. C. D. Cruz, M. S. Verdadero, J. C. Centeno and J.

M. Novelero, "Hand Gesture Recognition Based on Deep
Learning," 2023 International Conference on Digital Applications,
Transformation & Economy (ICDATE), Miri, Sarawak, Malaysia,
2023, pp. 250-254, doi: 10.1109/ICDATE58146.2023.10248500.
.