1

Problem Definition
The lack of efficient and accessible tools for real-time sign language
recognition poses a significant communication barrier for individuals with
hearing impairments, hindering their ability to interact seamlessly in
various social and professional settings. Current solutions often lack
accuracy, limited in language support, and are not easily deployable for
widespread use.
To address this issue, this project aims to develop a comprehensive real-
time sign language recognition system capable of accurately interpreting
simple signs, including greetings, in Marathi, Hindi, and English. The
system will leverage advanced deep learning techniques to achieve robust
recognition performance across multiple languages and will be designed
for easy integration into daily communication scenarios.
 3
Literature Survey

S.No. Authors Full title of the Inference from the paper Open Problem (For
name(s) paper with year proposed work)

1 • A. S "Action Detection It acknowledges the Developing robust and

Sushmitha for Sign Language challenges of efficient automated
Urs, Using Machine misinterpretation and systems capable of
• V. B Raj, P. Learning," 2023 language variations. It accurately recognizing
S, proposes a solution and interpreting
• P. Kumar utilizing automation for sign diverse sign signals
• K, M. B R V. recognition, enabling users from images or video
Kumar S to capture hand signals inputs remains an
through a camera and open problem in the
receive accurate field.
identification of the signs,
thereby enhancing
accessibility and inclusivity.
 4
Literature Survey
S.No. Authors Full title of the Inference from the paper Open Problem (For
name(s) paper with year proposed work)
2 • K. S. "An approach to By employing real-time Despite
Vikash, Generation of video processing and the advancements in sign
• K. sentences using SSD MobileNet V2 model language recognition
Jayakrishna Sign Language for sign language using computer vision
n, Detection," 2023 detection, coupled with and machine learning,
• S. TensorFlow Object achieving seamless
Ramanatha Detection API, the system real-time
n, demonstrates high communication
• G. Rohith accuracy and efficiency in between sign
• V. recognizing and translating language users and
Hanumara, sign language into text in non-users remains a
real-time scenarios. This challenge. Current
advancement holds systems may lack the
significant potential in ability to accurately
facilitating smoother interpret nuances in
interactions between the sign language, leading
signing and non-signing to potential
communities. miscommunications.
 5
Literature Survey

S.No. Authors Full title of the Inference from the paper Open Problem (For
name(s) paper with year proposed work)
3 • M. Zhang, "Deep Learning- The proposed system However, recent
• S. Yang Based Standard demonstrates superior advancements in hand
• M. Zhao, Sign Language performance compared to gesture recognition
Discrimination,“ existing methods when systems, effectively
2023 evaluated on a challenging addressing the
dataset, indicating its complexities of hand
effectiveness in dynamic segmentation, local
hand gesture recognition. feature representation,
global body
configuration
representation, and
gesture sequence
modeling remains a
challenge.
 6
Literature Survey
S.No. Authors Full title of the Inference from the paper Open Problem (For
name(s) paper with proposed work)
year
4 • M. "Exploring The proposed deep learning Even though, there are deep
Boonda LSTM and CNN architecture, combining CNN learning frameworks,
mnoen, Architectures for feature extraction and particularly utilizing
• K. for Sign LSTM for spatio-temporal convolutional neural
Thongsri, Language information capture, networks (CNN) and long
• T. Translation," demonstrates promising short term memory (LSTM),
Sahaban 2023 results for sign language to accurately recognize and
toegnsin recognition, particularly in the interpret sign language
• K. context of the Indian Sign gestures continues to be an
Woraratp Language (ISL) dataset. The open research challenge.
anya, scarcity of research utilizing
deep learning models to
effectively capture temporal
information underscores the
significance of this study in
addressing a pertinent open
problem in the field of sign
language recognition.
 7
Literature Survey
S.No. Authors Full title of the Inference from the paper Open Problem (For
name(s) paper with proposed work)
year

5 • B. G. J. "Sinhala Sign The proposed work aims to This gap presents an open
Gamage, Language address the gap in existing problem that motivates the
• R. P. S. Translation literature by developing a current research. The
D. through comprehensive system research aims to integrate
Paranag Immersive 3D specifically tailored for SSL various aspects, including
ama, Avatars and users in Sri Lanka. This learning methodologies,
• R. M. S. Adaptive system will integrate various dynamic sign detection,
H. Learning" 2023 components such as learning audio/video to sign
Ranawee modules, dynamic sign conversion, and vocal
ra, detection, audio/video to sign training, into a unified
• A. V. R. conversion, and vocal platform. Such a platform
Dilshan training into a unified would serve to facilitate
platform. The goal is to effective communication and
facilitate effective language development
communication and language within the SSL community in
development for the SSL Sri Lanka.
community.
 8
Literature Survey
S.No. Authors Full title of the Inference from the paper Open Problem (For
name(s) paper with year proposed work)

6 • S. Patil, Conversion of The paper addresses the The system's effectiveness is

S. Indian Sign communication challenges underscored by its impressive
Gulave, V. Language to faced by individuals with accuracy of up to 98%,
Gawai, P. Speech by Using hearing and speaking demonstrating the potential of
Gode and Deep Neural disabilities by proposing a neural networks in bridging
P. Network system to convert Indian Sign communication gaps for
Mudme, Language (ISL) into speech individuals reliant on sign
Year- 2022 through deep neural networks. language.
Acknowledging the widespread
JOURNAL difficulty in interpreting sign
language among the general
population, the study
emphasizes the necessity of
eliminating the reliance on
interpreters for communication
with mute and deaf individuals.
on steps facilitate the
interpretation of hand signs,
with speech synthesis
achieved through the Google
text-to-speech API.
 9
Literature Survey
S.No. Authors Full title of the Inference from the paper Open Problem (For
name(s) paper with proposed work)
year

7 • Z. Deep Learning- The study addresses the Training datasets, sourced

Mustaffa, Based challenges faced by from Kaggle and personal
N. A. Technique for individuals in learning Malay camera captures, are
Farihin Sign Language Sign Language (MSL) by meticulously divided for
Mohd Detection presenting an object robust training, validation,
Zulkifli, detection model tailored to and testing phases. The
M. H. Year- 2023 detect MSL signs in real study's successful
Sulaiman time. Highlighting the development of a real-time
, F. JOURNAL significance of sign and accurate MSL
Ernawan languages for the deaf and recognition system using the
and Y. A. hard-of-hearing community, SSD-Mobilenet model holds
Adam, the research focuses on promising implications for
developing a system capable enhancing communication
of recognizing static signs, access for individuals within
simplifying the learning the deaf and hard-of-hearing
process. Leveraging the community.
Single Shot Detector (SSD)
and Mobilenet architectures,
the model achieves real-time
detection accuracy.
 10
Literature Survey
S.No. Authors Full title of the Inference from the paper Open Problem (For
name(s) paper with proposed work)
year

8 B. A. Arabic Sign The paper addresses the Employing logistic

Dabwan and Language communication challenges regression for binary
M. E. Detection Using faced by the global classification and a 2-layer
Jadhav Deep Learning population of individuals who Artificial Neural Network
are deaf and mute, (ANN), the model aims to
Year -2023 numbering approximately accurately predict the
466 million. With meaning behind the
JOURNAL communication being a detected hand signs.
fundamental aspect of
human interaction, the study
highlights the importance of
providing accessible
communication methods for
this demographic.
Leveraging hand gestures
and symbols as their primary
means of communication, the
research proposes a model
for detecting and interpreting
these gestures using a Deep
Neural Network (DNN).
 11
Literature Survey
S.No. Authors Full title of the Inference from the paper Open Problem (For proposed
name(s) paper with year work)

9 • N. Hand sign The paper addresses the ResNet-34 is a convolutional

Titarmare language communication challenges neural network (CNN)
detection faced by the global population architecture that belongs to the
of individuals who are deaf and ResNet (Residual Network)
Year- 2022 mute, numbering approximately family, introduced by Microsoft
466million.. By utilizing Research in 2015. ResNet-34
JOURNAL advanced machine learning is known for its depth, featuring
techniques, the proposed 34 layers, hence the name. It
system endeavors to bridge builds upon the concept of
communication gaps and residual learning, which
empower individuals who rely enables the training of very
on hand sign language for deep networks without
communication. encountering the vanishing
gradient problem.
 12
Literature Survey
S.No. Authors Full title of the Inference from the paper Open Problem (For
name(s) paper with proposed work)
year

10 Abhishek Real Time Sign The application predicts Through transfer learning, it
Wahane Language these sentences and words attains an accuracy of
Recognition in real time, with an average 89.91% with Google's
using Deep accuracy of 75% each word. Inception v3. The system's
Learning A dual-model system that effectiveness and accuracy
Techniques operates a machine learning in recognizing and
model using the user's 2D- interpreting ASL sentences
Year -2020 Pose coordinates at runtime and gestures are ensured by
and recognizes gestures the fact that the application's
JOURNAL using the Single Shot machine learning and deep
Multibox Detector (SSD).In learning models are all
addition, the program may trained on a special dataset
generate custom phrases by created specifically for this
real-time ASL alphabet purpose.
recognition.
 13

Justification for the Proposed Problem

➢ The proposed problem addresses the need for inclusive communication
by developing a real-time sign language recognition system capable of
interpreting simple signs in multiple languages. This project aims to
bridge the communication gap for individuals who rely on sign
language by providing a robust and accessible tool that recognizes and
translates signs into text and audio in Marathi, Hindi, English, Telugu,
and Kannada. By leveraging deep learning techniques and real-time
integration, this system offers a practical solution to facilitate seamless
communication for the deaf and hard of hearing community.
 14

Software/Tools Requirements
Software requirements:-

• Python 3

Hardware requirements:-

• Computer (CORE - iUU78IU766TY7U8I5)

 15

Architecture / Design
 MODELS
• CNN - Convolutional Neural Network

• CNNs are a class of deep neural networks commonly used

for analyzing visual imagery.
• They consist of multiple layers, including convolutional
layers, pooling layers, and fully connected layers.
• CNNs have achieved state-of-the-art performance in various
computer vision tasks, including image classification, object
detection, and image segmentation.
 MODELS
• EfficientNet:

• EfficientNet is a family of convolutional neural network

architectures that aim to achieve better accuracy and
efficiency by scaling the network width, depth, and resolution
in a balanced way.
• It was introduced by Tan et al. in the paper "EfficientNet:
Rethinking Model Scaling for Convolutional Neural
Networks" in 2019.
• EfficientNet models are known for their superior performance
on various image classification tasks while being
computationally efficient compared to other architectures.
 MODELS
• ResNet50

• ResNet50 is a variant of the ResNet (Residual Network)

architecture introduced by Microsoft Research in the paper
"Deep Residual Learning for Image Recognition" in 2015.
• ResNet50 specifically refers to a ResNet architecture with 50
layers, consisting of residual blocks that help mitigate the
vanishing gradient problem during training.
• It has been widely used for various computer vision tasks,
including image classification, object detection, and image
segmentation.
 MODELS
• InceptionV3

• InceptionV3 is a convolutional neural network architecture

introduced by Google researchers in the paper "Rethinking the
Inception Architecture for Computer Vision" in 2016.
• It is an evolution of the original Inception architecture, designed
to improve computational efficiency and performance.
• InceptionV3 incorporates various modules such as convolutional
layers with different kernel sizes (including 1x1, 3x3, and 5x5),
max-pooling layers, and parallel concatenated operations.
• It has been widely used for image classification, object
detection, and image segmentation tasks, particularly in
scenarios where computational resources are limited.
 20
Implementation

IMAGE
 21

Implementation
REAL-TIME
 22

Timeline
Abstract • <00/02/2024>
Submission

Literature • <20/02/2024>
Survey

Design • <10-03-2024>

Implementation • <00-03-2024>

Testing • <00-04-2024>

Document • <Date of
Submission Completion>
 23

References
[1] A. S Sushmitha Urs, V. B Raj, P. S, P. Kumar K, M. B R and V. Kumar S, "Action
Detection for Sign Language Using Machine Learning," 2023 International
Conference on Network, Multimedia and Information Technology (NMITCON),
Bengaluru, India, 2023, pp. 1-6, doi: 10.1109/NMITCON58196.2023.10275950.
[2] K. S. Vikash, K. Jayakrishnan, S. Ramanathan, G. Rohith and V. Hanumara, "An
approach to Generation of sentences using Sign Language Detection," 2023
International Conference on Signal Processing, Computation, Electronics, Power and
Telecommunication (IConSCEPT), Karaikal, India, 2023, pp. 1-6, doi:
10.1109/IConSCEPT57958.2023.10170218.
[3] Y. Zhang, L. Long, D. Shi, H. He and X. Liu, "Research and Improvement of
Chinese Sign Language Detection Algorithm Based on YOLOv5s," 2022 2nd
International Conference on Networking, Communications and Information
Technology (NetCIT), Manchester, United Kingdom, 2022, pp. 577-581, doi:
10.1109/NetCIT57419.2022.00137.
 24

References
[4] M. Zhang, S. Yang and M. Zhao, "Deep Learning-Based Standard Sign Language
Discrimination," in IEEE Access, vol. 11, pp. 125822-125834, 2023, doi:
10.1109/ACCESS.2023.3330863.
[5] M. Boondamnoen, K. Thongsri, T. Sahabantoegnsin and K. Woraratpanya,
"Exploring LSTM and CNN Architectures for Sign Language Translation," 2023 15th
International Conference on Information Technology and Electrical Engineering
(ICITEE), Chiang Mai, Thailand, 2023, pp. 198-203, doi:
10.1109/ICITEE59582.2023.10317660.
[6] K. Aryasa and A. Rusydi, "Design and Build a Sign Language Detection
Application with Tensorflow Object Detection and SSD Mobilenet V2," 2023 5th
International Conference on Cybernetics and Intelligent System (ICORIS),
Pangkalpinang, Indonesia, 2023, pp. 1-5, doi: 10.1109/ICORIS60118.2023.10352247.
 U !
Y O
N K
H A
T