Volume 8, Issue 3, March – 2023 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Deep Learning for Sign Language Recognition

Bhagyashri Pagar Rutuja Shelar
Department of Information Technology, Sinhgad Department of Information Technology, Sinhgad
College of Engineering Pune, Maharashtra, India College of Engineering Pune, Maharashtra, India

Soumya Sheelavant Avinash A. Utikar

Department of Information Technology, Sinhgad Prof., Department of Information Technology,
College of Engineering Sinhgad College of Engineering
Pune, Maharashtra, India Pune, Maharashtra, India

Abstract:- A sign language is a way of communicating by Due to lack of ease in sign languages, non-sign language
using the hands instead of spoken words. Sign language is interpreter speaks or pays less attention to communicate with
used by deaf and dump people to communicate with other dump and deaf people. Most of people know nothing about
individuals. People who are speech-impaired and also sign language except for people engaged in special education
some people who have autism spectrum disorder face and most people will not take initiative to spend time and
problem while communicating with normal as they can learn sign language. The Indian sign language (ISL) is one of
converse using only sign language. So, it becomes difficult the popular languages among Sign languages. It uses gestures
for other individuals to understand this sign language. for communication. Gestures are various movements used in
Each country usually has its own native sign language. the process of communication either hand or body.
The Indian Sign Language recognition application
proposed here aims at solving the communication Sign languages uses gestures which usually make use of
problem between people. System will capture the different visually transmitted patterns. ISL presents various hand
hand gestures through camera of mobile phone. The signs movements using both right and left hands. This ISL gestures
used in sign language are identified by the features can be classified into two categories static and dynamic. In
extracted from the hand gestures. Then, processing of the static gesture there is no movement of hand. Most of gestures
image takes place by using convolution neural network of alphabets in ISL are static signs. Dynamic gestures involve
algorithm. After all the processing we finally get the hand movements during performing gestures. The gesture of
output as a text which can be easily understood by all hand recognition includes hand detection and recognition of
people. We are keeping purpose of developing system that that hand sign during processing. People find it very difficult
will make communication between Deaf and Dumb to understand sign language hence it become important to
person and normal person easy and convenient. design vision-based sing language translator. Many
researchers try to combine recent models for deaf community
Keywords:- Sign Language Recognition Application, to communicate more easily using AI in real world scenario
Convolution Neural Network, Deep Learning. which includes machine learning models and some dataset
collection. There are two major methods for sign language
I. INTRODUCTION translation. 1)Vision based 2) Glove based which includes
sensors and gloves for implementation. Glove-based hand
The history of sign language in Western societies dates gesture recognition system has introduced as a beginning, due
back to the 17th century, as a visual language although to wearing heavy devices and cable connections, it lacks
references to the forms of communication using sign naturalness. This research proposes a vision-based sign
language date back to the 5th century BC Greece. Sign language device capable of translating ISL to text. It includes
language is made up of a system of general gestures, to basic models like object detection, feature extraction and
represent the letters of the alphabet and emotions. Many sign classification. The training model built for classifying hand
languages are native languages, distinct from the structure of gestures using ML. The hand gestures are captured
spoken languages used near them, and are mainly used by dynamically using camera.
deaf people to speak. Hearing is the important sense among 5
human senses. Deafness hinders a person from understanding A Convolutional Neural Network (CNN) is a Deep
spoken languages Unlike spoken languages, where grammar Learning algorithm which is applied to Image and Video
is expressed using punctuation-based symbols, feature, recognition, Image Analysis and Classification etc. We use
attitude and syntax sign languages use gesture, punctuation, Convolutional Neural Network to increase the accuracy of
and body and facial expressions to form grammar. Sign sign classification. Therefore, this system balances the
language is for community that helps to interact with each communication between deaf people and normal people
other. They are used for conveying meaning through visual without any requirement of intermediate translator by
modality. converting gestures into text, which achieves our objective
and reduces the barrier in communication. The aim of this
paper is to recognize signs used in Indian Sign Language. It

IJISRT23MAR066 www.ijisrt.com 278

Volume 8, Issue 3, March – 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
is well studied subject in American Sign Language, but it has Hand gesture recognition framework was proposed for
received little attention in Indian Sign Language. We want to Bangla sign language to minimize the communication gap
solve this issue, but instead of using high-end technologies between deaf and dumb people. Hand detection and gesture
like gloves, we want to recognize gestures from photographs recognition from the detected hand is targeted output in this
and then use computer vision and ML techniques to extract study. To gain better accuracy, they have created a dataset of
specific features for classification. 3219 images. The development of this dataset will reduce the
limitation of a prospective study in this field. Hence, they
II. LITERATURE SURVEY worked with this dataset and emphasizing implementing a
more realistic real-time sign language recognition technique
All people communicate with each other through the [10]. A vision-based American sign language translator that
spoken language but, deaf and dumb people face problems incorporates three concatenated modules- SSD, CNN and
while communicating with other people. They use sign FCN was proposed. The translator is trained using cross
language to communicate with normal people. As there are validation technique. Several experiments were performed
many people who are not familiar with sign language and but the results of the experiment with the best setting were
unable to interpret the meaning of various signs used by deaf given in that paper. Experiments also showed that the sign
and dumb people. Also, unlike spoken languages sign labelling algorithm was capable of automatically detecting
languages users are limited and sign languages have their own and differentiating the different signs within a short period of
vocabulary and grammar, different than the corresponding time with very high accuracy [11].
spoken language. Deaf and Dumb people dependent on sign
language interpreters to communicate with others. However, In all these different methods they have developed
for all daily life work finding qualified interpreters is kind of model for converting sign language into text or they have
difficult task and also it is not affordable for those people. classify signs according to sign language used in their
Hence, there is need of some kind of interface which help to country. Although, these models provide better accuracy but,
ease the communication between deaf and dumb people and every time using laptop for sign language recognition is tough
normal people. So, various researchers used different in day-to-day life. Therefore, we need some application
approaches to tackle this problem. There are mainly two which can be implemented on the handheld device like
approaches for sign language recognition. One is vision based mobile phones. Nowadays, everyone has their own mobile
where different signs are converted into text or speech using phone that they can easily carry everywhere. So, we have
techniques like image processing and computer vision proposed an application which we can easily install in our
techniques. In this we compare different features of capture phones and through that we can easily recognize different
image with database of signs used in sign language. Light signs used in sign language which will ultimately help to
intensity and image resolution affects the accuracy of vision- bridge communication gap between deaf and dumb people
based system and dynamic gesture recognition is complex in with other individuals.
this type of approach. In second approach we used sensors
which can be placed directly on the hand and then we use III. PROPOSED SYSTEM
sensor values to classify gestures. This approach provides
better accuracy but, wearing heavy devices and cable According to the reference of available studies there are
connection lacks naturalness of sign language recognition. different models for sign language recognition Although,
Also, those sensors are very expensive and difficult to use in these models provide better accuracy but, every time using
day-to-day life. laptop for sign language recognition is tough in day-to-day
life. Therefore, we need some application which can be
Template matching algorithm which converts sign implemented on the handheld device like mobile phones.
language into text output. In this they used canny edge Nowadays, everyone has their own mobile phone that they
detection algorithm which removes the data which is not can easily carry everywhere. So, we have proposed an
useful and gives sign alphabet as output. So, we can extend application which we can easily install in our phones and
this implementation to mobile phones [1]. Deep CNN through that we can easily recognize different signs used in
architecture to discover and categorize signs used in sign sign language which will ultimately help to bridge
language from hand gesture images was used in another communication gap between deaf and dumb people with
model. However, the real time detection and classification of other individuals.
hand gestures has not been implemented in this paper.
Therefore, we should be directed towards development of An Artificial Neural Networks (ANNs) are the neural
intelligent methodologies for real time sign language networks which mimic the computational structure and
detection. In this regard, region-based CNN can be working of biological neurons. ANN is a computing system
implemented for more convenient detection of sign languages made up of number of simple, highly interconnected
[2]. Convolutional Neural Network (CNN) to classify processing elements, which process information by their
Myanmar Sign Language image datasets, discusses skin dynamic state response to external input. Neurons are
colour enhancement method for skin detection and Viola interconnected forming the layers, when there are more than
Jones algorithm for face detection was used in other paper. 3 layers of neurons it is called a Deep Neural Network. This
But Viola Jones algorithm can only detect frontal image. So neural network is trained over a dataset to form a model that
if we detect with unconstrained face, the system will be detects Indian Sign Language (ISL) hand gestures. ISL is a
difficult to detect the face and will not be able to work [3]. language of deaf community in India. ISL contains many

IJISRT23MAR066 www.ijisrt.com 279

Volume 8, Issue 3, March – 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
static gestures, in our main dataset we have focused on dataset gallery. Once the image is provided or captured the image is
of common ISL phrases. sent for pre-processing, the trained model detects the sign in
the image and gives corresponding output in the text format.

Fig 2. Phases of sign Recognition

In the phases of sign language recognition first, we

capture the input image then, we pre-process that input image
and perform segmentation to extract important feature from
Fig 1. System Architecture the image. After feature extraction we classify that image
according to the signs used in sign language and show the
System Architecture is shown in figure 1. The entire output as the text which can easily interpret the meaning of
system works in stages. particular sign that we have used as the input.

We have collected real-world samples for recognition  Pre-Processing:

system. The list of observed ISL were gathered via Kaggle Captured image is prepossessed, pre-processing is a step
website. Kaggle is a website which provides various datasets which do not increase image information, its aim is to
of various types. Our dataset contains both high and low improve the image data and supress undesired distortions or
resolution images. The dataset is divided into training and enhances some image features. We will be using the image
testing dataset. The dataset is trained using CNN algorithm. segmentation technique, segmentation partitions an image
It is important for the model to extract only valuable features into multiple parts based on pixels. Important features are
while ignoring the other objects or unimportant features in the grouped and data is sent to feature extraction.
background, such as person’s clothes, skin color and
shadows.  Feature Extraction:
The pre-processed data is taken as input. The important
 Convolution Neural Network: features are extracted by transforming raw data into
Convolution Neural Network (CNN) is a neural network numerical features that can be processed. The extracted
which is used in image processing. The model recognizes features are in numerical form called as feature vector.
what is in an image or what an image is. CNN has two layers
that are Convolution layer and Pooling layer.  Classification:
The feature vector is then sent for classification. Image
A. Convolution classification is the process of taking an input and outputting
Convolution layer maps or creates a feature map from a its class or probability that input is of a particular class. The
dataset. Feature map uses pixels, a filter of stride (n*n) is classification is done according to the class. The classified
predefined, which moves across the images and extracts the object is our output. The output is displayed into text format.
pixels which contains a value or feature.
IV. CONCLUSION
B. Pooling
Pooling layers down samples the output of previous Deaf and dumb people communicate with other using
layers by sliding the filter of stride size and calculates the sign language, however it is difficult for normal people to
maximum or average of the input. understand them. So, there is a need of a system which
recognizes the different signs, gestures and conveys the
The model is trained using CNN algorithm, this model information to the normal people. Hence, we are proposing
is saved in TensorFlow Lite (tflite) API. tflite model can be the idea of sign language recognition application which uses
accessed using Flutter, Flutter is a plugin for accessing convolution neural networks to recognize different signs used
TensorFlow Lite API in Android or iOS. After importing the in sign language. This application will be develop for
trained model in mobile application, the main function of individuals who cannot use gloves, sensors and other
detection uses camera. When a user wants to detect a sign, equipment for communication. In this proposal we have
he/she will start the application. Detection can be done in two designed an idea of application which will convert sign
ways firstly, using the camera we capture the image and language into text. We want to bridge the communication gap
second way, we can use a pre-existing photograph from between physically challenged people and normal people, but

IJISRT23MAR066 www.ijisrt.com 280

Volume 8, Issue 3, March – 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
instead of using high-end technologies like glove, we want to [11]. Rahib Abiyev, Rahib Abiyev, Murat Arslan:
recognize gestures from photographs (which can be accessed Reconstruction of Convolutional Neural Network for
from a webcam), and then use computer vision and machine Sign Language Recognition. Proc. of the 2nd
learning techniques to extract specific features and classify International Conference on Electrical, Communication
them and produce output as a text. and Computer Engineering (ICECCE) 12-13 June 2020,
Istanbul, Turkey
REFERENCES [12]. Rachana Patil1, Vivek Patil1,, Abhishek Bahuguna1,
Mr. Gaurav Datkhile: Indian Sign Language
[1]. Soma Shrenika, Myneni Madhu Bala: Sign Language Recognition using Convolutional Neural Network. ITM
Recognition Using Template Matching Web of Conferences 40, 03004 (2021) ICACC-2021
TechniqueUniversity College London July 05,2020 [13]. Ying Xie, Kshitij Bantupalli: American Sign Language
from IEEE Xplore. Recognition using Deep Learning and Computer Vision
[2]. Rajarshi Bhadra, Subhajit Kar: Sign Language Department of Computer Science Kennesaw State
Detection from Hand Gesture Images using Deep Multi- University Kennesaw, USA IEEE International
layered Convolution Neural Network. 2021 IEEE Conference on Big Data (Big Data) 2018
Second International Conference on Control,
Measurement and Instrumentation (CMI), India
[3]. Sai Myo Htet, Bawin Aye, Myo Min Hein: Myanmar
Sign Language Classification using Deep
Learning.California State University Fresno.
Downloaded on June 21,2021 from IEEE Xplore
[4]. Sarika Jain, Deepika Pahuja: Recognition of Sign
Language Symbols using Templates, 2020 8th
International Conference on Reliability, Infocom
Technologies and Optimization (Trends and Future
Directions) (ICRITO) Amity University, Noida, India.
June 4-5, 2020
[5]. Ebey Abraham, Akshatha Nayak, Ashna Iqbal: Real-
Time Translation of Indian Sign Language using LSTM
2019 Global Conference for Advancement in
Technology (GCAT) Bangalore, India. Oct 18-20, 2019
[6]. Ashok K Sahoo, Gouri Sankar Mishra and Kiran Kumar
Ravulakollu: Sign Language Recognition: State of the
Art. ARPN Journal of Engineering and Applied
Sciences. Vol. 9, No. 2, February 2014
[7]. Galib Ibne Haidar, Hasin Ishraq Reefat: Glove Based
American Sign Language Interpretation Using
Convolutional Neural Network and Data Glass. 2020
IEEE Region 10 Symposium (TENSYMP), 5-7 June
2020, Dhaka, Bangladesh
[8]. Necati Cihan Camgoz, Oscar Koller, Simon Hadfield,
Richard Bowden: Sign Language Transformers: Joint
End-to-end Sign Language Recognition and
Translation. 2020 IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR)
[9]. Dr.L.Priya, A.Sathya, S. Kanaga suba raja: Indian And
English Language To Sign Language Translator- An
Automated Portable Two Way Communicator For
Bridging Normal And Deprived Ones.2020 2nd
International Conference on Power, Energy, Control
and Transmission Systems May 19,2021 from IEEE
Xplore.
[10]. Dardina Tasmere, Boshir Ahmed: Hand Gesture
Recognition for Bangla Sign Language Using Deep
Convolution Neural Network. 2020 2nd International
Conference on Sustainable Technologies for Industry
4.0 (STI), 19-20 December, Dhaka

IJISRT23MAR066 www.ijisrt.com 281