Sign Language Recognition Reseach Paper-1
Sign Language Recognition Reseach Paper-1
Machine Learning
Avinash Kumar Sharma Abhyudaya Mittal Aashna Kapoor
Computer Science and Engineering Computer Science and Engineering Computer Science and Engineering
ABES Institute of Technology ABES Institute of Technology ABES Institute of Technology
Ghaziabad, India Ghaziabad, India Ghaziabad, India
[email protected] [email protected] [email protected]
Aditi Tiwari
Computer Science and Engineering
ABES Institute of Technology
Ghaziabad, India
[email protected]
Abstract - One of the major issues that our society is dealing with and orientations, as well as face expressions. Hearing loss
is the difficulty that people with disabilities have in sharing their affects roughly 466 million people globally, with 34 million
feelings with normal people. People with disabilities can of them being children. Individuals who are labelled as
communicate through sign (gesture) languages. This project aims "deaf" have very limited or no hearing capabilities.
to design a model which can recognize sign language alphabets
(hand gestures) and convert it into text and sound using machine Only a small percentage of the population is aware of sign
learning approach. The main goal of this project to break down language. It is also not an international language, contrary to
barriers to communication between people with disabilities and the common perception. Obviously, this makes communication
rest of society. The performance of this method is evaluated on between the Deaf population and the hearing majority even
publicly available ISL dataset. In our project, which is based on more difficult. Because the Deaf community is generally
Convolution Neural Networks, we used the Inception V3 deep less adept in writing a spoken language, the option of
learning model for image classification. Hand gestures are written communication is inconvenient [3]. Hearing or
captured as images by the webcams in this project, and the defined speech problems affect about 0.05 percent of the world's
model aids in the recognition of the corresponding alphabet for population, according to the United Nations Statistics
hand gestures. Division. The disability was present in 63 percent of these
We have tried to overcome the existing limitations in Sign patients at birth, whereas the rest acquired it as a result of an
Language Recognition and to increase the efficiency. accident. According to JICA disability statistics, hearing
impairments account for 8.36 percent of all disabilities in
Keywords — Sign Language, Hand Gesture, Gesture Recognition, India, while speech difficulties account for 5.06 percent. For
Human Computer Interaction, Sign Language Recognition.
a deaf population of around 7 million individuals in India,
there are only about 250 competent sign language
I. INTRODUCTION interpreters [4]. A department of Persons with Disabilities
Empowerment is part of the Ministry of Social Justice and
Communication is an indispensable tool for human Empowerment, and it deals with policies for people with
existence which is a basic and effective way to share
disabilities. The ISLRTC of the ministry is in charge of deaf
thoughts, feelings and opinion, but a significant portion of and dumb schools [5].
the world's population is lacking this ability. Hearing loss,
speech disability, or both affect a large number of people. It has ties to a number of Indian schools. Aside from that,
Hearing loss is defined as a partial or total inability to hear many groups, such as the Indian Deaf and Dumb Society, try
in one or both ears. Mute, on the other hand, is a disability to give various forms of assistance on their own. Schools for
that prevents people from speaking and makes them unable the deaf and dumb rely primarily on verbal communication
to communicate. If a child becomes deaf-mute throughout due to a dearth of skilled teachers. This is the state of large
childhood, their capacity to learn languages is hampered, centers, and when it comes to rural areas, there are no
resulting in language impairment, also known as hearing institutions or assistance for the deaf and dumb. As a result,
mutism. We discovered that those who are unable to residents in these locations are experiencing severe
communicate verbally and have hearing impairments have psychological distress and feel utterly cut off from the rest
difficulty in ordinary communication, and that this hearing of the world. Even when they reach adulthood, they remain
or speech disability results in a shortage of equal reliant on their relatives, or they struggle to make ends meet
opportunity for them [1] [2]. because they are unable to find suitable work.
People who are deaf or deaf-blind use sign language as a The biggest problem is that fit people are either unwilling to
means of communication. A sign language is made up of a learn sign languages or find them difficult to remember.
variety of gestures made up of diverse hand shapes, motions, Researchers have tried a variety of ways for recognizing
diverse hand gestures in order to allow normal people to
comprehend sign languages and to eliminate barriers in our
society for individuals with disabilities. With the
advancement of modern technology, we can find a variety of
ways to integrate these people into society. The availability
of sensors, cameras, and AI technologies like as deep ASL, like all languages, is a living language, it evolves
learning, CNN, ANN, and speech to voice programs, as well with time. Many high schools, colleges, and universities
as speech to text programs, has opened the way for the in the United States accept it as a modern and "foreign"
development of useful gadgets. We can undoubtedly make language requirement for academic degrees.
significant progress in engaging with people with the help of
these new technologies. C. Indian Sign Language (ISL)
A. Sign Language and Gestures Indian Sign Language (ISL) is India's most commonly
used sign language; it is referred to as the mother
To visually transmit sign patterns that convey meaning
tongue in some metropolitan regions due to its
in sign language, a sequence of facial expressions, an
orientation, hand shapes, and hand and body movement widespread use. ISL is a collection of authentic sign
are used. Hand gestures are very crucial for deaf and languages that have grown over time and are widely
mute people who use Sign Language to communicate used as shown in Figure 2 [7].
with the outside world. Sign language has been shown
to be useful in communicating a wide range of needs, India's sign language is very scientific, with its own
from basic necessities to complex concepts. There are grammar.
three types of sign languages which are as follows:
D. Inception Model
E. Classification
Alphabet - W
Many picture categorization models have been
developed to aid in the resolution of the most
pressing issue of identification accuracy. Image
categorization is a key subject in the field of
computer vision, having a wide range of practical
applications [28]. We have used transfer learning
mechanism to train our model. Inception V3 Model
is used in this project, which is an image classifier
model which works on CNN (Convolutional Neural
Network) and It is pre-trained on a very large data.
So, by transfer learning we mean that we have
trained the existing inception V3 model on our target Alphabet – B
dataset of sign languages. Now, we have used this
alphabet recognition model to predict the various Figure 6: Output Labels
labels of sign languages. The predict function takes F. Text to Speech Conversion
user image as input and map it to correct label
according to the trained model. Finally the correct Speech is one of the most ancient and natural ways
label is returned as output as shown in Figure 6. for humans to share information. Throughout the
years [23]. The process of turning words into a vocal
audio form is known as text-to-speech (TTS). The
programme, tool, or software takes a user's input text
and, using natural language processing methods,
deduces the linguistics of the language and does
logical inference on it. This processed text is then
sent to the next block, which performs digital signal
processing on it. This processed text is then
translated into a voice format using a variety of
techniques and transformations. Speech is
synthesised throughout the entire procedure.
Alphabet - R In this project for converting the text into speech we
have used gTTS module. Google Text-to-Speech
(gTTS) is a Python library and command-line utility
for interacting with the Google Translate text-to-
speech API [10]. The gTTS library, which can be
used for voice translation, will be imported from the
gTTS module [9].