10 V May 2022

https://doi.org/10.22214/ijraset.2022.43269
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue V May 2022- Available at www.ijraset.com

Audio to Indian Sign Language Translation

Harshita Mishra1, Mansi Sharma2, Muskan Ali Qureshi3, Shivani Chaudhary4
1, 2, 3, 4
Department of CSE, Krishna Engineering College, Ghaziabad

Abstract: This project's primary purpose is to bridge the gap between deaf and hearing persons, which will benefit those with
hearing impairments who employ a simple and effective way of sign language. Sign language is a visual language used by the
deaf community. It employs body language, hand gestures, and facial expressions. Indian Sign Language is one of the most
significant and commonly utilised modes of communication for individuals with speech and hearing difficulties. This web
application facilitates communication for deaf and speech-impaired individuals. The primary focus of these new web application
and natural language processing technologies is the conversion of spoken or written language into sign language. In this web
application, users can record their speech using a microphone or text as input utilising NLP-based speech recognition. If the
video is missing from the database, the word is spit out and the associated video is displayed. This technique has made
communicating with deaf individuals simple and practical.
Keywords: NLP, Speech to text, sign language translation.

I. INTRODUCTION
People who are deaf or hard of hearing often rely on sign language as their primary mode of communication because it is widely
used across the globe. Despite the fact that many countries utilise computers, relatively little research has been done to identify ISL.
The fundamental architecture of the vast majority of systems is based on: Translation techniques such as Direct Translation,
Transfer-based Architecture, and Statistical Machine Translation The system has been implemented. The focus is primarily on the
Indian Sign Language. According to the census completed in 2011, there are 63 million people in India who have hearing problems,
which accounts for 6.3 percent of the total population. It is estimated that between 76 and 89 percent of hearing-impaired Indians
have no knowledge of the language in either of its signed, spoken, or written forms. The low literacy rate can be attributed to a
number of problems, including the absence of sign language interpreters, a dearth of Indian Sign Language equipment, and a dearth
of research on the subject of Indian Sign Language.
The difficulties that are experienced by the deaf population in India are the impetus for the creation of the system that we have
recommended. This system is designed to perform a sign language translation for each individual word that is taken in as input. A
video that displays the sign language interpretation of the spoken sentence is accompanied by a voice recognizer that turns spoken
sentences into expressions and silences and recognises it as text - a sequence of words. This system is comprised of both of these
components. It is a flexible method that is capable of considerably cutting down on the amount of work and the parallel corpus that
are required for converting a speech to sign language translation system to a new area.

A. Problem Statement
A language known as sign language is one in which information is conveyed via the use of manual communication methods such as
facial expressions, hand gestures, and other bodily motions. In order to accomplish the task of translating written language into sign
language, this project utilises movies that are dedicated to specific terms.

B. Proposed System
This proposed system is developed to overcome the challenges faced by the deaf people. Basically, this system is planned to
translate words that is accepted as input to sign language. This model translates the words based on Indian Sign Language.
1) Natural Language Processing: Some unwanted words such as ‘are’, ‘as’, ‘is’, ‘was’, interjections, commas, symbols, non-root
verbs etc. Hence, this system takes away those stop words from the processed speech.
2) Root Words: The words may be in gerund form, plural form, or adjective form. The proposed system will remove these forms
of the words and find the root word from those words. These root words will be helpful in the effective conversion into sign
language.
3) Dataset: The system has a large dataset of Indian sign language words to map according to the text or text recognized from the
speech. So, it will be helpful to all deaf people in India. It makes the people understand most of the speech or text.

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 3904
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue V May 2022- Available at www.ijraset.com

II. LITERATURE SURVEY

As we know, India is a multilingual country. There are different state and all its state have their own regional language and al its
regional language have its own sign language. In India there is less research have done on this project.
The ISLRTC (Indian Sign Language Research & Training Center) have taken step to look on this type of application. For deaf
people in India provides practices to learn and understand American Sign Language ASL because it is very easy to learn and
understand and after this learn sign language. Approx. 98% of 1.1, million deaf people in India are illiterate. Deaf schools are trying
to implement this strategy with hearing aids, but this is not possible to apply on community. Research have tries to find a new
solution to this root problem. The strategy for producing gloss aligned with ISL grammar. The primary aim of this project is
transformation from audio to sign language is formation of phrases that help to identify the accurate American Sign Language ASL
grammar phrases. Each sentence from the input is converted into tokens, and after that each tokens particular actions are tokens
(Like each word are convert in its root word and helping words are removed). All those word from input which are not related to
ASL Gloss are removed. After this processing of input proper noun is outputted and separated by hyphen and remaining words are
also give output by applying various rules.
For example-INDIA, this alphabet of proper noun is I-N-D-I-A corresponding to the use of conventional sign language. This system
can cause destructive nature is only human error found. Gloss storing can be one of the reasons of human error.
As the second most prevalent cause of disability, Neha Poddar, Shrushti Rao, Shruti Sawant, Vrushali Somavanshi, and Professor
Sumita Chandak noted in their paper that the prevalence of deafness in India is fairly high. A portable interpreting device that
converts higher mathematics sign language into corresponding text and voice could be of great assistance to the deaf and solve
numerous problems.

III. METHODOLOGY
A. Forms of Input
Our project is intended to get inputs two formats. The inputs can be of forms:
1) Text input
2) Live speech input

B. Speech Recognition
The microphone in our system is where the live speech comes in. The Python package PyAudio is used to do this. PyAudio is a
package for Python that lets you record audio on many different platforms. Google Cloud Speech is then used to change the audio
into text.

C. Pre-processing of Text
In the part for providing input, sentences are broken up into tokens. Each sentence is given a classification based on the results of
machine learning and natural language processing tools. This module's goal is to transform input text that uses English grammar into
material that uses the grammar of International Sign Language. Following the tokenization step, the next step is the POS tagging,
which may be defined as the process of matching each word with its respective part of speech. Also known as point-of-sale (POS)
tagging in popular parlance. Included in the category known as Part of Speech are conjunctions, adjectives, verbs, adverbs, nouns,
pronouns, and adverbial phrases. After the POS tagging has been finished, It will then be possible for us to continue the process of
removing all of the stop words. Simply put, "stop words" are words that are used in language that do not contribute significantly to
the overall meaning of the sentence. They do not contribute significantly to the meaning of the statement; thus they can be
eliminated and the meaning will remain the same. In addition to this, once we have removed them, we will employ a process called
lemmatization to get rid of the inflectional ending and return the word to its base form, which is called the lemma. It will return
either saw or see depending on whether the token saw was used as a noun or a verb in the sentence while it was being lemmatized.
POS Tag POS Tag Full - Form
CC Coordinating Conjunction
NNS Noun, Plural
RP Particles
NNPS Proper Noun, plural
SYS Symbols
DT Determiners

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 3905
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue V May 2022- Available at www.ijraset.com

D. Result
Output from the ISL conversion phase of the input sentence is passed to the video generation phase, wherein each of the words in
the sentence is looked up in the database for its corresponding video file. Then these files are all concatenated to produce a more
structured, informative, and easy-to-understand visual depiction of Indian Sign Language.

Fig 1

Fig 2.

The final product of this unit will be a video clip containing text translated into ISL. The database will have a video for each
individual word, and the final video will be a compilation of all these words into a single video.

IV. FLOW CHART

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 3906
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue V May 2022- Available at www.ijraset.com

V. CONCLUSION
Here, we've tried to make a model that will let people with disabilities express themselves in their own way, which will make it
easier for them to fit in with the rest of the world. The audio we give our proposed model will be successfully turned into an
animation. As the ISL Dictionary grows, there will be many ways to improve this route. The number of words in the ISL is small, so
adding new words to their dictionary to make it bigger would help in many ways. Text-and-speech integration can also be done on a
project to make it easier for people to talk to each other. This will let users turn text into Indian Sign Language by typing it in by
hand.

VI. RECOMMENDATIONS FOR FUTURE STUDIES

In light of the fact that the ISL is still in its infancy and that very little research has been conducted on the subject, the dictionary
could benefit from the addition of a great number of fresh videos explaining the meanings of various words. This would broaden the
ISL's application and make it easier for people to communicate with one another using this language. Users will be able to convert
text to Indian Sign Language by manually entering the input sentence after the text-to-speech integration that can be performed in
the project to enable better communication strategies has been completed. Moreover, text to speech integration can be performed in
the project to enable better communication strategies.

REFERENCES
[1] F. Shaikh, S. Darunde, N. Wahie, and S. Mali, "Sign Language Translation System for Railway Station Announcements," 2019 IEEE Bombay Section
Signature Conference (IBSSC), 2019, pp. 1-6, DOI:10.1109/IBSSC47189.2019.8973041.
[2] K. Saija, S. Sangeetha and V. Shah, "WordNet Based Sign Language Machine Translation: from English Voice to ISL Gloss," 2019 IEEE 16th India Council
International Conference (INDICON), 2019, pp. 1-4, DOI: 10.1109/INDICON47234.2019.9029074.
[3] B.D. Patel, H. B. Patel, M. A. Khanvilkar, N. R. Patel and T. Akilan, "ES2ISL: An Advancement in Speech to Sign Language Translation using 3D Avatar
Animator," 2020 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), 2020, pp. 1-5, DOI: 10.1109/CCECE47787.2020.9255783.
[4] Sunitha K. A, Anitha Saraswathi.P, Aarthi.M, Jayapriya. K, Lingam Sunny, “Deaf Mute Communication Interpreter- A Review”, International Journal of
Applied Engineering Research, Volume 11, pp 290-296, 2016.
[5] Mathavan Suresh Anand, Nagarajan Mohan Kumar, Angappan Kumare- san, “An Efficient Framework for Indian Sign Language Recognition Using Wavelet
Transform” Circuits and Systems, Volume 7, pp 1874- 1883, 2016.
[6] Chandandeep Kaur, Nivit Gill, “An Automated System for Indian Sign Language Recognition”, International Journal of Advanced Research in Computer
Science and Software Engineering.

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 3907