2025 IEEE International Students' Conference on Electrical, Electronics, and Computer Science

SIGNBRIDGE -AUDIO TO SIGN LANGUAGE

TRANSLATOR
2025 IEEE International Students' Conference on Electrical, Electronics and Computer Science (SCEECS) | 979-8-3315-2983-3/25/$31.00 ©2025 IEEE | DOI: 10.1109/SCEECS64059.2025.10940434

Kammadanam shirisha E.Mani Deeksith M.S.S Sri Madhava

Department of Computer Science and Department of Computer Science and Department of Computer Science and
Engineering, Vardhaman College of Engineering, Vardhaman College of Engineering, Vardhaman College of
Engineering Engineering Engineering
Hyderabad, India Hyderabad, India Hyderabad, India
[email protected] [email protected] [email protected]

S. Surendhar R. Karthikeyan
Department of Computer Science and Department of CSE(AI&M)
Engineering, Vardhaman College of Vardhaman College of Engineering
Engineering Hyderabad, India
Hyderabad, India [email protected]
[email protected]

Abstract— The lack of communication options for Deaf and New web dev technologies, new ML capabilities… even
hearing people, some may say creates a significant social burgeoning NLP tech press the envelope when it comes to
disadvantage in accessing the often-bare essential services. In solving these involved problems. To facilitate
contrast to acoustically communicated sound patterns, sign communication between hearing (but non-signing) persons
language communicates ideas freely through manual and body
and deaf people, the team is working to develop an internet
language. It can be used by those who have trouble speaking, by
those who can hear but cannot talk, and by regular people to service that will translate spoken language into sign language
interact with those with hearing impairments. By automatically and vice versa. The software mainly employs machine
converting spoken words into hand gestures, this project creates learning models to convert different words into sign
a web-based interface that allows hearing-impaired individuals languages and speech-to-text APIs in conjunction with
to communicate with normal people in real time through sign natural language processing (NLP) to understand language.
language interpretation. Two major phases of the system The system is programmed to function in two major phases.
Speech-to-text technology interprets oral input into textual This involves recording and converting audio input into text
output. Next, the text is run through Natural Language using sophisticated speech recognition algorithms. The next
Processing (NLP) algorithms leveraging the Natural language
step would be to pre-process that text itself with NLP
toolkit (NLTK) to be syntactically parsed in the context of sign
natural rules. The last phase translates the parsed text into sign techniques, built using Natural Language Toolkit (NLTK)
language gestures which involve hand shapes, orientation, and which can help us understand sentence structures and convert
body movements to convey the visual meaning of a message. them into the equivalent sign. Ultimately, the system provides
This system could be used to greatly reduce the communication real-time sign language signs indicating hand movements as
barriers experienced by people with hearing loss and deafness well as orientation and body alignment against spoken words.
using Machine Learning (ML) to continuously improve Over time, the technology becomes more accurate and acts as
accuracy, enhance quality of life, and provide a more inclusive a communication bridge between deaf individuals with the
society between the deaf community in our daily interactions hearing community in an effective manner.
within an ever-growing world.
Keywords—Sign Language, Machine Learning, Natural The Unique feature of the 'Real-Time' translation system in
Language Processing (NLP), Natural language toolkit (NLTK) this project assures to achievement of a path-breaking
Speech-to-Text, Indian Sign Language, Hand Gestures. revolution by enabling the deaf community and making them
feel more powerful, regarding communication from their
I. INTRODUCTION perspective. Furthermore, it enhances community
engagement and helps battle the isolation many Deaf
Communication is key to socializing, learning, and gaining individuals think daily.
access to important services. However the limited number of
hearing people who know sign language makes a sound-proof II. RELATED WORK
lifestyle challenging for deaf individuals. Through sign For the last couple of decades, several projects and research
language, the most common form of communication among done to bridge the communication between Deaf people
the deaf, many people can preserve their feeling of culture hearing community through technology. The majority of the
and community with other members (it employs hand forms, research in these fields focused on text-to-sign translation
gestures, facial expressions, and body movement orientation systems and sign language recognition.
to transmit meaning). Even though communication through The majority of these models are constrained by the
sign language is very important, not everyone in the hearing technology they operate on or fail to recognize grammatical
world can understand it — which creates a huge social gap intricacies, which further complicate the language, despite
instead of one that helps them integrate into society.

979-8-3315-2983-3/25/$31.00 ©2025 IEEE

Authorized licensed use limited to: VTU Consortium. Downloaded on April 07,2025 at 16:34:21 UTC from IEEE Xplore. Restrictions apply.
 Criteria Voice to sign English to Indian Sign Real-Time Sign Language HamNoSys Sign To Speech
Language Language Translation Voice to Sign Converter Representation Converter
Converter (2020) (2020) Language (2015) (2004) (2020)
Conversion
(2021)
Technology Used Voice NLP & Gesture Real-Time Image Symbolic Speech
Recognition & Recognition Voice Analysis Processing & Notation For Recognition &
Sign Animation Accuracy:88% & Sign Gesture Sign Language Gesture
Accuracy:85% Language Mapping Accuracy:82% Conversion
Rendering Accuracy:80% Accuracy:87%
Accuracy:90%

Primary Objectives Bridging Domain-Specific Sign Instant Simplified Sign Sign Language Converting Sign
Communication Language Translation Translation of Language in Linguistic Language to
for Deaf-Mute Accuracy:88% Voice to Sign Interpretation Contexts Verbal Speech
Individuals Language Accuracy:80% Accuracy:82% Accuracy:87%
Accuracy:85% Accuracy:90%
System Architecture Voice Text NLP-based Translation Real-Time Gesture Computational Integrated
Engine with System with Gesture Processing Detection& Model for Sign System with
Animation Output Accuracy:88% Unit with Sign Visual Mapping Language Data Speech&
Accuracy:85% Animation System Accuracy:82% Gesture
Accuracy:90% Accuracy:80% Recognition
Accuracy:87%
Advantages Supports Multiple Focus on Indian Sign Fast Processing Low Cost, Easy Precise Efficient, Bi-
Languages, User- Language, High Speed, Real- Implementation Representation Directional
Friendly Accuracy, Accuracy:88% Time Accuracy:80% of Sign Communication
Accuracy:85% Capabilities Language Accuracy:87%
Accuracy:90% Grammar
Accaracy:82%
Challenges/Limitations Limited Gesture Restricted to Specific Hardware Limited to static Complex Gesture
Library, Speech Domains, Complexity in Limitations, Gestures, No Notation Recognition
Accuracy Issues Expansion Accuracy:85% Environment Dynamic System, Errors, Limited
Accuracy:83% Noise Interpretation Requires Vocabulary
Sensitivity Accuracy:78% Expertise Accuracy:84%
Accuracy:88% Accuracy:80%
Application Area Communication Education and Public Places Health Research, Interactive
Aid for Deaf Mute Interpretation Services Educational Education Linguistic Learning,
Community Accuarcy:88% Institutions Accuracy:80% Analysis Accessibility
Accuracy:85% Accuracy:90% Accuracy:82% Solutions
accuracy:87%
Unique Features Supports Real- Specialized for Converts Voice Focuses on Detailed Converts Sign
Time Animation Indian Sign Language to Animated simplified Symbol System Gestures Back to
of signs Accuracy:88% Signs Instantly Gesture-Based for Sign Speech
Accuracy:85% Accuracy:90% Communication Language Accuary:87%
Accuracy:80% accuracy:82%

notable advancements in text-to-sign translation and sign have additionally been used to enhance gesture prediction by
language recognition. having the system learn new signs and variations over time in
Recent approaches include NLP and ML to improve sign language usage. Such methods can cope with lexical
understanding of spoken or written language translation into ambiguity, word order, and the non-stationary structure of
sign language. Examples include systems that are built on sign language.
the Natural Language Toolkit (NLTK) or Stanford NLP such
as deeper linguistic analysis, sentence parsing, and applying
sign language–specific grammatical rules. ML algorithms

British Sign Language (BSL) with English in written form.

The TESSA system was developed in the late 1990s and early
parts of this century as part of ongoing research into a
multimedia learning environment that would support literacy
among deaf children, by combining visual (videos), textual,
and sign language information.
The flow appears to transition from an audio input (human
speech) to textbases, pulling words out of a phrase database
for animated signers.
III. DATAFLOW METHODLOGY
Since we are translating the source language into another one,
it is required a bilingual dictionary. So, we are going to create
Fig.1: Structure of TESSA System. a dictionary of such words which is its English word and
Indian Sign. Or, for an English word the translation could be
Tessa System (Text and Signa Linguistics) – a system a picture or video, those translated to sign language would be
developed to support the education of deaf-mute children. It glossed. This system is the best performing of all three but
was developed to support bilingual education by combining each strategy does have its own pros and cons with this being

SCEECS 2025
Authorized licensed use limited to: VTU Consortium. Downloaded on April 07,2025 at 16:34:21 UTC from IEEE Xplore. Restrictions apply.
most suitable as a video copyright checker. While it is giving rearrangement is then done to meet the grammatical
some attractive results, I prefer the writing images themselves requirements of ISL texts. After that, superfluous words are
as there are realities returned in Videos. If you have time, we removed since ISL only employs meaningful words; articles,
d like to develop a system based on artificial animations that linking verbs, and other useful terms are not used.The
consume less memory, are easy to replicate, and tie lemmatization module receives the result and reduces each
nationaranslate and comb. This is inspired by the avatar- word to its most basic form. Words that are not included in
based approach that can be easily individualized. In the dictionaries are substituted with synonyms.
proposed translation system offlips-based translation systems Main Features
rely on the audio text as input. Its duly into to be English text Transforms Speech into Sign Language.
in one step, then further to ISL grammar ranks from a parse •Makes use of text as a bridge.
tree structure. A machine will take the English and perform •Uses ISL rules for text tokenization.
rule-based translations for it into ISL grammar. With its al • Web-based interface that doesn't require installations
English parse tree. This is translated using bi-linguistic rules •Architecture of the client/server.
according to the rules of ISL grammar. This altered tree •Machine Learning: NLP tools are used.
undergoes additional conversion to text before being • Fully responsive: You don't need to worry about how this
processed to create the ISL language. website will seem on a desktop, tablet, or mobile device
1. Audio to Text: This process uses voice recognition because everything is responsively ready.
technology to translate spoken English (audio) into written •Simple to operate.
text. By giving spoken words a visual representation in • Easy to use.
written form, it makes spoken language understandable to The system consists of the following modules:
people with hearing impairments. • Audio-to-text converter
2. Audio to Sign: Converting audio to sign language allows • Input Parser
people who rely on sign language to communicate with each • ISL Generator
other more effectively by giving them a clear visual • Graphics Generator
depiction. • Audio to Text
3. Text to sign: Converting text to sign language makes • Audio to Sign
communication more accessible by giving those who use sign • Text to sign
language a visual representation of textual content. 1. Audio to text Translator
Any Personal Digital Assistant (PDA) can have an internal or
external microphone to provide input for the module. Google
Cloud Speech is used by the module to convert audio to text.
Any language can be used to provide the input. An English
text string is the output that is provided..
2. Input Parser
Sentences from the input paragraph are tokenized. All
sentences are tokenized into words using Natural Language
Processing and machine learning techniques. A list of tokens
for every line of text is the module's output.
Tokenization: The process of tokenization involves dividing
a string of characters into smaller units called tokens and
perhaps eliminating certain characters, such as punctuation,
in the process.
Results:

Fig.2: System Architecture.

Fig.3: Result.
Pseudo code: Tokenizer: A tokenizer breaks up text into a series of tokens
There are five modules in the system: that approximately translate to "words".
1. An English parser to process text in English 3. ISL Generator
2. A module for rearranging sentences according to ISL This module's goal is to translate an input text written in
grammatical rules English grammar into an ISL grammar document. Through
3. The eliminator that gets rid of stop words. the use of transfer-based translation, this has been
4. Using stemming to obtain each word's root and a accomplished.
synonym for any words that aren't in a dictionary. Transfer-based Translation: The input that Transfer-Based
5. Module for converting videos. Translation systems receive is plain text. The system then
The system receives a written English text, parses it, and uses analyzes it syntactically and semantically. It is then translated
the grammatical representation to generate a phrase structure. into sign language. This method uses certain linguistic
Since English writings follow the Subject-Verb-Object principles to translate an intermediate abstract text from the
structure while ISL texts follow the same framework but source language into the destination language. As a result of
include variances in negative and interrogative phrases, the distinct sets of rules used to read information from the

SCEECS 2025
Authorized licensed use limited to: VTU Consortium. Downloaded on April 07,2025 at 16:34:21 UTC from IEEE Xplore. Restrictions apply.
source language and construct a syntactic or semantic The outcome of this text mapping will resemble this: The
structure in the target language, translation is sometimes boy's automobiles have distinct colors, thus his car has a
called "rule-based translation." In our approach, a parser different color.
converts the raw English into phrase structure trees. A When we refer to stemming, we usually imply a basic
computer software that interprets sentence structure, heuristic process that, in most cases, aims to accomplish this
including which words belong to "phrases" and which are the goal by removing derived affixes from words. The technique
subject or object of a verb, is known as a natural language of appropriately applying a vocabulary and word
parser. Using linguistic information extracted from hand- morphological analysis is known as lemmatization. Usually,
parsed sentences, probabilistic parsers try to produce the most the objective is to recover each word's lemma, or dictionary
likely analysis of novel phrases. Our system used very form, while removing just the inflectional ends.
efficient PCFG and lexicalized dependency parsers, a Generation of Synonyms
lexicalized PCFG parser, and the Stanford parser, which is an Since our dictionary is small, not every word that exists in the
implementation of probabilistic natural language parsers. source language will be in the ISL dictionary. Each token in
After converting the source language into phrase structure the string may represent either of the following:
trees, we modify the modified tree's structure to better 1. Contains the word-corresponding SIGML file.
represent the grammar and structure of ISL by applying the 2. There isn’t a corresponding SIGML file.
grammar rules of ISL. In the second scenario, we search Wordnet for synonyms for
Grammar Rules for Converting English Sentences to ISL the term. We then determine whether a synonym for a word
Sentences: exists in the current database for each potential synonym, and
If the grammar rules of two spoken languages disagree, if it does, that synonym takes the place of the original word.
translating one spoken language to another might be Visual Composer:
challenging. When spoken language is used as the source The ISL text string is the module's input. Graphics Generator
language and sign language is used as the target language, the maps every text token to a database in a module.
complexity increases significantly. There is a file present in the database for each word that
Work Flow: exists. For example, the SIGML file for the word bird is
Rearranging the words in English sentences. a bird. sigml.
Converting the English SVO structure to the ISL SOV Each word is therefore mapped to a matching SIGML
structure is the first step in this module. Finding file. We give the words that cannot be mapped as a
interrogative sentences or those that contain W-questions is concatenation of SIGML files in every Alphabet.
the next step. For instance:
Managing WH-Interrogatives Rahul + R.sigml + a.sigml + h.sigml + u.sigml + l.sigml.
WH-Who, what, when, and why are interrogative indicators The avatar receives these SIGML files as input and generates
that are always inserted following a phrase. animated movements in the SIGML file.
In English, "When is your birthday?" .Proposal improvements are high-quality animations,
ISL: "QUESTION + YOUR BIRTHDAY TIME". integration of non-manual components, dynamic gesture
This is accomplished by analyzing the text's phrase structure rendering , real time optimisation
tree, removing any nodes that contain WHinterrogative
terms, and then appending the removed node as the node's IV. RESULT AND DISCUSSION
final child. Figure 3 shows the homepage of the website which contain
Eliminator the title, navigation links (Home, Converter, Log Out,
ISL guidelines state that we are never to utilize articles or the About), and a prominent green button labeled "Click to
connecting verbs am/is/are/was/were/. (a, one, a few, the). Start." An avatar image is included to represent the signing
We refer to these words as stop words. Although these words aspect of the tool.
are widely used in the source language, they have no meaning
and are not included in the target language's dictionary, so
they ought to be removed. In this module phase, we remove
stop words from tokens that have been rearranged.
Lemmatization and Stemming
In ISL, we never utilize word ends, suffixes, or words that
end in gerunds (-ing) for any term. In ISL, any word may be
found in its most basic form. In this section of the module, we
use stemming and lemmatizing to transform each token into
its root form. Documents written in the original language will
employ variants of words, like organize, organize, and
organizing, for grammatical reasons.
Additionally, certain word families, such as democratic,
democratic, and democratization, have related derivatives
with similar meanings. Reducing a word's inflectional and
sometimes derivatively related forms to a single base form is Fig.3: Home page.
the goal of lemmatization and stemming.
Consider the case that follows: cars, cars'=>car, am, are,
is=>be a car.

SCEECS 2025
Authorized licensed use limited to: VTU Consortium. Downloaded on April 07,2025 at 16:34:21 UTC from IEEE Xplore. Restrictions apply.
 VI. REFERENCES
[1] Voice to Sign Language Converter by Akshay Kishore,
Akshita Chauhan, Pooja Verma, Shivam Veraksatra, Meerut
Institute of Engineering & Technology, 250005 Meerut, Uttar
Pradesh India. International Journal of Emerging Technology
in Computer Science and Electronics April 2020.
[2] INGIT: Limited Domain Formulaic Translation from
Hindi strings to Indian Sign Language, Indian Institute of
Technology, Kanpur.
[3] Domain Bounded English to Indian Sign Language
Translation Model, CSE, Sharda University, Noida 16, June
2022.
Fig.4: Time Taken for Conversion. [4] HamNoSys to SiGML Conversion System for Sign
Language Automation, Multiconference on Information
In figure 4 the graph shows the Time Taken for Conversion Processing 12, December 2023
which illustrates the duration required for each stage of the [5] Automatic Translate Real-Time Voice to Sign Language
audio-to-sign language translation process. This analysis can Conversion for Deaf and Dumb People Prof. Abhishek
guide future improvements in the system's efficiency and Mehta, Dr. Kamini Solanki, Prof. Trupti RathodGujarat
responsiveness. Technological University, Bardoli, Gujarat, India. 2021
[6] SIGN LANGUAGE CONVERTER Taner Arsan and
Oğuz Ülgen, Kadir Has University, Istanbul, Turkey-2015.
[7] International Journal of emerging Technology in
Computer Science and Electronics (IJETCSE) ISSN: 0976-
1353 Volume 21 Isuue 4, April 2020. [3] HamNoSys to
SiGML Conversion System for Sign Language Automation,
Multi conference on Information Processing 12, December
2023.
[8] T. Hanke, HamNoSys – Representing Sign Language
Data in Language Resources and Language Processing
Contexts, University of Hamburg, Binderstraße 34, 20146
Hamburg, Germany, (2004).
[9]Youhao Yu, "Research on Speech Recognition
Technology and Its Application", IEEE 2012.
[10] "A proposed framework for Indian Sign Language
Recognition" by Ashok Kumar Sahoo, Gouri sankar Mishra
Fig.5: User Satisfaction.
& Pervez Ahmed, International Journal Of Computer
Application, October 2012.
In Figure 5 the User Satisfaction graph reflects the feedback
[11] Mohammed Elmahgiubi; Mohamed Ennajar; Nabil
from users regarding the Audio to Sign Language Tool across
Drawil: Mohamed Samir Elbuni "Sign language translator
four key aspects: ease of use, accuracy, speed, and clarity.
and gesture recognition", IEEE December 2015.
Overall, users express high satisfaction with the tool,
[12] SIGN TO SPEECH CONVERTER Deepali Yewale,
particularly appreciating its clarity and ease of use.
Shweta Jadhav, Sayali Mahale, Ruchika Bhor, E&TC
V. CONCLUSION Engineering AISSMS IOIT, PUNE. Dec-2020.
This article proposes an efficient translation from English [13] "Indian Sign Language Recognition System", by
audio to ISL sign language. Rule-based systems may be Yogeshwar L. Rokade, Prashant M. Jadav in July 2017
employed, and syntactic and semantic analysis can be used to International Journal of Engineering and Technology (UET).
get the correct translation because sign languages, such as [14] Anand Ballabh, Dr. Umesh Chandra Jaiswal," A study
BSL and ASL, have unique grammars. It is difficult to of Machine translation methods and their challenges",
perform a syntactic and semantic analysis of Indian Sign Published 2015.
Language as there are no particular grammatical rules to [15] M Mahesh, Arvind Jayaprakash, M Geetha, Sign
compare the English text to. Consequently, an accurate language translator for mobile platforms", IEEE September
translation of the English text is impossible. Facial 2017.
expressions in ISL indicate interrogative and negative [16] D. Bhavani, K. F. K. Reddy, S. M. Ananthula, S.
statements. The phrases alter to reflect the sentence's Soundararajan, H. Shanmugasundaram and V. Kukreja,
questioning and negation as the ISL animation for the verb "Flask Powered Heart Health Predictor Using Machine
clause plays. The system still hasn't managed to complete this Learning Algorithms," 2024 IEEE International Conference
functionality. Current limitations in the visual rendering are on Contemporary Computing and Communications (InC4),
low resolution animations Limited, signed variation, minimal Bangalore, India, 2024, pp. 1-5, doi:
non-manual features, legacy in real time rendering 10.1109/InC460750.2024.10649211.

SCEECS 2025
Authorized licensed use limited to: VTU Consortium. Downloaded on April 07,2025 at 16:34:21 UTC from IEEE Xplore. Restrictions apply.