Scribd
Upload
18 views22 pages

Sign Language Recognition

Uploaded by

2023aspire121
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
18 views22 pages

Sign Language Recognition

Uploaded by

2023aspire121
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 22

INDIAN SIGN LANGUAGE

RECOGNITION SYSTEM
BASED TEXT TO
ANIMATION CONVERSION
Team Members
Rimjhim
(2300520140053)
Manishka Singh
(2300520140032)
Payal Singh
(2300520140044)
Rashi Chaudhary Under the
(2300520140052) auspices of
Dr Upendra Kumar
Mr. Sandeep Yadav
INTRODUCTION
Sign language bridges communication gaps by
incorporating gestures, expressions, and lip movements,
adding meaning and context to conversations. It is vital
for nearly 6 million people in India who rely on Indian
Sign Language.
Our application leverages Natural Language Processing
(NLP) to convert spoken English into Indian Sign
Language animations. It acts as both a translator for
seamless communication and a learning tool for those
interested in ISL.
Special focus is given to capturing the unique phonetic
characteristics and linguistic nuances of Indian languages,
ensuring precise speech recognition and accurate
conversion into sign language.
MOTIVATION
Challenges Faced by the Hearing-Impaired
Individuals with hearing impairment face significant
communication barriers in daily life.
Lack of Accessible Sign Language Education
Only a small percentage of the deaf population in India
receive formal education in sign language, and sign language
training centers are often costly.
Prevalence of Hearing Impairment
Around 63 million people in India suffer from significant
auditory impairment.
Need for Better Communication Solutions
An estimated 5 million deaf and hard-of-hearing people in
India use sign language, highlighting the need for more
accessible communication tools.
RELATED WORK
METHODOLOGY
The methodology involves converting speech to text using JavaScript's
Speech Recognition library. The text is then tokenized, parsed, and
transformed into Indian Sign Language (ISL) syntax, following specific
grammatical rules. Stemming, lemmatizing, and removal of stop words
ensure accuracy in translation. Finally, the system scrapes an ISL
dictionary for sign language videos, which are merged into a cohesive
video sequence to visually represent the spoken sentence in ISL.

Convert speech to text using JavaScript's Speech Recognition.


Tokenize, parse, and generate ISL syntax based on grammar rules.
Apply stemming, lemmatizing, and remove stop words for accurate
translation.
Use web scraping to obtain ISL videos and merge them into a final
video representation.
Speech Animations
Major steps in the proposed system
Speech-to-Text Conversion:
This step involves using a built-in or external microphone to receives speech as an input from the user. The
system uses the JavaScript library, Speech Recognition, to perform this operation
It includes:
Pre-processing: The input is split into small frames for analysis.
Feature Extraction: Mel Frequency Cepstral Coefficients (MFCC) are used for feature representation.
Phonetic Unit Recognition: Phonemes are predicted using a pre-trained acoustic model.
Hidden Markov Model (HMM): HMMs model temporal dependencies in speech signals.
Parsing
The parsing step tokenizes the input text into smaller units and uses
the Stanford Parser to analyze its grammatical structure. It generates a
Context-Free Grammar (CFG) representation and creates a new parse
tree based on ISL rules, processing noun and verb phrases to ensure
accurate mapping for Indian Sign Language (ISL) syntax.

Tokenization: Splits text into smaller units (tokens) like words


and sub-words.
Stanford Parser: Analyzes grammar, tags parts-of-speech, and
generates CFG representation.
ISL Syntax Generation: Creates a parse tree based on ISL rules
for accurate mapping.
Recursive Parsing: Handles nested structures in verb phrases.
Stemming and Lemmatizing
Stemming and Lemmatizing are used to reduce words to their root forms in ISL. Stemming
removes suffixes to simplify words, while Lemmatizing groups different word forms into a
single root word. The system uses NLTK's stemming algorithms and WordNet Lemmatizer
to accurately process words like "playing," "played," and "plays" into "play," improving the
model's accuracy.

Stemming: Reduces words to their root form by removing suffixes.


Lemmatizing: Converts inflected words to their dictionary form.
NLTK Algorithms: Uses NLTK stemming and WordNet Lemmatizer for
processing.
Accuracy Improvement: Enhances model performance by
standardizing word forms.
Filtering Stop words
The ISL syntax is free of stop words. Sentences in ISL are formed using
main words only. Linking words, like ‘am’, ‘are’, ‘is’, ‘be’, ‘to’ and ‘for’, and
articles, like ‘the’, ‘a’ and ‘an’, are removed.

Generated ISL Syntax


The generated ISL syntax follows the subject-object-verb sentence pattern,
unlike Atlantis Highlights in Computer Sciences, volume 4 280 English, which
follows the subject-verb-object pattern. It does not have any inflections, like
gerunds, suffixes or any other forms of the word, like plurals or different tenses.
Only the root form of every word is utilized. The generated ISL sentence is free
from conjunctions, articles and linking verbs.
Web Scrapping
Web scraping is the process of utilizing bots to extract the underlying HTML code from
any website, as well as the data held in a database. To obtain the sign language videos
depicting every word, the system scrapes a web-based ISL dictionary. To automate this
process, Selenium is utilized. Selenium is a portable web framework that provides tools
to write functional tests without having to learn test scripting languages.

Merging Videos
Once the sign language videos are obtained, they are merged or concatenated into
a single video. This video is essentially a stream of videos that represents the
spoken sentence in ISL.
Data Flow Diagram
0 level DFD:

1 level DFD:
Use Case Diagram
Class Diagram
Sequence Diagram
Speech Recognition Sign Language
Service Database
User

Speech Input
Speech Input

Text Output

Request for Gestures

Gestures Output

Send to Animation

Animation Output

Animated Output

Animation
Rendering
Engine
Tech Stack Used:
Speech to text: IBM Watson speech-to-text
Text Preprocessing: Spacy
Text to ISL Glass mapping: OpenNMT(Python)
ISL Gesture Animation: Unity 3D
Optional Enhancement: Power BI for visual
analysis
Database: Indian Sign Language Research &
Training Center
CHALLENGES
Speech Recognition Accuracy: Achieving high accuracy
in converting spoken language to text can be difficult
due to accents, background noise, and speech
variations.
ISL Syntax Complexity: Translating text into ISL
involves complex grammar rules, which can be hard to
map accurately from spoken language.
Tokenization Issues: Handling ambiguous words and
context-dependent phrases during tokenization can lead
to errors in processing.
Video Synchronization: Merging multiple sign language
videos into a cohesive stream without losing context or
meaning is a technical challenge.
Future Potential and Innovation
The system's future potential includes improving speech recognition
accuracy and expanding sign language video databases. With AI-
driven personalized learning and multilingual support, it can enhance
communication, offering greater independence and inclusion for the
deaf and speech-impaired community.

Improve speech recognition accuracy for better ISL translation.


Expand sign language video databases for more comprehensive
coverage.
Integrate AI-driven personalized learning for tailored educational
experiences.
Support multilingual communication to cater to diverse user
needs.
CONCLUSION
• Accuracy and Real-Time Performance i.e. System should deliver
accurate results in real-time to ensure effective communication.
• Ability to accommodate different sign languages and dialects to
cater to diverse user needs.
• Interface design prioritizing ease of use and accessibility for all
users.
• Framework should be scalable to handle increasing data volume
and user demand.
• Successful implementation can significantly improve the lives of
individuals with hearing impairments, fostering inclusivity and
accessibility in society.
Thank You!

You might also like