SIGN LANGUAGE RECOGNITION FOR HINDI VARNAMALA

A Report Submitted

In Partial Fulfillment of the Requirements

for the Degree of

Bachelor Of Technology
In

COMPUTER SCIENCE & ENGINEERING

By
PRATIBHA GUPTA (1805610075)
PRIYA RAJPUT (1805610076)
SHAIVYA SHUKLLA (1805610100)
SRISHTI SHARMA (1805610116)
PRIYANKA KATITYAR (1900560109007)

Under the Supervision of

DR. UMESH DWIVEDI

(FACULTY-CSE)

BBDNIIT

DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

BABU BANARASI DAS NORTHERN INDIA INSTITUTE OF TECHNOLOGY,
LUCKNOW

Affiliated
To the

DR. APJ ABDUL KALAM TECHNICAL UNIVERSITY LUCKNOw

(Formerly Uttar Pradesh Technical University, Lucknow)
May, 2022
 DECLARATION

We hereby declare that the work presented in this report entitled "SIGN LANGUAGE
RECOGNITION FOR HINDI VARNAMALA", was carried out by us. We have not submitted
the matter embodied in this report for the award of any other degree or diploma of any other
University or Institute.

We have given due credit to the original authors/sources for all the words, ideas, diagrams,
graphics, computer programs, experiments, results, that are not my original contribution. We
have used quotation marks to identify verbatim sentences and given credit to the original
authors/sources.
We affirm that no portion of my work is plagiarized, and the experiments and results reported in
the report are not manipulated. In the event of a complaint of plagiarism and the manipulation of
the experiments and results, We shall be fully responsible and answerable.

Name: Pratibha Gupta, Priya Rajput, Shaivya Shukla, Srishti Sharma, Priyanka Katiyar

Roll. No.: 1805610075, 1805610076, 1805610100, 1805610116, 1900560109007

Branch: Computer Science and Engineering

Bcbibha Gupta- tstiha

Biyo Rag put- K.u
a
Shoivyoa Shukla
Souglni shoowa- Sishh
Bigonka kodiyos-A
(Candidate Signature)
 CERTIFICATE

Certified that Pratibha Gupta(1805610075), Priya Rajput(1805610076), Shaivya

Shukla(1805610100), Srishti Sharma(18056101 16), Priyanka Katiyar(1900s60109007)
carried out the research work presented in this thesis entitled "SIGN LANGUAGE
RECOGNITION FOR HINDI VARNAMALA" for the award of Bachelor of Technology
rom Dr. APJ Abdul Kalam Technical University, Lucknow under my supervision. The thesis
embodies results of original work, and studies are carried out by the students themselves and the
Contents of the thesis do not fom the basis for the award of any other degree to the candidate or
to anybody else from this or any other University/Institution.

Signature:

(Dr. Umesh Dwivedi)

Assistant Profcssor

BBDNIIT, Lucknow

Date: 10/05/2022

ii
 ABSTRACT

The Sign language is the only tool of communication for the person who is not able to speak and
hear anything. Sign language is a boon for the physically challenged people to express their
has been
thoughts and emotion. In this work, a novel scheme of recognition
sign anguage
proposed for identifying the alphabets and gestures in sign language. With the help of computer
VISion and neural networks we can detect the signs and give the respective text output. The
project aims at building a machine learning model that will be able to classify the various hand
gestures used for fingerspelling in sign language. n this user independent model, classification
machine learning algorithms are trained using a set of image data and testing is done on a
completely different set of data. For the image dataset, depth images are used, which gave better
results than some of the previous literatures, owing to the reduced pre-processing time. Various
machine learning algorithms are applied on the datasets, including Convolutional Neural
Network (CNN). An attempt is made to increase the accuracy of the CNN model by pre-training
it on the dataset. However, a small dataset was used for pretraining, which gave an accuracy of
15% during training. Our team has developed a project that would enable deaf people to get
more involved. The problem revolves around the idea of a camera-based sign language
recognition system that would be in use for the deaf for converting sign language gestures to text
and then text to speech. Our objective is to design a solution that is intuitive and simple. To
design a real time software system that will be able to recognize ISL hand-gestures using deep
learning techniques.

iv
 ACKNOWLEDGEMENT

We would like to express my profound gratitude to Dr. Anurag Srivastava (Head of

Department, Computer Science and Engineering, Babu Banarasi Das Northern India Institute Of

Technology, Lucknow) and Dr. V.K. Singh (Director, Babu Banarasi Das Northern India
Institute Of Technology, Lucknow) for their contributions in the completion of our project titled
"SIGN LANGUAGE RECOGNITION FOR HINDI VARNAMALA". We would like to
our special thanks to our mentor Dr. Umesh Dwivedi for the
time and efforts he has
express
provided throughout the year. His useful advice and suggestions were really helpful for us during
the project completion.

We would like to thank our parents for their encouragement throughout our project. At last we
thank everyone for supporting us directly or indirectly in completing this project successfully.
2 N N m m LO
N N
 2.1.3.2 Capture Video from Camera
12
2.1.3.3 Extracting image from ROI region 14
2.1.3.4 Feature Extraction
14
2.1.3.4.1 Converting color image to grayscale image 14
2.1.3.4.2 Applying Filter 14
2.1.3.4.3 Identifying moving object 14
2.1.3.4.4 Identifying Shape 15
2.2 Training CNN on captured dataset 16
2.2.1 CNN (Convolutional Neural Network) 16
2.2.2 Tensorflow 17
2.2.3 Keras
17
2.3 Predicting the data 18
2.4 Design 18
2.4.1 External Entity 19

2.4.2 Process 19
2.4.3 Data Flow 19
2.4.4 Data Store 19
2.5 Functional Modelling 20

2.5.1 Context Level DFD/ Level 0 DFD 20

2.5.2 Level 1 DFD 21
2.5.3 Level 2 DFD 22
CHAPTER 3:RESULTs

3.1 Use case Diagram 23

3.2 Test Cases 25

3.3 Future Scope 29

3.4 Conclusion 29
REFERENCES 30

vii
 TL
L mm m m m m m m
N N NON N N NN
LL L
 CHAPTER 1

INTRODUCTION

peech impaired people use

hand signs and gestures to communicate which
people often find difficult to use. Hence normal
there is a need for a system that
different signs, and recognizes the
gestures and conveys the information to normal
the gap between people. It bridges
physically challenged people and others.

1.1 Description

Language is a medium to communicate with a

person or group. The spoken
language is communication media for those who can
many more oral
speak and listen. The world has
languages in different countries.
Sign language is a method of
communication for people who face
problems with
Simultaneous orientation and movement of hand speaking
and hearing. It uses
shapes instead of
conveyed sound patterns. Such people communicate with each other or a acoustically
different signs and gestures. Normal group through
people find it difficult to understand and interpret
the meaning of
sign language, expressed by the verbally impaired, so it becomes
inevitable to have an interpreter for the translation of
sign language. There are very few
technologies that help to connect this social group to the world.

Using the native language Hindi' in this Sign Language Recognition project,
makes this project unique and helpful. This will make the communication more
effective and convenient for the hearing/verbally
impaired people and others. Hindi as a
sign language will remove the barrier of understandability and will also
give the feeling
of oneness to the people living in India. It will be
adaptable and feasible too at the time
of emergency.

We propose the design of a basic yet extensible

system that is able torecognize
static and dynamic gestures of Hindi Sign Language. We choose the Hindi
alphabet for
our project as there are multiple
projects made using American Sign Language but none
was made for the linguistic commute people who do not have
any idea of the English
alphabet but solely rely upon their native language for expressing their thoughts.
 REFERENCES

[11. Pigou L., Dicleman S., Kindermans PJ.,

Schrauwen (2015) Sign Language Recognition B.
Using Convolutional Neural Networks. In:
Vision ECCV 2014
Agapito L., Bronstein M., Rother C. (eds) Computer
Workshops. ECCV 2014. Lecture Notes in Computer Science, vol 8925.
Springer, Cham.
https://doi.org/10.1007/978-3-319-16178-5 40.
21. Tolentino, Lean Karlo & Serfa Juan, Ronnie &
Thio-ac, August & Pamahoy, Maria &
Forteza, Joni & Garcia, Xavier. (2019). Static
Sign Language Recognition Using Deep
Learning. International Journal of Machine Learning and 9. 821-827. Computing.
10.18178/ijmlc.2019.9.6.879.
3. Mehreen urroo , Mohammad Elham, 2020, Sign Language Recognition System using
Convolutional Neural Network and Computer Vision, INTERNATIONAL JOURNAL OF
ENGINEERING RESEARCH & TECHNOLOGY (LJERT) Volume 09, Issue 12 (December
2020).

[4]. Vijay More, Sanket Sangamnerkar, Vaibhav Thakare, Dnyaneshwari Mane, & Rahul Dolas.
(2021). SIGN LANGUAGE RECOGNITION USING IMAGE PROCESSING. JournalNX - A

Multidisciplinary Peer Reviewed Journal, 85-87.

Kusuma
[5]. Suharjito, Suharjito & Anderson, Ricky & Wiryana, Fanny & Ariesta, Meita &
for Deaf-Mute
Negara, I Gede Putra. (2017). Sign Language Recognition Application Systems
Science. 116. 441-448.
People: A Review Based on Input-Process-Output. Procedia Computer
10.1016/j.procs.2017.10.028.

using neural networks and kNN

6. Saho0, Ashok. (2014). Indian sign language recognition
Sciences. 9. 1255-1259.
classifiers. Journal of Engineering and Applied

T7. https://analyticsindiamag.com/hands-on-guide-to-sign-language-classification-using-cnn

1 8 . h t t p s : / / d a t a - f l a i r .t r a i n i n g / b l o g s / s i g n - a n g u a g e - r e c o g n i t i o n - p y t h o n - m l - o p e n c v /

19. https://github.com/PacktPublishing/Hands-On-Computer-Vision-with-TensorFlow-2

SHELTER
MANNING
CHOLLET;
FRANÇOIS
with Python, by
LdDeep Learning
ISLAND