A PRELIMENERY REPORT ON

REAL TIME SIGN LANGUAGE RECOGNIZER USING AI

SUBMITTED TO THE SAVITRIBAI PHULE PUNE UNIVERSITY, PUNE
IN THE PARTIAL FULFILLMENT OF THE REQUIREMENTS
FOR THE AWARD OF THE DEGREE

OF

BACHELOR OF ENGINEERING (COMPUTER

ENGINEERING)
SUBMITTED BY
MR. KUMAWAT SATTYAM SUNIL Exam No :71915281L
MR. DAREKAR ROHIT MOHAN Exam No :72004571B
MR. JAGDALE VISHAL JAYVANT Exam No :71915261F
MR. MURTADAK SURAJ JAGANNATH Exam No :72004579H

GUIDED BY
PROF. CHANDGUDE A. S.

DEPARTMENT OF COMPUTER ENGINEERING

SND COLLEGE OF ENGINEERING AND RESEARCH CENTER
YEOLA, NASHIK 423401
SAVITRIBAI PHULE PUN UNIVERSITY
2021-2022
 CERTIFICATE
This is to certify that the project report entitles

”REAL TIME SING LANGUAGE RECOGNIZER USING AI”

Submitted by

MR. KUMAWAT SATTYAM SUNIL Exam No :71915281L

MR. DAREKAR ROHIT MOHAN Exam No :72004571B
MR. JAGDALE VISHAL JAYVANT Exam No :71915261F
MR. MURTADAK SURAJ JAGANNATH Exam No :72004579H

is a bonafide student of this institute and the work has been carried out by him/her
under the supervision of Prof. Chandgude A. S. and it is approved for the partial fulfill-
ment of the requirement of Savitribai Phule Pune University, for the award of the degree
of Bachelor of Engineering (Computer Engineering).

(Prof. Chandgude A. S. ) (Prof. Chandgude A. S. )

Project Guide Head of Department

(Dr. Patil P. M.)

Principal

Internal Examiner External Examiner

—————– —————–

Place :
Date :

1
 ACKNOWLEDGEMENT

It gives us great pleasure in presenting the project report on ‘REAL TIME SIGN
LANGUAGE RECOGNIZER USING AI.’

We would like to take this opportunity to thank our internal guide Prof. Chandgude
A. S. for giving us all the help and guidance we needed. So really grateful to him for their
kind support. Their valuable suggestions were very helpful.

We are thankful to our Head of Department Prof. Chandgude A. S. for providing

various resources such as laboratory with all needed software platforms, continuous in-
ternet connection, for our project. Beside we are thankful to Dr. Patil P. M. Principal of
our college and management.

We would like to extend our sincere thanks to our family members. It is privilege
to acknowledge their cooperation during the course of this dissertation. We express our
heartiest thanks to our known and unknown well-wishers for their unreserved coopera-
tion, encouragement and suggestions during the course of this dissertation report.

We would like to thanks to our all teachers, and all our friends who helped with the
ever daunting task of gathering information for the dissertation.

MR. KUMAWAT SATTYAM SUNIL

MR. DAREKAR ROHIT MOHAN
MR. JAGDALE VISHAL JAYVANT
MR. MURTADAK SURAJ JAGANNATH

2
 ABSTRACT

Sign Language detection by technology is an overlooked concept despite there being a

large social group which could benefit by it. There are not many technologies which help
in connecting this social group to the rest of the world. Understanding sign language is
one of the primary enablers in helping users of sign language communicate with the rest
of the society. Image classification and machine learning can be used to help computers
recognize sign language, which could then be interpreted by other people. Convolutional
neural networks have been employed in this paper to recognize sign language gestures.
The image dataset used consists of static sign language gestures captured on an RGB
camera. Pre-processing was performed on the images, which then served as the cleaned
input. The paper presents results obtained by retraining and testing this sign language
gestures dataset on a convolutional neural network model using Inception v3. The model
consists of multiple convolution filter inputs that are processed on the same input. The
validation accuracy obtained was above 90reviews the various attempts that have been
made at sign language detection using machine learning and depth data of images. It
takes stock of the various challenges posed in tackling such a problem, and outlines future
scope as well
Keywords: Sign Language Recognition, Deep Learning, OpenCV, Convolutional Neural
Network, Image Processing.

3
Contents

1 INTRODUCTION 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 LITERATURE SURVEY 3

3 SOFTWARE REQUIRMENT SPECIFICATION 4

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3.1.1 Project Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3.1.2 User Classes and Characteristics . . . . . . . . . . . . . . . . . . 5

3.1.3 Assumptions and Dependencies . . . . . . . . . . . . . . . . . . 5

3.2 Functional Requirement . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3.2.1 Image Processing System: . . . . . . . . . . . . . . . . . . . . 5

3.2.2 Clasification System: . . . . . . . . . . . . . . . . . . . . . . . 5

3.3 External User Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4
 3.3.1 User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.3.2 Software Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.3.3 Hardware Interface . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.4 NonFunctional Requirements . . . . . . . . . . . . . . . . . . . . . . . . 6

3.4.1 Performance Requirements . . . . . . . . . . . . . . . . . . . . . 6

3.4.2 Safety Requirements . . . . . . . . . . . . . . . . . . . . . . . . 8

3.4.3 Security Requirements . . . . . . . . . . . . . . . . . . . . . . . 9

3.4.4 Software Quality Attributes . . . . . . . . . . . . . . . . . . . . 9

3.5 System Requirement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.5.1 Software Requirements . . . . . . . . . . . . . . . . . . . . . . . 9

3.5.2 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . 11

3.6 Analysis Model: SDLC Model to be applied . . . . . . . . . . . . . . . . 11

3.7 System Implementation Plan . . . . . . . . . . . . . . . . . . . . . . . . 13

4 SYSTEM DESIGN 15

4.1 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2 Data Flow Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2.1 Data Flow Level-0 . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2.2 Data Flow Level-1 . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.2.3 Data Flow Level-2 . . . . . . . . . . . . . . . . . . . . . . . . . 16

5
 4.3 UML Diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4.3.1 Sequence Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 18

4.4 Class Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.5 Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.6 Use Case Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

5 Other Specification 22

5.0.1 Advantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

5.0.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

5.0.3 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

6 Conclusions 23

APPENDIX A
APPENDIX B
APPENDIX C

6
 LIST OF ABBREVIATIONS

ABBREVIATION ILLUSTRATION

IOT Internet of Thing

QA Quality Assurance
SRS Software Requirements Specification
UML Unified Modeling Language
DFD Data Flow Diagram
ER Entity Relationship
IEEE Institute of Electrical and Electronics Engi-
neers

7
List of Figures

3.1 Performance measures‘ . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.2 SDLC Life Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.3 SLR- System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 14

4.1 SLR- System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2 DFD-Level 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.3 DFD-Level 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.4 DFD-Level 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4.5 Sequence Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4.6 Class Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.7 Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.8 Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

6.1 ReLU Activation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

8
 Chapter 1

INTRODUCTION

The sign language is used widely by people who are deaf-dumb these are used as a
medium for communication. A sign language is nothing but composed of various ges-
tures formed by different shapes of hand, its movements, orientations as well as the facial
expressions. There are around 466 million people worldwide with hearing loss and 34
million of these are children. ‘Deaf’ people have very little or no hearing ability .They
use sign language for communication. People use different sign languages in different
parts of the world. Compared to spoken languages they are very less in number. India
has its own sign language by the name Indian Sign Language (ISL). In developing coun-
tries there are only very few schools for deaf students. Unemployment rate among adults
with hearing loss are very high in developing countries . Data from Ethnologies states
that among deaf population in India, which is about 1 percent of total population, literacy
rate and number of children attending school is very less. It goes on to state that official
recognition of sign languages, increasing the availability of interpreters and providing
transcription in sign languages greatly improve accessibility. Signs in sign languages are
the equivalent of words in spoken languages Signed languages appear to favour. The as-
sociate editor coordinating the review of this manuscript and approving it for publication
was Weiyao Lin. Simultaneous sign internal modification , rather than the concatena-
tion of morphemes. But learners in the initial stages of SL learning use iconicity as a
mnemonic aid to remember new signs. Butthe lack of iconicity makes it difficult to learn
new signs for those who learn SL as a new language . Finger spelling is the representa-
tion of the letters of a writing system and sometimes numeral systems. Sign Language
(ISL) can represent English alphabets A-Z using finger spelling .It can be one handed
or two handed and ISL follows two handed style. It issued to represent words that have
no sign equivalent or used to emphasize a word Though finger spelling usage is less in
casual signing, they are an important component in sign i language learning. This project
aims at identifying alphabets in Indian Sign Language from the corresponding gestures.

SNDCOE&RC,Department of Computer Engineering,2021-22 1

Gesture

recognition and sign language recognition has been a well researched topic for Amer-
ican Sign Language(ASL), but few research works have been published regarding Indian
Sign Language(ISL). But instead of using high-end technology like gloves or kinect, we
aim to solve this problem using state of the art computer vision and artificial intelligence
algorithms.

1.1 Motivation

Highly influenced by the teachings of the Shrimad Bhagavad-Gita we decided work for
the people with speaking disability. During academics we had met people with speaking
disability, we had seen their problem closely. So that was the biggest motivation for us to
create this project.

1.2 Problem Definition

In the world of technology, everything and everyone in the world is growing so rapidly,
so we think disability of deaf people should not be barrier in the growth of deaf people.

As technology is changing world so rapidly, so technology can also change the life
of such people, artificial intelligence can play major role in improving communication of
such people.

SNDCOE&RC, Department of computer Engineering 2021-22 2

 Chapter 2

LITERATURE SURVEY

This chapter contains the existing and established theory and research in this report
range. This will give a context for work which is to be done. This will explain the depth
of the system. Review of literature gives a clearness and better understanding of the ex-
ploration/venture. A literature survey represents a study of previously existing material
on the topic of the report. This literature survey will logically explain this system.

Real Time Sign Language Interpreter: In this paper Geethu Nath and Arun C.S. de-
veloped a system using ARM CORTEX A8processor for recognizing the ASL symbols.

Continuous Indian Sign Language Gesture Recognition and sentence Formation:

In this paper Kumud Tripathi designed a system for recognizing continuous ISL gestures
using Principal Component Analysis (PCA) with various distance classifiers.

Implementation of Real Time Hand Gesture Recognition: : In this paper used the
codebook algorithm for background subtraction and generated binary images from the
given image frames.

Automatic Indian Sign Language Recognition for Continuous Video Sequence: An

eigenvector-based technique is used for feature extraction, and eigenvalue weighted Eu-
clidean distance technique is used for Classification of 24 different ISL alphabets.

Dynamic Hand Gesture Recognition and Novel Sentence Interpretation Algorithm

for Indian Sign Language Using Microsoft Kinect Sensor: In this paper Archana S.
Ghotkar and Gajanan K. Kharate explored rule based and dynamic time warping (DTW)
based method to recognize ISL words.

SNDCOE&RC,Department of Computer Engineering,2021-22 3

 Chapter 3

SOFTWARE REQUIRMENT SPECIFICATION

3.1 Introduction

A Software Requirements Specification (SRS) is a document that describes the nature of a

project, software or application. In simple words, SRS document is a manual of a project
provided it is prepared before you kick-start a project/application. This document is also
known by the names SRS report, software document. A software document is primarily
prepared for a project, software or any kind of application. There are a set of guidelines
to be followed while preparing the software requirement specification document. This
includes the purpose, scope, functional and nonfunctional requirements, software and
hardware requirements of the project. In addition to this, it also contains the information
about environmental conditions required,safety and security requirements, software qual-
ity attributes of the project etc.

3.1.1 Project Scope

We are planning to achieve higher accuracy even in case of complex backgrounds by

trying out various background subtraction algorithms. We are also thinking of improving
the preprocessing to predict gestures in low light conditions with a higher accuracy.The
proposed sign language recognition system used to recognize sign language letters can
be further extended to recognize gestures facial expressions. Instead of displaying letter
labels it will be more appropriate to display sentences as more appropriate translation
of language.This also increases readability.The scope of different sign languages can be
increased. More training data can be added to detect the letter with more accuracy. This
project can further be extended to convert the signs to speech.

SNDCOE&RC,Department of Computer Engineering,2021-22 4

3.1.2 User Classes and Characteristics

User of the Sign Language Recognition system are either the non profitable organization
or the individuals with vocal disabilities. They have facility to translate and spread the
thoughts of the deaf and dumb peaople.

3.1.3 Assumptions and Dependencies

Let’s assume that this system used in the gatherings or seminars where every person with
vocal disability can give his/her valueble thoughts to all the person or audience. assuming
this point we have developed this system.

3.2 Functional Requirement

3.2.1 Image Processing System:

This system requires the real-time system which had ability to capture video and con-
vert them into frames and after that the system should convert the frames into respective
numpy array to proceed further.

3.2.2 Clasification System:

After gettiing the numpy arrays of frame the classification system required to apply the
CNN on arrays and analyze the data and classify the image and gie the correct output.

SNDCOE&RC, Department of computer Engineering 2021-22 5

3.3 External User Interface

3.3.1 User Interfaces

User has to interface with system to access the features and to provide easy communica-
tion with system.
Frontend Interface: Tkinter Graphical User interFace, KivyMD
Backend Interfqace: Python ML

3.3.2 Software Interfaces

Python Tkinter and Kivy Library provides the software interface for the user.

3.3.3 Hardware Interface

A good working webcam or an external webcam will require to capture thhe frame of the
user.

3.4 NonFunctional Requirements

3.4.1 Performance Requirements

Following measures are taken for improving the performance of the Sign Language Recog-
nision system.

SNDCOE&RC, Department of computer Engineering 2021-22 6

 Figure 3.1: Performance measures‘

True positive and true negatives are the observations that are correctly predicted and
therefore shown in green. We want to minimize false positives and false negatives so they
are shown in red color. These terms are a bit confusing. So let’s take each term one by
one and understand it fully.

True Positives (TP):These are the correctly predicted positive values which means that
the value of actual class is yes and the value of predicted class is also yes.

True Negatives (TN) -These are the correctly predicted negative values which means
that the value of actual class is no and value of predicted class is also no.

False Positives (FP) - When actual class is no and predicted class is yes.

False Negatives (FN) -When actual class is yes but predicted class in no.

Once you understand these four parameters then we can calculate Accuracy, Preci-
sion, Recall and F1 score.
Accuracy - Accuracy is the most intuitive performance measure and it is simply a ratio of
correctly predicted observation to the total observations. One may think that, if we have
high accuracy then our model is best. Yes, accuracy is a great measure but only when
you have symmetric datasets where values of false positive and false negatives are almost
same. Therefore, you have to look at other parameters to evaluate the performance of
your model. For our model, we have got 0.803 which means our model is approx. 80
percent accurate.

Accuracy = T P + T N/T P + F P + F N + T N

SNDCOE&RC, Department of computer Engineering 2021-22 7

Precision - Precision is the ratio of correctly predicted positive observations to the total
predicted positive observations. The question that this metric answer is of all passengers
that labeled as survived, how many actually survived? High precision relates to the low
false positive rate. We have got 0.788 precision which is pretty good.

P recision = T P/T P + F P

Recall (Sensitivity) - Recall is the ratio of correctly predicted positive observations to the
all observations in actual class - yes. The question recall answers is: Of all the passengers
that truly survived, how many did we label? We have got recall of 0.631 which is good
for this model as it’s above 0.5.

Recall = T P/T P + F N

F1 score - F1 Score is the weighted average of Precision and Recall. Therefore, this
score takes both false positives and false negatives into account. Intuitively it is not as
easy to understand as accuracy, but F1 is usually more useful than accuracy, especially
if you have an uneven class distribution. Accuracy works best if false positives and false
negatives have similar cost. If the cost of false positives and false negatives are very
different, it’s better to look at both Precision and Recall. In our case, F1 score is 0.701.

F 1Score = 2 ∗ (Recall ∗ P recision)/(Recall + P recision)

3.4.2 Safety Requirements

A good working Camera is the integral parat Sign Language Recognition system. So if
the damage camera damage occures then the system cam crash completely. So camera
must be good for the safety of the good working of the system.

SNDCOE&RC, Department of computer Engineering 2021-22 8

3.4.3 Security Requirements

There should be only one person in the camera frame at a time.

3.4.4 Software Quality Attributes

Software testing is a critical element of software quality assurance and represents the ul-
timate review of specification, design and coding. In fact, testing is the one step in the
software engineering process that could be viewed as destructive rather than constructive.
A strategy for software testing integrates software test case design methods into a well-
planned series of steps that result in the successful construction of software. Testing is
the set of activities that can be planned in advance and conducted systematically. The un-
derlying motivation of program testing is to affirm software quality with methods that can
economically and effectively apply to both strategic to both large and small-scale systems.

Correctness: The system should classify the langauge and predict output very quickly.
The low accuaracy can result in low reliability.

Usability: The system is very useful for the organization of the deaf and dumb peo-
ple. Very reliable for the making change in the life of the deaf and dumb people.

3.5 System Requirement

3.5.1 Software Requirements

• Visual Studio Code:

Visual Studio Code is a lightweight but powerful source code editor which runs
on your desktop and is available for Windows, macOS and Linux. It comes with
built-in support for JavaScript, TypeScript and Node.js and has a rich ecosystem of

SNDCOE&RC, Department of computer Engineering 2021-22 9

 extensions for other languages. Visual Studio Code is a code editor redefined and
optimized for building and debugging modern web and cloud applications.

• Python: Python is a computer programming language often used to build websites

and software, automate tasks, and conduct data analysis. Python is a general pur-
pose language, meaning it can be used to create a variety of different programs and
isn’t specialized for any specific problems.

Following packages are used for the development of the project

– NumPy: NumPy can be used to perform a wide variety of mathematical op-

erations on arrays. It adds powerful data structures to Python that guarantee
efficient calculations with arrays and matrices and it supplies an enormous
library of high-level mathematical functions that operate on these arrays and
matrices.

– Scikit-Learn: Scikit-learn (Sklearn) is the most useful and robust library

for machine learning in Python. It provides a selection of efficient tools for
machine learning and statistical modeling including classification, regression,
clustering and dimensionality reduction via a consistence interface in Python.

– Tensorflow: TensorFlow is an open-source software library. It was originally

developed by researchers and engineers working on the Google Brain Team
within Google’s Machine Intelligence research organization for the purposes
of conducting machine learning and deep neural networks research, but the
system is general enough to be applicable in a wide variety of other domains
as well!

• Pycharm IDE: PyCharm is an integrated development environment (IDE) used in

computer programming, specifically for the Python language. It is developed by the
Czech company JetBrains (formerly known as IntelliJ).It provides code analysis, a
graphical debugger, an integrated unit tester, integration with version control sys-
tems (VCSes), and supports web development with Django as well as data science

SNDCOE&RC, Department of computer Engineering 2021-22 10

 with Anaconda. PyCharm is cross-platform, with Windows, macOS and Linux ver-
sions. The Community Edition is released under the Apache License,and there is
also Professional Edition with extra features – released under a proprietary license.

3.5.2 Hardware Requirements

• Camera : min.720p

• Processor : min. intel core i3 or AMD Ryzen 3

• RAM : min 4gb

• HDD/SSD : 256gb

• Power Supply

3.6 Analysis Model: SDLC Model to be applied

Figure 3.2: SDLC Life Cycle

1. Planning : - This is the first phase in the systems development process. It identifies
whether or not there is the need for a new system to achieve a business”s strategic
objectives. This is a preliminary plan (or a feasibility study) for a company”s busi-
ness initiative to acquire the resources to build on an infrastructure to modify or
improve a service. The company might be trying to meet or exceed expectations

SNDCOE&RC, Department of computer Engineering 2021-22 11

 for their employees, customers and stakeholders too. The purpose of this step is to
find out the scope of the problem and determine solutions. Resources, costs, time,
benefits and other items should be considered at this stage.

2. Systems Analysis and Requirements :- The second phase is where businesses

will work on the source of their problem or the need for a change. In the event
of a problem, possible solutions are submitted and analyzed to identify the best fit
for the ultimate goal(s) of the project. This is where teams consider the functional
requirements of the project or solution. It is also where system analysis takes place
or analysing the needs of the end users to ensure the new system can meet their
expectations. Systems analysis is vital in determining what a business’s needs are,
as well as how they can be met, who will be responsible for individual pieces of the
project, and what sort of time line should be expected.

3. Systems Design :- The third phase describes, in detail, the necessary specifica-
tions, features and operations that will satisfy the functional requirements of the
proposed system which will be in place. This is the step for end users to discuss
and determine their specific business information needs for the proposed system.
It”s during this phase that they will consider the essential components (hardware
and/or software) structure (networking capabilities), processing and procedures for
the system to accomplish its objectives.

4. Development :-The fourth phase is when the real work begins in particular, when a
programmer, network engineer and/or database developer are brought on to do the
major work on the project. This work includes using a flow chart to ensure that the
process of the system is properly organized. The development phase marks the end
of the initial section of the process. Additionally, this phase signifies the start of
production. The development stage is also characterized by instillation and change.
Focusing on training can be a huge benefit during this phase.

5. Integration and testing:- The fifth phase involves systems integration and system
testing (of programs and procedures) normally carried out by a Quality Assurance
(QA) professional-to determine if the proposed design meets the initial set of busi-
ness goals. Testing may be repeated, specifically to check for errors, bugs and

SNDCOE&RC, Department of computer Engineering 2021-22 12

 interoperability. This testing will be performed until the end user finds it accept-
able. Another part of this phase is verification and validation, both of which will
help ensure the program’s successful completion.

6. Implementation :- The sixth phase is when the majority of the code for the pro-
gram is written. Additionally, this phase involves the actual installation of the
newly-developed system. This step puts the project into production by moving
the data and components from the old system and placing them in the new system
via a direct cut over. While this can be a risky (and complicated) move, the cut
over typically happens during off-peak hours, thus minimizing the risk. Both sys-
tem analysts and end-users should now see the realization of the project that has
implemented changes.

7. Operations and Maintenance :- The seventh and final phase involves maintenance
and regular required updates. This step is when end users can fine-tune the system,
if they wish, to boost performance, add new capabilities or meet additional user
requirements.

3.7 System Implementation Plan

The stages described in the guide include:

• exploration

• installation

• initial implementation

• full implementation

• expansion and scale-up

SNDCOE&RC, Department of computer Engineering 2021-22 13

 Figure 3.3: SLR- System Architecture

SNDCOE&RC, Department of computer Engineering 2021-22 14

 Chapter 4

SYSTEM DESIGN

4.1 System Architecture

Figure 4.1: SLR- System Architecture

4.2 Data Flow Diagram

4.2.1 Data Flow Level-0

• DFD Level 0 : A level 0 data flow diagram (DFD), also known as a context di-
agram, shows a data system as a whole and emphasizes the way it interacts with
external entities. This DFD level 0 example shows how such a system might func-
tion within a typical retail business.

SNDCOE&RC,Department of Computer Engineering,2021-22 15

 Figure 4.2: DFD-Level 0

4.2.2 Data Flow Level-1

• DFD Level 1 : A level 1 data flow diagram (DFD) is more detailed than a level 0
DFD but not as detailed as a level 2 DFD. It breaks down the main processes into
sub processes that can then be analyzed and improved on a more intimate level.

Figure 4.3: DFD-Level 1

4.2.3 Data Flow Level-2

• DFD Level 2 : A level 2 data flow diagram (DFD) offers a more detailed look at
the processes that make up an information system than a level 1 DFD does. It can

SNDCOE&RC, Department of computer Engineering 2021-22 16

be used to plan or record the specific makeup of a system. You can then input the
particulars of your own system.

Figure 4.4: DFD-Level 2

SNDCOE&RC, Department of computer Engineering 2021-22 17

4.3 UML Diagrams

4.3.1 Sequence Diagram

• Sequence Diagram : A sequence diagram shows object interactions arranged

in time sequence. It depicts the objects and classes involved in the scenario and
the sequence of messages exchanged between the objects needed to carry out the
functionality of the scenario.

Figure 4.5: Sequence Diagram

SNDCOE&RC, Department of computer Engineering 2021-22 18

4.4 Class Diagram

• Class Diagram : The class diagram is the main building block of object-oriented
modeling. It is used for general conceptual modeling of the structure of the appli-
cation, and for detailed modeling translating the models into programming code.
Class diagrams can also be used for data modeling.

Figure 4.6: Class Diagram

SNDCOE&RC, Department of computer Engineering 2021-22 19

4.5 Activity Diagram

• Activity Diagram : Activity diagram is another important behavioral diagram

in UML diagram to describe dynamic aspects of the system. Activity diagram is
essentially an advanced version of flow chart that modeling the flow from one ac-
tivity to another activity. Activity Diagrams describe how activities are coordinated
to provide a service which can be at different levels of abstraction.

Figure 4.7: Activity Diagram

SNDCOE&RC, Department of computer Engineering 2021-22 20

4.6 Use Case Diagram

• Use Case Diagram : A UML use case diagram is the primary form of system/software
requirements for a new software program underdeveloped. Use cases specify the
expected behavior (what), and not the exact method of making it happen (how).
Use cases once specified can be denoted both textual and visual representation (i.e.
use case diagram). A key concept of use case modeling is that it helps us design a
system from the end user’s perspective.

Figure 4.8: Use Case

SNDCOE&RC, Department of computer Engineering 2021-22 21

 Chapter 5

Other Specification

5.0.1 Advantages

• Normal people no need to learn sign language he/she talk by using sign language
recognizer.

• Dumb deaf people interact with any people

• Easy to use.

5.0.2 Limitations

• The image shoulb must clean and clear.

• noisy data not allowed.

5.0.3 Applications

• In colleges for deaf people

• In seminar hall for deaf people

SNDCOE&RC,Department of Computer Engineering,2021-22 22

 Chapter 6

Conclusions

Conclusion

In this work, we presented a vision-based system to interpret isolated hand gestures

from the sign language. This work used two different approaches to classify on the spatial
and temporal features. CNN was used to classify on the spatial features, whereas RNN
was used to classify on the temporal features. We obtained an accuracy of 95.217that
CNN along with RNN can be successfully used to learn spatial and temporal features and
classify sign language gesture videos.

SNDCOE&RC,Department of Computer Engineering,2021-22 23

 APPENDIX A

SNDCOE&RC, Department of computer Engineering 2021-22 24

 Project problem : In the world of technology, everything and everyone in the
world is growing so rapidly, so we think disability of deaf people should not be
barrier in the growth of deaf people. As technology is changing world so rapidly,
so technology can also change the life of such people, artificial intelligence can
play major role in improving communication of such people.

Objctive: Breaching communication gap between disable people and normal people. •
To create the software which will help deaf people to communicate with other people. •
Making complete use of artificial intelligence and latest technologies to make human life
easier.

Introduction

Artificial Intelligence: Artificial intelligence (AI) is the ability of a computer or a robot

controlled by a computer to do tasks that are usually done by humans because they re-
quire human intelligence and discernment.
Neural Network: A neural network is a series of algorithms that endeavors to recognize
underlying relationships in a set of data through a process that mimics the way the human
brain operates. In this sense, neural networks refer to systems of neurons, either organic
or artificial in nature.

Methodology and Algorithm:

1. Input Image: OPENCV has features which are used to perform various operations
and hence convert the hand images and show the required results.

1.1 Camera module: This module is subject for interfacing and capturing input through
the different sorts of picture markers and sends this picture to the detection module for
handling as frames. The generally utilized techniques of capturing and recognizing input
are hand belts, data gloves and cameras. In our framework, we use the inbuilt webcam
which is financially savvy to see both static and dynamic signs.

SNDCOE&RC, Department of computer Engineering 2021-22 25

1.2 Detection Module: This module is liable for the image processing. The output from

SNDCOE&RC, Department of computer Engineering 2021-22 26

camera module is presented to different image handling methods, for instance, color con-
version, noise removal, thresholding following which the image goes through contour
extraction.[9] In the event that the image contains defects, at that point convexity defects
are found by which the gesture is identified. In the event that there are no defects, at that
point the image is classified utilizing Har cascade to recognize the gesture.

Convolutional Neural Network (CNN): Neural networks, as its name suggests, is a

machine learning technique which is modeled after the brain structure. It comprises of
a network of learning units called neurons. These neurons learn how to convert input
signals (e.g., picture of a cat) into corresponding output signals (e.g., the label “cat”),
forming the basis of automated recognition. convolutional neural network (CNN, or Con-
vent) is a type of feedforward artificial neural network in which the connectivity pattern
between its neuron is inspired by organization of the animal visual cortex.

1. Convolution:- The first layers that receive an input signal are called convolution
filters. Convolution is a process where the network tries to label the input signal by
referring to what it has learned in the past. If the input signal looks like previous
cat images it has seen before, the “cat” reference signal will be mixed into, or
convolved with, the input signal. The resulting output signal is then passed on to
the next layer.

2. Subsampling: Inputs from the convolution layer can be “smoothened” to reduce

the sensitivity of the filters to noise and variations. This smoothing process is called
subsampling, and can be achieved by taking averages or taking the maximum over a
sample of the signal. Examples of subsampling methods (for image signals) include
reducing the size of the image, or reducing the color contrast across red, green, blue
(RGB) channels.

3. Pooling: Its function is to progressively reduce the spatial size of the representa-
tion to reduce the number of parameters and computation in the network. Pooling
layer operates on each feature map independently .The most common approach
used in pooling is max pooling in which maximum of a region taken as its repre-
sentative. For example, in the following diagram a 2x2 region is replaced by the

SNDCOE&RC, Department of computer Engineering 2021-22 27

 maximum value in it.

4. Activation: The activation layer controls how the signal flows from one layer
to the next, emulating how neurons are fired in our brain. Output signals which
are strongly associated with past references would activate more neurons, enabling
signals to be propagated more efficiently for identification. CNN is compatible with
a wide variety of complex activation functions to model signal propagation, the
most common function being the Rectified Linear Unit (ReLU), which is favored
for its faster training speed.

ReLU Activation Function: The Rectified Linear Unit is the most commonly
used activation function in deep learning models. The function returns 0 if it re-
ceives any negative input, but for any positive value xx it returns that value back.
So, it can be written as

f (x) = max(0, x)

Figure 6.1: ReLU Activation

SNDCOE&RC, Department of computer Engineering 2021-22 28

 APPENDIX B

SNDCOE&RC, Department of computer Engineering 2021-22 29

• Joyeeta Singha and Karen Das described a novel approach to recognize alphabets of
ISL. An eigenvector-based technique is used for feature extraction, and eigenvalue
weighted Euclidean distance technique is used for Classification of 24 different ISL
alphabets

• Archana S. Ghotkar and Gajanan K. Kharate explored rule-based and dynamic time
warping (DTW) based method to recognize ISL words. Their experiments proved
that the performance of DTW is very much higher for continuous word recognition.

• M.K. Bhuyan segments frame into video object planes (VOPs), to obtain a seman-
tically meaningful hand position. Key VOPs and temporal information are tracked
to form a complete gesture sequence. The test results concluded that by using
keyframes, a gesture could be uniquely represented as a finite state machine with
keyframes and corresponding frame duration as states.

• M.M.Gharasuie and H.Seyedarabi proposed a system that recognizes the real-time

hand gestures of numbers from 0 to 9 using Hidden Markov Models (HMMs).In
this proposed system, preprocessing and tracking steps took place in hand trajectory
extraction phase and feature extraction is taking place in the classification phases.
The system is capable of providing 93 precent recognition rate.

SNDCOE&RC, Department of computer Engineering 2021-22 30

 Bibliography

1. Geethu G nath and arun C S, ”real time sign language interpreter,” 2017 interna-
tional conference on electrical, instrumentation, and communication engineering
(ICEICE2017).

2. Kumud tripathi, neha baranwal and G. C. Nandi, ”continuous indian sign language
gesture recognition and sentence formation”, eleventh international multiconfer-
ence on information processing2015 (IMCIP-2015), procedia computer science 54
(2015) 523 – 531.

3. Manasa srinivasa h s and suresha h s, ”implementation of real time hand gesture

recognition,” international journal of innovative research in computer and commu-
nication engineering, vol. 3, issue 5, may 2015.

4. Joyeeta Singha and Karen Das, ”Automatic Indian Sign Language Recognition
for Continuous Video Sequence,” ADBU Journal of Engineering Technology 2015
Volume 2 Issue 1.

5. Archana S. Ghotkar and Gajanan K. Kharate, ”Dynamic Hand Gesture Recogni-

tion and Novel Sentence Interpretation Algorithm for Indian Sign Language Using
Microsoft Kinect Sensor,” Journal of Pattern Recognition Research 1 (2015) 24-38.

6. M.K. Bhuyan, ”FSM-based recognition of dynamic hand gestures via gesture sum-
marization using key video object planes,” World Academy of Science, Engineering
and Technology Vol: 6 2012-08-23.

7. M.M.Gharasuie and H.Seyedarabi, ”Real-time Dynamic Hand Gesture Recognition

using Hidden Markov Models,” 2013 8th Iranian Conference on Machine Vision

SNDCOE&RC, Department of computer Engineering 2021-22 31

 and Image Processing (MVIP).

8. Kairong Wang, Bingjia Xiao, Jinyao Xia, and Dan Li, ”A Dynamic Hand Gesture
Recognition Algorithm Using Codebook Model and Spatial Moments,” 2015 7th
International Conference on Intelligent Human-Machine Systems and Cybernetics.

9. Francke H., Ruiz-del-Solar J. and Verschae R., ”Real-Time Hand Gesture Detec-
tion and Recognition Using Boosted Classifiers and Active Learning,” Advances in
Image and Video Technology. PSIVT 2007. Lecture Notes in Computer Science,
vol 4872. Springer, Berlin, Heidelberg.

10. Hari Prabhat Gupta, Haresh S Chudgar, Siddhartha Mukherjee, Tanima Dutta, and
Kulwant Sharma, ”A Continuous Hand Gestures Recognition Technique for Human-
Machine Interaction using Accelerometer and Gyroscope sensors,” IEEE Sensors
Journal (Volume: 16, Issue: 16, Aug.15, 2016) Page(s): 6425 – 6432.

SNDCOE&RC, Department of computer Engineering 2021-22 32