Sign Language Recognition Using Machine Learning Algorithm

SIGN LANGUAGE RECOGNITION USING MACHINE

LEARNING ALGORITHM

A Project report submitted in a partial fulfillment of the requirements for the award of the
degree of

BACHELOR OF TECHNOLOGY

IN

COMPUTER SCIENCE AND ENGINEERING

Submitted by

M Sai Manasa 321710302032

C.M. Mokshitha 321710302033

N Anil Kumar 321710302034

K Devi Charan 321710302022

Under the esteemed guidance of

Mrs.Mala B M

Assistant Professor

DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

GITAM School of Technology

(Deemed to be University)

BENGALURU

2020-21

Dept of CSE 1 2020-21

Sign Language Recognition Using Machine Learning Algorithm

DECLARATION

I/We, hereby declare that the project report entitled “SIGN LANGUAGE RECOGITION
USING MACHINE LEARNING ALGORITHM” is an original work done in the
Department of Computer Science and Engineering. GITAM School of Technology. GITAM
(Deemed to be University) submitted in partial fulfillment of the requirements for the award of
the degree of B.Tech in Computer Science and Engineering. The work has not been submitted
to any other college or University for the award of any degree or diploma.

Date: 28-01-2021

321710302032 M Sai Manasa

321710302033 C.M.Mokshitha

321710302034 N Anil Kumar

321710302022 K Devi Charan

Dept of CSE 2 2020-21

Sign Language Recognition Using Machine Learning Algorithm

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

GITAM SCHOOL OF TECHNOLOGY

GITAM

(Deemed to be University)

CERTIFICATE

This is to certify that the project report entitled “SIGN LANGUAGE RECOGNITION
USING MACHINE LEARNING ALGORITHM” is a bonafide record of work carried out
by M Sai Manasa (321710302032), C.M.Mokshitha (321710302033), N Anil Kumar
(321710302034), K Devi Charan (321710302022) students submitted in partial fulfilment of
requirement for the award of degree of Bachelors of Technology in Computer Science and
Engineering.

Project Guide Head of the Department

Mrs.Mala B M Vamshidhar Yendapalli

Assistant Professor Professor

Dept of CSE 3 2020-21

Sign Language Recognition Using Machine Learning Algorithm

ACKNOWLEDGEMENT

We had been able to complete our project successfully. However, it would not have been possible
without the kind support and help of many individuals. We would like to extend our sincere
thanks to all of them. Thanksgiving will not end without thanking the parents where even the
word Thanks is not enough for the sacrifices they made. I would like to thank my parents for
everything they have given and have made my path easier and to reach this point. We are highly
indebted to GITAM (Deemed to be University), Bangalore for their guidance and constant
supervision as well as for providing necessary information regarding the project and also for their
support in completing the project. Would like to express our gratitude towards Prof. S Dinesh
(Director of GST), Prof. Vamsidhar Y (HOD of CSE, GST), Mrs. Mala B M (Assistant Professor,
CSE, GST) and also to all the other supporting faculty and staff for their kind co-operation and
encouragement which helped us in the completion of this project.

Student Name’s Registration No.

M Sai Manasa 321710302032

C.M.Mokshitha 321710302033

N Anil Kumar 321710302034

K Devi Charan 321710302022

Dept of CSE 4 2020-21

Sign Language Recognition Using Machine Learning Algorithm

ABSTRACT

Since most people don't understand sign language and interpreters are hard to come by, we've
devised a real-time approach that employs neural networks. We tend to produce a symbol detector
that detects text via hand movements during this signing recognition project. We usually try to
analyse and recognise a variety of terms or sentences from a database of sign pictures. We often
try to investigate various machine learning techniques, such as Convolution Neural Networks
(CNN). One of the ways to communicate with deaf people is by sign language. During this work
collection, the enclosed choices and variance within the language with the neck of the woods are
the most significant obstacles that have led to lead to very little research being worn out ISL. To
interact with them, one should learn to sign. In certain cases, learning takes place in peer groups.
There are only a few research materials available for learning signs. As a result, learning to sign
learn can be a difficult task. Finger spelled sign learning is the first stage of sign learning which
is used when no equivalent sign occurs or the signer isn't paying attention. The majority of the
current tools for signing learning depend on expensive external sensors. Our project aims to
expand the field of exploration by collecting a dataset and using various feature extraction
techniques to extract useful data, which is then fed into various supervised learning techniques.

Dept of CSE 5 2020-21

Sign Language Recognition Using Machine Learning Algorithm

TABLE OF CONTENTS

Titles Page no

Declaration 2
Certificate 3
Acknowledgement 4
Abstract 5
Table of contents 6
List of Figures 8
1. Introduction 9
1.1 Sign Language Recognition
1.2 Solution implemented in the project
1.3 Justification of the project selected
2. Literature review 11
3. Problem Statement 13
3.1 Objectives
4. Designing 14
4.1 Feasibility study
4.1.1 Performance analysis
4.1.2 Technical analysis
4.1.3 Economical analysis
4.2 Development Methods
5. System Methodology 16
5.1 Dataset
5.2 Flow diagram
5.3 System Architecture
5.4 Sequence diagram
5.5 System requirements
5.5.1 Hardware requirements
5.5.2 Software requirements

6. Overview of Technology 22
6.1 Convolutional neural network (CNN)
6.2 Types of layers

Dept of CSE 6 2020-21

Sign Language Recognition Using Machine Learning Algorithm

6.3 Pre-training a CNN model

6.4 Software tools
6.4.1 Python
6.4.2 Python IDLE
7. Implementation 28
7.1 Packages
7.2 Functions
8. Experimental Results 37
8.1 Screenshots of output
8.2 Result
8.3 Discussion
9. Conclusion 39
9.1 Future scope
References 41

Dept of CSE 7 2020-21

Sign Language Recognition Using Machine Learning Algorithm

LIST OF FIGURES
Title Page no

1. Dataset 18
2. Flow diagram 18
3. Architecture diagram 29
4. Sequence Diagram 20
5. CNN vs Regular neural network 23
6. Fully connected layer inside CNN 25
7. Code(i) 28
8. Code(ii) 29
9. Code(iii) 30
10. Code(iv) 31
11. Code(v) 32
12. Code(vi) 32
13. Code(vii) 33
14. Output(i) 37
15. Output(ii) 37
16. Output(iii) 38
17. Output(iv) 38
18. Output(v) 38
19. Output(vi) 39
20. Output(vii) 39

Dept of CSE 8 2020-21

Sign Language Recognition Using Machine Learning Algorithm

CHAPTER 1
INTRODUCTION

Sign language is a key tool in order to survive in society for hearing and handicapped persons.
Linguistic communication minimises discomfort through visual speech instead of hearing in their
day-to-day lives. Communication is extremely important for men, as it allows the US to
determine itself. Speech is one of the most unknottable things used among them, which is
communicating through speech, gesture, visual communication, reading, writing or visual
support.
Unfortunately, however, there is a communication gap for the speech and hard-to-hear minority.
Area unit used for human activity with the visual aids, or interpreters of AN. These ways,
however, unit area rather cumbersome and expensive, and cannot be used in an emergency.
Linguistic communications mainly transmit what means by manual communication. This
combines the hand forms, directions and the movement of the hands with the thoughts of the
speaker. They are much less varied than spoken languages. India has its own language
communication under the Indian language communications name (ISL). Only a few colleges for
deaf students can be found in developing countries. Government rates for adults with surface
areas in developing countries are terribly high. Ethnologue's information says that the overall
population, performance rate and variety of teaching children among the deaf people in India are
incredibly smaller. It continues to state that official recognition of sign languages increases
significantly the availability of interpreters and transcripts in sign languages. Signs in the area of
sign languages unit word equivalent Languages that have been signed appear favourable.
The area of sign languages has mainly been developed to assist deaf and dumb people. They use
a combination of hand movements, hand forms, and orientation to provide explicit information
in a chronical and specific way. One of the languages used in Southeast Asia is the Indian
linguistic communication (ISL) system. The boundary between ISL and alternative sign
languages is that ISL empties any temporal inflections in the fingerprint system chart, as well as
the use of each hand.
With the presence of unnaturally smart algorithms, not to mention the provision of massive
information and enormous procedural resources, LEDs have increased considerably in tending,
robotics, automotive, HLI, etc.
The detection of hand gestures could be a hard task, significantly because of its use of each hand
the ISL recognition is difficult. During this respect, however, several studies are conducted with
the victimisation sensors (such as a glove sensor) and alternative imaging methods (such as edge

Dept of CSE 9 2020-21

Sign Language Recognition Using Machine Learning Algorithm

detection technique, etc.). But the performances in this field have increased considerably with
the new deep learning techniques like CNN which led to many new perspectives for the future.
Many people in India have discomfort in speech and/or hearing and use hand gestures to talk to
others. But, apart from a few, most are not attentive to this linguistic communication and may
need a non-convenient and expensive an interpreter. This project seeks to narrow this
communication gap by developing a package that can predict the alphanumeric gestures of the
ISL hand in real time.

1.1 SIGN LANGUAGE RECOGNITION

The language of signs may be a visual language used by people with disabilities. It makes it easier
for people to communicate. Because of the lack of signature recognition, it is of great social
importance for people with hearing impairments to develop a human-computer interaction system
that offers a platform to speak to others. The language of the sign is that with completely different
gestures of hand, the only medium that especially capable individuals can connect to the rest of
the world. The recognition of gesticulations has become extremely important with the progress
made in machine learning.

1.2 SOLUTION IMPLEMENTED IN THE PROJECT

The input aspect a video camera employs or with integrated image processing system allows the
calculation of vectors options for capturing an imagery to overcome the inconveniences of the
existing system. The process of segmentation takes place compared to images on the train sign
recognition, sign can realise a proof language interface to alphabet, which is converted to
completely different classifiers and the accuracy is high and user friendly, the main benefit of
this project is that the user can't wear anything and that we can use the integrated camera.
• The goal is to create a simple human interface wherever the laptop perceives human
signature
• This uses completely different classifiers to convert a text to a sign language
• Results square measure in terms of maximum precision

1.3 JUSTIFICATION OF THE PROJECT SELECTED

From time immemorial communication has been the means for building relationships,
understanding individuals, perceiving technology, and allowing the world to grow and develop.
Traditional people communicate their ideas and thoughts through speech to others. One important
element is that the use of signing is one of communication methodologies for the deaf community.
The people's lakhs use signing as their principal means of communication.

Dept of CSE 10 2020-21

Sign Language Recognition Using Machine Learning Algorithm

CHAPTER 2
LITERATURE REVIEW

The section which shows the various analyses and analyses created in the fields of interests and
also the already revealed results, taking into compete the different project parameters and also
the extent of the project is a literature study or literature review in the very project report.

MS Bhavsar, Dr Jeegar Trivedi [1]'s proposed framework defines and categorises the hand area
from the image contour and separates the handshape from the skin with skin colour function
using the Neuro-Fuzzy Classifier. The result after running the testing database is 92 percent.

Kusuma Krori Dutta, Sunny Arokia Swamy Bellary [2] proposed system, Indian sign language
classification by machine learning. A Principal Component Analysis (PCA) and Artificial neural
network (ANN) algorithms in MATLAB provide the system with dual-handed sign language
training. The single- and two-handed Indian language of the sign is acquired and classified by
means of KNN and Back propagation methods for English letters and numbers, where PCA is
used to reduce dimension. For K-NN techniques with K=1, 100% of the recognition rate was
achieved, while 94-96% was achieved with the back propagation technique.

Yangho Ji, Sunmok Kim, and Ki-Bak Le [3] are developing their new system of learning sign
language based on 2D image sampling and concatenating to solve problems with conventional
sign recognition. In this system the data sampled from the sign language demonstration video
were learned from a personal convolutionary neural network. 86% of the final accuracy of the
test.

Xinyun Jiang, Wasim Ahmad [4] has designed a new sign language education system for the
resolution of conventional sign recognition based on 2D image sample and concatenation. In this
system, a personalised convolutional neural network has learned the data from the demonstration
video of the sign language sample.

Mekala Priyanka Gao Ying Jeffrey FanAsadDaivar [5] requires at least one computer 1 GHz
processor and 256 MB free RAM for this sign language recognition approach. All A through Z
alphabets are included in the training set (26 patterns). Mr. Kishore, Mr. Prasad,

Dept of CSE 11 2020-21

Sign Language Recognition Using Machine Learning Algorithm

Mr. Kiran Kumar & D. Anil Kumar [6] has shown sign language identification strategies in the
Indian language. Based on the 2 frail-edge data framework wavelet combination. One is the
subtraction model of graphical-based morphs, while the other is an administrator of shrewd
angles. Descriptors based on Fourier curved give form components using a streamlined number
of shape parameters. Standard parts have chosen to discover the component vector to a basis for
the record clip grouping for each case. The neural network with the back propagation algorithm
finishes the grouping of the sign markers. The rate of recognition is still 92.34%. Recognition of
dynamic hand motion

Mengyi Xie,XinMa*[7] in this work sign language dataset consists of 36 static sign language
words including number and letters, the project uses different pre-processing techniques and
different classifiers like SVM which gives around 95% accuracy when we performed .This paper
is different from traditional methods which are used before. The identification is end to end
framework which provides convenient method of saving, reduces the loss and improves the
accuracy.

Leela Surya Teja Mangamuri, Lakshay Jain, Abhishek sharmay [8],In this work we use Indian
sign language dataset validation on various grading models of different accuracy using different
pre-processing techniques and dilution methods to minimise noise and unwanted backgrounds.
The accuracy is calculated and it is around 91%.

Chen[9] ,This paper examines various segmentation and non-controlled learning algorithms to
create an accurate model for sign language recognition. We have tried with just up to 10 different
classes/letters in our self-made dataset instead of to deal with this problem and achieve reasonable
results. Every 26 letters possible. A randomly selected set of test data using our trained model
have achieved a classification accuracy of 98%.

Dept of CSE 12 2020-21

Sign Language Recognition Using Machine Learning Algorithm

CHAPTER 3
PROBLEM STATEMENT
Understanding the exact context of the symbolic manifestations of deaf and stupid is the
difficult task in actual life until it is correctly specified.

A model which can recognise the gestures of hands using profound learning techniques. This
project predicts the text in symbolic terms.

The main context of this project is the symbolic expression of sorry and stupid people, which
is really challenging work in daily life. Thus, this project converts symbolic expression, like
text format, into understandable human language. With the use of algorithms such as CNN. We
intend to help deaf and dumb people communicate behind the project with others using this
sign language

3.1 OBJECTIVE
Our project aims to bridge the gap between people with speech impairment and ordinary
individuals. The main idea for this project is to create a system that stupid people can use
their ordinary gestures to communicate significantly with all others.
SLR aims at developing algorithms and methods for properly identifying and understanding
a sequence of the signs produced. Many methods of SLR treat the problem correctly as
gesture recognition.
The goal is to develop an easy-to-use interface to understand the language of human signs.
The interface uses different classifiers to convert the sign language into text The results are
shown in the highest precision

➢ The main purpose of this project is to thoroughly identify hand gestures, images in order
to achieve a communication gap between an individual with a normal hearing impairment
➢ The main aim of this project was to establish the usability of the system for gesture
recognition
➢ Pre-processing the extracted data to eliminate unnecessary information (like removing
any unnecessary information and keeping the required database
➢ The algorithm with different features will be trained to produce the correct output
➢ And spell correction of a word will be generated
➢ And we also add additional feature like text to speech conversion

Dept of CSE 13 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

CHAPTER 4

DESIGNING

Provided an endless number of resources and time, every device is feasible. Unfortunately, in the
real world, this condition does not exist. As a result, it's both important and wise to assess the
system's viability as soon as possible. If an unwell developed structure is identified early within
the description part, months or years of effort, thousands of rupees, and a great deal of professional
embarrassment can be avoided. In several ways, practicability and risk analysis are related. In cases
of a low project risk, a high-quality package can be produced feasibly. In this situation, there are
three key issues of concern.

4.1 FEASIBILITY STUDY

When given limitless resources and endless time, all systems are feasible. Unfortunately, in the
real world, this situation does not exist. As a result, assessing the system's viability as soon as
possible is both appropriate and wise. If the early definition of an impermeable system, months or
years of work are identified, thousands of rupees and unspeakable professional humiliations can
be avoided. In several cases, feasibility and risk analysis are related. When project risk is high, the
likelihood of delivering high-quality applications is lowered. There are three main fields of concern
in this situation.

4.1.1 Performance Analysis

The project is run with the help of a safe networking setup for the entire practicality of the project
work. The operating system is typically Windows. The main goal of this initiative is to help deaf
and dumb people communicate with others. The aim of performance analysis is to retrieve
information in a safe and reliable manner. It is important that the performance measurement and
definition methods be carried out in parallel.

4.1.2 Technical Analysis

The technical requirements of the company Simply stated, this check of practicability determines
whether the scheme will succeed until designed and implemented, as well as whether there are any
significant obstacles to implementation. There are some things to consider in relation to these
issues in technical analysis:

Dept of CSE 14 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

• Changes to the method to be implemented: All improvements should be in a constructive

direction, resulting in a higher degree of quality and customer support.
• Required abilities: This project's platforms and technologies are commonly used. As a
result, the professional workforce is immediately available inside the business.
• Acceptability: The system's configuration is unbroken enough that there shouldn't be any
problems.

4.1.3 Economical Analysis

Economical analysis is used to assess the importance of growth in comparison to the ultimate
financial gain or benefits resulting from the developed environment. We don't want to use high-
performance servers to operate this technique. Package modules are used to enforce all of the
features. We don't seem to be using any physical equipment for affiliation during this system. That
the scheme is viable from a financial standpoint.

4.2 DEVELOPMENT METHODS

The development of software is usually a process step by step. It involves the software design
process before the implementation process. A software design is a description of the component
structure, the data that is part of the system, the interfaces between the components and the
algorithms used sometimes. Designers do not immediately achieve a finished design but iteratively
develop the design in a number of different versions. The design process involves adding
formalities and details in the development of the design with constant retrofitting to correct
previous designs.
Software design is an ad hoc process in many software development projects. An informal design
is prepared based on the set of requirements, usually in natural language. The coding starts and
when the system is implemented the design stage will be modified. When the stage of
implementation is complete, the design usually has changed from the initial specification to an
incorrect and incomplete description of the system by the original design document. The design
phase offers various advantages. Some of the following are listed:
• The design stage helps user needs understand and helps user demands to be mapped into
the implementation stage.
• Design iterations help to incorporate in the final software as many user requirements as
possible.
Different design options need to be considered at each stage during the design process. Design
phase aims at producing the software's overall design.

Dept of CSE 15 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

CHAPTER 5
SYSTEM METHODOLOGY
The aim is to provide a simple human interface that can be used anywhere that ce transforms a sign
language to text. The results are categorised according to the maximum level of precision.
The system's goal was to recognise hand gestures for linguistic contact.
As seen in the diagram, the system of gesture recognition is usually divided into four stages: data
collection, pre-processing, feature extraction, and classification. Data collection, data pre-
processing, feature extraction, and gesture classification are all stages in the expected scheme.
➢ Data Acquisition: An OpenCV and a regular USB camera are connected to a PC with a
viewing SLR system and no auxiliary objects(such a gloves are need to collect the data.
➢ Data Pre-processing: The aim of data pre-processing is to remove as much unnecessary
context as possible while still leaving sufficiently useful linguistic contact hand gesture
information. Hand segmentation, image cropping, reorientation, and normalisation are the
four stages that make up this level.

Canny edge detection:

The algorithmic software detection of a large range of edges is used in images canny edge detection
perform cv2 is a built-in function in OpenCV. Canny () is a function that takes our input image as
the first argument and its aperture size (min and max values) as the second and third arguments.

Erosion and Dilation:

Erosion and Dilation are the most basic morphological operations.

They need a wide range of applications, such as:

✓ Getting rid of noise

✓ Finding intensity bumps or gaps in a picture, as well as isolating individual parts and
connecting disparate parts in a picture.

A collection of operations for displaying supported shapes in a system. Morphological operations

produce an output image by applying a structuring component to an input image. As applied to
Gray-scale or binary images, dilation and erosion are simple morphological process operations that
provide contrasting effects.

Erosion:

Dept of CSE 16 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Erosion is a morphological process that aims to reduce the size of the foreground object.It's similar
to eroding, except it erodes the foreground object's boundary.In this operation, the kernel is passed
through the image and only considers pixels with a price of one. It's scoured otherwise. The pixels
near the boundary are discarded in this process, resulting in a shrunken foreground object in the
picture.

Dilation:

Dilation is the polar opposite of abrasion; rather than shrinking the foreground object, it extends
it. The structuring factor (kernel) falls through the picture during this process. However, a
component has a price of one if at least one other component has a price of one. As a result, the
object extends along the boundary, resulting in an enlarged image.

➢ Feature Extraction: When recognising gestures, the options extracted should include
relevant information from the input of gestures, which is the AN identity to be classified
separately from different gestures. Feature Extraction aims to scale back the amount of
options in an exceedingly dataset by making new options from the prevailing ones (and
then discarding the first features).

➢ Gesture Classification: During this Classification method we have a tendency to apply

some classification algorithms to extract sophisticated options.

In the Hand Gesture Classification, a dataset that contains pictures of different gestures
of the hand, like a hand, palm, thumb, and others, which may also be used in this guide
to display letters.

There are many algorithms available in machine learning, of which we use the
Convolution Neural Network (CNN), as discussed above, with the support of Keras ( a
library in Python written on the open-source neural network

Keras is quick and easy and continues to support CNN too though, trying to run
seamlessly on both CPU and GPU.

Dept of CSE 17 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

5.1 DATASET

Fig 1: Dataset

The dataset consists of Sign Language Hand Gestures containing 26 signs with different variations
Each sample consists of a person signing the corresponding letter while facing directly at the
camera

5.2 FLOW DIAGRAM

Data Data pre Feature

acquisition processing extraction

Classification

Hand
recognition

Fig 2: Flow diagram

Dept of CSE 18 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

• The dataset for sign language is taken and loaded.

• The data were pre-processed to eliminate unnecessary background and noise from the
dataset.
• The model is built using machine learning algorithms such as Convolutional neural
networks.
• The model is trained on pre-processed data, and the CNN algorithm is used to recognise
sign language.

5.3 SYSTEM ARCHITECTURE

Sign language Dataset

Train Data Test Data

Train the Model

Test the Model Model Evaluation

Recognising sign

User Input Final Model

Output

Fig 3: Architecture diagram

Dept of CSE 19 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

• Taking and loading the data set for the sign language.
• In order to remove unnecessary background and noise from the data set, the data were pre-
processed.
• The model is made using algorithms such as the neural networks of Convolution.
• The model is pre-worked and used to recognise sign language with the CNN algorithm.

5.3 SEQUENCE DIAGRAM

Fig 4: Sequence diagram

Dept of CSE 20 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Sequence Diagram
In Sign Language Recognition data set is taken and loaded the data set consists of 26 signs which
represents 26 different alphabets and word for each letter. The dataset is pre-processed in order
reduce unwanted background and noise in the dataset taken. Machine learning algorithms such as
Convolutional neural network is used to construct the model. Keras is recommended because it
increases deep-neural network architecture by providing certain optimization functions like
backpropagation, gradient descent, etc. The model is trained using pre-processed information and
CNN algorithms recognise the sign language.

5.4 SYSTEM REQUIREMENTS

5.4.1 HARDWARE REQUIREMENTS

Hardware requirements for the implementation and testing the project are mentioned as
below
➢ System : Pentium IV 2.4 GHz.
➢ Hard Disk : 500 GB.
➢ Ram : 4 GB.
➢ Any desktop / Laptop system with above configuration or higher level.

5.4.2 SOFTWARE REQUIREMENTS

Software requirements for the implementation and testing the project are mentioned as below

➢ Operating system : Windows 10

➢ Coding Language : Python
➢ Interpreter : Python 3.6
➢ IDE : Python IDE
➢ ML APIS : Sklearn, numpy, pandas,
matplotlib, keras, tensorflow

Dept of CSE 21 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

CHAPTER 6

OVERVIEW OF TECHNOLOGY

6.1 CONVULATIONAL NEURAL NETWORK (CNN)

A CNN is a deep-learn algorithmic programme that can absorb the input image, allocate weights
and biases in different aspects/objects in the image, and distinguish between them. The CNN is an
algorithmic programme of deep learning. The pre-processing requirement of a ConvNet is
significantly less than other classification algorithms. While filters are traditionally manufactured
by hand, ConvNets can learn these features with sufficient training.
The visual field design impressed the ConvNet design, which is similar to the pattern of Neurons
in the Human Brain. Only a small area of the visual field called the receptive field is used to
respond to stimuli by individual neurons. A group of such fields overlap the whole cortical region.

By using relevant filters, an image of abstraction and temporal dependencies can be captured
successfully in a ConvNet. The design is more robustly aligned with the image data set because of
the reduction in the number of parameters involved and the reusability of the weights. In other
words, the Network can be equipped for a higher level of image advancement.

Standard neural networks have an architecture different from the Convolutionary neural networks.
Regular Networks (RNNs) restructure data through a hidden layer sequence. Every layer consists
of a range of neurons and every layer is linked to one or all neurons in the preceding layer. Finally,
a fully connect layer and an output layer represent the forecasts.
CNNs are a bit different from others. CNNs are a little different. The layers are arranged in three
dimensions first and foremost: width, height and depth. In addition, the neurons in a single layer
are not connected in the next layer to all of the neurons, but only a few of them. Finally, a single
vector of shift values along the axis will reduce the final output.
Square Convolutionary Neural Networks measure an entirely different small amount. 1st of all, the
three-dimensional square layers: width, height and depth. In addition, neurons in one layer do not
hook up to only a small low area with all neurons within the next layer. Finally, the final output is
reduced to an organised, depth-dimensional vector of changing scores.

Dept of CSE 22 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Fig 5: CNN vs Regular neural network

The three circles in the input layer in the above figure represents input layer and it is noted as X.
The next four circles are hidden layers and usually noted as W they represent activation nodes.

There are three types of layers in a standard Neural Network:

1. Input Layers: This is the layer where we typically provide input to our model. The total
number of options in our knowledge is sufficient due to the number of neurons in this layer
(number of pixels in case of an image)
2. Hidden Layer: The hidden layer receives the data from the input layer. Depending on our
model and the size of the data, there may be many hidden layers. A different number of
neurons per secret layer usually exceeds the number of decisions. Each layer's output is
calculated by combining the previous layer output with the learnable weights of that layer,
then adding learning differences and activating the network, thus making it nonline.
3. Output Layer: The hidden layer's output is then fed into a supplying function like sigmoid
or Soft Max, which transforms each category's output into a probability score for each
category.

Dept of CSE 23 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

6.2 TYPES OF LAYERS:

There are four major operations in a CNN model:

1. Convolution,
2. Non-Linearity (Relu),
3. Pooling
4. Classification (Fully-connected layer).

CONVOLUTION LAYER: The main building block of CNN is the convolution layer. It is
responsible for the majority of the network's procedure load.
Convolution's main goal is to derive options from the input such as edges, colours, and corners.
When we progress further into the network, we'll notice that it begins to include more advanced
choices such as shapes, digits, and face components.
To encourage a 3x3x1 convolved function, convolution a 5x5x1 image with a 3x3x1 kernel.
This layer performs an internal product between two arrays, with one array representing the set of
parameters learned (known as a filter / kernel) and the other representing the restricted area of the
picture.
If the image is RGB, the height and width of the filter may be less then that of the image, but its
depth (high x wide x 3) will be the same as the image. We have a matrix at the top of the
convolution method with lower sizes as well as simpler choices than the original image. We will
therefore work with our featured matrix from now on.
POOLING LAYER: This layer only reduces the operating capacity necessary for processing the
data. The scale of the displayed matrix is further reduced. We are trying to extract from a small
number of neighbours the most important options during this sheet. Let us use an example to
demonstrate that. So if the method moves to another matrix area, we take as many numbers as
possible within the pooling area and transfer the pooling region.
AVERAGE-POOLING AND LIQUID ECSTASY POOLING are the two types of pooling
techniques.
The difference among these two is that in AVERAGE pooling we take all the values in the pooling
region on average, while in MAX pooling, the values in the pooling region actually rank the
highest.

Dept of CSE 24 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

So, when we pool a sheet, we get a matrix containing the image's key choices, and this matrix has
even smaller dimensions, which helps a lot in the next step. .

Fig 6: Fully connected layer inside CNN

FULLY CONNECTED LAYER: Until now, we haven't done anything to classify completely
different pictures; instead, we've highlighted some options in a picture and dramatically reduced
the size of the image.
From here on out, we'll be paying close attention to the classification system.
Now that our input image has been converted into a format that is suitable for our fully connected
architecture of several levels, we are going to flatten it into one column vector. Each coaching
iteration uses back propagation to feed the planat output into a neural feed system. In images over
a series of epochs, the model can differentiate between the dominant and the bundled low-level
options.
1. Create a convolution layer with the input image.
2. Use featured kernels/filters for convolution.
3. To reduce the size, apply a pooling layer.
4. Repeat the layering process.
5. Flatten the output and feed it into a layer that is fully connected.
6. Train the model

Dept of CSE 25 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

6.3 CNN MODEL PRE-TRAINING

Here, the concept of Transfer Learning is used, in which the model is first pre-trained on a
dataset that is completely different from the first. The model gains information in this way,
which is then passed to various neural networks. The model's information is saved and can be
loaded into another model using a range of "weights." By layering fully-connected layers on
top of the pre-trained model, it will be used as a function extractor. When loading the saved
weights, the model is trained with the first dataset.

6.4 SOFTWARE TOOLS

6.4.1 Python

Python is an interpreted object-oriented, complicated semantic high-level programming language.

It is ideal in screening and rapid application development with high-level built-in data structures
coupled with dynamic timing and dynamic binding.
Python's easy to learn syntax puts readability premium on it, reducing maintenance costs for
software. Python supports modules and bundles, enabling modularity of software and reuse of
code.
The Python parser is open source, as is a wide standard library. Python is widely considered to be
the best language for teaching and learning. The word ML (Machine Learning) refers to the process
of learning something new.

Python is a major reason why machine learning is so widely known. It is easy to read and is loved
by experienced developers and experimental students both. It is easy to read. Python's simplicity
allows developers to focus rather than spend their entire time on solving the machine learning
problem. Python is a major reason why machine learning is so popular. It is easy to read and is
loved by experienced developers and innovative students both. It is easy to read. Python's
simplicity means that developers can concentrate on fixing the situation of machine learning rather
than spending time. Due to its portable and extendable character, many cross-language operations
can easily be carried out on Python. Python learners can easily improve their knowledge of
machine learning, which only makes them more popular.

Dept of CSE 26 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Features:

• Interpreted Language
• Large Standard Library
• High-Level Language
• Dynamically Typed Language
• Easy to code
• Free and Open Source
• Object-Oriented Language
• Extensible feature
• Python is Portable language
• Python is Integrated language

6.4.2 Python IDLE

An integrated development environment (IDE) is a software development programme that
integrates a number of tools designed specifically for software development. These
methods can be used in a variety of ways.
o A code editor is a piece of software that lets you edit code.
o Source control software
o Creation, execution, and debugging tools When you install Python, IDLE is already
installed.
o Its core capabilities include the Python shell window, self finishing, syntax
emphasis, smart indentation and a simple built-in Debugger.
o Python 3.6.8 was used in this project.

Dept of CSE 27 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

CHAPTER 7
IMPLEMENTATION

Fig 7: Code(i)

Dept of CSE 28 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Fig 8: Code(ii)

Dept of CSE 29 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Fig 9: Code(iii)

Dept of CSE 30 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Fig 10: Code(iv)

Dept of CSE 31 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Fig 11: Code(v) Fig 12: Code(vi)

Dept of CSE 32 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Fig 13: Code(vii)

Dept of CSE 33 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

7.1 PACKAGES:

➢ NumPy:
▪ What is NumPy, exactly?

• NumPy is a Python package containing a strong n-dimensional array object.

NumPy is a Python package.
• It's used for array computing on the market.
• NumPy strives to be 50 times faster than regular Python lists when it comes to
array objects.

▪ The following are some of NumPy’s features:

▪ Advanced (broadcasting) capabilities

▪ C/C++ and Fortran code integration tools
▪ An effective N-dimensional array object
▪ Linear algebra, Fourier transform, and random numbers awareness

▪ NumPy's Applications

o NumPy includes a high-performance multidimensional array as well as basic

computation and manipulation methods.
o It's a Python replacement for lists and arrays that consumes very little memory.
o We are capable of performing a wide range of mathematical operations.
o Shape Manipulation
o It's compatible with Pandas, SciPy, Tkinter, and other Python libraries.

➢ Open CV:
o What is Open CV?
OpenCV is a cross-platform library for building computer vision applications in real
time. It focuses on image processing, video grabing and analytics, including the
detection of faces and objects.
OpenCV was developed to provide common infrastructure for computer vision
applications and accelerate the use of machine perception in commercial products.
OpenCV, a BSD-licensed product, makes it easier for companies to use and modify the
code. The library has more than 2500 optimised algorithms, including a full range of
both classic and advanced computer vision and master learning algorithms. These

Dept of CSE 34 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

algorithms can be used for identification of faces, objects identification, classification

of human activities into videos, camera tracking, moving objects and 3D object models
deleted. Their images can be stitched to produce highly-intensive 3D point clouds.
Color segmentation or colour filtration is commonly used for identifying certain
objects/regions of a certain colour in OpenCV. The most common colour area is RGB.
It is called an additive colour space since the three colour shades add to the image.

o Features of Open CV:

• Write and read images
• Take photos and videos
• Work with photographs (filter and transform)
• Analyse the video or images
• Perform feature extraction
o Applications of Open CV:
The computer vision is used widely in different domains such as
• Robotics
• Medical
• Security
• Automation
➢ Keras
• What is keras?
▪ Keras is a lightweight python library that makes building deep learning models
based on Tensorflow simple. Keras was created with the aim of rapidly define
deep learning models.
▪ Keras contains numerous implements on common neural network compilation
layers, targets, activation features, optimizers and a host of tools to simplify the
coding needed to create deep neural network code, making working with image
and text data easier. A great approach for deep learning is to create a
convolutions neural network to classify images (CNN). The Python Keras
library makes the construction of a CNN quite simple.
• Features of keras:
▪ Minimal Structure
▪ Consistent, simple
▪ High Scalability and computation

Dept of CSE 35 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

▪ User Friendly
▪ It supports on multiple platforms

7.2 FUNCTIONS
GAUSSIAN BLUR: The picture is convolved with a Gaussian filter instead of the box filter in the
Gaussian Blur process. The Gaussian filter is a low-pass filter that eliminates high-frequency
components from audio signals.
• The Gaussianblur() method is used to perform this operation on an image.
• The GaussianBlur() method of OpenCV can be used to blur an image.
• GaussianBlur() is a function that uses the Gaussian kernel. The kernel's height and width
should both be positive and odd numbers. Then you must define the X and Y directions,
which are denoted by the letters sigmaX and sigmaY, respectively. Both are considered the
same if only one is mentioned.
Syntax: blur_image = cv2.GaussianBlur(img, (7,7), 0)

➢ CONVERTING INTO GRAYSCALE: Gray scaling is the method of converting an

image from other colour spaces to shades of grey, such as RGB, CMYK, HSV, and so
on. It can be completly black and complete white.

Importance of Gray scaling:

● Reduced Dimensions
● Reduces Model Complexity

This function takes the original image as its first input. It receives the colour space c
conversion code as a second input. We use the COLOR BGR2GRAY code to transform our
original image from BGR to grey.

Gray = cv2.cvtColor (image, cv2.COLOR_BGR2GRAY)

➢ DETECTING EDGES: The Canny() method cv2 which implements the Canny edge
detector can be used to detect the edges of a picture. The Canny edge detector is also
called the perfect detector
Canny() has the following syntax:
cv.Canny(image,minvalue,maxvalue)
The minimum and maximum gradient intensity values here are minVal and maxVal
respectively.

Dept of CSE 36 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

CHAPTER 8
EXPERIMENTAL RESULTS

8.1 SCREENSHOTS OF OUTPUT

Fig 14: output(i) In this hand gesture showing a sign ‘A’ so its displaying letter ‘B’ in the box as it is in char mode

Fig 15: output(ii) In this hand gesture showing a sign ‘B’ so it is displaying the letter ‘B’ in the box as it is in char
mode

Dept of CSE 37 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Fig 16: output(iii) In this, sign is showing ‘U’ so it is displaying letter ‘U’ in the box as it is in char mode

Fig 17: output (iv) In this no sign is showing so it is displaying nothing in the box

Fig 18: output(v) In this, sign is showing letter ‘A’ as it is in word mode the speech output will be ‘I am hungry’

Dept of CSE 38 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Fig 19: output(vi) In this, sign is showing letter ‘B’ as it is in word mode the speech output will be ‘I need help’

Fig 20: output(vii) In this, sign is showing letter ‘L’ as it is in word mode the speech output will be ‘What are you
doing?’

Dept of CSE 39 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

8.2 RESULT

Simple gestures or signals are recognised by the machine. The architecture is

st9olraightforward and does not necessitate the use of gloves. The device is also adaptable to a
variety of backgrounds. This sign language recognition method requires a computer with a 1 GHz
processor and at least 256 MB of free RAM.
• All alphabet A to Z are in the training set (26 patterns). As an end to our project, we
managed to implement our entire system in real time, so that hand gestures took place
Directly the sign via the web camera, the segmented hand gesture and the classifier's output,
which is a letter of ten in our dataset, were shown on the webcam linked to our computer.
• Each frame's assessment takes less than 2 seconds. The screenshots in Figure 8.1 depict our
real-time implementation and the results obtained. The original image is shown in each
screenshot, along with the segmentation result and the projected result to the right.

8.3 DISCUSSION

• Real-time operation: The sign language output will be displayed in text form in real time.
As a result, the system is more effective, and hearing and speech disabled people can
communicate more easily.
• The images taken with the webcam are compared, and the outcome of the comparison is
shown simultaneously.
• As a result, this function of the device makes communication very easy and without delay.

Dept of CSE 40 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

CHAPTER 9

CONCLUSION

We used various tools to go through an automated sign language gesture recognition framework
in real time in this project. The proposed system's main goal is to understand sign language and
translate it to text format. For people who can't talk, a gesture-based communication recognition
device is a really useful technology in public. There has been a lot of research done there, and
we've discovered a lot of different ways to get signals, such as picture-based approach, sensor-
based device, and so on. The data set is split into two parts, one for training and the other for
research. The training set contains 70% aggregate data and the remaining 30% is used for research.
We used a convolutional neural network to train these datasets. The algorithm's advantage is its
high processing speed, which allows it to generate results in real time. Because of the neural
network architecture, processing speed is increased. Production of words and sentence gestures
could be used in potential extensions. In this case, grammar and syntax play a significant role in
determining the structure's efficiency and speed. The accuracy of the findings in this experiment
is about 95%. In the future, we intend to expand our work to include a larger number of gestures
and dataset validation using deep learning algorithms. The suggested method can also be used to
recognise Indian sign language, which is much more complicated since it requires two hand
gestures. Additionally, with better camera and graphics support, the system's response time can be
decreased.

9.1 FUTURE SCOPE

Only static ISL numeral signs can benefit from the system. The ISL recognizer system cannot be
regarded as a complete system since we need to include ISL alphabets, words, and sentences to
fully understand sign language. These signals may be used in the future. Additional extraction
algorithms, such as transform Wavelet, invariant moments, Shape lets descriptors and other
existing methods, can also be used to enhance the results of the experiments. Other classifications
can be used in experiments for increasing recognition rates, such as Multi-class Support Vector
Machine (SVM), Principal Component Analysis (PCA) or Linear Discriminant Analysis (LDA),
or a combination of them.
✓ Use Dynamic Loading for the Dataset: Our original dataset was very big and would require
a server with a lot of RAM and disc space to run. Splitting the file names into preparation,

Dept of CSE 41 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

validation, and test sets and dynamically loading images in the Dataset class might be one
solution.
✓ We might train the model on more samples in the dataset if we used such a loading
technique.
✓ Sign language recognition is a diverse field of study.
✓ More methods for shading detection, image separation, and grouping can also be used. We
plan to focus on more approaches in the future, as well as build a framework for continuous
sign language recognition for more terms and sentences.
✓ We used tools learned in computer vision and machine learning to implement an automated
sign language gesture recognition system in real-time in this project.
✓ We discovered that often simple solutions are better than complicated solutions. Despite
using a sophisticated segmentation algorithm, the most effective skin masks were extracted
using a simple skin segmentation model. We also recognised the time constraints and
challenges of building a dataset from the ground up.
✓ Other symptoms, such as the Indian Sign Language, as currently available for American
Sign Language, should be added to our model.
✓ Further training of the neuro-network to recognise symbols that effectively involve two
hands in the model's recognition of common terms and expressions. Using linear classifiers
to improve the performance.

Dept of CSE 42 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

REFERNCES

1. Hand Gesture Recognition for Indian Sign Language using Skin Color Detection and
Correlation-Coefficient algorithm with Neuro-Fuzzy Approach[1], Ms. Hemina Bhavsar, Dr.
2. Jeegar Trivedi.

3. Real-time Sign Language Recognition based on Neural Network Architecture[2],

4. Kusumika Krori Dutta, Sunny Arokia Swamy Bellary
5. Sign Language Learning System with Image Sampling and Convolutional Neural
6. Network[3], Yangho Ji, Sunmok Kim, and Ki-Baek Le

7. Sign Language Recognition Using Deep Learning on Custom Processed Static Gesture
8. Images[4], Aditya Das, Shantanu Gawde, Khyati Suratwala and Dr. Dhananjay Kalbande.

9. Hand Gesture Detection based Real-time American Sign Language Letters Recognition using
10. Support Vector Machine[5], Xinyun Jiang, Wasim Ahmad
11. End to end Residual Neural Network with Data Augmentation for sign Language
12. Recognition[6], Mengyi Xie, Xin Ma

13. Ian Lim, Joshua Lu, Claudine Ng, Thomas Ong and Clement Ong, "Sign language
14. Recognition through Gesture & Movement Analysis (SIGMA)[7]." Computer

15. Two hand Indian Sign language dataset for benchmarking classification models of machine
Learning[8], Leela surya teja manga muri, Lakshay jain, Abhishek

Dept of CSE 43 2020-21

 Sign Language Recognition Using Machine Learning Algorithm

Dept of CSE 44 2020-21