1

EasyCom

Submitted by:

101603274 Ravi Singla

101603276 Rishab
101603278 Rishabh Gupta
101610077 Sabhan Goyal

BE Third Year, COE

CPG No. 117

Under the Mentorship of

Dr. Karamjeet Singh

Assistant Professor

Computer Science and Engineering Department

Thapar Institute of Engineering and Technology, Patiala

Jan-Dec 2019
2

TABLE OF CONTENTS

• Mentor Consent Form 3

• Project Overview 4
• Need Analysis 5
• Literature Survey 6
• Assumptions and Constraints 8
• Standards 9
• Objectives 9
• Methodology 9
• Work Plan 10
• Project Outcomes & Individual Roles 10
• Course Subjects 11
• References 11
 3

Mentor Consent Form

I hereby agree to be the mentor of the following Capstone Project Team
Project Title:

Roll No Name Signatures

NAME of Mentor: ……… . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SIGNATURE of Mentor: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4

Project Overview

Communication is one of the fundamental step for the existence and survival of humans.
Moreover, it is mandatory for good and sustainable of an organisation. Everyone needs to
communicate in order to live a good and healthy life. But it is often very difficult for the
hearing-impaired community to communicate their ideas and creativity to the normal
humans. Our project “EasyCom” aims to aid these special group of people who have
difficulty in communicating in verbal form. There is several software to recognize
gestures but there is no dedicated device which can convert gestures to speech.

Body language is an important way of communication among humans. Normal people

can communicate their thoughts and ideas to others through speech. The only means of
communication for the hearing-impaired community is the use of sign language. Most
people don’t understand sign language, therefore speech impaired people face a lot of
difficulty while communicating to normal people. This project aims to bridge this
communication gap by providing a device which can convert sign gestures to speech and
text.

We aim to build a device which consists of a camera which will record the signs
performed by a differently abled person and then convert the signs into text and speech.
The text will be displayed on a screen attached to the device and an audio output will be
produced using a speaker. We will train a model on a dataset of a standard sign language
and the model will be installed in our device.
5

Need Analysis

Body language is an important way of communication among humans. Ideally, people

can communicate their thoughts and ideas to others through speech. In today’s world, the
only means of communication for the speech impaired community is the use of sign
language.

Most people don’t understand sign language, therefore it is observed in social interactions
that impaired people face a lot of difficulty while communicating to normal people. This
can lead to miscommunication and conflict between these people.

This project aims to bridge this communication gap by providing a device which can
convert sign gestures to speech and text. This can be a huge step as it can help in
communication between speech impaired and normal people. The device will contain a
screen so that people can read the text on it and understand what the user of the device is
trying to say using gestures. It will also contain a speaker so that people can also hear the
audio translation of the gestures.

Existing /Related Work

Action Recognition is a project related to our proposed project. In this project, we

recognise an action simply using deep learning. But there are some limitations associated
with afore mentioned project which are briefly discussed below:

● More Generalist Approach: It focuses on the broad classes of all actions, not
specifically concentrating on the hand gestures.
● No Hardware Implementation: There is no hardware implementation of the above
mentioned project.
6

Literature Survey
A literature survey provides a quick summary/overview of recent research in a topic. In
our project we will talk about the existing research papers related to this.

3-D Hand Gesture Recognition with Different Temporal Behaviours

Using HMM and Kinect: Vision based hand gesture recognition is getting
increasingly popular due to its intuitive and effective interaction between man and
machines. However, there are not sufficient means of support for deployment, research
and execution for these tasks. In this thesis, we present 3-D hand gesture recognition
system to recognize, especially when dealing with gestures of similar shape but different
temporal pattern using Hidden Markov Models (HMMs). This is facilitated by recording
gestures using Microsoft Kinect sensor. Recorded gestures include different shapes but
with varying temporal patterns. We train the Hidden Markov Model (HMM) models
based on feature vectors extracted from the recorded gestures. Training and classification
process was implemented using machine learning framework Accord.Net. We analyse the
feasibility of HMMs in detecting and recognizing different gestures in the classification
process. We implemented five-fold cross validation strategy to determine accuracy rates
of different gestures for different training parameters. We also analysed the results of
classification process for gestures drawn by specific users. Our results suggest that our
algorithm can achieve an average best of 63.6% in recognizing gestures with different
temporal patterns and an average best of 86.8% in recognizing gestures drawn by a
specific user. This shows that within limits, HMMs can be applied to distinguishing user
gestures with similar shape and different temporal patterns.

A Method for Static Hand Gesture Recognition Based on Non-Negative

Matrix Factorization and Compressive Sensing: Hand gesture recognition is
becoming a more and more popular research field in human computer interaction.
However, the recognition rate still remains to improve. In this paper, we present a novel
static gesture recognition method based on non-negative matrix factorization (NMF) and
compressive sensing (CS). Firstly, original images are projected to low-dimensional
subspace by using NMF, and then gesture is recognized by the classifier which is
designed by CS theory. Experimental results which are done on two gesture databases
show that the CS classifier performs better than some other classifiers no matter on the
recognition rate or resisting occlusion. The NMF also has a better occlusion resistance
than the principal components analysis (PCA). This method can increase the gesture
recognition rate in comparison to some previous methods.
7

Hand Gesture Recognition Using Wavelet Neural Network: This paper

presents a method for hand gesture recognition through Statistic hand gesture which is
namely, a subset of American Sign Language (ASL). The proposed technique presents an
image of hand gesture by passing it through four stages, pre-processing, normalization,
feature extraction, and classification. The wavelet neural network is used to build
information models because it can handle very complex interactions. The practical
system is simulated using MATLAB. The performance of suggested method is excellent
compared with traditional approaches and the final results of selected patterns recognition
are powerful

Real Time Gesture Recognition for Easy Load Transport: Motion of the
body is interacting with other things for communication is called gesture recognition. For
more accurate results, we need to know the gesture information which is taken is possibly
same to provide to receiver. In this paper we are usually focus on control machine using
movement of the user body. Kinect is used to track the Body skeleton of a user or
operator. For easy to carry a load we used the system contains Kinect, Arduino & matlab.
We integrate the Kinect sensor and MATLAB image processing mechanism to process
video input which is captured by Kinect. The machine capable of carrying out a complex
series of action, and control device which is used to control the motor drivers which
implementing human skeleton tracking method. Real time moving position of an
Operator is captured using Kinect sensor having three cameras one is RGB, IR emitter,
IR depth measurement camera then captured body movements are providing for
processing and instruction of body movements are used to perform vehicle according to
the user actions. Proposed system will consist of process to provide a facility to user or
operator to ease their field work to drive a vehicle, with the help of Kinect sensor and
processing mechanism to follow the movement of body.
8

Assumptions

1. The user should know different gestures that can be recognised and successfully
decoded by the device.
2. The user must know about the device like battery life, range of the speaker,
position of the camera relative to the user and position of the screen.
3. The user must know optimal position of hands while performing the gestures for
best recognition by the camera.
4. The user cannot communicate with a person who is both deaf and blind using this
device.

Constraints

1. The device must be affordable.

2. The device must be portable and light-weighted.
3. The device should have sufficient battery backup for longer usage.
4. Camera quality should be good for accurate recognition of gestures.
5. The device must have good quality speaker.
6. The screen resolution must be good so that the text is easily readable.
7. Gesture recognition and conversion time must be as low as possible for effective
communication.
9

Standards

• IEEE
• Industry 4.0

Objectives

1. To build a Deep learning model which can accurately recognise the limited set of
gestures of a standard sign language.
2. To design and develop a portable hardware device which can run the above
mentioned model.
3. To install the model on the hardware device and then test the final product.

Methodology

1. Successfully build an efficient and accurate deep learning model which can
convert the gestures to text and speech.
2. To train this model we need an ample dataset to increase the efficiency and the
accuracy of the model. So, we need to record the dataset of the gestures. There are
a lot of gestures but we will only concentrate on a limited set of these gestures.
3. Assemble all the hardware components to create the final hardware device and
install the trained model on it.
4. Test the final product.
10

Work Plan

Project Outcomes & Individual Roles

A portable and compact device which can help a hearing-impaired person to

communicate with a normal person through sign language.

Students\Tasks Designing Model Documentatio Testing and

and Training n and Optimization
Assembling Diagrams
Hardware

Rishab Garg

Rishabh Gupta

Ravi Singla

Sabhan Goyal
11

Course Subjects

1. Python
2. Keras
3. OpenCV
4. Numpy
5. Scipy
6. Computer Architecture
7. Electronics
8. Machine Learning
9. Deep Learning

References

https://arxiv.org/abs/1608.00859

https://www.talkinghands.co.in/

https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up

https://www.researchgate.net/publication/319406874_Text_-_To_-
_Speech_Synthesis_TTS