See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/344063213

Implementation of Virtual Assistant with Sign Language using Deep Learning

and TensorFlow

Conference Paper · July 2020

DOI: 10.1109/ICIRCA48905.2020.9183179

CITATIONS READS

15 418

4 authors, including:

Vismay Chaudhari Swati Nadkarni

Rochester Institute of Technology Shah And Anchor Kutchhi Engineering College
1 PUBLICATION 15 CITATIONS 14 PUBLICATIONS 39 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Swati Nadkarni on 19 August 2023.

The user has requested enhancement of the downloaded file.

 Proceedings of the Second International Conference on Inventive Research in Computing Applications (ICIRCA-2020)
IEEE Xplore Part Number: CFP20N67-ART; ISBN: 978-1-7281-5374-2

Implementation of Virtual Assistant with Sign

Language using Deep Learning and TensorFlow
Dipanshu Someshwar
Student, Department of Dharmik Bhanushali Vismay Chaudhari
Information Technology, Student, Department of Student, Department of
Shah and Anchor Kutchhi Information Technology, Information Technology,
Engineering College, Chembur, Shah and Anchor Kutchhi Shah and Anchor Kutchhi
Engineering College, Chembur, Engineering College, Chembur,
Mumbai- 400088, India
[email protected]. Mumbai- 400088, India Mumbai- 400088, India
in [email protected] [email protected]
n

Swati Nadkarni
Associate Professor,
Department of Information
Technology, Shah and Anchor
Kutchhi Engineering College,
Chembur,

Mumbai- 400088, India

[email protected]

Abstract— The paper is all about the system and interface Ho me Automation. Since these assistants are purely Vo ice
developed, that allows deaf mutes to make use of various voice Automated, Deaf-Mutes find it hard to make use of such
automated virtual assistants with help of Sign Language. technology as observed in [8]. The agenda of the pro ject is
Majority of Virtual Assistants work on basis of audio inputs
to develop an interface that will help the Deaf-mutes to use
and produces audio outputs which in turn makes it impossible
these Virtual Assistants easily with easy. As of now, it
to be used by people with hearing and speaking disabilities.
might seem irrelevant to design such a system but in a
The project makes various voice controlled virtual assistants
respond to hand gestures and also produces results in form of longer run it might help deaf-mutes to equally enjoy their
text outputs. It makes use of concepts like Deep Learning, social and personal life. Designing such an interface will
Convolutional Neural Network, Tensor Flow, Python Audio make them find their freedo m wh ile using such technologies
Modules. A webcam first captures the hand gestures, then and might boost their confidence in this Dig ital Age. This
Convolutional Neural Network interprets the images produced paper focuses on a research that gives an idea of combining
and produces rational languages. These languages are then two modern technologies that are Hand Gesture Recognition
mapped to pre-defined datasets using Deep learning. For this and Virtual Voice Assistants in order to make it possible fo r
purpose, Neural Networks are linked with Tensor flow library. people with hearing/speaking difficulties to interact with
The designed system will then produce audio input for the Dig ital Gadgets and also communicate with the outside
Digital Assistant, using one of the Python text to speech world. This research work has imp lemented Alexa which is
module. The final audio output of the Digital Assistant will be an audio based Virtual Assistant. The proposed system has
converted into text format using one of the Python speech to
been successful in rep lacing Speech Recognition technique
text module which will be displayed on the viewing screen.
with Hand Gesture Recognition technique. The proposed
Keywords— Deep Learning, Virtual Assistants, Ten sor Flow, system makes use of follo wing technologies: TensorFlow
Convolutional Neural Network Hand Gestures, Sign Languages. which is the most important lib rary used for designing and
developing the model of these system, Convolutional Neural
I. INT RODUCT ION Network, is an Deep Learn ing Algorith m that have been
Nowadays, Virtual Assistant devices have been part and used for serving the purpose of Image Recognition, that
parcel of our lives, but most of them are Vo ice Automated . helps in converting the images in form o f matrix that can be
Most common ly used Virtual Assistants are Alexa, Google understood by the model and making it Classifier ready, and
Ho me, Apple Siri and Microsoft Cortana. These assistants lastly OpenCV that will act as an Eye of the system that will
listen to user’s queries and respond accordingly making capture and process Real-t ime Hand Gestures and predict
results with help of Classifier.
there life easier, thus they have been a very important part of

978-1-7281-5374-2/20/$31.00 ©2020 IEEE 595

Authorized licensed use limited to: Shah and Anchor Kutchhi Engineering College. Downloaded on August 19,2023 at 21:35:34 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Second International Conference on Inventive Research in Computing Applications (ICIRCA-2020)
IEEE Xplore Part Number: CFP20N67-ART; ISBN: 978-1-7281-5374-2

With increasing trends in technology, personal assistant It has been understood by using a block d iagram shown in
devices are becoming mo re and mo re popular. But such Fig.1
devices are voice automated. They need audio inputs and
provide audio outputs. So what if someone does not have
their own voice or are not in a condition to speak properly,
that’s where this project co mes into light. Such people can
easily commun icate with these devices using an interface
that takes hand gestures as an input and provides audio as
well as text output. This project has the capacity to bridge
the gap between such impaired people and booming
technology.

II. LIT ERAT URE REVIEW

Every existing Virtual Assistant in today’s date is found to
be Voice Automated thereby making it unusable by Deaf-
mutes and people with certain d isabilities. This leads to the
need of a system wh ich can help people with speaking or
listening disabilities to make use of such Virtual Personal
Fig. 1. Implemented System Workflow
Assistants [8]. Artificial Neural Netwo rk is used in majority
cases where static recognition is performed as shown in [1],
Now to understand the technical workflow of the co mplete
but there are few drawbacks related to the efficiency of
system, a brief exp lanation goes as follows - The very first
recognizing distinctive features from images which can be
step is to provide training dataset and train the system with a
improved by using Convolutional Neural Netwo rk .
variety of hand gestures named with their respective labels.
Convolutional Neural Network when co mpared to its
This is the most time consuming step. Better the underlying
predecessors, recognizes important distinctive features more system software specifications, lesser the time required fo r
efficiently and without any human supervision. Artificial
training will be. Once training is co mplete, the next phase is
Neural Net work uses one-to-one mapping wh ich increases prediction mode. It now uses the input image fro m a
the number of nodes required thereby degrading the
webcam and runs it through the classifier to find its closest
efficiency whereas Convolutional Neural Network uses one- neighbours based on the training examp les and labels
to-many, keeping the number of nodes low and greatly
provided in the previous step. If a certain p rediction
improving the efficiency [5]. Many systems designed with
threshold is crossed, it will append the label on frame as
such objectives tend to make use of more of physical
suggesting that system recognised the following hand
hardware like the design observed in Cyber Glove thereby
gesture accordingly. Then Python Text-to-Speech module is
leading to need of manufacturing of such hardware gadgets
used for speech synthesis to speak out the detected label to
and also making it mandatory for the users to wear it while
the digital assistant, after collecting series of hand gestures
accessing the Virtual Assistants [11]. Many systems are and execute it on presence of certain call co mmands which
designed in such a way that there application is limited to
will be declared previously. If the spoken word is ‘Alexa’ it
only certain Sign language or series of Hand gestures [9] causes the nearby Echo to awaken and begin listening for a
whereas the proposed system is designed in such a way that
query and then Alexa responds to query in voice format.
it gives us the flexibility of changing to any standard sign Meanwhile system will start the Voice to Text python
language just by changing the dataset and training the model
module which will listen to the Alexa responded query and
for the same.
then convert it into text fo rmat and display it in output frame
III. M ET HODOLOGY on the Display Screen. The entire process can be repeated
mu ltip le nu mber of t imes. But the hand gestures will be only
The most basic explanation of workflow of the system goes recognised if they meet the trained data set standards.
as follows - A hand gesture is performed in front of the
webcam just as observed in [10]. Th is sign gesture is
converted to text and the text output is converted to audio
and is served as an input to the assistant. The assistant
processes the question and responds in audio format. This
audio format is converted to text output. The text output will
be then displayed on the display screen.

978-1-7281-5374-2/20/$31.00 ©2020 IEEE 596

Authorized licensed use limited to: Shah and Anchor Kutchhi Engineering College. Downloaded on August 19,2023 at 21:35:34 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Second International Conference on Inventive Research in Computing Applications (ICIRCA-2020)
IEEE Xplore Part Number: CFP20N67-ART; ISBN: 978-1-7281-5374-2

It can be understood by using a Flow chart shown in Fig.2 higher is the success rate. But it is required to keep a note
that any changes made with the labels folder before training
will lead to the system that is being t rained for the very first
time. In simp le terms if it is required to make any changes in
the labels folder that is adding new labels or replacing the
existing labels, the model will need to be trained again fro m
the beginning. It was observed that for training of around 15
labels on an average configured system, it takes about 12-15
hours straight of model training for the first time. Ho wever
retrain ing of same set of labels requires comparat ively lesser
amount of time.

Fig. 3. T esting Accuracy

Fig. 3, is the part of training a dataset that shows the training

accuracy obtained at respective number of step, along with
the Cross Entropy value and Validation Accuracy for the
same. Basically train accuracy is the value which let us
understand how well the training is taking place wh ile the
Validation Accuracy let us know how the model will react
while predicting the data it has not seen before.

B. Tensor Flow
Fig. 2. Flow of the system
The best part of using TensorFlow library is that it is an
A. Training Dataset open Source Library with lots of pre designed models ,
Dataset is the most fundamental element of any Machine useful in Machine Learning and especially Deep Learning.
Learn ing Model. As it is a process of feeding into For understanding the conceptual use of Tensor Flow is
Machine’s memory to help classify whatever it insights in required to understand the meaning of two terms, where the
future for the designed application. Since our system is an Tensor here is considered as N-Dimensional Array and Flo w
interface for Real-Time Classification of Hand Gestures our refers to graph of operations. Every mathematical
Dataset will purely consis t of large number of Images in computation in TensorFlow is considered as graph of
form of .jpeg, .jpg, these are the only two extensions that out operations where Nodes in the Graph are operations and
model is accepting. The designed model makes use of a Edges are nothing but tensors.
Labelled dataset method for training our system, thus Any mathematical computation is written in form of data
assigning labels to folder names will simply use sub-files of flow diagram in Python Frontend or C++ or Java, as in our
images to be trained under assigned labels. Each label is case Python is used. Then, TensorFlow Execution Engine
being trained with about more than 2000 images captured at comes into picture and makes it deployable on any of the
various possible angles in order to make system learn better hardware of Embedded System let it be CPU or A ndroid or
and classify more accurately and quicker as observed in [2]. IOS. TensorFlo w is a Machine learning framewo rk that
Once the model is co mpletely trained for a set of particular comprises of uses the dataset to train Deep learning models
labelled images it gets Classifier ready and can be u sed for and helps in prediction and also improvise future results.
testing the system’s prediction rate. However it was noticed
that retraining the same set of labels tends to give better The biggest advantage of using TensorFlow is it’s feature o f
results in terms of accuracy and speed of predicting the providing Abstraction, that is the developer does not need to
Hand gestures as observed in [3]. Basically the model will work on every small aspects of designing the model as it is
be trained more nu mber of t imes for the same set of labels, managed by the lib rary itself, thus giving the developer the

978-1-7281-5374-2/20/$31.00 ©2020 IEEE 597

Authorized licensed use limited to: Shah and Anchor Kutchhi Engineering College. Downloaded on August 19,2023 at 21:35:34 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Second International Conference on Inventive Research in Computing Applications (ICIRCA-2020)
IEEE Xplore Part Number: CFP20N67-ART; ISBN: 978-1-7281-5374-2

freedom to focus on logic building, which was clearly Inception-v3 Convolutional Neural Network has been
explained in [7] . implemented, while designing this system. Inception v3 is a
48 layers deep Neural network. Inception Network is better
TensorFlow in our system helps us in training the model than most of Convolutional Neural Networks because it just
using the provided dataset. TensorFlow object recognition does not dig deeper and deeper in the layers like other
algorith ms helps us classify and identify different hand
Convolutional Neural Networks instead it believes in
gestures when combined with use of Open CV. By analysing
working wider on the same layer before going deeper into
thousands of photos, Tensorflow can help classifying and
the next layer. This is the reason, bottlenecks are used while
identifying real-time hand gestures. It makes possible to
training the model. Bottleneck in Neural Net work is just a
develop a model wh ich can help identify 3D images and
layer having less neurons as compared to the layers above or
classify it on basis of 2D images fro m its feed dataset.
below it. TensorFlow bottleneck is the last step of pre-
TensorFlow is capable of processing more info rmation and
spot more patterns. processing phase that starts before actual training of dataset
starts.
C. Deep Learning
E. OpenCV
Deep Learn ing is basically a subset of Machine Learning
model wh ich consists of algorith ms that make use of mu lti- OpenCV is an open source library for Co mputer Vision.
layer neural networks. Deep Learn ing makes use of Neural Now since all the train ing and classification is ready to be
Network most of the t imes to imp lement its functioning. A executed when it needed an eye for the designed system to
Neural Netwo rk is a collection of layers that transforms the capture real-time images of Hand Gestures which can then
input in some way to produce output. be sent for classification and identification. OpenCV adds
intelligence to Deep Learning models for visualization
Image can be termed as matrix o f pixel values so it may image processing. Here images are considered over 2
seem that classification can be an easier task simp ly based channels as: RGB Channel and Grey Scale Channel so once
on matrix classification but that is not the case with co mplex the image is captured by OpenCV it first converts into Grey
matrix images or images with similar forms of mat rix o r a channel so it can then undergo morphological processing as
very huge dataset of images with min imal changes in the shown in [9]. OpenCV makes use of Nu mpy Library fo r
matrix. Th is may lead to clash in prediction scores and numerical co mputation of Images in form of matrix o f
thereby affecting the accuracy and speed of classifier model.
pixels.
This is where Neural Net work co mes into picture and thus it
is required to use deep learning over mach ine learning. A blue bo x of particular dimension has been designed with
Machine Learning works with lesser number of layers when help of OpenCV in a way that it will consider hand gestures
compared with Deep Learning as observed fro m [12] and present inside this blue box. It then converts the image over
thus not preferred for technologies like Image Recognition different channels and then convert the image into
which requires need of Convolutional Neural Networks. convoluted form of matrix so the Classifier model can
D. Convolutional Neural Network compare it with previously learned labelled images. It will
then predict a suggestion of gesture on basis of the score
A convolutional Neural Network is nothing but a Deep
generated. As OpenCV is converting real-t ime hand gesture
Learn ing algorith m that is capable of assigning biases and
it will be continuously suggesting predictions because of
weights to different objects in an Image and on basis of the
slightest of motion of real-t ime hand gesture. The confirmed
same it can differentiate one image fro m another. It consists
prediction with highest score will enter the sequence until a
of processing different layers of Image Classification and it
CA LL COMAND is executed. Then the entire sequence will
is designed with means of representing functioning of
enter the next stage of designed interface that is it will be
Neurons in Human Brain as explained in [4].
converted into Audio format which will then wake the
Even if the most minimalist pixelated image is considered, it Virtual Voice Assistant and become the Input Query.
still needs 4x4 mat rix and required to consider the same
F. Python Text and Speech APIs
image in different channels of colour fo rmats like RGB,
Greyscale, HSV, etc so it is very difficult to process The Python text-to-speech library that used is very simp le
thousands of images in high rates of pixels for instance and easy to use. It makes use of modules like pyttsx3 and
1020x1980 pixels. Here co mes the need of Convolutional engine.io which let us change different properties like rate
Neural Network that convolutes every image into its basic and intervals of text to speech conversion and outflow.
reduced form of matrix wh ich can be differentiable at the The Python speech-to-text library by which pract icing
same time. These increases the Accuracy and Speed and makes use of speech recognition module. It let us adjust the
also reducing the processing of Classifier model. The amb ient noise and also helps in recording the audio in form
convolutional layer is also supported with Pooling layer to of mp4 files.
decrease the processing need of classifier model. It also
convolutes the matrix but on basis of dominant features .
Pooling is majorly of two types; MAX Pooling and A VG
Pooling, this is clearly explained in [6].

978-1-7281-5374-2/20/$31.00 ©2020 IEEE 598

Authorized licensed use limited to: Shah and Anchor Kutchhi Engineering College. Downloaded on August 19,2023 at 21:35:34 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Second International Conference on Inventive Research in Computing Applications (ICIRCA-2020)
IEEE Xplore Part Number: CFP20N67-ART; ISBN: 978-1-7281-5374-2

IV. RESULTS
A short demonstration of our project is g iven below with
help of images. Here, it performed a Hand Gesture by
representing a term “Weather” and once CALL
COMMAND is received, a query asking weather will be
sent to Virtual Vo ice Assistant and the real-time output will
be converted into text and displayed on output frame.

Fig. 7. Output Frame

Fig. 7, shows the output frame which d isplays the real-time

response generated by Vo ice Assistant, but converted in text
format and displayed on the screen.

Fig. 4. Hand Gesture for term Weather

Fig. 4, shows how the designed system will capture real-

time hand gestures, only the ones inside blue box, using
Web Camera. In the above figure person is performing a
hand gesture representing word ‘Weather’.

Fig. 8. Interface

Fig. 8, is a view of real-time interface of the designed

system with all the window frames tied up together,
performing their expected roles. Now, again if no gesture is
being performed the sequence window will print DEL,
which is also another CALL Command.

Fig. 5. Sequence frame

Fig. 5, represents a window frame wh ich will be displaying

system’s suggestions for captured hand gesture.

Fig. 9. Classifier’s Accuracy

Fig. 9, shows us the score which indicates the accuracy by

Fig. 6. Pre-Defined Call Command
which the gesture is being classified by the model. It ranges
between 0 to 1 as shown above.
Fig 6, represents the system capturing a real-time hand
gesture for CALL Co mmand, so that the previously V. CONCLUSION
captured words in the sequence frame can now be converted
The designed system was successfully able to capture Hand
into Audio format as an input for Assistant .
Gestures using the integrated Web Camera and process and
convert into text format and display it onto the Input frame
and then converted into Audio format on receiving a CALL
COMMAND. The audio becomes a query for the Virtual
Assistant and again the audio output was being successfully
converted into Text format and displayed on the screen as
shown above in results. Most of the times in the preferred

978-1-7281-5374-2/20/$31.00 ©2020 IEEE 599

Authorized licensed use limited to: Shah and Anchor Kutchhi Engineering College. Downloaded on August 19,2023 at 21:35:34 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Second International Conference on Inventive Research in Computing Applications (ICIRCA-2020)
IEEE Xplore Part Number: CFP20N67-ART; ISBN: 978-1-7281-5374-2

conditions, the system was able to provide the accurate and favourable for better results, also presence of good amount
best of its results. However sometimes in poor light of light wh ile presenting the hand gestures. So it is required
conditions and in absence of proper background the system to overcome these difficu lties in order to make system
struggled to produce correct and expected results. perform better.

While developing th is system, many difficult ies are faced A CKNOWLEDGMENT

like in recent updates these virtual assistants have stopped We wish to express our profound gratitude to our Principal
responding to digital voices of certain frequency and Dr. Bhavesh Patel and our project guide Ms. Swat i Nadkarn i
bandwidth, in order to not wake these devices in cases of for allo wing us to go ahead with this project and giv ing us
commercial advertisements (like in case of ALEXA) so it is the opportunity to explore this domain. We would also like
required to correctly select the Digital Vo ice, which is used to thank the Review Co mmittee for their invaluable
in our Python text-to-speech library to avoid this issue.
suggestions, constant encouragement and support towards
Also as the designed system is trained co mpletely using the achieving this goal. Finally, we would also like to thank
chosen Dataset, thus selection of Dataset should be done on Mumbai University for believing in our project’s scope and
basis of the Standard Sign Language used by the deaf-mutes providing us the Grant for the requirements of the Project.
in targeted region as these languages tend to change locally REFERENCES
and globally.
[1] Yusnita, L., Rosalina, R., Roestam, R. and Wahyu, R., 2017.
Implementation of Real-T ime Static Hand Gesture Recognition Using
Basically the system can be cons idered as a boon to people Artificial Neural Network. CommIT (Communication and Information
with hearing disabilities or speaking disabilities or both at Technology) Journal, 11(2), p.85.
the same time. These system would not only bring [2] Rathi, P., Kuwar Gupta, R., Agarwal, S. and Shukla, A., 2020. Sign
technology into their Personal lives but also give rise of Language Recognition Using ResNet50 Deep Neural Network
Architecture. SSRN Electronic Journal
opportunities in their professional life.
[3] V. Adithya, P. R. Vinod and U. Gopalakrishnan, "Artificial neural
VI. FUT URE SCOPE network based method for Indian sign language recognition," 2013
IEEE Conference on Information & Communication Technologies,
The current designed system co mp letely works on basis of T huckalay, Tamil Nadu, India, 2013, pp. 1080-1085.
underlying Dataset which is used to train the system, thus [4] Guru99.com. 2020. Tensorflow Image Classification:
CNN(Convolutional Neural Network). [online] Available at:
making its use limited to certain g roup of people which <https://www.guru99.com/convnet-tensorflow-image-
communicates using the similar Sign Language. However it classification.html>.
is found out that there are various forms of Sign Languages [5] Guo, T., Dong, J., Li, H. and Gao, Y., 2017. Simple Convolutional
Neural Network on Image Classification. IEEE 2nd International
globally, so dataset used needs to be changed according to Conference on Big Data Analytics, pp.1-2.
the standard Sign Languages used in that nation or region.
[6] Medium. 2020. A Comprehensive Guide To Convolutional Neural
Networks — The ELI5 Way. [online] Available at:
Since the current system’s Graphical User Interface is <https://towardsdatascience.com/a-comprehensive-guide-to-
simp le but not visually soothing, further it is aimed at convolutional-neural-networks-the-eli5-way-3bd2b1164a53>
designing and building a better Visual Interface that makes [7] Medium. 2020. Deep Learning With Tensorflow: Part 1 — Theory
it even more vibrant and eye-catching eventually making it And Setup. [online] Available at:
<https://towardsdatascience.com/deep-learning-with-tensorflow-part-
easier as well interesting to use, for our targeted audience. 1-b19ce7803428>
[8] Issac, R. and Narayanan, A., 2018. Virtual Personal
The existing Virtual Vo ice Assistants are basically a fo rm o f Assistant. Journal of Network Communications and Emerging
Smart Speakers which are Vo ice Automated. Our designed Technologies (JNCET), Volume 8(Issue 10, October (2018).
interface is an add-on to make them accessible by deaf- [9] Lai, H. and Lai, H., 2014. Real-T ime Dynamic Hand Gesture
mutes with help of a Web Camera and a laptop. So if this Recognition. International Symposium on Computer, Consumer and
Control, pp.658-661.
system gets a good response by the users, it can even take it [10] Pankajakshan, P. and Thilagavathi B, 2015. Sign language
to a larger scope by being able to integrate our system recognition system. 2015 International Conference on Innovations in
completely with those Virtual Assistant speakers in a better Information, Embedded and Communication Systems (ICIIECS).
form where the speakers itself will consist of a Web [11] K. A. Bhaskaran, A. G. Nair, K. D. Ram, K. Ananthanarayanan and
H. R. Nandi Vardhan, "Smart gloves for hand gesture recognition:
Camera, wh ich will act as an eye and respond to Hand Sign language to speech conversion system," 2016 International
Gestures, if combined with processing abilities. Conference on Robotics and Automation for Humanitarian
Applications (RAHA), Kollam, 2016, pp. 1-6, doi:
As the current system can still not be called a co mplete 10.1109/RAHA.2016.7931887.
Error Free product, it still needs to make it better by means [12] Ertham, F. and Aydin, G., 2017. Data Classification with Deep
Learning using T ensorflow. IEEE 2nd International Conference on
of it ’s Overall Accuracy and Productivity in terms o f Computer Science and Engineering, pp.757-7
generating end results. Our system has few limitations such
as need of a plain background, white being the most

978-1-7281-5374-2/20/$31.00 ©2020 IEEE 600

Authorized licensed use limited to: Shah and Anchor Kutchhi Engineering College. Downloaded on August 19,2023 at 21:35:34 UTC from IEEE Xplore. Restrictions apply.

View publication stats