Proceedings of 2018 Eleventh International Conference on Contemporary Computing (IC3), 2-4 August, 2018, Noida, India

Automated Sign Language Interpreter

Hardik Rewari, Vishal Dixit, Dhroov Batra, Hema N

Department of Computer Science and Information Technology, JIIT, Noida
{hardik.rewari007, vishal.dixit2897, dhroovbatra28}@gmail.com, [email protected]

Abstract— Taking huge leaps in technologies with each exist fail to compete with the international level when it comes
passing year, the humans are making smart inventions every to the ease of use.
year to help themselves and for the ones who are affected by any
disability. We aim to make the communication for dumb people The major problem faced by Indian SLI until now was the
easy and hence proposing a sign interpreter, which automatically ambiguity in the sign language community. The words that
converts sign language into audio output. For the dumb people, need to be covered in sign language dictionary are business,
sign language is the only way of communication. With the help of leisure, emotions, alphabets, numbers, ask-learn, colors,
sign language, physically impaired people express their thoughts family, food, relations, games, interview, medical, health,
to the other people. It is difficult for common people to place, money months, sentences, travel, animal, basic, days,
understand the specific sign language therefore communication fruits, government etc. Proposed sign interpretation uses
becomes difficult. The sign language recognition has become an India's first-of-its-kind dictionary [5], which aims to bring
empirical task, as it consists of various movements and gesture of uniformity in sign languages across the nation.
the hands and therefore getting the right accuracy at a low-cost is
a mammoth task. Instrumented gloves with audio out are the Indian Sign Language Research and Training Centre
solution to this problem. The gloves attached with various developed “The Indian Sign Language” (ISL) dictionary
sensors are worn for sign interpretation. Hence, the proposed which consists of 3,000 Hindi and English words with the
system solves the problem and helps the dumb people in corresponding representation of their signs [11]. The ISL
communication with the rest of the world at low cost. dictionary was created with a focus on removing the
ambiguity and redundancy in the Indian sign language
Keywords—Automated Interpreter; Indian Sign language; prevailing throughout the nation. The dictionary is updated
Instrumented Gloves; Sensors; Audio output; time to time in order to expand the vocabulary used by SLI
[4]. It will also enable government officials, teachers,
I. INTRODUCTION
professionals, community leaders and the public at large to
Sign language is the medium of communication for the learn, use and implement sign language.
people who cannot speak. It uses gestures to convey what they
want to say. Sign language is a combination of hand-shapes, The proposed sign interpreter uses Arduino Uno based
hand-orientation and hand, arm or body -movement, facial microcontroller as a core controlling components. The
expressions and lip-patterns for conveying messages. American proposed sign language interpreter is wearable gloves.
Sign Language (ASL), British Sign Language (BSL), Indian Interpreter gloves have a microcontroller with various other
Sign Language (ISL) etc. are some of the common sign sensors to capture the hand gesture and hand movements.
languages in the world [1]. As per the census 2011, there are Sensors connected to the gloves include flex sensors and the
19,98,535 people with speech disability in India [2]. accelerometer integrated with a gyroscope sensor, which
Communication is a way of exchanging thoughts and feeling provides the input to the microcontroller. Both the gloves
and should not be restricted by usage of words or speech. Sign communicate with each other through Bluetooth module to
language interpreters (SLI) are required to translate sign produce various signs, which are further communicated
language to spoken language [1]. Sign interpreters have been through the speakers.
developed in the recent decade and have gained a lot of The proposed system aims to keep the sign language
significance in the field of research and development. Over the interpreter as simple, affordable and efficient. Further,
years, several hand-movement trackers have been developed, presented paper discusses the existing literature of sign
that are different in their performance parameters like accuracy, language interpreter in section II, in addition to the pros and
jitter, drift, and latency [3]. cons of it. In section III, the architecture of the proposed
However, generic automated sign interpreter system is discussed. It briefly discusses the software and
methodologies include the use of sensors such as optical fibers hardware components used in the proposed system. In section
and flex sensors to detect the movement of the palm and the IV, the implementation of the proposed system is discussed in
fingers for mapping various letters. Every country has its detail. In section V, the results of the proposed system are
specific sign language; however, the sign for the same word discussed along with cost analysis and its limitations. Section
varies from country to country [4]. Significant research on the VI concludes the work with the future scope.
sign language interpreters can be noted in countries like USA,
UK, Singapore, Canada etc. However, no significant research
has been done in India until now. The products that already

978-1-5386-6835-1/18/$31.00 ©2018 IEEE

 Proceedings of 2018 Eleventh International Conference on Contemporary Computing (IC3), 2-4 August, 2018, Noida, India

II. LITERATURE SURVEY x Programming through SD card

This section discusses broad issues and background of the x Push pop socket Slightly over PCB.
SLI and its various existing SLI algorithms. x Serial communication
A. Broad Issues and Background
The basic issue that is being faced by the majority of the
existing SLI is that they are only capable of detecting single
alphabets from the hand gestures which is commonly known
as finger spelling. Finger spelling is not a practical approach
as it compromises the efficiency of SLI due to repeated delay
[5][1].
The second most concerning issue is the pertaining
ambiguity regarding the authenticity of the sign for different
words and letters for Indian sign language [6]. Many of the
existing systems use signs made by the convenience of the
developer of those products. Such ambiguity in the gestures
leads to a chaotic atmosphere where the user has to learn
product specific signs, which is not prescribed [6][7][8].
B. Sign Language Interpretation Algorithms
x Hidden Markov Models (HMM)
A Hidden Markov model [9] consists of finite states and
each state consisting of two sets of probabilities: (i) A
Fig. 1. Proposed Sign Language Interpreter (SLI) architecture
transition probability, (ii) either a discrete output probability
distribution or continuous output probability density function.
B. Flex Sensors
Each state is characterized by the given state. HMM is a
doubly stochastic model and is appropriate for coping with the Flex sensors are nothing but resistive carbon substrates. On
stochastic properties in gesture recognition. The best gestures bending the substrate, a resistance output correlated to the
are recognized by evaluating the trained HMMs. bend radius is produced [12]. The proposed SLI uses flex
sensors to get the inputs from the fingers in the form of
x Artificial Neural Networks(ANN) resistance, which is further used to calculate the angle. The
An artificial neural network [10] involves a network of calculated angle values, are further mapped to the words and
simple processing elements (artificial neurons) which can words are outputted on speaker.
exhibit complex global behaviour, determined by the
C. Gyroscope with accelerometer
connections between the processing elements and element
The MPU6050 has an embedded 3-axis MEMS gyroscope,
parameters. An interconnected group of artificial neurons
a 3-axis MEMS accelerometer. MPU6050 has been used to
using a connectionist approach processes the signs. As ANN is
calculate the acceleration and rotation angle to distinguish
an adaptive system, it changes its structure based on external
between various levels at which the glove has been rotated.
or internal information that flows through the network. The
effectiveness of ANN models lies in the fact that they are used D. Bluetooth
to infer a sign from the observations. For wireless communication between the gloves,
x Support Vector Machine(SVM) Bluetooth module called the HC-05 is used [13]. HC-05 uses
The SVM [11] uses supervised learning for sign Serial Port Protocol for transparent wireless serial connection
recognition. SVM can handle unknown signs by dividing the setup [13]. Bluetooth module helps in the communication
feature space for each glove. The SVM predicts the output on between the two gloves in turn giving a fixed output by
the basis of the input sign. evaluating the combination of the two inputs provided by the
output.
III. ARCHITECTURE
E. SD-Card Module
The proposed system uses architecture as shown in Fig1. This Micro SD Card is used for transferring data to and
The architecture comprises hardware and software from a standard SD Card. The pinout is directly compatible
components. During experiments, sign language data set has with Arduino and also can be used with other micro-
been created according to the sensors value ranges observed controllers. It allows us to add mass storage and data logging
because of the hand-movements. Following are the hardware to the system [11].
components used in the proposed system.
A. Arduino uno
Arduino is an open-source development board that has on- IV. METHODOLOGY
board microcontrollers, which can be interfaced with various The implementation design and block diagram of the
analog and digital sensors. Some of the features of Arduino proposed system is as shown in Fig 2. Each glove has sensors
are listed below and is activated by powering the microcontroller.
x Input Voltage: 3.3V/5V
 Proceedings of 2018 Eleventh International Conference on Contemporary Computing (IC3), 2-4 August, 2018, Noida, India

At a first step, the flex sensors acquire the various

resistances generated by the movement of the fingers. In the
second step, analog input of flex sensors is provided to the
microcontroller. As the third step, the gyro angle and the
accelerometer angle recorded by MPU6050 (calibration takes
around 10 seconds) are input to the microcontroller. In the
fourth step, microcontroller makes a decision based on the
angles and resistance obtained by sensors of both the gloves.
The decision would be a corresponding sign from the stored
data set. In the fifth step, microcontroller maps appropriate
audio file from SD card. A detailed description of system
working has been discussed in section A to section E.

Fig. 3. Capturing variable finger angle using resistance

After the calculation of the final resistance for that

particular finger the angle is calculated by an inbuilt function
“MAP” of arduino. The dataset entries for a few sample words
have been shown in Table I.

TABLE I. FLEX SENSOR DATASET FOR A FEW SAMPLE WORDS

Angle(in degrees)
Flex-1 Flex-2 Flex-3 Flex-4
Word
Min Max Min Max Min Max Min Max
YOUR 600 800 950 1150 1500 1800 900 1200
NAME 600 800 900 1100 900 1200 1000 1300
LITTLE 550 750 800 1100 1100 1300 1200 1500

B. Mapping the orientation of hand using MPU6050

Fig. 2. Block diagram of proposed SLI

MPU6050 consists of gyroscope and accelerometer and
provides us with the acceleration produced in the x, y, and z-
A. Capturing the finger movement direction (acc _x, acc _y, acc _z) by the hand glove and the inbuilt
gyroscope helps in the x, y, and z-axis calculation (gyro_x,
Flex sensors tap the input from the varying resistance gyro_y, gyro_z). The MPU6050 gives analog input to the
( ) and voltage ( ) produced by the movement of the microcontroller. The block biagram of MPU6050 is shown in
fingers and calculates the angle. This data is fed to the Fig 4. Algorithm-2 is to convert the sensor value to angle and
microcontroller as an analog input. The flow chart of flex Table II gives mapping range of sample words for MPU6050.
sensors is shown in Fig 3. Algorithm-1 used to calculate the Algorithm-2
angle based on the resistance and voltage provided by the flex
sensors. x Step1: calculation of the angle pitch for x, y, z axis, is
given by (3)
Algorithm-1
_ ℎ += _ ∗ 0.0000611 (3)
Intialization: VCC = 4.98v,
R_DIV = 47500.0, x Step2: calculation of the total accelerometer vector is
STRAIGHT_RESISTANCE = 37300.0 given by (4)
BEND_RESISTANCE = 90000.0, _ _ = (( _ ∗ _ )+( _ ∗
Flex ADC as analog input. _ )+( _ ∗ _ )) (4)
Step1:Calculation of the flex voltage( ) is given by (1)
∗
x Step3: calculation of the angle roll output by the formula:
= (1)
. _ _ = _ _ ∗ 0.9 +
Step2:Calculation of the flex resistrance( ) is given by _ ∗ 0.1 (5)
(2)

= R_DIV ∗ (2)
 Proceedings of 2018 Eleventh International Conference on Contemporary Computing (IC3), 2-4 August, 2018, Noida, India

V. RESULTS
The proposed system, interprets the Indian Sign Language
and is represented as an application for dumb people
especially for Indians.
Few samples were tested around 90 words during the
experimentation out of the 3000 words in the Indian Sign
Language. Further, remaining words can be interpreted using
same algorithmic techniques. Initial development of the
proposed system is as shown in Fig.6 and the final
development of the system is as shown in Fig. 7.

Fig. 4. Mapping the orientation of MPU6050

C. Recording audio files

After validating the values from sensors, the

microcontroller maps the appropriate audio file on the SD card
and further outputs the audio file from the speaker.

TABLE II. MPU6050 DATASET MAPPING

Word Orientation(axis angle)

Min value Max value
YOUR 7.8 13.91
NAME 4.56 9.99
Fig. 6. Initial development of the proposed SLI
LITTLE 5.71 10.38
Cost Analysis
D. Bluetooth Connectivity
A detailed description of the cost and quantity of the
HC 05 handles the input of the two gloves by sending the components used in the proposed SLI is shown in Table III.
input of one glove to the other using the pins RX and TX,and The estimated cost is much lower than the existing cost of
then, checking for a combination value from dataset that has SLIs.
been produced. The working details of Bluetooth is as shown
in Fig. 5.

Fig. 7. Final Development of the proposed SLI

Fig. 5. Bluetooth module workflow

VI. CONCLUSION AND FUTURE WORK
Our proposed project, which is the sign interpreter, is a
useful tool for the people who cannot speak as this gives them
 Proceedings of 2018 Eleventh International Conference on Contemporary Computing (IC3), 2-4 August, 2018, Noida, India

a medium to communicate with the rest of the world without [12] HC-05 Bluetooth Module, Itead Studio, online data sheet available
any hesitation or any inconvenience. With the help of various online at http://www.electronicaestudio.com/docs/istd016A.pdf
components like flex sensors, MPU6050, HC-05, SD-Card [13] “How To Use SD Card As Internal Storage On Android | Adoptable
Storage On Android”, Fossbytes, available online at
module etc., we have tried to replicate a few signs from the “https://fossbytes.com/android-sd-card-internal-storage-adoptable-
Indian Sign Language. As our project is in developing stage storage/”, April 23, 2017.
right now we will try to minimize the errors as the flex sensors
may change values when used for a long time. The Indian
Sign Language also has some words which detect wrist
movement and we would work on it so that those signs can be
expressed using our glove. Similarly, elbow movement and
facial recognition would be the two areas where we would
work in the future so that all 3000 words can be expressed
easily using our gloves.

TABLE III. COST ANALYSIS

Component Quantity Price
Flex Sensor 8 Rs.4360
MPU6050 5 Rs.1400
HC-05 2 Rs. 600
Arduino Uno 2 Rs.1000
Speaker 2 Rs.400
SD Card Module 2 Rs.200
Miscellaneous - Rs.500
Total 21 Rs.8460

REFERENCES
[1] Nair, A.V. and Bindu, V., “A review on Indian sign language
recognition”, International Journal of Computer Applications, 73(22),
2013.
[2] Sailee Brahme, “READING THE SIGNS:SIGNS LANGUAGE”, April
21, 2017, available online at
http://forreadingaddicts.co.uk/culture/reading-signs-sign-
language/18395
[3] Republic World, Press Trust of India, "India’s First Sign Language
Dictionary Released”, Mumbai, March 23, 2018
[4] Lucy Sweeney, “Gloves that convert sign language to speech ‘to
empower the deaf community’ ”, ABC News, April 2016.
[5] Divyangjan, “Indian Sign Language Research and Training Centre
(ISLRTC), Department of Empowerment of Persons with Disabilities,
Ministry of Social Justice & Empowerment, Government of India, online
available at http://205.147.97.190/islrtcapp/
[6] Sanjay Kumar, “First Indian Sign Language Dictionary of 3000 Words”,
Press Information Bureau, Government of India, Ministry of Social
Justice & Empowerment, 23-March-2018
[7] Strong, M. and Prinz, P.M., “A Study of the Relationship Between
American Sign Language and English Literacy”, The Journal of Deaf
Studies and Deaf Education, 2(1), pp.37-46, 1997
[8] Divya Deora, D. and Bajaj, N., “Indian sign language recognition”, In
Emerging Technology Trends in Electronics, Communication and
Networking (ET2ECN), 2012 1st International Conference on (pp. 1-5).
IEEE, December 2012.
[9] Yang, J. and Xu, Y., “ Hidden markov model for gesture recognition”,
(No. CMU-RI-TR-94-10), Carnegie-Mellon Univ Pittsburgh Pa
Robotics Inst, 1994.
[10] Kishore, P.V.V. and Kumar, P.R., “A model for real time sign language
recognition system”, International Journal of Advanced Research in
Computer Science and Software Engineering, 2(6) pp. 30-35, 2012.
[11] Lilha, H. and Shivmurthy, D., “Evaluation of features for automated
transcription of dual-handed sign language alphabets”, In Image
Information Processing (ICIIP), 2011 International Conference on (pp.
1-5). IEEE, November 2011.