w w w .ijir d.

com May, 2013 Vol 2 Issue 5

ISSN: 2278 – 0211 (Online)

Indian Sign Language (ISL) Translation System For
Sign Language Learning
M.Jerin Jose
PG Student, Applied Electronics, S.K.P Engineering College, Tiruvannamalai, India
V.Priyadharshni
PG Student, Applied Electronics, S.K.P Engineering College, Tiruvannamalai, India
M.Suresh Anand
Asst. Professor, CSE, Sri Sairam Engineering College, Chennai, India
A.Kumaresan
Head of dept of Computer Science and Engineering, S.K.P Engineering College,
Tiruvannamalai, India
Dr.N.MohanKumar
Professor/Head of Dept of Electronics and Communication Engineering, S.K.P
Engineering College, Tiruvannamalai, India

Abstract:
Sign language is a language which uses visually transmitted sign patterns to convey meaning.
It is the combination of hand shapes, orientation and movement of hands, arms or body, and
facial expressions. Our System is capable of recognizing sign-language symbols can be used
as a means of communication with hard of hearing people. Our paper proposes a system to
help normal people can easily communicate with hard of hearing people. Instead we are
using a camera and microphone as a device to implement the Indian Sign Language (ISL)
system. The ISL translation system has translation of voice into Indian Sign Language. The
ISL translation system uses microphone or USB camera to get images or continuous video
image (from normal people) which can be interpreted by the application. Acquired voices are
assumed to be translation, scale and rotation invariant. In this process the steps of
translation are acquisition of images, binarized, classification, hand shape edge detection
and feature extraction. After getting vectors feature extraction state then pattern matching
done by comparing existing database. The GUI application is displaying and sending the
message to the receiver. This system makes normal people to communicate easily with
deaf/dumb person. Also in video calling or chatting this application helps the hard speaking
and hearing people.

Key words: Indian sign language (ISL), translation, image processing, hard hearing and
hard speaking

INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH & DEVELOPMENT Page 358

 w w w .ijir d.com May , 2013 Vol 2 Issue 5

1.Introduction
In recent years, research has progressed steadily in regard to the use of computers to
recognize and render sign language. Technology is rapidly changing and improving the
way the world operates. Barriers for people who are hard hearing are diminishing as
projects of the past two decades have unfolded. Through the use of image processing,
artificial intelligence and pattern matching techniques, researchers are striving to develop
hardware and software that will impact the way hard hearing individuals communicate
and learn. Using sign language hard speaking and hearing people could communicate
among them and with normal people.
Practically, world’s normal people has been in a difficult situation in the society because
of their inability to communicate vocally with hard hearing people in connection with
that the indifference of others to learn their language, the sign language. With the arrival
of multimedia, animation and other computer technologies, it is now becoming possible
to bridge the communication gap between the normal and hearing-impaired person.
Sign language is a visual/gestural language that serves as the primary means of
communication for hard hearing individuals, just as spoken languages are used among
the hearing. Hard hearing individuals encounter the difficulty that most hearing
individuals communicate with spoken language [12].Generally, there is no problem
when two hard hearing persons communicate using their common sign language. The
real difficulties arise when a normal (non-deaf) person wants to communicate with a hard
hearing person [9].In such scenario there is need of sign language. Sign language (SL) is
the native language for the hard hearing people. Although they successfully
communicate with each other when using sign language. They face many difficulties
when they try to communicate with hearers, especially those who are incompetent in sign
language .Solutions such as pointing, writing, or lip-reading, combined with some new
technological means, i.e., faxes, computers (e.g., e-mails), mobile phones (e.g., SMSs),
facilitate such communication.
There are 700,000,000 deaf or hard of hearing people worldwide (World Health
Organization) and the 143 existing different sign languages (types with dialects). Being
as complex as any spoken language, Sign Language has many thousands of signs formed
by specific gestures (related to the hands) and facial expressions, each differing from
another by minor changes in hand motion, shape, position, and facial expression [17].
What is Sign Language? Sign language is a language which uses visually transmitted
sign patterns to convey meaning. It is the combination of hand shapes, orientation and

INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH & DEVELOPMENT Page 359

 w w w .ijir d.com May , 2013 Vol 2 Issue 5

movement of hands, arms or body, and facial expressions. Sign languages are not
international. Every country has unique sign language. Ex: American Sign Language
(ASL) has its own grammar and rules—it is not a visual form of English. Sign language
is unique for every nation [1].Countries like Arabic, Bangla, and Malay, Taiwanese,
Chinese, Japanese, Spanish and many has their own sign language. There are
approximately 70 million people with hearing deficiencies in the world [1]. Sign
language translator is system which converts voice to text or sign language in any native
language. These systems are called as human computer interaction systems (HCI).It
could be done in two ways (i) rule based translation (ii) statistical based translation.
A sign language translation system would make it possible to communicate with hard
hearing people. Sign languages primarily consist of hand gestures performed by the
hearing impaired as a means of communication. Machine vision methods for sign
language spotting, i.e., detecting and recognizing signs in a known vocabulary, in videos
of sentences and stories produced by native signers. The difficulty of sign language
spotting is that instances of signs vary in both motion and appearance. For dealing with
motion, previous vision-based methods have demonstrated some successes using Hidden
Markov Models [15].
Gesture description involving hand shapes, movement, position, and palm orientation
have been employed to recognize and segment sign image sequences. These methods
focus on image processing more than machine translation and sign language analysis and
have limited vocabularies [12]. Sign Language Translation System or software that
translates voice or text into sign language animations could considerably improve hard
hearing lives especially in communication and accessing information [13]. Sign language
is a visual language and consists of 3 major components they are,
(i) finger-spelling: used to spell words letter by letter
(ii) word level sign vocabulary: used for the majority of communication
(iii) non-manual features: facial expressions and tongue, mouth and body position
In our proposed system we focus spelling alphabets letter by letter, Then natural
language translation module converts a word sequence into a sign sequence..

INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH & DEVELOPMENT Page 360

 w w w .ijir d.com May , 2013 Vol 2 Issue 5

Figure1: Indian Sign Language Alphabet

Now-a-days voice calling-chatting and many facilities are offered for communication.
Hard speaking and hearing peoples couldn't use these facilities effectively or fully. These
people find difficulty in communicating with normal people. Our paper discusses
solution to this problem and proposed a system for translation of sign language using mic
in laptops or multimedia smart mobile phones.

2.Related Works
Gestures are expressive, significant body motions involving physical movements of the
fingers, hands, arms, head, face, or body with the intent of conveying meaningful
information or interacting with the background. The importance of gesture recognition
lies in building efficient human–machine interaction. Its applications range from sign
language recognition through medical rehabilitation to virtual reality. The tools surveyed
for gesture recognition are HMMs, particle filtering and condensation algorithm, FSMs
(Finite State Machines).The sign language translation system Speech to Sign. A practical
low cost visual communication system for sign language translation using low
bandwidth. It is made up of a speech recognizer (for decoding the spoken utterance into a
word sequence), a natural language translator (for converting a word sequence into a
sequence of signs belonging to the sign language), and a 3D avatar animation module
(for playing back the hand movements).

3.Propopsed Methodlogy (ISL Translation System)

In this paper we are proposing our Indian Sign Language translation system. In this
system we are giving two modes of translation. First one is if a deaf-dumb person can
communicate with normal person (who doesn’t know sign language). And the second
one is reverse of first one i.e. a normal person speaks, it is converted to text and then
appropriate text meant sign will be displayed.

INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH & DEVELOPMENT Page 361

 w w w .ijir d.com May, 2013 Vol 2 Issue 5

Figure2: Speech into sign language translation system

3.1.Speech To Sign Translation

The speech from normal person is taken via micro phone of cellular phone or computer.
For the need of good quality of voice signal it will be sent for noise removal. Voice data
is converted to text by speech-recognition module that is voice to text conversion with
the use of trained voice database. The converted text is compared with database for
finding meaning and symbol (sign). Display the Sign Symbol with text to the receiver
(hard speaking and hard hearing person).

Figure3: Speech to sign translation system architecture

INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH & DEVELOPMENT Page 362

 w w w .ijir d.com May , 2013 Vol 2 Issue 5

3.2.Noise Removal
The speech from normal person is taken via micro phone of cellular phone or computer
and it will be sent to noise removal. There are several types of noise removal techniques
like Filtering Techniques (spectral subtraction method, weiner filter, least mean square
filter and kalman filter)[23] ,Spectral restoration(minimum mean square error short time
spectral amplitude) and many. There are another two types of noise removal methods
also, Modulation detection and Synchrony detection. This algorithm speech detector
analyses in signal amplitude. Speech modulations are slow and have big amplitude
fluctuations. Noise modulations are more constant with rapid and smaller fluctuations.

3.3.Speech Recognition
After noise removal the voice sent for speech recognition module. Here the speech
recognizer converts voice into single letter (latter words and sentences).The system uses
context-dependent continuous Hidden Markov Models (HMMs) built using decision tree
state clustering: 1807 states and seven mixture components per state. These models have
been trained with more that 40h of speech from the SpeechDat database. Although
SpeechDat is a telephone speech database, the acoustic models can be used in a
microphone application because CMN and CVN techniques have been used to
compensate the channel differences. The influence of this aspect in the speech
recognition results is small. For new user voice training has to be done for making
trained voice database. In testing phase or real time the voice database used for quick and
easy recognition .This paper proposes for single letter or alphabet into Indian way of
Sign Language. Then we planned to implement this idea for individual words and then
continuous word spoken.

3.4.Domain And Database

The experimental framework is restricted to a limited domain that consists of sentences
spoken by normal people. In this context, a speech to sign language translation system is
very useful since most of the normal persons do not know sign language and have
difficulties when interacting with Deaf people. Once the ISL encoding was established, a
professional translator translated the original set into ISE making use of more than 320
different signs. Then, the 416 pairs were randomly divided into two disjoint sets: 266 for
training and 150 for testing purposes. The speech-to-sign translation purpose the same

INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH & DEVELOPMENT Page 363

 w w w .ijir d.com May, 2013 Vol 2 Issue 5

test set has been used. The test sentences were recorded by two speakers (1 male and 1
female).

3.5.Rule Based Text To Sign Matching

Recognized voice is now in the form of text that is voice to text conversion module gives
us text output. Then text to sign matching is done by rule based technique. Here the
relationship between text and sign has been defined. Here the natural language
translation module considering a bottom-up strategy. In this case, the relationship
between signs and words are defined by an expert hand. In a bottom-up strategy, the
translation analysis is carried out by starting from each word individually and extending
the analysis to neighborhood context words or already-formed signs (generally named
blocks). This extension is made to find specific combinations of words and/or signs
(blocks) that generate another sign.
The translation process is carried out in two steps. In the first one, every word is mapped
to one or several syntactic– pragmatic tags. After that, the translation module applies
different rules that convert the tagged words into signs by means of grouping concepts or
signs (blocks) and defining new signs. These rules can define short and large scope
relationships between the concepts or signs. At the end of the process, the block
sequence is expected to correspond to the sign sequence resulting from the translation
process.
The rule-based translation module contains 153 translation rules. For evaluating the
performance of the systems, the following evaluation measures have been considered:
SER (Sign Error Rate), PER (Position Independent SER), BLEU (BiLingual Evaluation
Understudy), and NIST. The first two measures are error measures (the higher the value,
the worse the quality) whereas the last two are accuracy measures (the higher, the better).
Forward and continuous matching we have to more because of word meanings and
context meaning in sentence (continuous speech). The relationship rules are defined
carefully with extreme logic to achieve good translation.

Table 1: Results obtained with the rule-based translation system

INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH & DEVELOPMENT Page 364

 w w w .ijir d.com May , 2013 Vol 2 Issue 5

3.5.1.Sign Confidence Measure

The translation module generates one confidence value for every sign: a value between
0.0 (lowest confidence) and 1.0 (highest confidence). This sign confidence is computed
from the word confidence obtained from the speech recognizer. This confidence
computation is carried out by an internal procedure that is coded inside the proprietary
language interpreter that executes the rules of the translation module.
In other more complex cases, the confidence for the generated signs may be dependent
on a weighted combination of confidences from a mixture of words and/or internal or
final signs. This combination can consider different weights for the words or concepts
considered in the rule. These weights are defined by the expert as the same time the rule
is coded.

3.6.Delays Between The Spoken Utterance And The Sign Animation

One important aspect to be considered in a speech to sign language translation system is
the delay between the spoken utterance and the animation of the sign sequence. This
delay is around 1–2 s and it slows down the interaction. In order to reduce this delay, the
speech recognition system has been modified to report partial recognition results every
100ms. These partial results are translated into partial sign sequences that can be
animated without the need to wait until the end of the spoken utterance. Our system 40%
delay reduction is achieved without affecting the translation process performance.

4.Conclusion
Our paper is to help those people to communicate with hard hearing people without
sophisticated devices like power and etc. The ISL translation system is speech to sign a
normal person speaks, it is converted to text and then appropriate text meant sign will be
displayed. We planned to extend this idea for words and sentences by using new
methodologies. We planned to implement the complete idea of this paper in smart
mobile phones also. The Challenge in implementing this idea in mobile phone is
implementing the methods used for image processing.

INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH & DEVELOPMENT Page 365

 w w w .ijir d.com May , 2013 Vol 2 Issue 5

5.Reference
1. O. Aran, I. Ari, L. Akarun, B. Sankur, A. Benoit, A. Caplier, P. Campr,A.H. Carrillo,
and F.-X. Fanard, “SignTutor: An interactive system for sign language tutoring,”
IEEE Multimedia, vol.16, pp. 81–93, 2009.
2. R. San-Segundo , R. Barra , R. Co´rdoba , L.F. D’Haro , F. Ferna´ndez , J. Ferreiros
,J.M. Lucas , J. Macı´as-Guarasa , J.M. Montero , J.M. Pardo “Speech to sign
language translation system for Spanish” ,Speech Communication 50 (2008) 1009–
1020
3. S. Mitra and T. Acharya, “Gesture recognition: A survey,” IEEE Transactions on
Systems, Man, and Cybernetics, Part C: Applications and Reviews, vol. 37, pp. 311-
324, 2007
4. M. D. Manoranjan and John A. RobinsonAsda, “Practical low-cost visual
communication using binary images for deaf sign language”, IEEE Transactions on
Rehabilitation Engineering ,vol. 8,pp 81-88 ,March 2000.
5. Omar Al-Jarrah,Alaa Halawani, ”Recognition of gestures in Arabic sign language
using neuro-fuzzy systems ”Artificial Intelligence :ElseVier Artificial Intelligence -
AI , vol. 133, no. 1-2, pp. 117-138, 2001.
6. David M. Saxe and Richard A. Foulds, “Robust Region of Interest Coding for
Improved Sign Language Telecommunication”, IEEE Transactions on Information
Technology in Biomedicine, vol. 6, no. 4, pp.310-316, December 2002
7. Gaolin Fang, Wen Gao, and Debin Zhao,“Large Vocabulary Sign Language
Recognition Based on Fuzzy Decision Trees “IEEE Transactions on Systems, Man,
and Cybernetics —Part A: System and Humans, vol. 34, NO. 3, pp. 305-314, May
2004.
8. Chung-Hsien Wu, Yu-Hsien Chiu, and Chi-Shiang Guo, “Text Generation From
Taiwanese Sign Language Using a PST-Based Language Model for Augmentative
Communication”, IEEE Transactions on Neural Systems and Rehabilitation
Engineering, vol. 12, no. 4, pp.441-454, December 2004.
9. Khaled Assaleh, M. Al-Rousan, “Recognition of Arabic Sign Language Alphabet
Using Polynomial Classifiers” EURASIP Journal on Applied Signal Processing
2005:13,2136–2145
10. M. Mohandes, S. Buraiky, “Automation of the Arabic Sign Language Recognition
using the PowerGlove”, AIML Journal, Volume 7, Issue 1, June, 2007.

INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH & DEVELOPMENT Page 366

 w w w .ijir d.com May , 2013 Vol 2 Issue 5

11. Sylvie C.W. Ong and Surendra Ranganath, “Automatic Sign Language Analysis: A
Survey and the Future beyond Lexical Meaning”, IEEE Transactions on Pattern
Analysis and Machine Intelligence, vol. 27, no. 6,pp.873-891, June 2005
12. Yu-Hsien Chiu, Chung-Hsien Wu, Hung-Yu Su, and Chih-Jen Cheng, “Joint
Optimization of Word Alignment and Epenthesis Generation for Chinese to
Taiwanese Sign Synthesis IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 29, no. 1,pp.28-39, January 2007
13. Dr.Sami M.Halawani, “Arabic Sign Language Translation System On Mobile
Devices”, IJCSNS International Journal of Computer Science and Network Security,
vol.8,no.1, pp. 251-256,January 2008.
14. R. San-Segundo , R. Barra, R. Co´rdoba, L.F. D’Haro, F. Ferna´ndez, J.
Ferreiros,J.M. Lucas, J. Macı´as-Guarasa , J.M. Montero , J.M. Pardo, “Speech to sign
language translation system for Spanish”, Speech Communication 50 (2008) 1009–
1020, www.sciencedirect.com.
15. Hee-Deok Yang,Stan Sclaroff,andSeong-Whan Lee,“Sign Language Spotting with a
Threshold Model Based on Conditional Random Fields”, IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 31, no. 7,pp.1264-1277, July 2009
16. Hung-Yu Su and Chung-Hsien Wu, “Improving Structural Statistical Machine
Translation for Sign Language with Small Corpus Using Thematic Role
Templates as Translation Memory ”, IEEE Transaction on Audio ,Speech and
language Processing ,vol. 17, no. 7,pp.1305-1315 September 2009.
17. Vasiliki E. Kosmidou, and Leontios J. Hadjileontiadis, “Sign Language
Recognition Using Intrinsic-Mode Sample Entropy on sEMG and Accelerometer
Data”, IEEE Transaction on Biomedical Engineering, vol. 56, no. 12,pp.2879-
2890, December 2009.
18. Ruiduo Yang, Sudeep Sarkar, and Barbara Loeding, “Handling Movement
Epenthesis and Hand Segmentation Ambiguities in Continuous Sign Language
Recognition Using Nested Dynamic Programming”, IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 32, no. 3,pp. 462-477, March
2010
19. Biswajit Sarkar, Kaushik Datta, C. D. Datta, Debranjan Sarkar , Shashanka J.
Dutta, Indranil Das Roy, Amalesh Paul, Joshim Uddin Molla, Anirban Paul, “A
Translator for Bangla Text to Sign Language”,India Conference INDICON
2009,Annaul IEEE feb 2010.

INTERNATIONAL JOURNAL OF INNOVATIVE RESEARCH & DEVELOPMENT Page 367