2012 International Conference on Computer Communication and Informatics (ICCCI -2012), Jan.

10 – 12, 2012, Coimbatore, INDIA

Hand Gesture Recognition for Indian Sign Language

1
Archana S. Ghotkar, 2Rucha Khatal , 3Sanjana Khupase, 4Surbhi Asati & 5Mithila Hadap
Department of Computer Engineering,
Pune Institute of Computer Technology, Pune, India.
E-mail : [email protected] , [email protected], [email protected], [email protected],
5
[email protected]

Abstract- In this paper, we introduce a hand gesture II. LITERATURE SURVEY

recognition system to recognize the alphabets of
Indian Sign Language. In our proposed system there In general there are 3 main categories of object tracking-
are 4 modules: real time hand tracking, hand point tracking, kernel tracking and silhouette tracking. A
segmentation, feature extraction and gesture comparative study of different object tracking methods
recognition. Camshift method and Hue, Saturation, has been listed in [2]. The various state of art tracking
Intensity (HSV) color model are used for hand methods and techniques has been discussed in [1]. From
tracking and segmentation. For gesture recognition, this paper it is seen that Camshift method has real time
Genetic Algorithm is used. We propose an easy-to-use performance and it can track accurately in presence of
and inexpensive approach to recognize single handed distracters and noise. Kalman filter method has error
as well as double handed gestures accurately. This classification rate of 86% [2].Speeded-Up-Robust-
system can definitely help millions of deaf people to Features (SURF) is able to detect rotating object but it
communicate with other normal people. fails to detect highly scaled object. KLT is only for affine
motion but not suitable for translational, rotational nor
Keywords- Indian Sign Language, Hand Tracking, scaling motion [1]. There are other methods of tracking
Segmentation, Feature Extraction, Gesture Recognition, like robust multi-cue hand tracking achieved by velocity
Single handed Gestures, Double handed Gestures. weighted features and color cue implemented by
Zhigeng[3]. Liu [4] uses Viola-Jones method for hand
I. INTRODUCTION detection. In [5], Tower method and skin color are used
for hand gesture recognition tracking and control. From
Sign language is the most natural and expressive way for this literature survey we found out that Camshift method
the hearing impaired. The Indian Sign Language was is best suited for our System since it is inexpensive i.e. it
proposed by Government of India so that there is a can be used with the in-built web camera and gives good
uniform sign language that can be used by all the deaf and result in terms of speed and accuracy.
dumb people in the country. Till now very little work has
been done to develop software that translates this sign For image segmentation there are various methods like
language into corresponding text and voice. Automatic The Adaptive Color Image Segmentation System (ACIS)
sign language recognition offers enhancement of [6] uses HSV color model and neural network for
communication capabilities for the speech and hearing segmentation. The given method does not require a prior
impaired. It promises improved social opportunities and knowledge of the image and its performance is efficient
integration in the society to these people. on noisy images. The Automatic Seeded Region Growing
Algorithm (ASRGC) [7] uses YCbCr model for image
The idea of our project is to design a gesture recognition segmentation along with the techniques of automatic seed
system that will automatically capture, recognize and selection, region growing corresponding to each pixel and
translate the alphabets of Indian Sign Language into merging of similar regions. YCbCr model has been used
corresponding text and voice in a vision based setup. In in [8] to design a self adaptive skin color model and a
our project we propose to recognize single handed as well Gaussian model classifier using BP neural network.
as double handed gestures accurately with a single normal Different region based and object based segmentation
webcam using bare human hands. The aim of our project approaches have been discussed in [9]. The HSV, color
is to recognize the gestures with highest accuracy and in space describes color with intuitive values. The
least possible time. To recognize the gestures, our system intuitiveness of the color space components and explicit
consists of 4 modules: Hand Tracking and Segmentation, discrimination between luminance and chrominance
Feature Extraction, Gesture Recognition, Application properties works effectively for skin color segmentation
Interface. [10]. So we have chosen HSV color model for hand
segmentation.

978-1-4577-1583-9/ 12/ $26.00 © 2012 IEEE

 2012 International Conference on Computer Communication and Informatics (ICCCI -2012), Jan. 10 – 12, 2012, Coimbatore, INDIA

After segmentation, we have to search the input image in have been used for Gesture Recognition system. Genetic
a database. The image has to be described or represented algorithms [15][16][17] are a class of parallel adaptive
by certain features. Shape is an important visual feature of search algorithms based on the mechanics of natural
an image. There are various methods used in Feature selection and natural genetic system. Over the last twenty
Extraction. [20] describes those methods: Contour shape years, it has been used to solve a wide range of search,
techniques only exploit shape boundary information. optimization, and machine learning. GA has great
Methods based on the Hausdorff distance are useful for advantages over other techniques like it is easy to
locating objects in an image or sub-image matching. The understand, it supports multi-objective optimization, it is
retrieval performance of the Generic Fourier descriptor good for “noisy” environments. GA is inherently parallel
(GFD) demonstrates it is a desirable solution to generic and easily distributed.
shape representations. If storage is a concern, Fourier
Descriptor (FD) can be considered. III.PROPOSED SYSTEM
There have been varied approaches of gesture recognition The Gesture Recognition System takes the input hand
ranging from statistical modeling such as Principal gestures through the in-built web camera. Hand tracking is
component Analysis (PCA)which is used to extract the done using Camshift method. Then the Segmentation of
features from the hand gesture’s images, Hidden Markov hands is carried out by using HSV color model. The
Model (HMM)[11][12]which is a double stochastic segmented hand image is represented using certain
process governed by a Markov chain and is used to model features. These features are used for gesture recognition
wide range of data, to approaches based on soft using the Genetic Algorithm which gives optimized
computing tools such as Artificial Neural results. The final result obtained is converted into
Networks(ANN) [13][14]useful for complex pattern corresponding text and voice.
recognition and classification tasks, Genetic Algorithms
(GA)[15][16] is used for feature selection problem. Our system consists of 4 modules: Hand Tracking,
Techniques like Support Vector Machines (SVM)[18] Segmentation, Feature Extraction and Gesture
employed for classification, particle filtering and Recognition.
condensation algorithm, Finite State Machine(FSM)[19]

Figure 1.System Overview

A. Hand Tracking B. Segmentation

For hand tracking from the captured image, we have After tracking the hands, our next step is to segment the
chosen the general Camshift algorithm which is hands from the background. The HSV color model is used
summarized in following steps: for this purpose.
1. Chose the initial region of interest which contain
the hands we want to track C. Feature Extraction
2. Make a color histogram of that region as the In feature extraction several general purpose features are
object model. extracted and the relationship between features and
3. Make a probability distribution of frame using classes is inferred by an appropriate classifier. These
the color histogram. features have been selected from shape representation
4. Based on the probability distribution image find proposed in [20] and are listed below: Hausdoff Distance,
the center mass of the search window using Fourier Descriptor.
mean-shift method.
5. Center the search window to the point taken from D. Gesture Recognition
step 4 and iterate step 4 until convergence. After extracting features of the input character, we search
6. Process the next frame with the search window its features into the database and consider the most similar
position from the step 5. features as the result.
 2012 International Conference on Computer Communication and Informatics (ICCCI -2012), Jan. 10 – 12, 2012, Coimbatore, INDIA

Genetic Algorithm is a good option for managing the Fk € F where 1 <= k <=26
randomness of natural samples, and seems to be suitable Following functions can be mapped onto the elements
for Hand gestures analysis. Genetic Algorithm [7][8] is of the set:
summarized in following steps:- 1. Let fe be a rule of I to O such that for given input; it
returns output text.
1. Initialize: create an initial population. fe(I) |
O.
2. Evaluate: calculate the fitness of each individuals For example,
in the population. fe(I0) |
{B}
O.
3. Choice parents: select individuals that have
fitness between a given ranges from the IV. CONCLUSION
population to be the parents.
4. Cross2x: perform 2-point crossover (2x) on the In this paper a real time hand gesture recognition system
parents to create new individuals. is proposed by means of Camshift method, HSV color
5. Mutation: it takes the new individuals created by model and Genetic algorithm. The proposed gesture
cross2x procedure and randomly changes some recognition system can handle different types of hand
genes from it. gestures in a common vision based platform. The system
6. Replacement: it takes five elements randomly is suitable for both single handed and double handed
from the population, choose worst one and then gestures.
replace it with a new individuals.
7. Check stop: stop the system if best solution The system makes use of bare hands for interacting with
found. the computer and is inexpensive, therefore facilitating the
deaf-dumb people to use it. The proposed system is
III.MATHEMATICAL MODEL effective in recognizing the alphabets of Indian Sign
Language. This system can definitely help millions of
The mathematical model of our system is: deaf people to communicate with other normal people.
Let G be a hand gesture recognition system that
recognizes hand gesture; such that V. FUTURE WORK
G = {I, SG, DG, F, O |
s}
where In our work, we showed that our proposed gesture
I is a set of input hand gestures; recognition system can recognize only the alphabets of
I = {I1, I2…} Indian Sign Language. The future work includes
SG represents a set of single-handed gestures of manual recognizing different other gestures in Indian Sign
alphabets; Language.
SG I
SG ={C, I, J, L, O, U, V}; ACKNOWLEDGEMENT
DG represents a set of double-handed gestures of manual We are sincerely thankful to Prof. (Dr.) G. K. Kharate for
alphabets; his valuable guidance and suggestion for the proposed
DG I work.
DG ={A,B,D,E,F,G,H,K,M,N,P,Q,R,S,T,W,X,Y,Z};
F represents Feature Set ; REFERENCES
F = {f1, f2…f26}
O is a set of output alphabets in form of text; [1]. Alper Yilmaz, Omar Javed, Mubarak Shah, “Object Tracking: A
Survey”, ACM Computing Surveys, Vol. 38, No. 4, Article 13,
O = {A…Z |
} December 2006.
Initialization: We initialize the feature set consisting of [2]. Fariborz Mahmoudi, Mehdi Parviz, “Visual Hand Tracking
features of 26 alphabets. Algorithms,” Proceedings of the Geometric Modeling and
F = {f1, f2…f26} Imaging – New Trends (GMAI’06), 2006.
[3]. Zhigeng Pan, Yang Li, Mingmin Zhang, Chao Sun, Kangde
The output set is initialized to NULL. Guo, Xing Tang, Steven Zhiying Zhou, “A Real-time Multi-cue
O= {

Hand Tracking Algorithm Based on Computer Vision,” IEEE
The success state of the system is: Virtual Reality 2010, 20-24 March, Waltham, Massachusetts,
Ii=Fj where Ii € I USA.
[4]. Liu Yun, Zhang Peng, “An Automatic Hand Gesture
Fj € F where 1 <= j <=26 Recognition System Based on Viola-Jones Method and SVMs”.
The failure state of the system is: 2009 Second International Workshop on Computer Science and
1. Ii≠Fj where Ii € I Engineering.
Fj € F where 1 <= j <=26 [5]. Pham The Bao, Nguyen Thanh Binh, “A New Approach To
Hand Tracking and Gesture Recognition by a new feature type
2. Ii=Fj and Ii=Fk where Ii € I and HMM”, 2009 Sixth International Conference on Fuzzy
Fj € F where 1 <= j <=26 and Systems and Knowledge Discovery.
 2012 International Conference on Computer Communication and Informatics (ICCCI -2012), Jan. 10 – 12, 2012, Coimbatore, INDIA

[6]. Deshmukh K.S., Shinde G. N, “An Adaptive Color Image [16]. Ho-Duck Kim, Chang-Hyun Park, Hyun-Chang Yang, and
Segmentation", Electronic Letters on Computer Vision and Kwee-Bo Sim,” Genetic Algorithm Based Feature Selection
Image Analysis 5(4):12-23, 2005. Method Development for Pattern Recognition”, SICE-ICASE
[7]. Frank Y. Shih, Shouxian Cheng, “Automatic seeded region International Joint Conference 2006 Oct. 18-2 1, 2006.
growing for color image segmentation”, Image and Vision [17]. Angel Dacal-Nieto1,2, Esteban Vázquez-Fernández1,3, Arno
Computing 23(2005) 877-886. Formella2, Fernando Martin3, Soledad Torres-Guijarro1,
[8]. Guoliang Yang, Huan Li, Li Zhang and Yue Cao,”Research On Higinio González-Jorge1,”A genetic algorithm approach for
a Skin Color Detection Algorithm Based on Self-adaptive Skin feature selection in potatoes classification by computer vision”,
Color Model”, 2010 International Conference On 2009 IEEE.
Communications And Intelligence Information Security. [18]. Majida Ali Abed , Ahmad Nasser Ismail and Zubadi Matiz
[9]. Noel O’Connor, Sorin Sav, Tomasz Adamek, Vasileios Mezaris, Hazi,” Pattern recognition Using Genetic Algorithm”,
Ioannis Kompatsiaris, International Journal of Computer and Electrical Engineering,
[10]. Tsz Ying Lui, Ebroul Izquierdo, Christian Ferran Bennström, Vol. 2, No. 3, June, 2010
Josep R Casas,” Region and Object Segmentation Algorithms in [19]. Yu Yuan and Kenneth Barner, ”Hybrid Feature Selection for
the Qimera Segmentation Platform”. Gesture Recognition using Support Vector Machines”, ICASSP
[11]. Ryzard S. Choras, “Hand Shape and Hand Gesture 2008.
Recognition”, 2009 IEEE Symposium on Industrial Electronics [20]. Sushmita Mitra and Tinku Acharya, “Gesture Recognition: A
and Applications (ISIEA 2009), October 4-6, 2009. Survey”, IEEE Transactions on Systems, MAN, and
[12]. Nguyen Dang Binh, Toshiaki Ejima, ”Real-Time Hand Gesture Cybernetics—Part C: Applications and Reviews, Vol. 37, No. 3,
Recognition using Pseudo 3-D Hidden Markov Model”, Proc. May 2007.
5th IEEE Int. Conf. on Cognitive Informatics (ICCI'06), 2006. [21]. Dengsheng Zhang, Guojun Lu,“Review of shape representation
[13]. Assoc Prof Abd Manan Ahmad, Dr Abdullah Bade, et al. and description techniques” 2003 Pattern Recognition Society.
”Using Principal Component Analysis And Hidden Markov Pattern Recognition 37 (2004) 1 – 19.
Model For Hand Recognition Systems”, 2009 International
Conference on Information and Multimedia Technology.
[14]. Stefan Oniga, Alin Tisan, Daniel Mic, Attila Buchman and
Andrei Vida-Ratiu, ”Hand Postures Recognition System Using
Artificial Neural Networks Implemented in FPGA”, 30th ISSE
2007.
[15]. Md. Rezwanul Ahsan, Muhammad Ibn Ibrahimy, Othman O.
Khalifa, ”Electromygraphy (EMG) Signal based Hand Gesture
Recognition using Artificial Neural Network (ANN)”, 2011 4th
International Conference on Mechatronics (ICOM), 17-19 May
2011.