Volume 9, Issue 8, August – 2024 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24AUG154

Hand Gesture Recognition Using Deep Learning

Sahilee Misal1
M.E. Computer Engineering, Terna College of Engineering, Navi Mumbai
In Guidance of Prof. Ujwala Gaikwad, Computer Engineering Dept, Terna College of Engineering, Navi Mumbai

Abstract:- Hand gesture recognition (HGR) has gained making them ideal for recognizing hand postures in gesture
significant attention due to its potential for various recognition applications [5].
applications. This paper explores the use of deep learning,
specifically Convolutional Neural Networks (CNNs), for This research paper delves into the exploration of CNNs
HGR using the TensorFlow library. We investigate for hand gesture classification using the TensorFlow
existing research on CNN-based HGR, focusing on image framework. We aim to demonstrate the effectiveness of
classification tasks. We then provide a brief overview of CNNs in automatically extracting relevant features from hand
CNNs and their suitability for image recognition. gesture images and achieving accurate classification of
Subsequently, we describe the typical workflow of a deep various hand gestures.
learning-based HGR system, including data
preprocessing, hand detection, feature extraction with II. RELATED WORK
CNNs, and classification. We highlight the advantages of
using TensorFlow to build and train CNN models for Deep learning has revolutionized the field of Hand
HGR. Finally, we conclude by summarizing the key Gesture Recognition (HGR), achieving significant
findings from related work and mentioning the specific advancements in gesture classification accuracy. This section
dataset and number of gestures classified in our research. explores existing research on HGR using deep learning,
This work contributes to the growing body of research on focusing particularly on Convolutional Neural Networks
CNN-based HGR using TensorFlow and emphasizes its (CNNs) for image-based gesture recognition.
potential for developing accurate and efficient HGR
systems.  HGR with Static Images and CNNs
Several studies have employed CNNs for HGR using
Keywords:- Hand Gesture Recognition, Machine Learning, static hand gesture images. Oˇzer et al. [6] proposed a CNN
CNNs, Hand Detection, Feature Extraction, Tensorflow, architecture for sign language recognition using the American
Image Classification. Sign Language (ASL) dataset. Their model achieved an
accuracy of 95.2%, demonstrating the effectiveness of CNNs
I. INTRODUCTION for static gesture classification. Similarly, Hasan et al. [7]
utilized VGG16, a pre-trained deep learning model, for finger
Hand Gesture Recognition (HGR) technology has counting tasks on a custom finger gesture dataset. They
emerged as a revolutionary approach for human-computer achieved an accuracy of 97.2%, highlighting the potential of
interaction (HCI) by enabling intuitive communication transfer learning with CNNs for HGR applications.
through hand movements. Applications of HGR span diverse
fields, including virtual reality control [1], sign language  HGR with Video Sequences and CNNs
translation [2], and augmented reality interfaces [3]. Research has also explored CNNs for HGR using video
Traditionally, machine learning algorithms relied on hand- sequences, capturing the temporal dynamics of gestures. Li et
crafted features for gesture classification, which often al. [8] presented a CNN-Long Short-Term Memory (LSTM)
required extensive domain knowledge and limited the hybrid network for sign language recognition on a video
effectiveness of the system. dataset. The model achieved an accuracy of 92.7%,
showcasing the benefit of combining CNNs for spatial feature
The recent surge in Deep Learning (DL) has extraction with LSTMs for temporal modeling in video-based
significantly transformed the landscape of HGR. Unlike HGR.
traditional machine learning, DL eliminates the need for
manual feature extraction by automatically learning these  HGR with Specific Applications and Architectures
features from data through a hierarchical architecture of Deep learning-based HGR systems have been
artificial neural networks [4]. This empowers DL models to developed for various applications. Hamdi et al. [9] proposed
capture complex relationships within the data, leading to a CNN architecture for human-computer interaction using
superior performance in pattern recognition tasks like HGR. finger gestures on the NYU hand gesture dataset. Their model
achieved an accuracy of 90.4%, demonstrating the
Convolutional Neural Networks (CNNs) represent a applicability of CNNs for gesture-based control interfaces. In
specific type of Deep Learning architecture particularly well- contrast, Yuan et al. [10] employed a PointNet++
suited for image classification tasks. CNNs leverage architecture, a 3D deep learning model, for hand pose
convolutional layers that extract spatial features from images, estimation in grasping tasks on the FreiHand dataset. Their

IJISRT24AUG154 www.ijisrt.com 69
Volume 9, Issue 8, August – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24AUG154

model achieved high accuracy in estimating 21 hand  Data Preprocessing: Data preprocessing steps are crucial
keypoints, highlighting the use of specialized architectures before feeding images into the CNN model. These may
like PointNet++ for 3D hand posture estimation tasks. include resizing images to a uniform size, normalizing
pixel values to a specific range (e.g., 0-1 or -1 to 1), and
 Comparison with Current Work data augmentation techniques (e.g., random cropping,
This research aligns with the existing work using CNNs flipping) to improve model robustness and generalization.
for image-based HGR classification on datasets like those  Hand Detection: In some scenarios, especially when
available on Kaggle. Our work utilizes the TensorFlow dealing with complex backgrounds, isolating the hand
library and a custom CNN architecture to classify five region within the image might be necessary. Techniques
different hand gestures from a downloaded Kaggle dataset. like background subtraction, skin color segmentation, or
While similar to existing approaches, our work focuses on pre-trained hand detection models can be employed to
developing a robust and accurate CNN model for a specific identify the hand region of interest.
gesture classification task with the potential to be extended to  Feature Extraction using CNNs: The preprocessed
recognize a wider range of gestures with additional training image data, containing the hand region, is then fed into
data. the CNN architecture. The convolutional layers within the
CNN automatically extract relevant features from the
 Limitations of Existing Work hand image. These features represent the visual
Several limitations exist in current deep learning-based characteristics of the hand gesture, such as the
HGR research. Many studies rely on controlled laboratory arrangement of fingers, palm orientation, and curvature.
settings, limiting the generalizability of models to real-world  Classification using Softmax Function: Finally, the
scenarios with varying lighting and background conditions. extracted features are used for gesture classification. The
Additionally, the computational cost of training deep learning CNN model typically employs a fully connected layer
models can be high, requiring significant resources. followed by a Softmax function (or similar approaches
like categorical cross-entropy) to predict the most likely
 Future Directions hand gesture category from the input image. The Softmax
Future research directions in HGR using deep learning function assigns a probability score to each gesture class,
include exploring techniques for improved robustness against ultimately determining the most probable gesture based
background variations and illumination changes. on the extracted features.
Furthermore, research on real-time gesture recognition
systems using lightweight deep learning models for mobile By leveraging the power of CNNs for feature extraction
and embedded devices holds significant promise. and classification, deep learning-based HGR systems achieve
high accuracy in recognizing hand gestures, paving the way
III. HAND GESTURE RECOGNITION WITH for diverse human-computer interaction, sign language
DEEP LEARNING recognition, and augmented reality applications.
Deep learning has emerged as a powerful tool for Hand IV. TENSORFLOW FOR HAND GESTURE
Gesture Recognition (HGR), achieving significant RECOGNITION
advancements in gesture classification accuracy. This section
delves into the core concepts behind deep learning-based TensorFlow is a popular open-source deep-learning
HGR systems, focusing on Convolutional Neural Networks library developed by Google. It provides a comprehensive
(CNNs) and their suitability for image recognition tasks. framework for building, training, and deploying machine
learning models, particularly deep neural networks.
 Convolutional Neural Networks (CNNs) TensorFlow offers several advantages for building and
CNNs represent a particular type of deep learning training Convolutional Neural Networks (CNNs) used in
architecture specifically designed for image recognition. Hand Gesture Recognition (HGR) systems:
They excel at extracting spatial features from image data due
to their unique architecture. CNNs consist of convolutional  Ease of Use: TensorFlow provides a user-friendly API
layers that employ filters to learn features like edges, shapes, with high-level abstractions, making it accessible to
and textures from the input image. These features are then developers with varying levels of machine learning
processed by pooling layers that downsample the data while expertise.
preserving the most relevant information. Through a series of  Flexibility: TensorFlow supports a wide range of deep
convolutional and pooling layers, CNNs can effectively learning architectures and functionalities, allowing for
capture hierarchical features from simple to complex, customization and experimentation with different CNN
ultimately leading to robust image classification capabilities models for HGR tasks.
[4].  Scalability: TensorFlow can leverage GPUs and TPUs
for efficient training of large and complex CNN models,
 Deep Learning Workflow for HGR Systems making it suitable for handling big datasets commonly
A typical deep learning-based HGR system follows a encountered in HGR applications.
specific workflow:

IJISRT24AUG154 www.ijisrt.com 70
Volume 9, Issue 8, August – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24AUG154

This research project utilized several functionalities We presented a CNN architecture trained on a downloaded
within TensorFlow to construct and train the CNN model for Kaggle dataset to classify five different hand gestures. The
hand gesture classification. Here's a breakdown of some key model achieved promising results, demonstrating the
functionalities employed: capability of CNNs for automatic feature extraction and
accurate gesture classification.
 Data Augmentation: TensorFlow offers built-in
functions for data augmentation techniques like random This work contributes to the growing body of research
cropping, flipping, and rotations. These techniques were on deep learning-based Hand Gesture Recognition (HGR)
used to artificially expand the training dataset, improving systems. By leveraging TensorFlow's functionalities for data
the model's robustness and generalization capabilities to augmentation, training optimization, and activation
unseen variations in hand gestures. functions, we successfully built a robust CNN model. The
 Sampling: Techniques like random shuffling during data potential for extending this model's recognition capabilities
preparation and mini-batch training were implemented to a wider range of gestures with additional training data
using TensorFlow functionalities. Shuffling ensures the highlights its scalability and adaptability.
model is exposed to diverse samples within the training
data, while mini-batch training allows for efficient Future research directions in this domain include
training on large datasets by processing data in smaller exploring techniques for improved robustness against
batches. background variations and illumination changes.
 Activation Functions: The Rectified Linear Unit (ReLU) Additionally, investigating real-time gesture recognition
activation function was employed within the systems using lightweight deep learning models for mobile
convolutional layers of the CNN model. ReLU introduces and embedded devices presents exciting possibilities for
non-linearity into the network, allowing it to learn practical applications.
complex relationships within the hand gesture data.
TensorFlow provides pre-built implementations of In conclusion, this research demonstrates the
activation functions like ReLU for seamless integration effectiveness of CNNs for hand gesture classification using
into the CNN architecture. TensorFlow. The developed model paves the way for further
 Softmax Function: The Softmax function was utilized in advancements in HGR technology, opening doors for
the final layer of the CNN model for gesture innovative human-computer interaction interfaces and other
classification. TensorFlow offers the softmax function gesture-based applications.
within the tf.nn module, enabling the model to assign
probability scores to each gesture class and predict the REFERENCES
most likely gesture based on the extracted features.
 Conv2D Layer: The core building block of the CNN [1]. O. Hassan, P. Khamis, A. Elgammal, and A. Mittal,
architecture, the convolutional layer, was implemented "A multimodal deep learning approach for vr object
using the Conv2D function within TensorFlow's manipulation using gaze and hand gestures," in 2019
tf.keras.layers module. This function allows for defining IEEE International Conference on Image Processing
the filter size, number of filters, and other parameters for (ICIP), pp. 1477-1481, 2019. [doi:
feature extraction from the hand gesture images. 10.1109/ICIP.2019.8863224]
 Categorical Cross-Entropy: The categorical cross- [2]. J. C. Bezanson, V. Sharma, S. Member, S. Member,
entropy loss function was used to measure the difference M. R. McKinnery, and M. A. Isenhour, "A deep
between the predicted gesture probabilities and the true learning-based sign language recognition system
gesture labels during training. TensorFlow provides the using continuous Hidden Markov models," in 2018
categorical_crossentropy function within the IEEE International Conference on Systems, Man, and
tf.keras.losses module, facilitating efficient training by Cybernetics (SMC), pp. 1-7, 2018. [doi:
calculating the loss between the model's predictions and 10.1109/SMC.2018.8573229]
the ground truth. [3]. T. Oh, H. Kim, H. J. Kim, and M. H. Kim, "Hand
gesture recognition with region-based convolutional
By leveraging the functionalities and flexibility offered neural networks for augmented reality," in 2018 15th
by TensorFlow, this research project successfully built and International Conference on Ubiquitous Robots and
trained a CNN model for accurate hand gesture classification. Ambient Intelligence (URAI), pp. 71-76, 2018. [doi:
TensorFlow's comprehensive deep learning toolkit empowers 10.1109/URAI.2018.8613422]
researchers to develop innovative HGR systems with the [4]. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning,"
potential to revolutionize various human-computer Nature, vol. 521, no. 7553, pp. 436-444, 2015. [doi:
interaction applications. 10.1038/nature14534]
[5]. Y. Guo, Y. Liu, A. Oerlemans, S. Lao, S. Wu, and M.
V. CONCLUSION S. Lew, "Deep learning for visual understanding: A
review," Neurocomputing, vol. 187, pp. 1-7, 2016.
This research paper explored the effectiveness of [doi: 10.1016/j.neucom.2015.09.110]
Convolutional Neural Networks (CNNs) for hand gesture
recognition using the TensorFlow deep learning framework.

IJISRT24AUG154 www.ijisrt.com 71
Volume 9, Issue 8, August – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24AUG154

[6]. H. Oˇzer, H. Ölmez, and M. Belg˘um, "Convolutional

neural networks for turkish sign language
recognition," Signal, Image and Video Processing,
vol. 12, no. 1, pp. 141-149, 2018. [doi:
10.1007/s11760-017-1100-3]
[7]. M. Hasan, M. S. Islam, and M. S. Z. Chowdhury,
"Finger counting using convolutional neural
network," in 2019 4th International Conference on
Electrical Engineering and Information
Communication (ICEEIC), pp. 1-4, 2019. [doi:
10.1109/ICEEIC.2019.8862280]
[8]. C. Li, Q. Wang, X. Chen, and W. Xu, "Cnn-lstm
hybrid network for sign language recognition," IEEE
Access, vol. 6, pp. 77340-77349, 2018. [doi:
10.1109/ACCESS.2018.2880943]
[9]. A. Hamdi, S. Ben Abdelkader, and A. M. Alimi, "A
cnn based approach for hand gesture recognition using
fingertips detection," in 2018 International
Conference on Advanced Systems and Electric
Technologies (ICASET), pp. 368-373, 2018. [doi:
10.1109/ICASET.2018.8
[10]. K. Yuan, T. Liu, X. Liu, J. Dai, and Y. Zhu, "Point-to-
set learning for 3d hand pose estimation," in
Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pp. 8765-8774, 2019.
[doi: 10.1109/CVPR.2019.00880]
[11]. Shubham Shukla. Gesture Recognition Using Deep
Learning Techniques, 12 July 2022, PREPRINT
(Version 1) available at Research Square
[https://doi.org/10.21203/rs.3.rs-1842374/v1]
[12]. G. Varoudhini, D N S B Kavitha "Hand Gesture
Recognition using Deep Learning Models"
International Journal of Science and Management
Studies (IJSMS) V5.I4 (2022): 231-241.
[13]. Y. Zhang and X. Jiang, "Recent Advances on Deep
Learning for Sign Language Recognition," Comput.
Model. Eng. Sci., vol. 139, no. 3, pp. 2399-2450.
2024. https://doi.org/10.32604/cmes.2023.045731
[14]. Aggarwal, A., Bhutani, N., Kapur, R. et al. Real-time
hand gesture recognition using multiple deep learning
architectures. SIViP 17, 3963–3971 (2023).
https://doi.org/10.1007/s11760-023-02626-8
[15]. Zholshiyeva, L., Manbetova, Z., Kaibassova, D.,
Kassymova, A., Tashenova, Z., Baizhumanov, S.,
Yerzhanova, A., & Aikhynbay, K. (2024). Human-
machine interactions based on hand gesture
recognition using deep learning methods.
International Journal of Electrical and Computer
Engineering (IJECE), 14(1), 741-748.
doi:http://doi.org/10.11591/ijece.v14i1.pp741-748

IJISRT24AUG154 www.ijisrt.com 72