JMC Research Journal, Vol : XII. No.

1, July 2023 1
Breaking Barriers:
Hand Gesture Vocalizer for the Deaf and Mute
Ashish Jha1, Kamal Pathak2
Lecturer
Department of Mathematics and ICT
1, 2

Janamaitri Multiple Campus, Kathmandu, Nepal

Email: [email protected], [email protected]

ABSTRACT
This study addresses communication challenges faced by deaf and mute
LQGLYLGXDOVE\H[SORULQJWKHIHDVLELOLW\RIXWLOL]LQJÀH[VHQVRUVDQGWKH5DQGRP
Forest algorithm for hand gesture vocalization. The background underscores the
VLJQL¿FDQFHRIDFFHVVLEOHFRPPXQLFDWLRQWRROVLQHQKDQFLQJWKHOLYHVRIWKRVHZLWK
KHDULQJDQGVSHHFKLPSDLUPHQWV7KHVWXG\ VSXUSRVHLVWRDVVHVVWKHHৼHFWLYHQHVV
RI ÀH[ VHQVRUV LQ GHWHFWLQJ KDQG JHVWXUHV DQG WKH 5DQGRP )RUHVW DOJRULWKP V
potential to generate vocalized speech corresponding to these gestures. The
PHWKRGRORJ\LQYROYHVGDWDFROOHFWLRQIURPÀH[VHQVRUVWKURXJK$UGXLQR5DQGRP
Forest model training, and accuracy evaluation in gesture recognition. Promising
results indicate the model's high accuracy in classifying diverse hand gestures.
The study emphasizes the technology-driven solution's importance in bridging
FRPPXQLFDWLRQJDSVIRUWKRVHZLWKLPSDLUPHQWV&RPELQLQJÀH[VHQVRUVDQGWKH
5DQGRP )RUHVW DOJRULWKP RৼHUV DQ LQWXLWLYH FRPPXQLFDWLRQ WRRO WUDQVIRUPLQJ
LQWHUDFWLRQV IRU GHDI DQG PXWH LQGLYLGXDOV &RQVLGHUDWLRQ IRU UHDOZRUOG
scenarios and user diversity during system development is highlighted, crucial for
practical accuracy. Beyond individual communication, the study's implications
VSDQHGXFDWLRQHPSOR\PHQWDQGVRFLDOLQWHJUDWLRQIRUSHRSOHZLWKGLVDELOLWLHV
Implementing this technology in education fosters inclusive environments,
HPSRZHULQJ GHDI DQG PXWH VWXGHQWV WR HQJDJH DFWLYHO\ 7KH LQWHJUDWLRQ RI ÀH[
VHQVRUVDQGWKH5DQGRP)RUHVWDOJRULWKPKROGVLPPHQVHSURPLVHUHYROXWLRQL]LQJ
FRPPXQLFDWLRQDQGOLIHTXDOLW\$VDQDFFHVVLEOHJHVWXUHEDVHGYRFDOL]DWLRQWRRO
it can reshape societal perspectives, fostering inclusivity and empathy. The study
DGYRFDWHVFRQWLQXRXVUHVHDUFKDQGGHYHORSPHQWXUJLQJZLGHVSUHDGWHFKQRORJ\
adoption to create an inclusive society valuing diversity.
Keywords: Hand Gesture, Flex Sensor, Sign Language, Random Forest

Introduction
(൵HFWLYHFRPPXQLFDWLRQLVHVVHQWLDOWRKXPDQFRQQHFWLRQEXWWKRVHZKRKDYH
VSHHFK RU KHDULQJ LPSDLUPHQWV ¿QG LW GL൶FXOW WR H[SUHVV WKHPVHOYHV DQG XQGHUVWDQG
others. For the deaf and mute community, sign language has long been an essential form
of communication, but it can be challenging for non-sign language users to completely
understand. In response to these challenges, this study aims to explore the feasibility of
 2 Ashish Jha, Kamal Pathak : Breaking Barriers: Hand Gesture Vocalizer for the Deaf and Mute

XVLQJÀH[VHQVRUVLQFRQMXQFWLRQZLWKWKH5DQGRP)RUHVWDOJRULWKPWRGHYHORSD+DQG
Gesture Vocalizer. The purpose of this research is to provide an accessible and intuitive
communication tool that enables deaf and mute individuals to express themselves
through hand gestures, with the system generating corresponding vocalized speech in
UHDOWLPH7KHH൵HFWLYHQHVVRIWKHVXJJHVWHGKDQGJHVWXUHYRFDOL]HUKDVEHHQHYDOXDWHG
WKURXJKSUHOLPLQDU\WHVWV'DWDFROOHFWHGIURPÀH[VHQVRUVDWWDFKHGWRYDULRXVKDQG
JHVWXUHVZHUHXWLOL]HGWRWUDLQWKH5DQGRP)RUHVWPRGHO7KHLQLWLDO¿QGLQJVLQGLFDWH
promising results, with the model achieving a high accuracy rate in accurately
FODVVLI\LQJGL൵HUHQWKDQGJHVWXUHV7KHV\VWHP VFDSDFLW\WRSURGXFHYRFDOL]HGVSHHFK
that corresponds to identify movements demonstrates considerable promise for bridging
the gap in communication for those with hearing and speech impairments. However,
further research and real-world testing are required to validate the system's accuracy,
robustness, and usability in practical settings. The study is ongoing, and future work
ZLOOIRFXVRQUH¿QLQJWKH+DQG*HVWXUH9RFDOL]HUDGGUHVVLQJSRWHQWLDOFKDOOHQJHVDQG
exploring applications in educational and social environments.
7KHDELOLW\WRFRPPXQLFDWHH൵HFWLYHO\LVDIXQGDPHQWDOKXPDQQHHG,WFDQEH
TXLWHGL൶FXOWIRUWKRVHZKRDUHGHDIRUKDUGRIKHDULQJWRFRPPXQLFDWHZLWKWKHKHDULQJ
population. Sign language has long been recognized as a crucial tool for deaf individuals
to communicate with one another and with those who have learned sign language. Still, it
FDQEHFKDOOHQJLQJIRULQGLYLGXDOVZKRDUHQRWSUR¿FLHQWLQVLJQODQJXDJHWRXQGHUVWDQG
DQGFRPPXQLFDWHH൵HFWLYHO\ZLWKGHDILQGLYLGXDOV7HFKQRORJLHVKDYHEHHQFUHDWHGWR
KHOSGHDISHRSOHDQGWKHKHDULQJSRSXODWLRQFRPPXQLFDWHPRUHH൵HFWLYHO\LQRUGHUWR
DGGUHVVWKLVGL൶FXOW\2QHVXFKWHFKQRORJ\LVWKH+DQG*HVWXUH9RFDOL]HUZKLFKXVHV
machine learning algorithms to recognize hand gestures and generate corresponding
vocalized speech. The ability of this method to help deaf and mute people communicate
EHWWHUKDVVKRZQVLJQL¿FDQWSURPLVH
However, there is another approach that combines sign language communication
ZLWKPRGHUQVHQVRUWHFKQRORJ\%\DGGLQJÀH[VHQVRUVWRWKH¿QJHUVRIDVLJQODQJXDJH
XVHULWLVSRVVLEOHWRGHWHFWWKHPRYHPHQWVRIWKH¿QJHUVDQGJHQHUDWHFRUUHVSRQGLQJ
speech. With this strategy, sign language communication might become more
approachable and convenient for non-sign language users. In this paper, we will explore
WKHXVHRIÀH[VHQVRUVDQG$UGXLQRPLFURFRQWUROOHUVLQFUHDWLQJD+DQG*HVWXUH9RFDOL]HU
for deaf and mute individuals. We will discuss the challenges involved in developing
VXFKDV\VWHPDQGH[SORUHWKHSRWHQWLDOEHQH¿WVRIXVLQJÀH[VHQVRUVLQVLJQODQJXDJH
communication. We will also talk about the possible uses of this technology and how
LW ZLOO D൵HFW WKRVH ZKR KDYH VSHHFK DQG KHDULQJ GL൶FXOWLHV )XUWKHUPRUH WKH +DQG
*HVWXUH9RFDOL]HUZLWKÀH[VHQVRUVDQG$UGXLQRWHFKQRORJ\FDQPDNHFRPPXQLFDWLRQ
more intuitive and natural for individuals with hearing impairments. It provides a means
of communication that is not only easier to understand but also promotes inclusivity and
JMC Research Journal, Vol : XII. No. 1, July 2023 3
XQGHUVWDQGLQJEHWZHHQLQGLYLGXDOVZLWKGL൵HUHQWDELOLWLHV:LWKWKHLQFUHDVLQJDYDLODELOLW\
DQGD൵RUGDELOLW\RIVHQVRUWHFKQRORJ\WKH+DQG*HVWXUH9RFDOL]HUKDVWKHSRWHQWLDOWR
be widely adopted and used by those who need it. Additionally, this technology may
VLJQL¿FDQWO\D൵HFWWKHHGXFDWLRQDODQGHPSOR\PHQWSURVSHFWVRIWKRVHZKRVX൵HUIURP
hearing and speech problems. It can improve their ability to communicate with hearing
individuals, leading to greater social integration and increased access to education and
HPSOR\PHQW RSSRUWXQLWLHV 0RUHRYHU WKH +DQG *HVWXUH9RFDOL]HU XVLQJ ÀH[ VHQVRUV
DQG$UGXLQRPLFURFRQWUROOHUVFDQEHHDVLO\FXVWRPL]HGDQGDGDSWHGWRGL൵HUHQWXVHUV 
needs and preferences. This system can recognize a wide range of hand gestures and
generate speech quickly and accurately in response to those gestures. As such, It might
prove to be a useful tool for enhancing accessibility and communication for those with
speech- and hearing-impaired.
Literature Review
Numerous research has been done on the use of technology to enhance
communication for those with speech and hearing impairments. One such study by
Shrestha et al. (2020) explored using machine learning algorithms to understand hand
motions and produce speech for those who are deaf. The study found that the system was
able to recognize a wide range of hand gestures and generate accurate speech responses.
Another study by Han et al. (2020) developed a hand gesture recognition system
using a deep learning algorithm and showed that it could accurately recognize hand
gestures in real-time. The system was also able to generate corresponding speech,
which improved the communication abilities of individuals with hearing and speech
impairments.
,QDUHODWHGVWXG\/LDRHWDO  GHYHORSHGDZHDUDEOHGHYLFHXVLQJÀH[VHQVRUV
to detect hand movements and generate speech in response to those movements. The
VWXG\IRXQGWKDWWKHV\VWHPZDVH൵HFWLYHLQJHQHUDWLQJVSHHFKUHVSRQVHVLQUHDOWLPH
and had the potential to be used as a communication tool for individuals with hearing
and speech impairments.
Furthermore, research has also been conducted on the use of sensor technology
in improving the accessibility of sign language communication. One study by Zou et al.
(2017) developed a system that used motion sensors to recognize sign language gestures
DQGJHQHUDWHFRUUHVSRQGLQJVSHHFK7KHVWXG\IRXQGWKDWWKHV\VWHPZDVH൵HFWLYHLQ
recognizing sign language gestures and had the potential to improve the communication
abilities of individuals with hearing and speech impairments.
Nalawade and Bodhe (2019) reviewed various machine learning algorithms used
in sign language recognition systems. They highlighted the challenges in developing
 4 Ashish Jha, Kamal Pathak : Breaking Barriers: Hand Gesture Vocalizer for the Deaf and Mute

such systems and discussed the potential for using deep learning algorithms to improve
accuracy.
Qin et al. (2020) developed a wearable sign language recognition system using
deep learning. To recognize sign language motions in real-time, they combined recurrent
neural networks (RNNs) and convolutional neural networks (CNNs). The system
achieved high accuracy and demonstrated the potential for wearable sign-language
recognition systems.
Chakraborty and Dutta (2021) developed a hand gesture recognition system using
ZHDUDEOHÀH[VHQVRUVDQGPDFKLQHOHDUQLQJ7KH\WUDLQHGD&11PRGHOWRUHFRJQL]H
hand gestures and generate corresponding speech in real time. The system was tested on
a dataset of American Sign Language gestures and achieved high accuracy.
Xiao et al. (2019) reviewed various wearable sensor technologies used in sign-
ODQJXDJHUHFRJQLWLRQV\VWHPVLQFOXGLQJDFFHOHURPHWHUVJ\URVFRSHVDQGÀH[VHQVRUV
They discussed the strengths and limitations of each technology and highlighted the
potential for combining multiple sensors to improve accuracy.
Wu et al. (2019) developed a sign language recognition system using a combination
of deep learning and computer vision techniques. They used a CNN model to extract
features from video data and applied a sliding window approach to recognize signs in
continuous signing. The system achieved high accuracy on a dataset of Chinese Sign
Language gestures.
Raptis et al. (2020) developed a sign language recognition system using a wearable
GHYLFHWKDWFRPELQHGDÀH[VHQVRUDQGDQDFFHOHURPHWHU7KH\XVHGDVXSSRUWYHFWRU
machine (SVM) algorithm to classify hand gestures and demonstrated the potential for
using wearable devices to improve the accessibility of sign language communication.
Raptis and Katsouros (2021) further improved their wearable sign language
recognition system by incorporating a neural network-based approach. They used a
combination of CNN and long short-term memory (LSTM) networks to recognize signs
in real time. The system achieved high accuracy and demonstrated the potential for
wearable sign language recognition systems in improving accessibility.
Yang et al. (2018) developed a sign language recognition system using a
combination of motion sensors and surface electromyography (sEMG) sensors. They
used a decision tree algorithm to classify signs and achieved high accuracy. They also
highlighted the potential for combining multiple sensor modalities to improve the
accuracy of sign language recognition systems.
JMC Research Journal, Vol : XII. No. 1, July 2023 5
Fuentes-Hurtado et al. (2021) developed a sign language recognition system
using a Kinect sensor and a deep learning algorithm. They used a combination of
CNN and RNN models to recognize signs in continuous signing. The system achieved
high accuracy and demonstrated the potential for using depth cameras to improve the
accuracy of sign language recognition systems.
Table 1
List of Literature

Article Methodology Key Results

Shrestha et al. Review of sign language Overview of various machine
(2020) detection methods using learning algorithms and their
machine learning applications in sign language
recognition.
Han et al. (2020) Deep learning and hand (൵HFWLYH UHDOWLPH UHFRJQLWLRQ
tracking for real-time sign of sign language gestures using
language recognition deep learning and hand-tracking
techniques.
Liao et al. Wearable device with Development of a wearable
(2021) ÀH[ VHQVRUV IRU JHVWXUH device that accurately recognizes
recognition and speech hand gestures and generates
synthesis corresponding speech output.
Zou et al. (2017) Wearable sign language Successful implementation of
recognition system based on a wearable system utilizing
motion sensors motion sensors for sign language
recognition.
1DODZDGHDQG Review of machine learning Comprehensive overview of
Bodhe (2019) algorithms for sign language machine learning algorithms
recognition applied to sign language
recognition, highlighting
challenges and techniques.
Qin et al. (2020) Novel wearable sign language Introduction of a new wearable
recognition system based on system leveraging deep learning for
deep learning accurate sign language recognition.
Chakraborty and Hand gesture recognition Development of a hand gesture
Dutta (2021) XVLQJ ZHDUDEOH ÀH[ VHQVRUV recognition system using wearable
and machine learning ÀH[VHQVRUVDQGPDFKLQHOHDUQLQJ
potentially aiding speech-impaired
individuals.
6 Ashish Jha, Kamal Pathak : Breaking Barriers: Hand Gesture Vocalizer for the Deaf and Mute

Xiao et al. Review of wearable sensor Evaluation and comparison of

(2019) technologies for sign wearable sensor technologies and
language recognition their potential in sign language
recognition.
Wu et al. (2019) Combination of deep learning Successful integration of deep
and computer vision for sign learning and computer vision
language recognition techniques for enhanced sign
language recognition.
5DSWLVHWDO Wearable sign language Design and implementation of a
(2020) recognition system based on ZHDUDEOHV\VWHPXVLQJÀH[VHQVRUV
ÀH[VHQVRUDQGDFFHOHURPHWHU and an accelerometer for accurate
gesture recognition.
5DSWLVDQG Wearable sign language Development of a wearable
Katsouros recognition system based on system utilizing neural networks
(2021) neural networks for precise sign language gesture
FODVVL¿FDWLRQ
Yang et al. Sign language recognition Exploration of multi-sensor fusion
(2018) based on multi-sensor fusion techniques to improve the accuracy
of sign language recognition.
Fuentes-Hurtado Continuous sign language Successful implementation
et al. (2021) recognition with Kinect of continuous sign language
sensor and deep learning recognition using Kinect sensor
and deep learning.
Mahajan et al. Hand gesture recognition Achievement of high accuracy in
(2020) XVLQJ ÀH[ VHQVRUV DQG KDQGJHVWXUHUHFRJQLWLRQXVLQJÀH[
machine learning sensors and a machine learning
algorithm.
Kumbhar et al. Sign language recognition Successful design of a sign
(2021) V\VWHPXVLQJÀH[VHQVRUVDQG language recognition system
machine learning XWLOL]LQJÀH[VHQVRUVDQGPDFKLQH
learning techniques.
Methodology
• 7KLV VWXG\ ZKLFK LQYROYHV XWLOL]LQJ ÀH[ VHQVRUV DQG WKH 5DQGRP )RUHVW
DOJRULWKP IRU KDQG JHVWXUH YRFDOL]DWLRQ LV MXVWL¿HG E\ VHYHUDO IDFWRUV WKDW
FRQWULEXWHWRLWVH൵HFWLYHQHVVDQGUHOHYDQFHLQDGGUHVVLQJWKHFRPPXQLFDWLRQ
challenges faced by deaf and mute individuals.
• Accuracy and Reliability: Flex sensors are capable of accurately capturing
JMC Research Journal, Vol : XII. No. 1, July 2023 7
hand movements and gestures, providing a reliable source of input data.
The Random Forest algorithm is known for its robustness and accuracy in
FODVVL¿FDWLRQWDVNVPDNLQJLWDVXLWDEOHFKRLFHIRUUHFRJQL]LQJDQGLQWHUSUHWLQJ
various hand gestures.
• Real-time Interaction: The chosen methodology aims to achieve real-time
hand gesture recognition and vocalization, enabling seamless and immediate
FRPPXQLFDWLRQ5HDOWLPHLQWHUDFWLRQLVHVVHQWLDOIRUH൵HFWLYHFRPPXQLFDWLRQ
DVGHOD\VRUODWHQF\FDQKLQGHUWKHQDWXUDOÀRZRIFRQYHUVDWLRQ
• $FFHVVLELOLW\DQG8VHU)ULHQGOLQHVV)OH[VHQVRUVR൵HUDQRQLQWUXVLYHDQGXVHU
friendly method of capturing hand gestures, ensuring that the communication
tool is accessible and comfortable for users. This aligns with the objective of
developing an inclusive communication aid.
• /RZ &RVW DQG 3RUWDELOLW\ )OH[ VHQVRUV DUH UHODWLYHO\ D൵RUGDEOH DQG
OLJKWZHLJKW PDNLQJ WKHP D FRVWH൵HFWLYH DQG SRUWDEOH VROXWLRQ 7KLV LV
particularly important for widespread adoption, especially in resource-
constrained environments.
• Potential for Customization: The methodology allows for the customization
of gesture recognition and vocalization, accommodating individual user
preferences and needs. This adaptability enhances the user experience and
promotes a sense of ownership and control over the communication process.
• Previous Success in Gesture Recognition: The Random Forest method has
VKRZQH൵HFWLYHQHVVLQDQXPEHURIJHVWXUHUHFRJQLWLRQWDVNVLQFOXGLQJVLJQ
language recognition. Leveraging the algorithm's capabilities in this study
HQKDQFHVWKHOLNHOLKRRGRIDFFXUDWHJHVWXUHFODVVL¿FDWLRQ
• Integration with Existing Technology: The use of an Arduino microcontroller
and the Random Forest algorithm can facilitate seamless integration with
H[LVWLQJ WHFKQRORJ\ HQDEOLQJ FRPSDWLELOLW\ ZLWK GL൵HUHQW GHYLFHV DQG
platforms.
• Scalability: The methodology is scalable, allowing for future enhancements
and improvements. As technology advances and more data becomes available,
WKHV\VWHP VSHUIRUPDQFHFDQEHIXUWKHUUH¿QHGDQGRSWLPL]HG
• Research Gap Addressing: The methodology addresses a research gap by
H[SORULQJ WKH SRWHQWLDO RI FRPELQLQJ ÀH[ VHQVRUV ZLWK WKH 5DQGRP )RUHVW
DOJRULWKP VSHFL¿FDOO\ IRU KDQG JHVWXUH YRFDOL]DWLRQ 7KH GHYHORSPHQW RI
communication tools for people with hearing and speech impairments is aided
by this innovative method.
,QVXPPDU\WKHFKRVHQPHWKRGRORJ\R൵HUVDEDODQFHGFRPELQDWLRQRIDFFXUDF\
DFFHVVLELOLW\FRVWH൵HFWLYHQHVVDQGUHDOWLPHLQWHUDFWLRQDOLJQLQJZLWKWKHVWXG\ VJRDO
RIGHYHORSLQJDQH൵HFWLYHDQGSUDFWLFDOKDQGJHVWXUHYRFDOL]DWLRQV\VWHP
 8 Ashish Jha, Kamal Pathak : Breaking Barriers: Hand Gesture Vocalizer for the Deaf and Mute

Flex Sensors: Flex Sensors are the primary input devices for the system. They can
EHRIYDULRXVVL]HVDQGVKDSHVDQGDUHW\SLFDOO\PDGHRIDÀH[LEOHPDWHULDOWKDWFKDQJHV
resistance when bent. The resistance change is proportional to the degree of bending
and can be measured using an analog input pin of the Arduino.

Figure 1: Flex Sensor

Arduino: The Arduino board is the microcontroller that processes the sensor
readings and runs the Random Forest algorithm. The Arduino can be a low-cost, open-
source board that provides various digital and analog input and output pins. It can also
communicate with other devices using serial communication or other protocols.

)LJXUH$GUXLQR0HJD

Accelerometer: The accelerometer can be a low-cost, small-sized device that

measures acceleration in three axes (x, y, and z). It can be a MEMS (Micro-Electro-
Mechanical Systems) sensor that is integrated with the Arduino or connected via an
external breakout board.
JMC Research Journal, Vol : XII. No. 1, July 2023 9

)LJXUH*<';/$FFHOHURPHWHU

)LJXUH7LOWDQGLQFOLQDWLRQRIWKH$FFHOHURPHWHU>@

6LJQDO FRQGLWLRQLQJ FLUFXLW 'HSHQGLQJ RQ WKH VSHFL¿F UHTXLUHPHQWV RI WKH
application, a signal conditioning circuit may be required to improve the accuracy and
stability of the Flex Sensor readings. The signal conditioning circuit can consist of an
RSHUDWLRQDODPSOL¿HUFDSDFLWRUVDQGUHVLVWRUVDQGFDQEHGHVLJQHGWRDPSOLI\¿OWHU
and stabilize the Flex Sensor signal.
Power supply: The system requires a power supply to operate the Arduino and
other components. The power supply can be a battery, a USB port, or an external power
DGDSWHUGHSHQGLQJRQWKHVSHFL¿FUHTXLUHPHQWVRIWKHDSSOLFDWLRQ
Output devices: The system may require output devices to display the results of
the Random Forest algorithm. The output devices can be LEDs, LCD displays, or other
types of displays. They can also be actuators that perform an action based on the degree
of bending, such as a servo motor or a solenoid.
10 Ashish Jha, Kamal Pathak : Breaking Barriers: Hand Gesture Vocalizer for the Deaf and Mute

)LJXUH+DUGZDUH,QWHJUDWLRQ

)LJXUH6\VWHP'LDJUDPRI+DQG*HVWXUH
JMC Research Journal, Vol : XII. No. 1, July 2023 11
$ FODVVL¿FDWLRQ DOJRULWKP FDOOHG 5DQGRP )RUHVW LV D PHPEHU RI WKH IDPLO\ RI
models built using decision trees. It is an ensemble learning technique that integrates
various decision trees to create a prediction model that is more reliable and accurate.
$OWKRXJK5DQGRP)RUHVWFDQEHXVHGIRUERWKFODVVL¿FDWLRQDQGUHJUHVVLRQWDVNVLQWKLV
instance, we'll concentrate on its strengths in that area. In Random Forest, a subset of
the training data and a subset of the characteristics are used to build numerous decision
WUHHV7KHIRUHFDVWVIURPHDFKWUHHDUHWKHQDYHUDJHGWRFUHDWHWKH¿QDOSUHGLFWLRQ$
majority vote is taken on the forecasts from all the decision trees in the Random Forest,
HDFKRIZKLFKKDVEHHQWUDLQHGLQGLYLGXDOO\WRSURYLGHWKH¿QDOSUHGLFWLRQ
7KHFODVVL¿FDWLRQSURFHVVLQ5DQGRP)RUHVWLQYROYHVWKHIROORZLQJVWHSV
• Data preparation: The input data is preprocessed and cleaned to remove any
missing values, outliers, or noise.
• Feature selection: A subset of the features is randomly selected for each
GHFLVLRQWUHHWRUHGXFHRYHU¿WWLQJDQGLPSURYHDFFXUDF\
• Tree construction: A predetermined portion of the training data and
characteristics are used to build a decision tree. A stopping requirement, such
DVDPD[LPXPGHSWKRUDPLQLPXPQXPEHURIVDPSOHVSHUOHDILVIXO¿OOHGDV
the tree grows.
• Random Forest construction: Multiple decision trees are constructed using
GL൵HUHQWVXEVHWVRIWKHWUDLQLQJGDWDDQGIHDWXUHV7KHWUHHVDUHFRPELQHGWR
form the Random Forest by averaging their predictions.
• Prediction: The test data is fed into the Random Forest, and each decision tree
LQWKHHQVHPEOHPDNHVDSUHGLFWLRQ7KH¿QDOSUHGLFWLRQLVREWDLQHGE\WDNLQJ
a majority vote of the predictions from all the trees.
• 5DQGRP )RUHVW KDV VHYHUDO DGYDQWDJHV RYHU RWKHU FODVVL¿FDWLRQ DOJRULWKPV
including:
High accuracy: Random forests can achieve high accuracy on complex and large
datasets by combining multiple decision trees, Robustness: Random Forest is robust to
QRLVHDQGRYHU¿WWLQJDVHDFKWUHHLVWUDLQHGRQDGL൵HUHQWVXEVHWRIWKHGDWD)HDWXUH
selection: Random Forest can handle a large number of features and automatically
VHOHFWVDVXEVHWRIIHDWXUHVIRUHDFKWUHHUHGXFLQJRYHU¿WWLQJDQGLPSURYLQJDFFXUDF\
Interpretability: Random Forest provides feature importance scores, which can help in
interpreting the model and identifying the most important features, Overall, Random
)RUHVWLVDSRZHUIXODQGZLGHO\XVHGFODVVL¿FDWLRQDOJRULWKPWKDWFDQEHDSSOLHGWRD
variety of real-world problems, including hand gesture vocalization for deaf and mute
people.
12 Ashish Jha, Kamal Pathak : Breaking Barriers: Hand Gesture Vocalizer for the Deaf and Mute

The algorithm and working principle of Random Forest can be summarized as

follows:
Algorithm:
• Initialize the number of decision trees (n_trees) to be used in the ensemble,
the number of features (m) to consider at each split, and a stopping criterion
for the tree growth.
• For each decision tree i in the ensemble, repeat steps 3-5.
• Sample a subset of the training data (with replacement) to create a bootstrap
sample.
• Randomly select m features from the total set of features.
• Construct a decision tree using the bootstrap sample and selected features,
VWRSSLQJZKHQWKHWUHHUHDFKHVWKHVSHFL¿HGVWRSSLQJFULWHULRQ
• After all decision trees have been constructed, a new observation is fed into
the ensemble, and each decision tree predicts the class label of the observation.
• 7KH¿QDOFODVVODEHOLVGHWHUPLQHGE\DPDMRULW\YRWHRIWKHSUHGLFWLRQVIURP
all decision trees in the ensemble.
An ensemble of decision trees, each of which is built using a subset of the training
data and a subset of features, is created by the Random Forest algorithm. A technique
known as bootstrap aggregating (or bagging) is used to generate the trees, in which
several random samples (with replacement) are selected from the training data to
SURGXFHYDULRXVVXEVHWVRIWKHGDWDIRUHDFKGHFLVLRQWUHH7KLVOHVVHQVRYHU¿WWLQJDQG
increases the model's accuracy.
7RIXUWKHUGHFUHDVHRYHU¿WWLQJDQGERRVWWKHYDULHW\RIWKHWUHHVHDFKGHFLVLRQ
tree in the ensemble is built using a distinct subset of the features. The method develops
decision rules that decide the class label of an observation during tree construction and
divides the data into several subsets based on the chosen characteristics.
Once every decision tree has been built, a fresh observation is added to the
ensemble, and every decision tree predicts the observation's class label. A majority vote
RQWKHIRUHFDVWVIURPHDFKGHFLVLRQWUHHLQWKHHQVHPEOHLVXVHGWRFKRRVHWKH¿QDOFODVV
label. The Random Forest algorithm is adaptable and can handle both continuous and
FDWHJRULFDOGDWDPDNLQJLWDSSURSULDWHIRUDYDULHW\RIFODVVL¿FDWLRQLVVXHV
JMC Research Journal, Vol : XII. No. 1, July 2023 13

)LJXUH)ORZFKDUWRI5HDOWLPH$SSOLFDWLRQ
 14 Ashish Jha, Kamal Pathak : Breaking Barriers: Hand Gesture Vocalizer for the Deaf and Mute

Results
7KHUHVXOWVRIXVLQJWKH5DQGRP)RUHVWDOJRULWKPZLWKÀH[VHQVRUVIRUKDQGJHVWXUH
vocalization have been promising. In a study by Mahajan et al. (2020), the authors used
ÀH[VHQVRUVDQGDQ$UGXLQRPLFURFRQWUROOHUWRFROOHFWGDWDRQKDQGJHVWXUHVDQGWUDLQ
a Random Forest model to classify the gestures. The model achieved an accuracy of
LQFODVVLI\LQJGL൵HUHQWKDQGJHVWXUHV
6LPLODUO\LQDVWXG\E\.XPEKDUHWDO  WKHDXWKRUVXVHGÀH[VHQVRUVDQG
an Arduino board to collect data on hand gestures and trained a Random Forest model
to classify the gestures. The model achieved an accuracy of 93.33% in classifying 9
GL൵HUHQW KDQG JHVWXUHV 7KHVH UHVXOWV GHPRQVWUDWH WKH H൵HFWLYHQHVV RI WKH 5DQGRP
)RUHVWDOJRULWKPLQDFFXUDWHO\FODVVLI\LQJKDQGJHVWXUHVXVLQJÀH[VHQVRUV7KHKLJK
accuracy of the models suggests that this approach could be a viable solution for hand
gesture vocalization for deaf and mute individuals. However, it is important to note that
these results were obtained in a controlled laboratory setting, and further research is
needed to evaluate the performance of the models in real-world scenarios.
(YHQWXDOO\ ZH GHYHORSHG D V\VWHP WKDW FDQ UHFRJQL]H YDOXHV IRU VSHFL¿F XVHU
gestures, forecast the result of the gesture, show it on the laptop screen, and emit sounds
via a desktop GUI. All of the alphabets and widely used words were included in the
dataset. With those datasets, the machine was trained and the model was constructed.
Then, in order to train the machine with less variation and ensure that the model stays
JHQHUDODQGRYHU¿WVOHVVWKHGDWDVHWVIRUDOOWKHDOSKDEHWVZHUHSRROHGDQGVKX൷HG
When an alphabet or word was input via sign language, the input was run through a
trained computer, and the result was the closest value anticipated, which was displayed
and played as audio. The accuracy of the model was then predicted by comparing the
correlation plot of each letter with the other alphabet to determine how closely related
each alphabet was to the others. Our model's accuracy was found to be 96.8%.

Sign ASL

C
JMC Research Journal, Vol : XII. No. 1, July 2023 15

OKAY

STOP

HELLO

THANK YOU

CAN

BYE

AND

NO

EAT

I NEED TO GO TO

WASHROOM

Figure 8: Sign Language

16 Ashish Jha, Kamal Pathak : Breaking Barriers: Hand Gesture Vocalizer for the Deaf and Mute

Dataset Values for Some Gestures of Dataset

)LJXUH'DWDVHW9DOXHVIRU$OSKDEHW$

)LJXUH'DWDVHW9DOXHVIRU$OSKDEHW%

)LJXUH'DWDVHW9DOXHVIRU$OSKDEHW&
JMC Research Journal, Vol : XII. No. 1, July 2023 17

Figure 12: Dataset Values for Word Hello

Figure 13: Dataset Values for Word Okay

)LJXUH*ORYHZLWK6HQVRUDQG&LUFXLW'HVLJQRI$GXULQRDQG*ORYH
18 Ashish Jha, Kamal Pathak : Breaking Barriers: Hand Gesture Vocalizer for the Deaf and Mute

)LJXUH$OSKDEHW2XWSXWIURP$UGXLQR

Conclusion
7KH XVH RI ÀH[ VHQVRUV DQG WKH 5DQGRP )RUHVW DOJRULWKP IRU KDQG JHVWXUH
vocalization has shown promising results. The accuracy achieved by the models in
classifying hand gestures indicates that this approach could be a viable solution for
communication for deaf and mute individuals. The technology provides a simple and
FRVWH൵HFWLYHVROXWLRQWKDWFDQEHXVHGLQDYDULHW\RIVHWWLQJV
7KHVWXGLHVUHYLHZHGLQWKLVSDSHUGHPRQVWUDWHWKHH൵HFWLYHQHVVRIWKH5DQGRP
)RUHVWDOJRULWKPLQDFFXUDWHO\FODVVLI\LQJKDQGJHVWXUHVXVLQJÀH[VHQVRUV,WLVFUXFLDO
to remember that the experiments were carried out in controlled laboratory environments
and that additional study is required to assess how well the models work in real-world
conditions.
2YHUDOO WKH XVH RI ÀH[ VHQVRUV DQG WKH 5DQGRP )RUHVW DOJRULWKP KDV WKH
potential to revolutionize communication for deaf and mute individuals, enabling
them to communicate more easily with the world around them. The development of
more sophisticated algorithms and hardware could further improve the accuracy and
usability of this technology. In addition, the use of such technology could also lead
to better education and employment opportunities for individuals who are deaf and
PXWH,WFRXOGSURYLGHWKHPZLWKDPHDQVWRFRPPXQLFDWHH൵HFWLYHO\LQWKHFODVVURRP
or in a work setting, breaking down barriers to success. However, there are still some
limitations to this technology that need to be addressed. One limitation is that the
sensors may not be able to accurately capture the nuances of certain hand gestures,
JMC Research Journal, Vol : XII. No. 1, July 2023 19
OHDGLQJWRPLVFODVVL¿FDWLRQ)XUWKHUPRUHWKHDFFXUDF\RIWKHV\VWHPPD\EHD൵HFWHGE\
factors such as sensor placement, lighting conditions, and user variability. Despite these
OLPLWDWLRQVWKHGHYHORSPHQWRIWKLVWHFKQRORJ\LVDVLJQL¿FDQWVWHSWRZDUGVLPSURYLQJ
the quality of life for individuals who are deaf and mute. It has the potential to improve
their social interactions, education, and employment opportunities, and could lead to a
more inclusive society. With further research and development, this technology could
become an essential tool for individuals with disabilities, facilitating communication
and breaking down barriers to success.
References
Shrestha, B., Khadka, P., Shrestha, S., & Dhakal, S. (2020). Sign Language Recognition using
Machine Learning Techniques: $5HYLHZ-RXUQDORI(QJLQHHULQJDQG$SSOLHG6FLHQFHV
15(3), 839-844. https://doi.org/10.3923/jeasci.2020.839.844
Han, S., Lee, J., & Kim, Y. (2020). Real-Time Sign Language Recognition Using Deep Learning
and Hand Tracking. Electronics, 9(6), 969. https://doi.org/10.3390/electronics9060969
Liao, C., Lin, Y., Chen, C., & Lai, C. (2021). A Wearable Device for Gesture Recognition and
Speech Synthesis Based on Flex Sensors. Sensors, 21(7), 2548. https://doi.org/10.3390/
s21072548
Zou, Y., Wang, W., Zhu, H., & Zhang, Y. (2017). A Wearable Sign Language Recognition System
Based on Motion Sensors. Sensors, 17(5), 1007. https://doi.org/10.3390/s17051007
Nalawade, A., & Bodhe, S. K. (2019). Sign language recognition using machine learning
algorithms: A review. -RXUQDO RI $PELHQW ,QWHOOLJHQFH DQG +XPDQL]HG &RPSXWLQJ
10(9), 3599-3615. https://doi.org/10.1007/s12652-018-1095-5
Qin, C., Zhao, Y., & Liu, Y. (2020). A novel wearable sign language recognition system
based on deep learning. ,((( $FFHVV  203573-203582. https://doi.org/10.1109/
ACCESS.2020.3039891
&KDNUDERUW\ 5  'XWWD 7   +DQG JHVWXUH UHFRJQLWLRQ XVLQJ ZHDUDEOH ÀH[ VHQVRUV
and machine learning for communication aid of speech-impaired individuals. -RXUQDO
RI $PELHQW ,QWHOOLJHQFH DQG +XPDQL]HG &RPSXWLQJ    2563-2573. https://doi.
org/10.1007/s12652-020-02571-9
Xiao, R., Zhang, M., & Wang, Z. (2019). A review of sign language recognition using wearable
sensors. -RXUQDORI6HQVRUV 1-15. https://doi.org/10.1155/2019/2593042
Wu, H., Fan, J., Liu, J., & Chen, W. (2019). Sign language recognition using a combination
of deep learning and computer vision techniques. ,((($FFHVV  97888-97897. doi:
10.1109/ACCESS.2019.2926909
Raptis, K., Vasilopoulos, N., & Katsouros, V. (2020). A wearable sign language recognition
V\VWHPEDVHGRQDÀH[VHQVRUDQGDQDFFHOHURPHWHUSensors, 20(21), 6114. doi: 10.3390/
s20216114
 20 Ashish Jha, Kamal Pathak : Breaking Barriers: Hand Gesture Vocalizer for the Deaf and Mute

Raptis, K., & Katsouros, V. (2021). A wearable sign language recognition system based on
neural networks. 6HQVRUV   1695. doi: 10.3390/s21051695
Yang, J., Huang, Q., & Cui, L. (2018). Sign language recognition based on multi-sensor fusion.
,((($FFHVV 39413-39423. doi: 10.1109/ACCESS.2018.2855255
Fuentes-Hurtado, F., Medina-Carnicer, R., Morocho-Cayamcela, R., & Carrillo, R. (2021).
Continuous sign language recognition with Kinect sensor and deep learning. Sensors,
   2016. doi: 10.3390/s21062016
Mahajan, P., Kamal, N., & Shokeen, P. (2020). Hand Gesture Recognition Using Flex Sensors
and Machine Learning Algorithm. ,QWHUQDWLRQDO -RXUQDO RI 6FLHQWL¿F 5HVHDUFK LQ
&RPSXWHU6FLHQFH(QJLQHHULQJDQG,QIRUPDWLRQ7HFKQRORJ\   1145-1149.
Kumbhar, S., Khedkar, S., & Marathe, S. (2021). 6LJQ /DQJXDJH 5HFRJQLWLRQ 6\VWHP 8VLQJ
)OH[6HQVRUVDQG0DFKLQH/HDUQLQJWK,QWHUQDWLRQDO&RQIHUHQFHRQ,QYHQWLYH
Computation Technologies (ICICT), 694-699.