Deep_Learning_Approach_for_Sign_Language_Recognition_Using_DenseNet201_with_Transfer_Learning
Deep_Learning_Approach_for_Sign_Language_Recognition_Using_DenseNet201_with_Transfer_Learning
Learning
Yasir Altaf Abdul Wahid
Department of ComputerScience Department of Computer Science and Infromation Technology
Maulana Azad National Urdu University Maulana Azad National Urdu University
Hyderabad, India Hyderabad, India
yasiraltafwani66@gmail.com wahidabdul76@yahoo.com
Abstract—Transfer learning has been utilized to solve many people, (b) assisting speech and hearing loss people in their
complex real-world problems. Over the last several years, trans- daily life activities, such as studying, learning, travelling,
fer learning had many applications in image and video recogni- marketing, banking, and health quires; (c) accelerating the
tion. To get higher recognition rates, deep and wider architectures
of the convolutional neural networks (CNN) have been designed. teaching learning at schools meant for speech and hearing-
In this research, we proposed a novel transfer learning-based impaired students, and (d) improving the digital literacy among
model using a popular CNN architecture called DenseNet201 for deaf-mute people and also improving their lifestyle with
the recognition of Indian Sign Language (ISL) hand gestures. We enhanced communication and understanding. A number of
applied transfer learning to DenseNet201 by freezing some of its automatic sign language recognition and translation systems
layers to retain its knowledge of generalization and fine-tuning
the remaining layers for ISL dataset. Pre-trained DenseNet201 have been developed over the last several years to meet the
was used to extract the features of the gesture images. To classify communication needs of deaf and hearing-impaired individuals
the ISL gesture, custom layers were added to the pretrained [1], [2].This area is still an active area for research and requires
DenseNet201 model. The proposed model helped to achieve improvements to implement such systems in various sectors,
higher accuracy of 100%. such as education to teach and assist speech and hearing
Keywords—Transfer Learning, DenseNet201, Deep Learning,
Sign Language Recognition, Multilayer DenseNets, Image recog-
loss students, public places such as railway stations, airports,
nition. and bus stops to guide deaf-mute people and help them
by understanding their quires, providing feedback to them,
and bringing them closer to the hearing world [3]–[5].With
I. I NTRODUCTION
advancements in technologies researchers have adopted many
The language used by deaf and mute people is called approaches for sign language recognition, such as sensory-
sign language (SL). It is used to propagate messages among based, Vision-based and deep learning-based approaches are
them and also used by them to communicate with normal widely used. Traditionally, most sign language recognition and
people. In every country, people with speech and hearing translation systems utilize contact-based approaches, wherein
loss have their own ways of expressing ideas using sign the signer needs some sort of special wearable device (like
language. For example, people living in America use American glove equipped with sensors) to perform gestures, and such
Sign Language (ASL) and the people of India use Indian systems did not grow in large scale for the development of dig-
Sign Language (ISL). Similarly, all other countries have their ital assisting applications for deaf-mute people because of their
own sign language. However, due to the varied nature of obvious limitation with the hard-to-use circuitry of wearable
sign language across countries, people face difficulties in devices. To simplify such systems, vision-based approaches
exchanging their ideas when it is communication between two provide new, easy-to-use, robust, and adequate methods for
deaf-mute persons or between normal and deaf-mute persons. large-scale development of sign language-based assisting sys-
To bridge this communication gap, technology known as Sign tems. Vision-based systems for sign language recognition and
Language Recognition and Translation plays an important translation are easy to use and do not require any hard-to- use
role. The various benefits of such a technology are: (a) wearable devices; instead, they utilize high-quality cameras
removing the communication barrier between the deaf-mute to capture the signs performed by the signer. In recent years,
SCEECS 2023
Authorized licensed use limited to: M S RAMAIAH INSTITUTE OF TECHNOLOGY. Downloaded on March 10,2025 at 08:23:25 UTC from IEEE Xplore. Restrictions apply.
Fig. 1: A three-layer DenseNet architecture.
SCEECS 2023
Authorized licensed use limited to: M S RAMAIAH INSTITUTE OF TECHNOLOGY. Downloaded on March 10,2025 at 08:23:25 UTC from IEEE Xplore. Restrictions apply.
TABLE I: Description of the data set used in this study
Dataset Total No. of classes No. of images in train set No. of images in val. set No. of images in test set Total no. of images in data set
ISL 35 33,600 6,300 21,00 42,000
V. R ESULTS
TP + TN
Accuracy = (3)
TP + TN + FP + FN
TP
P recision = (4) Fig. 6: Classification report of the proposed model.
TP + FP
TP
Recall = (5)
TP + FN
2 × P recision × Recall
F 1 − score = (6)
P recision + Recall
SCEECS 2023
Authorized licensed use limited to: M S RAMAIAH INSTITUTE OF TECHNOLOGY. Downloaded on March 10,2025 at 08:23:25 UTC from IEEE Xplore. Restrictions apply.
TABLE II: Comparison of this study with existing related studies.
Study Model Dataset Accuracy (%)
Abdul et al. [32] Inception- BiLSTM Arabic sign language (KSU-ArSL) 84.20
Molchanov et at. [33] 3DCNN VIVA challenge dataset 77.05
Islam et al. [34] VGG16 BdSL 99.92
Mistree et al. [35] MobileNet ISL 97.26
Wangchuk et al. [36] CNN BSL 97.62
Suri et al. [37] CapsNet Constructed 94
Katoch et al. [8] CNN ISL 99
Singh et al. [38] 3DCNN ISL 88.24
Dhulipala et al. [39] CNN British Sign Language 97.40
Patil et al. [40] CNN ISL 95
Sharma et al. [9] CNN ISL 99.52
Nandi et al. [41] CNN ISL 99.76
This study DenseNet201 ISL 100
SCEECS 2023
Authorized licensed use limited to: M S RAMAIAH INSTITUTE OF TECHNOLOGY. Downloaded on March 10,2025 at 08:23:25 UTC from IEEE Xplore. Restrictions apply.
R EFERENCES [22] H. Farhana, T. Ahmed, H. Ahmad, K. Narasingamurthi , H. Harkat and
S. K. Phang, ”DF-WiSLR: Device-Free Wi-Fi-based Sign Language
[1] M. Sobhan, M. Z. Chowdhury, I. Ahsan, H. Mahmud and M. K. Hasan, Recognition,” Pervasive and Mobile Computing, vol. 69, p. 101289,
”A Communication Aid System for Deaf and Mute using Vibrotactile 2020.
and Visual Feedback,” 2019 International Seminar on Application for [23] M. G. Martı́n, R. S. Segundo, F. F. Martı́nez, and J. F. López, ”Improving
Technology of Information and Communication (iSemantic), 2019, pp. physical activity recognition using a new deep learning architecture
184-190, doi: 10.1109/ISEMANTIC.2019.8884323. and post-processing techniques,” Engineering Applications of Artificial
[2] K. Yousaf et al, ”A Novel Technique for Speech Recognition and Visual- Intelligence, vol. 92, p. 103679, 2020.
ization Based Mobile Application to Support Two-Way Communication [24] Y. Ma, G. Zhou, S. Wang, H. Zhao, and W. Jung, ”SignFi: Sign Lan-
between Deaf-Mute and Normal Peoples,” Wireless Communications & guage Recognition,” in Proc. ACM Interact. Mob. Wearable Ubiquitous
Mobile Computing (Online), p. 12, 2018. Technol, 2018.
[3] E.E. Abdallah and E. Fayyoumi, ”Assistive Technology for Deaf People [25] Y. Xu, W. Yang, M. Chen, S. Chen and L. Huang, ”Attention-Based
Based on Android Platform,” Procedia Computer Science, vol. 94, pp. Gait Recognition and Walking Direction Estimation in Wi-Fi Networks,”
295-301, 2016. IEEE Transactions on Mobile Computing, vol. 21, no. 2, pp. 465-479,
[4] M.A. Azizan, I. Zulkiflee and N. Norhashim, ”Development of Sign Lan- 2022.
guage Translator for Speech Impairment Person,” in Human-Centered [26] H. Xue, W. Jiang, C. Miao, F. Ma, S. Wang, Y. Yuan, S. Yao, A. Zhang,
Technology for a Better Tomorrow, Springer Singapore, 2022, pp. 115– and L. Su, ”DeepMV: Multi-View Deep Learning for Device-Free
131. Human Activity Recognition,” in Proc. ACM Interact. Mob. Wearable,
[5] S. Sundaramoorthy and B. Muthazhagan, ”Human-Computer Interaction 2020.
System for Improving Digital Literacy Among Speech- and Hearing- [27] G. Huang, Z. Liu, L. Van Der Maaten and K. Q Weinberger, ”Densely
Impaired in Developing Countries,” in Digital Literacy and Socio- Connected Convolutional Networks,” in 2017 IEEE Conference on
Cultural Acceptance of ICT in Developing Countries, Cham, Springer Computer Vision and Pattern Recognition (CVPR), 2017.
International Publishing, 2021, pp. 139-156. [28] M. Z. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga, ”Attention-
[6] T.W. Chong and B.J. Kim, ”American Sign Language Recognition Based Image Captioning Using DenseNet Features,” in Neural Informa-
System Using Wearable Sensors with Deep Learning Approach,” The tion Processing, Cham, 2019.
Journal of the Korea institute of electronic communication sciences, vol. [29] B. Aksoy , O. K. M. Salman , and Ö. Ekrem, ”Detection of Turkish
15, no. 2, p. 291–298, 2020. Sign Language Using Deep,” Applid Artifial Intelligence, vol. 35, no.
[7] N. Musthafa and C.G. Raji, ”Real time Indian sign language recognition 12, p. 952–981, 2021.
system,” Materials Today: Proceedings, vol. 58, pp. 504-508, 2022. [30] M. Kołodziej, E. Szypuła, A. Majkowski and R. Rak, ”Using deep
[8] S. Katoch, V. Singh and U. S. Tiwary, ”Indian Sign Language recog- learning to recognize the sign alphabet,” Przeglad Elektrotechniczny,
nition system using SURF with SVM and CNN,” Array, vol. 14, p. vol. 6, p. 32, 2022.
100141, 2022. [31] V. Thambawita, I. Strümke, S.A. Hicks, P. Halvorsen, S. Parasa and M.
[9] S. Sharma, and S. Singh, ”Recognition of indian Sign Language (ISL) A. Riegler, ”Impact of Image Resolution on Deep Learning Performance
using Deep Learning Model.,” Wireless personal communications, vol. in Endoscopy Image Classification: An Experimental Study Using a
123, pp. 671-692, 2021. Large Dataset of Endoscopic Images,” Diagnostics-Basel, vol. 11, no.
[10] C.K.M. Lee, Kam K.H. Ng, Chun-Hsien Chen, H.C.W. Lau, S.Y. Chung, 12, p. 2183, 2021.
and T. Tsoi, ”American sign language recognition and training method [32] W. Abdul, M. Alsulaiman, S.U. Amin, M. Faisal, G. Muhammad, F.
with recurrent neural network,” Expert Systems with Applications, vol. R. Albogamy, M. A. Bencherif and H. Ghaleb, ”Intelligent real-time
167, p. 114403, 2021. Arabic sign language classification using attention-based inception and
[11] S. Sharma, and K. Kumar, ”ASL-3DCNN: American sign language BiLSTM,” Computers and Electrical Engineering, vol. 95, p. 107395,
recognition technique using 3-D convolutional neural networks,” Mul- 2021.
timed Tools Appl, vol. 80, p. 26319–26331, 2021. [33] P. Molchanov, S. Gupta, K. Kim and J. Kautz, ”Hand gesture recogni-
[12] K.M. Peréz, D.M.C. Esparza, J. Terven, A.M.H. Navarro, T.G. Ramı́rez, tion with 3D convolutional neural networks,” in IEEE Conference on
and A.R. Pedraza , ”Automatic Recognition of Mexican Sign Language Computer Vision and Pattern Recognition Workshops (CVPRW), 2015.
Using,” Appl. Sci., vol. 12, p. 5523, 2022. [34] M. M. Islam, M.R. Uddin, M. N. AKhtar and K.M. R. Alam, ”Recogniz-
[13] Q. Xiao, M. Qin and Y. Yin, ”Skeleton-based Chinese sign language ing multiclass Static Sign Language words for deaf and dumb people
recognition and generation for bidirectional communication between of Bangladesh based on transfer learning techniques,” Informatics in
deaf and hearing people,” Neural Networks, vol. 125, pp. 41-55, 2020. Medicine Unlocked, vol. 33, 2022.
[14] J. Zhang, W. Zhou, C. Xie, J. Pu and H. Li, ”Chinese sign language [35] K. Mistree, D. Thakor and B. Bhatt, ”Indian Alphabets and Digits Sign
recognition with adaptive HMM,” in IEEE International Conference on Recognition Using Pretrained Model,” Smart Innovation, Systems and
Multimedia and Expo (ICME), 2016. Technologies, vol. 283, 2022.
[15] S.G.M. Almeida, F.G. Guimarães, and J.A. Ramı́rez, ”Feature extraction [36] K. Wangchuk, P. Riyamongkol and R. Waranusast, ”Real-time Bhutanese
in Brazilian Sign Language Recognition based on phonological structure Sign Language digits recognition system using Convolutional Neural
and using RGB-D sensors,” Expert Systems with Applications, vol. 41, Network,” ICT Express, vol. 7, no. 2, pp. 215-220, 2021.
no. 16, pp. 7259-7271, 2014. [37] K. Suri and R. Gupta, ”Continuous sign language recognition from wear-
[16] M. Elpeltagy,M. Abdelwahab,M. E. Hussein,A. Shoukry,A. Shoala, and able IMUs using deep capsule networks and game theory,” Computers
M. Galal, ”Multi-modality-based Arabic sign language recognition,” IET & Electrical Engineering, vol. 78, pp. 493-503, 2019.
Comput. Vis., vol. 12, pp. 1031-1039, 2018. [38] D. Singh, ”3D-CNN based Dynamic Gesture Recognition for Indian
[17] A. Tang, K. Lu, Y. Wang, J. Huang and H. Li, ”A Real-time Hand Sign Language Modeling,” Procedia Computer Science, vol. 189, pp.
Posture Recognition System Using Deep Neural,” ACM Trans. Intell. 76-83, 2021.
Syst. Techno, vol. 9, no. 4, p. 23, 2013. [39] S. Dhulipala, F. F. Adedoyin and A. Bruno , ”Sign and Human Action
[18] L. Pigou, S. Dieleman, P. J. Kindermans and B. Schrauwen, ”Sign Lan- Detection Using Deep Learning,” J. Imaging, vol. 8, no. 192, pp. 1-34,
guage Recognition Using Convolutional Neural Networks,” in Computer 2022.
Vision - ECCV 2014 Workshops, 2015. [40] R. Pati, V. Pati, A. Bahuguna and G. Datkhile, ”Indian Sign Language
[19] J. Duan, J. Wan, S. Zhou, X. Guo, and S. Z. Li, ”A Unified Framework Recognition using Convolutional Neural Network,” in ITM Web of
for Multi-Modal Isolated Gesture Recognition,” ACM Trans. Multimedia Conferences (ICACC), 2021.
Comput. Commun. Appl, vol. 9, no. 4, p. 17, 2017. [41] U. Nandi, A. Ghorai, M. M. Singh, C. Changdar, S. Bhakta and R.
[20] Gupta M, Thakur N, Bansal D, Chaudhary G, Davaasambuu B, and Hua K. Pal, ”Indian sign language alphabet recognition system using CNN
Q, ”CNN-LSTM Hybrid Real-Time IoT-Based Cognitive Approaches with diffGrad optimizer and stochastic pooling,” Multimedia Tools and
for ISLR with WebRTC: Auditory Impaired Assistive Technology,” J Applications, 2022.
Healthc Eng, 2022.
[21] L. Kraljević, M. Russo, M. Pauković, and M. Šarić, ”A Dynamic Gesture
Recognition Interface for Smart Home Control based on Croatian Sign
Language,” Applied Sciences, vol. 10, no. 7, 2020.
SCEECS 2023
Authorized licensed use limited to: M S RAMAIAH INSTITUTE OF TECHNOLOGY. Downloaded on March 10,2025 at 08:23:25 UTC from IEEE Xplore. Restrictions apply.