Volume 9, Issue 5, May – 2024 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24MAY1338

Sign Language Recognition System Using DL-CNN

Model Using VGG16 and Image Net with
Mobile Application
S Asrita Sreekari1 Bathi Venkata Varaha Durga Yamini 2
Department of Computer Science Engineering Department of Computer Science Engineering
(AI &ML) GVPCEW(JNTUK) (AI &ML) GVPCEW(JNTUK)
Visakhapatnam, India Visakhapatnam, India

Somayajula Venkata Thanmayi Sri3 Maram Naga Sireesha4

Department of Computer Science Engineering Department of Computer Science Engineering
(AI &ML) GVPCEW(JNTUK) (AI &ML) GVPCEW(JNTUK)
Visakhapatnam, India Visakhapatnam, India

Abstract:- In this project, a Deep Learning of creating an all- encompassing sign language recognition
Convolutional Neural Network (DL-CNN) model trained system. The main objective is to build a strong machine
on ImageNet and based on VGG16 is used to develop a learning model that can precisely understand a wide variety
Sign Language Recognition System incorporated into a of sign language movements in order to achieve high
mobile application. The technology recognizes a variety accuracy in gesture identification. To enable rapid
of hand gestures and movements that are inherent in interpretation and maintain smooth and dynamic
sign language, allowing for real-time interpretation of communication through a mobile device's camera, real- time
sign language gestures that are recorded by the device's processing skills are crucial. In order to adjust the model to
camera. Users can simply interact with the system by various signing styles and guarantee responsiveness to the
capturing motions in sign language and obtaining dynamic nature of sign language expressions, ongoing
corresponding written or aural outputs for better learning processes will be implemented. When paired with
communication through the app interface. Through an intuitive mobileapplication interface, users will be able to
improving accessibility and inclusivity for people with submit sign language movements and effortlessly obtain
hearing loss, this project seeks to close gaps and promote equivalent text or spoken output. Additionally, the initiative
understanding through technology by facilitating prioritizes accessibility and inclusivity with the goal of give
seamless communication in a variety of settings. people with hearing loss the resources they need to
communicate, obtain information, and take part completely
Keywords:- VGG16, ImageNet, Convolution Neural in everyday activities. All things considered, the Sign
Networks, Mobile Application. Language Recognition System is extremely important since
it fills a critical need in society, promotes inclusivity,
I. INTRODUCTION removes obstacles to communication, and may even
improve many people's lives by making society more
A vital component of human contact is communication, cohesive and inclusive.
but for those who are hard of hearing, using traditional
forms of communication can be very difficult. For the deaf A. Ease of use
and hard of hearing community, sign language is an
essential communication tool. Effective communication,  User Interface:
however, may be hampered by others who are not For sign language recognition systems, an interface
experienced with sign language misinterpreting it. With the that is easy to use and intuitive is essential. Gestures should
purpose of resolving this problem, the B.Tech student be an easy way for users to engage with the system.
project presents a "Sign Language Recognition System
Using Machine Learning withMobileApplication." By using  Real-Time Feedback:
cutting-edge machine learning algorithms to instantly Providing immediate feedback on recognized gestures
analyze and translate sign language motions, the project helps users adjust their signing if needed. Real-time
aims to close the communication gap. Through the feedback enhances the user experience and facilitates
incorporation of this technology into a smartphone smoother communication.
application, the project hopes to establish an approachable
and accessible platform that enables people with hearing loss
to smoothly communicate with the larger community.The
project's goals cover a number of crucial areas with the goal

IJISRT24MAY1338 www.ijisrt.com 1279

Volume 9, Issue 5, May – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24MAY1338

II. MATERIALS AND METHODS gaps and enable people with hearing impairments to
participate more fully in social interactions by utilizing the
An important development in assistive technology is the strength of DL-CNN models trained on ImageNet in
Sign Language Recognition System (SLRS), which makes conjunction with the practicality of mobile applications.
use of a Deep Learning Convolutional Neural Network (DL-
CNN) model trained on the ImageNet dataset and based on  Data Collection and Preprocessing:
the VGG16 architecture. The approach used in the creation The American Sign Language (ASL) dataset that was
and deployment of the SLRS is described in this study, with obtained from Kaggle was carefully curated. It was
a focus on how DL-CNN technology was integrated with a composed of video recordings that captured ASL motions
mobile application to improve accessibility and usability. using a well calibrated, top-notch camera setup. Every film
The data collecting approach, preprocessing methods, model was meticulously annotated and processed, with individual
architecture, training protocols, deployment strategies, and frames accurately capturing unique ASL movements
validation protocols used to create an efficient and user- extracted. The core components of the dataset were created
friendly SLRS are all described in depth in the materials and by converting these frames into picture files.
methods section. This SLRS seeks to close communication

Fig 1 Some Sample of our used Dataset is given in this Figure

 Model Training and Optimization: for accessibility, such text-to- speech, voice commands, and
Utilizing a deep learning framework such as movable font sizes, to meet the needs of different user types.
TensorFlow or PyTorch, implement the DL-CNN model For widespread accessibility, make sure it works with both
based on the VGG16 architecture. To take use of learnt the iOS and Android operating systems.
features, initialize the model with pretrained weights from
the ImageNet dataset. Utilizing the sign language dataset,  Sign Language Recognition:
adjust the model to the particular job of sign language Allow the smartphone app to use the camera on the
recognition.To enhance model performance, adjust device to record sign language gestures. Utilize the DL-CNN
hyperparameters like learning rate, batch size, and model to analyze acquired images in real time for
regularization strategies. interpretation.

 Mobile Application Development: Convert understood motions into spoken or written

Create and implement an intuitive mobile application communication outputs. Put feedback systems in place to
interface to record sign language movements. Provide deal with misidentifications and improve the interpretation
real-time camera capture, image processing, and DL-CNN over time.
model communication functionalities. Incorporate features

IJISRT24MAY1338 www.ijisrt.com 1280

Volume 9, Issue 5, May – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24MAY1338

 Testing and Evaluation: III. METHODOLOGY

Conduct thorough testing to assess the Sign Language
Recognition System's speed, accuracy, and usability. To In the paper, we proposed a new method a new method
evaluate the system's performance in practical situations, do for detecting sign language. We divided our whole model
validation testing on deaf people. Get userinput to determine into four sub steps. The are Data Preprocessing , Model
what needs to be optimized and improved. To evaluate the Building, Model Training ,and Real-time prediction . We
system's effectiveness and excellence, compare it to other gave a description of all steps of our “Sign Language
sign language recognition techniques already in use. Recognition System “in figure 3.

 Deployment and Maintenance:

Release the Sign Language Recognition System to the
public via app stores. For continuous maintenance and
updates, keep an eye on user input and system performance.
Iterative development cycles should be used to continuously
improve the system by adding new features and
improvements depending on user requests and technology
breakthroughs.

By adhering to this process, the Sign Language

Recognition System may be built and implemented
efficiently, offering a dependable and easily accessible
means of sign language communication for people with
hearing impairments

 Real-Time Prediction:
After fully training the CNN model with our
customized preprocess dataset we save the model for real-
time prediction.In the real-time prediction step,we draw the
hand landmarks and analyze the hand position with the
mobile and give them into the trained model. Now the
trained model finds the best match between the given sign
and the dataset sign. In figure 2 we given some of the
sample real-time prediction images of our model.

Fig 3 Architecture

IV. EXPERIMENTAL RESULTS

Different models were tested in an experimental

evaluation of sign language recognition to see how well they
could read sign language movements. According to the
Fig 2 Real-Time Prediction Result of our Trained Model
findings, the conventional CNN model obtained a
recognition accuracy of 82%, while the sdd_mobilenet
 Literature Review:
model achieved 88%. By putting the VGG16 architecture
The literature review will explore existing research and
into practice, gesture recognition accuracy increased to 92%.
developments in the following areas:
Pretrained weights from the ImageNet dataset were used to
initialize the VGG16 model, which produced the most
 Sign Language Recognition
accurate performance, with an accuracy rate of 96%. These
 Deep Learning and CNN Models results demonstrate the efficiency of deep learning models in
 VGG16 Architecture accurately capturing and interpreting the intricacies of sign
 ImageNet Dataset language gestures, especially the VGG16 architecture with
 Mobile Application Development ImageNet initialization.

IJISRT24MAY1338 www.ijisrt.com 1281

Volume 9, Issue 5, May – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24MAY1338

Fig 4 Comparision of Accuracies Fig 6 Accuracy of Our Sign Language Recognition System
as they Relate to the epoch in this Figure. It Shows
 Output of Our Project: Accuracy on y-Axis and epoch on x-Axis

Fig 5 Output of Our Project

Fig 7 Loss of Our Sign Language Recognition System asthey
V. RESULT ANALYSIS Relate to the epoch in this Figure. It Shows loss on y-Axis
and epoch on the x-Axis
 Accuracy & Loss Curve:
The performance of a deep learning model can be  Result :
illustrated by accuracy and loss curves. The following In this experiment, we detected English sign language
graph shows how deep learning models improve their using Convolutional Neural Network (CNN).For this
performance per epoch. In order to plot accuracy and loss research,we built a large number of sign language signs to
curves, we place accuracy and loss on the y-axis and epoch train our model.after training the model we tested our model
on the x-axis. Our model was trained over 5 epochs. Using using the testing the dataset.The result of our sign language
this curve, We can determine if a model is overfitted or recognition system is given in below table.After testing the
underfitted. We gaveour model’s accuracy and loss curves in model,if the model gives very good results we save it for
the below figures. making a real-time prediction system.Some of the real-time
prediction of our trained model is given figure 2. We clearly
see that our model classified all the sign language very
accurately andefficiently.

IJISRT24MAY1338 www.ijisrt.com 1282

Volume 9, Issue 5, May – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24MAY1338

Accessibility, inclusion, and communication in a range [4]. Sharma, A., Sawant, S., & Singhal, S. (2020). Sign
of situations and environments. With continued language recognition using deep learning techniques:
development, assessment, and cooperation, the project can A systematic review. International Journal of
advance and change. Machine Learning and Cybernetics, 11(7), 1623-
1650.
Table 1 In this Table, we given our Sign Language [5]. Chen, L., Han, Y., & Gao, S. (2020). A sign
Recognition System all Results language recognition method based on deep learning.
Architecure Accuracy Multimedia Tools and Applications, 79(9- 10), 5719-
CNN 98% 5736.
[6]. Drahansky, M., Klepal, M., & Hunka, F. (2019).
VI. CONCLUSIONS Real-time sign language recognition system based on
deep neural networks. In 2019 International
To sum up, the Sign Language Recognition System Conference on Applied Electronics (AE) (pp. 1-4).
project is a big step in the right direction towardimproving IEEE.
communication and accessibility for those who have [7]. Hassani, N. H., & Arifin, A. (2020). Real-time
hearing loss. The project intends to accurately understand American Sign Language recognition system using
sign language motions in real-time by utilizing deep machine learning. International Journal of Electrical
learning techniques, namely convolutional neural networks and Computer Engineering (IJECE), 10(5), 4691-
(CNNs) with VGG16 architecture and the ImageNet 4700.
dataset, integrated into a smart phone application. A range [8]. Huang, X., & Zhang, W. (2018). Sign language
of technologies and approaches, such as image processing, recognition based on a convolutional neural network.
machine learning, mobile application development, and IEEE Access, 6, 41819-41827.
accessibility features, have been used throughout the [9]. Hwang, S. W., & Kim, H. J. (2017). Sign language
project. A smooth and inclusive user experience is recognition using recurrent neural networks with
guaranteed by the system's architecture, which includes conditional random fields. Applied Sciences, 7(12),
modules for the user interface, image processing, 1312.
recognition, feedback, and integration. The project [10]. Tavares, A., & Dias, M. S. (2016). Real-time sign
prioritizes usability, reliability, performance, security, and language recognition systems: A review. Expert
accessibility while addressing both functional and non- Systems with Applications, 65, 259-273.
functional needs. In order to make sure the system satisfies [11]. Jumaah, F. M., & Abdulkareem, K. H. (2020). Real-
the requirements and expectations of its users, test cases time Arabic sign language recognition using machine
have been created to validate the system's functionality, learning techniques. IEEE Access, 8, 221862-
performance, and user experience. The Sign Language 221874.
Recognition System has a lot ofroom to grow in the future. [12]. Krejcar, O., & Jan, J. (2016). Sign language
Some of these improvements and additions include multi- recognition in videos with multiple instance learning.
gesture recognition, gesture translation and synthesis, International Journal of Machine Learning and
ongoing development, personalization, cross-platform Cybernetics, 7(3), 397-408.
compatibility, integration with augmented reality, [13]. Kowsari, K., Heidarysafa, M., Brown, D. E.,
expanding the gesture database, improving accessibility, Meimandi, K. J., & Barnes, L. E. (2019). A text
and working with research partners. All things considered, mining approach for capturing temporal and trends of
the Sign Language Recognition System project has the scientific research: An empirical case study using
potential to significantly improve the lives of people who are Medical Research papers. Expert Systems with
hard of hearing by promoting. Applications, 124, 60-73.
[14]. Yan, Y., & Wang, C. (2019). Sign language
REFERENCES recognition system using the Kinect sensor and a
convolutional neural network. IEEE Access, 7,
[1]. Li, D., Zhang, H., Liu, Y., & Du, Y. (2022). Real- 58919-58927.
time American Sign Language recognition using [15]. Zeinali, Y., Harandi, M. T., & Lovell, B. C. (2018).
convolutional neural networks on embedded Sign language recognition using 3D convolutional
platforms. IEEE Access, 7, 159465-159475. neural networks. IEEE Transactions on Human-
[2]. Puertas, E., Jara, C. A., & Pomares, J. (2020). Sign Machine Systems, 49(5), 463-474.
language recognition through machine learning:
current state of the art and challenges. Sensors,
19(20), 4400.
[3]. Starner, T., & Pentland, A. (2019). Real-time
American Sign Language recognition from video
using hidden Markov models. Technical Report
#357, MIT Media Laboratory Perceptual Computing
Section.

IJISRT24MAY1338 www.ijisrt.com 1283