Recognizing and Transforming Sign Language to Speech

BATCH MEMBERS

RIYAZ AHAMED S (812420104076)

SATHAM HUSSAIN A (812420104082)
MOHAMED IRFAN J (812420104316)
MOHAMED JULPER KHAN M(812420104318)
PROJECT GUIDE
Mr. S.SENTHIL NATHAN,M.E.,
ASSISTANT PROFESSOR
Department of Computer Science and Engineering
 OBJECTIVE

• The objective of the project is to develop a deep learning system, utilizing CNN
algorithms, to recognize sign language gestures and convert them into speech,
enabling communication between individuals who are hard of sighted and those who
are sighted.
 INTRODUCTION

• Leveraging deep learning techniques, particularly convolutional neural networks

(CNN)
• The project seeks to recognize and interpret sign language gestures captured through
visual input
• Upon recognizing the gestures, the system converts them into speech, enabling
seamless communication with individuals who don't understand sign language
• The project aims to enhance accessibility and inclusivity for individuals with hearing
impairments in various social and professional settings
 ABSTRACT
• People with impaired speech and hearing uses Sign language as a form of communication.
• Disabled People use this sign language gestures as a tool of non-verbal communication to
express their own emotions and thoughts to other common people.
• To address this problem, we can implement artificial intelligence technology to analyse the
user’s hand with finger detection.
• In this proposed system we can design the vision based system in real time environments.
• And then using deep learning algorithm named as Convolutional neural network algorithm to
classify the sign and convert into speech about recognized sign.
 LITERATURE SURVEY
TITLE AUTHOR YEAR TECHNIQUE MERITS DEMERITS
NAME

Real-time Vernacular Sign Arpita Halder 2021 Support Vector Machine Efficient in precise Require a large dataset and
Language Recognition using and robust complicated methodology
MediaPipe and Machine Learning

An Efficient Two-Stream Network Hamzah Luqman 2022 Machine Learning Provide an Rely on abstractions
for Isolated Sign Language efficient and
Recognition Using Accumulative accurate way
Video Motion
A comprehensive survey and Hamzah Luqman 2022 Deep Learning Handle varieties of Time consuming and more
taxonomy of sign language research data resources required

Continuous Sign Language Ilias Papastratis 2021 Generative adversarial More reliable and Does not support real time
Recognition through a Context- networks easy convergence applications
Aware Generative Adversarial
Network
An integrated mediapipe‑optimized Kil‑Houm Park 2022 Media-Pipe Optimized High learning High computational cost
GRU model for Indian sign Gated Recurrent Unit efficiency and and cannot manage long
language recognition Model (MOPGRU) faster convergence term sequential data
 EXISTING SYSTEM

• Gesture Recognition consists of two approaches

a) vision based

b) glove based

• Glove based approach uses sensors or gloves to identify the hand gesture.

• Some type of flex sensors, accelerometers etc are used in glove based approach.

• Static gestures use hand poses and the image is captured by using cameras.

• The images captured are given for analysis which is done using segmentation.
 DISADVANTAGES

• Need hardware control to detect the hands

• Hand segmentation become complex of various backgrounds

• Segmentation accuracy is less in hand tracking

• Blinds cannot see the neither gesture nor image

 PROPOSED SYSTEM

• The hand gesture images are captured from the vision based camera
• Using background subtraction technique to separate the hand from background
• Segmentation and classification technique to classify the finger postures
• Classification done by using deep learning algorithm named as Convolutional neural
network algorithm
• Provide the Speech about sign in real time video streaming with improved accuracy
rate
 ADVANTAGES

• Segmentation accuracy is high

• Easy to detect the finger postures

• Track fingers and sign recognition with less computational steps

• No need for additional hardware system

ARCHITECTURE DIAGRAM
Interface Creation

Hand gesture capturing

Webcam

Binarization Features Extraction

Background data Region of Finger

removal Detection

Foreground
Subtraction Classification using
CNN

Sign Recognition and

Speech Conversion
 HARDWARE REQUIREMENTS

• Processor : Dual core processor 2.6 GHZ

• RAM : 4GB
• Hard disk : 320 GB
• IO Devices : keyboard, Inbuilt Web Cam
 SOFTWARE REQUIREMENTS

• Operating System : Windows 7 or Above version

• Front End : Python
• IDE : PyCharm or VS Code
 MODULES

• Hand image acquisition

• Binarization
• Region of finger detection
• Classification of finger gestures
• Sign recognition to speech
 MODULE DESCRIPTION

HAND IMAGE ACQUISITION

• In this module, we can input the hand images from real time camera. The
inbuilt camera can be connected to the system.
 MODULE DESCRIPTION
BINARIZATION
• All detection techniques are based on modeling the background of the image i.e.
set the background and detect which changes occur.
• In addition to processing in real time, systems need to be able to adapt to these
changes.
• Then implement the techniques to extract the foreground from background
image.
• Using Binarization approach to assign the values to background and foreground.
Foreground pixels are identified in real time environments
 MODULE DESCRIPTION

REGION OF FINGER DETECTION

• Segmentation refers to the process of partitioning a digital image into multiple segments. In other

words, grouping of pixels into different groups is known as Segmentation.

• The division of an image into meaningful structures, image segmentation, is often an essential

step in image analysis, object representation, visualization, and many other image processing
tasks.

• But segmentation of a satellite image into differently textured regions (groups) is a difficult

problem.

• One does not know a priori what types of textures exist in a satellite image, how many textures
there are, and what regions have certain textures.
 MODULE DESCRIPTION

CLASSIFICATION OF FINGER GESTURES

• Input data along with their response values are the fundamental components of a training
dataset. In order to have higher predictive power and the ability to generalize for several
new datasets, the best way is to use larger training datasets.
• The fingers can be classified by using convolutional neural network algorithm.
• CNN is a common method of training artificial neural networks so as to minimize the
objective function.
• It requires a dataset of the desired output for many inputs, making up the training set. It is
most useful for feed-forward networks (networks that have no feedback, or simply, that
have no connections that loop).
 MODULE DESCRIPTION
SIGN RECOGNITION TO SPEECH
• Sign Language is a well-structured code gesture, every gesture has meaning assigned
to it.
• Sign Language is the only means of communication for blind people. With the
advancement of science and technology many techniques have been developed not
only to minimize the problem of impaired people but also to implement it in different
fields.
• From the classification of sign features, label the signs and convert into speech with
improved accuracy rate.
 CONCLUSION
• In conclusion by harnessing the power of deep learning, specifically CNN algorithms
• To recognize sign language gestures and convert them into speech, the system
provides a practical tool for bridging linguistic barriers.
• This technology has the potential to significantly improve accessibility and
inclusivity for individuals with hearing impairments in various social and
professional contexts
 FUTURE ENHANCEMENT
• In future, we can extend the framework to implement various deep learning
algorithms to recognize the signs and implement in real time applications.
 BOOK REFERENCE

• Van Rossum, Guido, and Fred L. Drake. The python language reference manual.

Network Theory Ltd., 2011.

• Dierbach, Charles. Introduction to Computer Science using Python: A Computational

Problem-Solving Focus. Wiley Publishing, 2012.

• James, Mike. Programmer's Python: Everything is an Object Something Completely

Different. I/O Press, 2018.

• Reges, Stuart, Marty Stepp, and Allison Obourn. Building Python Programs.

Pearson, 2018.
 WEBSITE REFERENCE
• https://docs.python.org/3/tutorial/
• https://www.w3schools.com/python/
• https://www.tutorialspoint.com/python/index.htm
• https://www.programiz.com/python-programming
• https://www.learnpython.org/
THANK YOU