Sign Language Detection
Sign Language Detection
Sign Language Detection
SUBMITTED BY
Amrit Sapkota (076 BCT 05)
Asmit Oli (076 BCT 43)
Nischal Maharjan (076 BCT 20)
Sakshyam Aryal (076 BCT 29)
SUBMITTED TO:
DEPARTMENT OF COMPUTER ENGINEERING
LALITPUR ENGINEERING COLLEGE
LALITPUR, NEPAL
SUPERVISED BY
Er. Hemant Joshi
December, 2023
TRIBHUVAN UNIVERSITY
INSTITUTE OF ENGINEERING
LALITPUR ENGINEERING COLLEGE
DEPARTMENT OF COMPUTER ENGINEERING
SUBMITTED BY
Asmit Oli (076 BCT 43)
Amrit Sapkota (076 BCT 05)
Sakshyan Aryal (076 BCT 29)
Nischal Maharjan (076 BCT 20)
SUPERVISED BY
Er. Hemant Joshi
August, 2023
ACKNOWLEDGEMENT
First and foremost, we would like to thank our supervisor, Er. Hemant Joshi sir,
who guided us in doing this project. He provided us with invaluable advice and
helped us in difficult stages. His motivations helped tremendously to the successful
completion of the project. We are really grateful to our project coordinator, Er.
Bisikha Subedi, for advising us and introducing the project to us in an easy-to-
understand way which has helped us to complete our project easily and effectively
on time. We would like to express our special thanks of gratitude to IOE as well
as our principal, Mr. Lallan Tiwari, who gave us the golden opportunity to do
this wonderful project on the topic, Sign language Detection, which also helped
us in doing a lot of research and we came to know about so many new things.
We are really thankful to them. Besides, we would like to thank all the teachers
who helped us by advising us and providing the equipment we needed. We are
overwhelmed in all humbleness and gratefulness to acknowledge our depth to all
those who have helped us to put these ideas, well above the level of simplicity and
into something concrete. Also, we would like to thank our family and friends for
their support. Without their support we wouldn’t have succeeded in completing
this project. Last but not the least, we would like to thank everyone who helped
and motivate us to work on this project.
Sincerely,
Asmit Oli (076 BCT 43)
Amrit Sapkota (076 BCT 05)
Sakshyan Aryal (076 BCT 29)
Nischal Maharjan (076 BCT 20)
i
ABSTRACT
ii
TABLE OF CONTENTS
ACKNOWLEDGEMENT i
ABSTRACT ii
TABLE OF CONTENTS iii
LIST OF FIGURES v
LIST OF ABBREVIATIONS vi
1 INTRODUCTION 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.5 System Requirement . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.5.1 Functional Requirements . . . . . . . . . . . . . . . . . . . 2
1.5.2 Non-functional Requirement . . . . . . . . . . . . . . . . . . 3
2 LITERATURE REVIEW 4
3 FEASIBILITY ANALYSIS 6
3.1 Technical Feasibility . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.2 Economic Feasibility . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.3 Operational Feasibility . . . . . . . . . . . . . . . . . . . . . . . . . 6
4 BLOCK DIAGRAM 7
4.1 System Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.2 Use Case Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.3 Level 0 DFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.4 Level 1 DFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.5 Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5 METHODOLOGY 12
5.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
5.2 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
5.2.1 Video Acquisition . . . . . . . . . . . . . . . . . . . . . . . . 12
5.2.2 Video Segmentation . . . . . . . . . . . . . . . . . . . . . . . 12
iii
5.2.3 Frame Extraction . . . . . . . . . . . . . . . . . . . . . . . . 13
5.2.4 Preprocessing Techniques . . . . . . . . . . . . . . . . . . . 13
5.2.5 Hand Segmentation . . . . . . . . . . . . . . . . . . . . . . . 14
5.2.6 Temporal Alignment . . . . . . . . . . . . . . . . . . . . . . 14
5.2.7 Data Augmentation . . . . . . . . . . . . . . . . . . . . . . . 14
5.3 Convolution Neural Network . . . . . . . . . . . . . . . . . . . . . . 14
6 IMPLEMENTATION PLAN 16
6.1 Gantt Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7 REQUIREMENT ANALYSIS 17
7.1 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . 17
7.2 Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 17
7.3 User Requirement Definition . . . . . . . . . . . . . . . . . . . . . . 18
8 RESULT AND DISCUSSION 19
8.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
8.2 Training Accuracy and Loss . . . . . . . . . . . . . . . . . . . . . . 20
8.3 Analysis Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
8.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
9 EPILOGUE 23
9.1 Task Completed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
9.2 Remaining Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
REFERENCES 24
iv
LIST OF FIGURES
v
LIST OF ABBREVIATIONS
vi
CHAPTER 1
INTRODUCTION
1.1 Background
With the rapid growth of technology around us, Machine Learning and Artificial
Intelligence has been used in various sectors to support mankind including gesture,
object, face detection etc. With the help of Deep Learning, a machine imitates
the way humans gain certain types of knowledge. Using Artificial Neural Network,
simulation of human brain is done and using Convolution layers, extraction of
selected important part from an image to make computation easy. ”Sign Language
Detection”, the name itself specifies the gist of the project. Sign language recog-
nition has been a major problem between mute disabilities people in community.
People does not understand sign language and also it is difficult for them to learn
those sign language. A part from the scoring grades form this minor project, the
core idea is to make communication easy for deaf people. We set the bar of the
project such that it would be beneficial to society as well. The main reason for us
to choose this project is to aid people using Artificial Intelligence.[1]
1
1.3 Scope
The field of sign language recognition includes the development and application of
techniques for recognizing and interpreting sign language gestures. This involves
using computer vision and machine learning techniques to analyze video input and
identify gestures of sign language users. Sign language recognition has a wide range
of potential applications, including communication aids for deaf people, automatic
translation of sign language into spoken or written language, and an interactive
platform for learning sign language. The scope also extends to improving the
accuracy and efficiency of sign language recognition systems through advances in
algorithms, sensor technology and data collection. Additionally, this scope also
includes addressing challenges related to sign language diversity, gestural variation,
lighting conditions, and the need for robust real-time performance in a variety of
environments. [3]
1.4 Objectives
• To design and implement a system that can understand the sign language of
Hearing-impaired people.
• To train the model with a variety of datasets using MediaPipe and CNN,
and provide the output in real-time.
The following is the desired functionality of the new system. The proposed project
would cover.
• Real-time Output.
2
• Data sets comment.
• Performance Requirement.
• Design Constraints.
• Reliability.
• Usability.
• Maintainability.
3
CHAPTER 2
LITERATURE REVIEW
There are many articles and papers that have been published regarding Sign
Language Detection. Many of them used different algorithms and data sets of
their own. In 1992, researchers developed a camera that could focus on a person’s
hand because the signer wore a glove with markings on the tip of each finger and
later, in 1994, on a ring of color around each joint on the signer’s hand (Starner,
1996).In 1995, Starner began the development of a system that initially involved
the signer wearing two different colored gloves, although eventually no gloves were
required. A camera was placed on a desk or mounted in a cap worn by a signer in
order to capture the movements (Starner, 1996). More recently, a wearable system
has been developed that can function as a limited interpreter (Brashear, Starner,
Lukowicz, and Junker, 2003).[4] To this end, they used a camera vision system
along with wireless accelerometers mounted in a bracelet or watch to measure
hand rotation and since the early 2000s, ConvNets have been applied with great
success to the detection, segmentation, and recognition of objects and regions in
images. These were all tasks in which labeled data was relatively abundant, such as
traffic sign recognition53, the segmentation of biological images54 particularly for
connectomics55, and the detection of faces, text, pedestrians, and human bodies
in natural images36,50,51,56–58.[5]
A major recent practical success of ConvNets is face recognition. Toshev and
Szegedy proposed a deep learning-based method, which localizes body joints by
solving a regression problem and further improves on estimation precision by using
a cascade of these pose regressors. Their work demonstrates that a general deep
learning-based network originally formed for a classification problem can be fine-
tuned and used to solve localization and detection problems.[4] Since the early 2000s,
ConvNets have been applied with great success to the detection, segmentation, and
recognition of objects and regions in images. These were all tasks in which labeled
data was relatively abundant, such as traffic sign recognition, the segmentation of
4
biological images particularly for connectomics, and the detection of faces, text,
pedestrians, and human bodies in natural images. A major recent practical success
of ConvNets is face recognition. [6]
5
CHAPTER 3
FEASIBILITY ANALYSIS
The project is based on how much we have trained the model. The more data
we provide in the data set, the more accuracy can be observed. We might face
challenges on the way but we have enough datasets and problem-solving skills,
making us technically eligible. And regarding the output of the software we would
be using google voice translator.
Since the project is totally based on software, the only expenditure is computational
power. The dataset we will be using are easily available and the computation
power would be from our personal computers and smartphones, so this project is
economically feasible.
The project is operationally feasible since after the completion of the project it
can be operated as intended by the user to solve the problems for what is has been
developed.
6
CHAPTER 4
BLOCK DIAGRAM
The overall workflow of the system is shown in the above block diagram. Data-set
are like the memory of the system. Each and every detection that we view in real
time are the results of the data-set. Data-sets are captured in real time from the
front camera of the laptop . Using media pipe live perception of simultaneous
human pose, face landmarks, and hand tracking in real-time various modern life
applications including sign language detection can be enabled. With the help of
the landmarks or let’s say key points of features (face, pose, and hands) we get
from the media pipe we train our model. All the data that we collected from the
data-sets and from deep learning models are considered as training data. These
data are provided to the system such that the system can detect the sign language
in real-time. Input to this system is real-time or say live video using the front
camera of the laptop. As the real-time input i.e. sign language is provided using
the front camera of laptop, simultaneously live output can be seen on the screen in
text format. It acts as an interface for the Sign Language System providing an
environment for input data to get processed and provide the output.
7
4.2 Use Case Diagram
8
4.3 Level 0 DFD
9
4.4 Level 1 DFD
10
4.5 Activity Diagram
11
CHAPTER 5
METHODOLOGY
In this project, we will collect sign language data from multiple sources to develop
a robust sign language detection system. We intend to work with sign language
schools, collect recorded videos of sign language interpreters, and make use of the
sign language datasets already available to the scientific community. High-quality
video recording tools, including cameras and lighting setups that allow for good
viewing of hand motions, will be used to collect the data
Data preprocessing is a process of preparing the raw data and making it suitable
for a machine-learning model. It is the first and crucial step while creating a
machine-learning model. Real-world data generally contains noises and missing
values and may be in an unusable format, which cannot be directly used for
machine, learning models. Data preprocessing is a required task for cleaning the
data and making it suitable for a machine-learning model, which also increases the
accuracy and efficiency of a machine-learning model.
The video data for sign language detection will be captured using a high-definition
camera with a resolution of 1920x1080 pixels and a frame rate of 30 FPS. The
camera will be positioned to capture the frontal view of the signer’s upper body,
focusing on the hand region.
Then the acquired video data will be segmented into individual sign language
gestures. We will employ an automatic gesture detection algorithm based on motion
12
and hand region analysis. This algorithm will then detect significant changes in
motion and will use hand-tracking techniques to separate consecutive gestures from
video sequences.
From the segmented video data, frames will be extracted at a rate of one frame
per second to capture key moments of each gesture. A sample set of frames will
thus be guaranteed for additional study.
Normalization
We will use min-max scaling to translate the intensity values from the [0, 255]
range to [0, 1], standardizing the pixel values across frames. By ensuring that the
input data has consistent ranges, this normalization step will help in convergence
during model training.
13
5.2.5 Hand Segmentation
We will use a hand segmentation technique based on color and region analysis be-
cause hand movements are important in sign language. To separate the hands from
the background and other unimportant items, this technique will use background
subtraction and skin color modeling.
Dynamic time warping (DTW) will be used to synchronize frames between various
sign language motions. Due to the temporal similarity of hand movements, DTW
will enable us to align the frames and account for changes in gesture duration and
space.
We will use data augmentation techniques to broaden the variety and amount
of the training dataset. These will consist of randomizing the frames’ cropping,
rotation, translation, and flipping. The model’s ability to recognize sign gestures
in a variety of situations will be strengthened with the aid of data augmentation.
14
The CNN layer is the most significant; it builds a convolved feature map by
applying a filter to an array of picture pixels. I developed a CNN with three layers,
each layer using convolution, ReLU, and pooling. Because CNN does not handle
rotation and scaling by itself, a data augmentation approach was used. A few
samples have been rotated, enlarged, shrunk, thickened, and thinned manually.
ReLU layer is a layer where data travels through each layer of the network, the
ReLU layer functions as an activation function, ensuring non-linearity. Without
ReLU, the dimensionality that is desired would be lost. It introduces non-linearity,
accelerates training, and reduces computation time.
Pooling layer is a layer that gradually decreases the dimension of the feature
and variation of the represented data. Decreases dimensions and computation,
speeds up processing by reducing the number of parameters that the network must
compute, reduces overfitting by reducing the number of parameters, and makes the
model more tolerant of changes and distortions. Pooling strategies include max
pooling, min pooling, and average pooling; I tried max pooling. The maximum
input of a convolved feature is used in max pooling.
Flatten is used to transform the data into a one-dimensional array for input to
the next layer.
15
CHAPTER 6
IMPLEMENTATION PLAN
16
CHAPTER 7
REQUIREMENT ANALYSIS
• CPU
• GPU
• Storage
TensorFlow
TensorFlow is a free open source library that can be used in the field of machine
learning and artificial intelligence. Including many other tasks, it can be used for
training purposes in deep learning.
Mediapipe
MediaPipe offers cross-platform, customizable machine learning solutions for live
and streaming media i.e. real time videos. Its features are End to end acceleration,
Build once deploy anywhere, ready to use solution, and Free and open source.
17
Figure 7.1: Hand Landmarks
The user requirement for this system is to make the system fast, feasible, less prone
to error, save time and improve the communication gap between normal people
and deaf people
18
CHAPTER 8
We have collected our data for the training model in real time using the Open-CV
library. We have used the Media-pipe library for key points extraction and these
key points are stored as data. Each data collected will be transformed into a
numpy array of 35 sequences and 20 sub-folders. The sample of data collection is
shown below.
19
8.2 Training Accuracy and Loss
Recall, F1-Score, and Error Rate. Accuracy refers to how closely the measurements
TP + TN
Accuracy = = 100 (8.1)
TP + FN + FP + TN
TP
P recision = = 100 (8.2)
TP + FP
20
In machine learning, recall, also referred to as sensitivity or true
positive rate, represents the likelihood that the model accurately
recognizes the detected anomaly.
TP
Recall = = 100 (8.3)
TP + FN
2 ∗ P recision ∗ Recall
F 1 − Score = = 100 (8.4)
P recision + Recall
21
8.4 Discussion
The above photo is the sign of the ’Hello’ word. The shown hand’s
landmarks are taken in NumPy array form, then it is fed into our
deep learning model. Our model classified the gesture of our hand.
Hence it outputs a class label corresponding to the gesture.
22
CHAPTER 9
EPILOGUE
23
REFERENCES
24