SILENT SPEECH INTERFACE USING FACIAL

RECOGNITION AND ELECTROMYOGRAPHY

seminar report submitted

in partial fulfillment of the requirement for award of the degree of

Bachelor of Technology
in
Computer Science & Engineering

by

M.KISHORE 19UECS0470
V.SIVA KUMAR 19UECS0920
K.V.KOUSHIK 19UECS0405

Under the guidance of

Ms. R. Sumathy, M.Tech.,
Assistant Professor

DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

SCHOOL OF COMPUTING

VEL TECH RANGARAJAN Dr.SAGUNTHALA R&D

INSTITUTE OF SCIENCE AND TECHNOLOGY
(Deemed to be University Estd u/s 3 of UGC Act, 1956)
CHENNAI 600 062, TAMILNADU, INDIA

May, 2021
 BONAFIDE CERTIFICATE
It is certified that the work contained in the seminar report titled ”SILENT SPEECH
INTERFACE USING FACIAL RECOGNITION AND ELECTROMYOGRAPHY”
by ”M.KISHORE (19UECS0470),V.SIVA KUMAR (19UECS0920),K.V.KOUSHIK
(19UECS0405)” has been carried out under my supervision and that this work has
not been submitted elsewhere for a degree.

Signature of Supervisor
Ms. R. Sumathy
Assistant Professor
Computer Science & Engineering
School of Computing
Vel Tech Rangarajan Dr.Sagunthala R&D
Institute of Science and Technology
May, 2021

Signature of Head of the Department

Dr. V. Srinivasa Rao
Professor & Head
Computer Science & Engineering
School of Computing
Vel Tech Rangarajan Dr.Sagunthala R&D
Institute of Science and Technology
May,2021

i
 DECLARATION
We declare that this written submission represents our ideas in our own words and
where others’ ideas or words have been included, we have adequately cited and ref-
erenced the original sources. We also declare that we have adhered to all principles
of academic honesty and integrity and have not misrepresented or fabricated or fal-
sified any idea/data/fact/source in our submission. We understand that any violation
of the above will be cause for disciplinary action by the Institute and can also evoke
penal action from the sources which have thus not been properly cited or from whom
proper permission has not been taken when needed.

(Signature)
M.KISHORE
Date: / /

(Signature)
V.SIVA KUMAR
Date: / /

(Signature)
K.V.KOUSHIK
Date: / /

ii
 APPROVAL SHEET

This seminar report entitled SILENT SPEECH INTERFACE USING FACIAL RECO-
GNITION AND ELECTROMYOGRAPHY by M.KISHORE (19UECS0470),V.SIVA
KUMAR (19UECS0920), K.V.KOUSHIK (19UECS0405) is approved for the de-
gree of B.Tech in Computer Science & Engineering.

Signature of Supervisor
Ms. R. Sumathy
Assistant Professor
Computer Science & Engineering
School of Computing
Vel Tech Rangarajan Dr.Sagunthala R&D
Institute of Science and Technology

Signature of Seminar Handling Faculty

Dr. J. Santhosh
Associate Professor
Computer Science & Engineering
School of Computing
Vel Tech Rangarajan Dr.Sagunthala R&D
Institute of Science and Technology

Date: / /
Place:

iii
 ACKNOWLEDGEMENT
We express our deepest gratitude to our respected Founder Chancellor and Pres-
ident Col. Prof. Dr. R. RANGARAJAN B.E. (EEE), B.E. (MECH), M.S (AUTO).
DSc., Foundress President Dr. R. SAGUNTHALA RANGARAJAN M.B.B.S.,
Chairperson Managing Trustee and Vice President.

We are very grateful to our beloved Vice Chancellor Prof. S. SALIVAHANAN,

Ph.D., for providing us with an environment to complete our seminar successfully.

We record indebtedness to our Dean & Head, Department of Computer Science

& Engineering Dr. V. SRINIVASA RAO, M.Tech., Ph.D., for immense care and
encouragement towards us throughout the course of this seminar.

We take this opportunity to express our gratitude to Our Internal Supervisor

Ms. R. SUMATHY, M.Tech., for her cordial support, valuable information and
guidance, he helped us in completing this seminar through various stages.

A special thanks to our Seminar Coordinator Dr.T.VEERAMAKALI Ph.D.,

for her valuable guidance and support throughout the course of the seminar.

We thank to our Dr. J. SANTHOSH M.E.,Ph.D., for the valuable information

shared in proceeding with our seminar.

We thank our department faculty, supporting staff and friends for their help and
guidance to complete this project.

M.KISHORE 19UECS0470
V.SIVA KUMAR 19UECS0920
K.V.KOUSHIK 19UECS0405

iv
 ABSTRACT

Silent Speech Interface (SSI) has been introduced to place an end

to noise pollution and to assist those folks that have lost their voice
and can’t speak on the mobile phone.This technology helps people to
communicate in noisy places and to reduce noise pollution to some
extent. This technology detects every lip movement and converts them
into electrical pulses then into speech sound that everyone can under-
stand.It uses Electromyography and Image Processing for monitoring
the tiny muscular movements that occur when we speak.SSI can only
be used for languages with simpler pronunciations like English and
French. This technology can provide some secrecy by avoiding others
overhear what you talk in mobile.

Keywords : Silent Speech Interface(SSI),Silent Sound Technol-

ogy(SST), Electromyogrpahy , Sensors, Image Processing.

v
 LIST OF FIGURES

Fig No Title Pg No
4.1 Electromyographic sensors attached to the face 8
4.2 Electromyographic Activity captured by the Sensors 8
4.3 Process model architecture 9
5.1 SSI using Image Processing 10

vi
 LIST OF ACRONYMS AND
ABBREVIATIONS

abbr Abbreviation
EMG Electromyography
SST Silent Sound Technology
FT Fourier Transform
IP Image Processing
AI Artificial Intelligence
EU European Union

vii
 TABLE OF CONTENTS

Page.No

ABSTRACT v

LIST OF FIGURES vi

LIST OF ACRONYMS AND ABBREVIATIONS vii

1 INTRODUCTION 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Aim of the Seminar . . . . . . . . . . . . . . . . . . . 2
1.3 Scope of the Seminar . . . . . . . . . . . . . . . . . . 2
1.4 Methodology . . . . . . . . . . . . . . . . . . . . . . 2

2 LITERATURE REVIEW 3

3 SEMINAR DESCRIPTION 5
3.1 Existing System . . . . . . . . . . . . . . . . . . . . . 5
3.2 Feasibility Study . . . . . . . . . . . . . . . . . . . . 6

4 METHODOLOGIES 7
4.1 ELECTROMYOGRAPHY . . . . . . . . . . . . . . . 7
4.2 IMAGE PROCESSING . . . . . . . . . . . . . . . . 9

5 RESULTS AND DISCUSSIONS 10

6 CONCLUSION AND FUTURE ENHANCEMENTS 11
6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . 11
6.2 Future Enhancements . . . . . . . . . . . . . . . . . . 11

References 11
 Chapter 1

INTRODUCTION

1.1 Introduction

A Sound is something that you hear or that can be heard. Nowadays

sound (or) noise pollution become one of the threat. Think of getting
surrounded by large gathering of people and receiving an important
call,at-least once time in our lifetime we have gone through this situa-
tion.One of the prime reasons why “Silent Speech Interface” technol-
ogy emerged.This SSI can also be termed as ”Silent Sound Technol-
ogy” among the researchers. Silent Sound Technology can reduce the
noise pollution caused by the humans while talking on their phones
to some extent. Silent Sound Technology also provides a solution
for those who’ve lost their voices somehow. This technology tends
to notice every movement of the lips and transform them into sounds,
which could help people who lost their voices, Rather than making any
sounds, your handset will convert the movements your mouth makes
by measuring muscle activity, then convert this into speech that the
person on the other end of the call can hear. The Silent Sound Tech-
nology was firstly developed at ‘Klaushure Institute of Technology’.
“We currently use electrodes which are glued to the skin. In the future,
such electrodes might for example by incorporate into cell phones”,
said Michael Wand, from the KIT

1
1.2 Aim of the Seminar

The main objective is to analyze and understand the every lip move-
ment and the facial expressions has been recorded and converted into
speech as well as text.

1.3 Scope of the Seminar

At-least 10 percentage of worlds population live with some kind of

disability. Many born with it, While others acquire later by an acci-
dent,by a trauma or by effects of ageing. Which makes the life of them
harder. To communicate with others we need ability of speech. Ac-
cording to an survey nearly a million of people in the world don’t have
the ability of speech, Whereas 63 percentage of them born without it
and 27 percentage lost it later. This has given a motivation to create a
technology that can produce speech according to the movements of lip
without physical voice. This provided a solution for the people who
lost voice and people with onset fluency disorder.

1.4 Methodology

SSI consists of Electromyography and Image Processing as two

main methodologies to acquire the speech and later the Signals will
be converted to Speech by Computer Algorithm.

2
 Chapter 2

LITERATURE REVIEW

The study area that concentrates on the implementation of EMG[1]

and image processing for developing the tech named Silent Sound
Technology. The first documented EMG done by Francesco Redi in
1666.There are many applications for the use of EMG. EMG has been
used for neuromuscular and neurovascular diagnosing. This the main
objective of EMG is to measure the electrical activity released by mus-
cle contradiction, the movements are stored as a electrical signals later
the signals can be converted[2].
This EMG can also be used for Silent Speech Interface by measuring
the electrical activity of the facial muscular movments happen when
speak. If we speak air will pass through the tongue and the larynx
and the different words are produced by the articulation muscle in the
mouth and jaw region[4]. It involves of electrodes sticked to the facial
muscles. Then recorded signals are compared with pre-recorded signal
pattern, If any signals matched with this the output spoken words will
be heard[3].
There the problem arises the pierced electrodes causes discomfort
and bruises when used for long period of time and also causes asymp-
tomatic haematoma. The speech interface consist of four Sensors Pres-
sure sensor,Vibrator sensor,Electromagnetic sensor,Motion sensor[6].

3
 The simplest form of image processing converting the video into
images with minimal distortions and changes. In the field of computer
science Image processing means giving an image as an input and the
output maybe the set of numbers, a matric or an equation related to the
image[7].
In image processing the lip movements are captured with a Ultra
sound device which couple with high resolutions optical camera fit-
ted in the gadget, then the images will converted into grayscale image
for analyzing[5]. After analyzing the sequence will be compared with
the pre-recorded sequences, The output speech will be based on the
matched sequence.

4
 Chapter 3

SEMINAR DESCRIPTION

3.1 Existing System

At present sign language and text-to-speech has been used as a tool

for communication by the people who can’t talk. The problem is
sign language its little hard to learn. There is no universal Sign lan-
guage, Different Sign languages are used in different Regions.There
are somewhere between 138 and 300 different types of sign language
used around the globe today.So,That requires an alternativs Technol-
ogy. That led the way to Silent Speech Interference.

3.1.1 Advantages

1. It reduces noise pollution to certain level.

2. It ensures some level of privacy.
3. Can getback the voice of lost.

3.1.2 Disadvantages

1. The output sound will always be like talking to a Robot.

2. The equipment are all high cost.
5
3.2 Feasibility Study

At present there is not a lot of companies which produces EMG

machines. So,there is a demand for raw materials. It Requires a tech-
nically skilled person to alter the EMG according to our purpose. Once
the sensors,electrodes and EMG are connected the implementation of
the System is ease. There is no widespread market for this and the
price of setting this up is bit expensive, But still there are people who
looking for this. EU countries show more interest towards SSI.

6
 Chapter 4

METHODOLOGIES

4.1 ELECTROMYOGRAPHY

Electromyography (EMG) is a technique used for recording and

evaluating the electrical activity of the muscular movements and rep-
resent it in audible signal as well as text. Electromyography contain
of fine wired electrode which is attached to the face and the electri-
cal signal produced by facial muscle is been recorded and compare
with the recorded signal of the spoken words which are the same. It
proved that electrical signal produced match with spoken words then
the electrical signal can now be transmitted and convert to an equiva-
lent sound signal at the listener side. The transducers involved converts
the pulses into electrical signals is the result of neuromuscular activa-
tion associated with muscle contraction.The EMG comprises of four
main methodologies as Data Corpus,Feature Extraction, Classification
using Decision Trees, Channel Reduction and Optimisation. The Sig-
nal obtained from the EMG’s are classified by the computer algorithm
for the relevent information, Which by time domain or frequency do-
main or by both. The number of the electrodes increases the efficiency
of the output,but it may cause some pain to the user.

7
 Fig 4.1: Electromyographic sensors attached to the face

Fig 4.2: Electromyographic Activity captured by the Sensors

8
4.2 IMAGE PROCESSING

Image processing is any sort of signal processing that the input is a

picture . Image processing consists of two techniques Analog image
processing and Digital image processing. Analog image processing
can only be used the pre-captured hard copy of the images. Camera
fitted with the gadget can be used to capture the lip moments and can
be processed by Digital image processing. Digital Image processing
make use of computer algorithms to process the image.Process model
is considered as a most suitable technique for this process. Process
model is said to be the iterative method,Speech Analysis can be done
by capturing only the lip region is achieved.If efficiency with detec-
tion technique isn’t proper then the edge value falls out of the defined
unique index value and retrial has got to be made.The captured im-
ages can be processed and the matching words will be fetched from
the database. Then it will be converted into Speech.

Fig 4.3: Process model architecture

9
 Chapter 5

RESULTS AND DISCUSSIONS

After spending a lot with this research of Silent Sound Technol-

ogy we came to know that this system has an high efficiency over
the pre-existing methods used. Unlike other methods its efficiency
is based on the electrodes.Higher and Sensitive the Electrodes higher
the efficiency of the System. The other technique Image processing is
based the computer’s processing speed and the algorithm.The image
processing Technique works by constantly taking pictures of the lip
movements and process them to get a sequence of words. By this it
can also generate texts which can lately translated into any language
by conversion tools.

Fig 5.1: SSI using Image Processing

10
 Chapter 6

CONCLUSION AND FUTURE

ENHANCEMENTS

6.1 Conclusion

This technology has said to have the high efficiency. Silent Sound
technology one of the recent trends in the field of information. It will
be a useful innovation for mere future will be used on our day to day
life.

6.2 Future Enhancements

There is a possibility of making these devices handy by incorporat-

ing nano technology and adding electrodes into mobile phones. There
is research going on developing an AI to capture lip movements and
convert it into word.

11
 Reference

[1] AsifAbdullah,KrishnanChemmangat, [2020] “A Computa-

tionally Efficient sEMG based Silent Speech Interface using
Channel Reduction and Decision Tree based Classifica-
tion”,ScienceDirect,Procedia Computer Science,Volume 171,
Pages 120-129 .

[2] Jose A. Gonzalez-Lopez; Alejandro Gomez-Alanis; Juan M.

Martı́n Doñas; José L. Pérez-Córdoba; Angel M. Gomez,
[2020],”Silent Speech Interfaces for Speech Restoration: A Re-
view”,IEEE,IEEE Access Volume-8,Page(s): 177995 - 178021.

[3] Sonal Sadanand Gaikwad, Jayash Ganesh Ambade, [2020]

“VOICE GENERATION WITHOUT VOICE REMOVAL: A
SILENT VOICE TECHNOLOGY”International Research Jour-
nal of Engineering and Technology (IRJET) e-ISSN: 2395-
0056,Volume: 07 Issue: 06,Page 5880-5883.

[4] Sanni Hafiz Oluwasola-[2017]-”Silent Sound Technol-

ogy”International Journal of Science and Research (IJSR)
Volume 6 Issue 4,Pages: 1732 - 1735

[5] Vibhu Varshney,Deeksha Singh,Ayush Tiwari-[2016],”Deep learn-

ing and its application in silent sound technology”IEEE,3rd Inter-
national Conference on Computing for Sustainable Global Devel-
opment.
12
[6] Deepak Balwani, Honey Brijwani,Somyata Rastogi,[2014] “Talk-
ing Without Talking” t. Journal of Engineering Research and Ap-
plications,2, Vol. 4, Issue 4( Version 9),pp.51-56.

[7] Priya Jethani, Bharat Choudhari,[2014], “Silent Sound Technol-

ogy: A Solution to NoisyCommunication”-International Journal of
Engineering Trends and Technology (IJETT) – Volume 9 Number
14.

[8] Denby B, Schut T. HondakHueber T, Gilbert J.M, BrumbergJS-

(2010) “slient speech interface surface electromyography signal
processing and application” A.N Norali, M.H. Mat Som, Journal
of electromyography and kinestology 10 (2000) 351 -360.

13