Facial Movements Based Mouse Cursor Control for Physically Disabled Individual

Abstract - The system we have described in able to move the cursor. An ‘eye tracking mouse’ will be
this which presents a hand-free interface of a lot of use to disabled and an amputee currently, this
between human and computer. Generally the technology is not available at a large scale, only a few
mouse which used by people has been companies are trying to develop this technology and
replaced in a new way which uses the human have made it available. This paper intend to develop an
facial expressions and features. It uses various eye tracking mouse where most of the functions which
image processing methods such as face present in the computer mouse will be available, so that
detection, eye extraction and interpretation of an amputee or disable people can operate mouse using
a sequence of eye blinks in real time for their eye. Eye tracking technology has become one of
controlling a nonintrusive human computer the most popular techniques within the human and
interface. An interaction with the computer computer interaction (HCI) this is very important for the
with the mouse is replaced with the human eye people who have difficulty with speech and movement
movements. It uses a typical webcam to disabilities, especially for the handicapped and
capture an input image. Mouse cursor control amputees person. The idea of computers with the eyes
can be done by facial movement by moving the will remove the help required by other person to handle
face towards left, right, up and down and the computer.
mouse events are controlled through eye
blinks. To perform these operations different 1.1 RELATED WORK
algorithms like Haar Cascade algorithm, Dlib
are used. Our system is mainly aimed for The existing system which was proposed by
physically challenged peoples to have an Margrit Betke introduced the camera mouse for
effective communication with computer. nonverbal and quadriplegic people. Here, by using a
camera the user movements can be tracked where
Key Words: Human Computer Interaction these can be plotted to the cursor points of the
(HCI), Conventional mouse, Nonintrusive, Eye mouse which can seen on the computer screen.
movements , Haar Cascade, Dlib. Furthermore Robert Gabriel Lupu proposed
another method for human computer interaction
which has made use of head mounted device to
1. INTRODUCTION trace eye movement and to convert it on screen.
Another technique by Prof. Prashant salunke
Moving the finger or the computer mouse has proposed a technique of eye tracking using Hough
become a common way to move the computer transform.
cursor along the computer screen in the present In this project, we develop a system which will use
technology. To map it to the movements of the only Webcam, and to use human eyes as a pointing
cursor the system detects any movement in the device for computer device and to provide user
computer mouse or the human finger. This friendly human-computer interaction. Designing a
current technology will not be able to make use system for tracking face and eye using camera and
by some people like ‘amputees’ as they do not also using facial landmark’s technique. This project
have their hands to operate. Hence, the amputee is mainly developed to help physically disabled
and other physically challenged people can be people. We can accurately detect the eye movement
able to operate, if their eyeball movement can be by detecting the exact location of iris present in our
mapped to the cursor. If the direction of the eyes and counters, here we use regression tree
eyeball movement is traced and direction algorithm or Decision trees algorithm.
towards which eye is looking at can be tracked,
the cursor will be mapped and the physically
challenged people can
 -

This paper introduced a system which is a human 5) Eye Tracking: We use coordinate system to detect
computer interface for performing the mouse the eye movement, which locates the center position
operations using their eyes. Our idea is to make of the eye.
mouse cursor hands free by using light-reflection 6) Control Signal: Depending on eye movement cursor
based systems with non-imaging sensors, thus is moved in different directions.
providing an alternative to computer mouse.

2. Methodology

The main contribution of our project is

building user friendly human and computer interaction.
Since our project is about detecting the features of the eyes
and facial movement and plotting them to the computer
cursor. Primarily the webcam wants to be accesses, which
means the webcam is on and starts capturing images in
video. Once webcam is started, the process begins and it
extracts all frames from the captured video. The video
captured by webcam has frame-rate around 30frames per
second, at every 1/30th of a second a frame will be
processed. Before the features of the frame are detected the
frame goes through a set of processes and then plotted to
the cursor. And this process interminably occurs as a part
of a loop for every frame.
Fig -2: Use Case diagram of the project
2.1 Modeling and Analysis
Use case consists of user and processor where user is to
provide the input to the system and processor is used to
process the input data and provide output.

3. ALGORITHM

Step 1: Run the program, where the webcam will be ON.

Step 2: Pinpoint the position of the face in the image.
Step 3: Recognize the key facial features on the face ROI.
Step 4: Once the face is detected, MAR (Mouth Aspect
Ratio) and EAR (Eye Aspect Ratio) will be calculate.
Step 5: Input will be read.
Step 6: The action of the eyeball can be plot to the cursor.
Step 7: Managing of mouse cursor is acquired by facial
movements as moving the face towards left, right, up and
Fig -1: Model of the project down and mouse events are managed through eye blinks.

There are many image processing operations carried out

in system, includes
1) BGR to Gray Conversion: To decrease the system
delay time an original colored image is converted
into gray colored image because for proper
processing a frame size of the image should be
minimum.
2) Feature Detection and Blurring Image: To detect the
exact corners of specific area of resized image, the
Gaussian blur filter performs blurring of the image.
3) Canny Edge Detection: It used a blurred image. It will
be put in for soft edges of the image.
4) Hough Circle Transform: To draw a circle on eye
pupil we use this Hough circle transformation
method.
Fig -3: Work Flow of the application
4. PROPOSED WORK
this relation called the eye aspect ratio (EAR).
In this paper, we have described a system that has facial
tracking, detection of face, detection of eye and
continuous eye blinks for controlling a nonintrusive
human computer interface. Human eye movement has
been used in the place of the conventional mouse. The
main aim of this system is to help physically challenged Fig -6: Eye Aspect Ratio
and amputee people especially person who do not have
Where p1…p6 are 2D facial landmarks locations. The
their arms.
distance between the vertical eye landmarks is computed
by the numerator of the eye aspect ratio while the distance
4.1 Face Detection
between horizontal eye landmarks is computed by the
The regions of the face has been detected by using facial
denominator. There is only one set of horizontal points but
landmarks such as eyebrows ends, eyes, mouth ends,
two sets of vertical points.
nose, jaw line. It is successfully applied to alignment of
face, estimation of head pose, face swapping, detection of
blink and much more.
4.3 Eye Movement Classification
The movement of eye can be classified using eye aspect
ratio where if horizontal ratio is more than set threshold
value then the user is looking left else user is looking right,
if the values do not meet both threshold values then user is
looking at the center, if EAR values is less than the set
threshold then user is blinking.
So using the values of EAR we can classify the eye
movement as it moves towards left, right, up and down.
Hence, if eyeball movement can be traced and in which
direction, the eye is staring and looking at can be tracked,
the eyeball movement can be plotted to cursor and disabled
Fig -4: Detection of Facial landmarks and physically challenged people will be able to move the
cursor at their will.
Shape prediction problems are subset of landmarks
detection of face. As an input the image is taken (and in
generally an ROI which describes an interest in object),
shape predictor make an effort of interest along the
shape of the face by localizing key points. Facial
landmarks detection which is of two step process: Step 1:
Pinpoint the position of the face in the image. Step 2: On
the face of ROI detecting key facial structures.

4.2 Eye Region Detection

Begins with the left corner of the eye as you were looking
at the person and it is working clockwise around the left
out region. Each eye is represented with coordinates (x,
y).

Fig -7: Movement Classification of Eye

4.4 Advantages
1) It will help in eliminating the other person who is
required to control the cursor and to handle the system
Fig -5: Eye Region Detection
(computer).
2) This is also helpful for amputee and physically
On looking an image, we should consider the key points
challenged people to operate the computer with the help
of an eye: As there is a relation between the height and
of their facial expressions and eyeball movements.
width of the points (coordinates). Soukupova and Cech
3) This measure will be the most useful for paralyzed
proposed a work in their paper, the detection of eye blink
people who can operate cursor only by their eyeball
in Real-Time using facial landmarks, by this they derived
movements.
an equation that reflects
4) There is no age limit to perform this technique. [5] Hossain, Zakir, Md Maruf Hossain Shuvo, and
Prionjit Sarker “Hardware and software
5. CONCLUSIONS implementation of real time electrooculogram
By implementing this process we can conclude that the (EOG) acquisition system to control computer
cursor movement control can be controlled by the facial cursor with eyeball movement” In 2017 4 th
expressions and eyeball movement i.e., without International Conference on Advances in Electrical
requirement of hands to operate the computer. This is Engineering (ICAEE), pp. 132-137.
useful for disabled and amputees to perform these cursor [6] Mohamed Nasor, K K Mujeeb Rahman, Maryam
operations by using eyeballs, without requirement of the Mohamed Zubair, Haya Ansari, Farida Mohamed
other person to operate the cursor. This technology in “Eye-controlled mouse cursor for physically
the future can be enhanced by inventing or modifying or disabled individual”, 2018 Advances in Science and
improving more techniques like clicking events, human Engineering Technology International Conferences
computer interface systems which uses eyeball (ASET), 2018.
movement, eye blinks to perform the cursor operation.
[7] K. H. Shibly, S. Kumar Dey, M. A. Islam and S.
Technology extended to this eye tracking technique to
Iftekhar Showrav, “Design and Development of
get the efficient and accurate movement.
Hand Gesture Based Virtual Mouse”, 2019 1 st
International Conference on Advances in Science,
REFERENCES
Engineering and Robotics Technology (ICASERT),
[1] SunitaBarve, DhavalDholakiya, Shashank Gupta, Dhaka, Bangladesh, 2019, pp. 1-5, doi:
DhananjayDhatrak, “Facial Feature Based Method 10.1109/ICASERT, 2019.8934612.
For Real Time Face Detection and Tracking I- [8] “Hand Gesture Recognition System Using Image
CURSOR”, International Journal of EnggResearch Processing” Sagar P. More and Abdul Sattar,
and App., Vol. 2, pp. 1406-1410, Apr (2012), IEEE, International Conference on Electrical, Electronics,
2012. and Optimization Techniques (ICEEOT) -2016.
[2] Yu-Tzu Lin Ruei-Yan Lin Yu-Chih Lin Greg C Lee [9] Neil Castellino and Michelle Alva, “An image-based
“Real-time eye-gaze estimation using a low- eye controlled assistive system for paralytic
resolution webcam”, Springer, pp. 543-568, IEEE, patients”, IEEE Conference Publications, 2017.
Aug (2012). [10] Shu-Fan Lin, Xuebai Zhang, Shyan-Ming Yuan,
[3] Samuel Epstein-Eric MissimerMargritBetke “Using “Eye Tracking Based Control System for Natural
Kernals for a video based mouse replacement Human-Computer Interaction”, Computational
interface”, Springer link, Nov (2012). Intelligence and Neuroscience, 2017
[4] Cech and Soukupova, “Real-Time Eye Blink
Detection using Facial Landmarks”, Center for
Machine Perception, February 2016.