See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/370785494

Mouse Cursor Movement and Control using Eye Gaze- A Human Computer
Interaction

Conference Paper · December 2022

DOI: 10.1109/ICAIoT57170.2022.10121742

CITATIONS READS
0 323

3 authors, including:

Nusrat Jahan
Daffodil International University
27 PUBLICATIONS 168 CITATIONS

SEE PROFILE

All content following this page was uploaded by Nusrat Jahan on 12 January 2024.

The user has requested enhancement of the downloaded file.

 Mouse Cursor Movement and Control using Eye Gaze- A Human Computer
Interaction
2022 International Conference on Artificial Intelligence of Things (ICAIoT) | 979-8-3503-9676-8/22/$31.00 ©2022 IEEE | DOI: 10.1109/ICAIoT57170.2022.10121742

Payel Miah Mirza Raina Gulshan Nusrat Jahan

Department of CSE Department of CSE Department of CSE
Daffodil International University Daffodil International University Daffodil International University
Dhaka, Bangladesh Dhaka, Bangladesh Dhaka, Bangladesh
e-mail: [email protected] e-mail: [email protected] e-mail: [email protected]

Abstract—An emerging technology is human computer the system that can play a great role for these persons so they
interaction (HCI). One of the most important HCI strategies is can utilize their knowledge to do something.
the eye gazing method, which enables the user to operate the This technology has been the subject of extensive
display without using their hands. Direct eye movement development in recent years with the goal of making it
detection, template, appearance, feature, hybrid, regression, accessible to the general population. First, three different
clustering 3D methods are all ways to categorize eye gaze technologies were used to construct eye trackers, which
detection techniques. Deep learning, a technology that mimics monitor eye movement. These three methods include optical
human behavior and features like speech recognition, image tracking, which uses light to measure the motion of eye, and
recognition, and language translation can make this possible. A
potential electric measurement, which is being used to
web camera was employed in this study to capture a frame of
an eye frame for mouse cursor movement. In connection with
calculate eye’s moment of dipole to determine motion of
the point previously mentioned, we must first concentrate on eye. Eye attached tracking involves attaching a device to the
the function of our eye. We are employing a web camera for eye, such as a contact lens.
pupil identification, which can manage the computer’s cursor. Then, using the information gathered by the tracker,
For this paper, an Aspect Ratio Eye (EAR) is determined that several methodologies were created in order to identifying
corresponds to the blinks of eye’s (right or left) applying the the x, y coordinates of the screen region the user is viewing.
library of Mediapipe which is open source and acts as a This information was then used to carry out a variety of
computer vision library. You can provide smart people with generic activities in human computer conversation, including
crippled limbs who are having trouble using computers a object selection, object movement, text scrolling, and other
chance to express their opinions. Here, the method’s objective [1]. The proposed technology is a hand free communication
is improving the experience of using computer for physically system between human and computer on the concept of
disabled people by assisting them in overcoming challenges Human-machine-interaction (HMI). External webcam is
using a mouse. being used to take input from human and control the cursor
based on this input. Taking input is a real time process [2].
Keywords-Eye Gaze; Human Computer Interaction; Feature- Also the technology will exchange traditional screen
based Classification; Conventional Mouse pointing method to a smarter and new process [3].
In this paper, we proposed a multimodal camera-face
I. INTRODUCTION human machine interface architecture where acknowledging
Moving a computer mouse using finger has become a eye movement tracking has been completed. Constant eye
typical method of moving the computer cursor currently input has traditionally been employed for individuals who
moving around the computer screen technology. To align it are unable to move their eyes but can able to use them as an
with the motions of the system notices any movement of the input.
cursor follow human finger or computer mouse. This is using
II. BACKGROUND STUDY
present technology will not be possible referred to by some
as “amputees” since they do not the ability to use their hands. Interest in creating a method for studying eye movements
As a result, the amputee and other people with physical has grown recently. Numerous algorithms, models, and
disabilities can if their eyeball movement can be controlled, strategies have been examined in this research survey. The
they will be to the cursor, mapped. Depending on how the findings of the survey have helped identify certain crucial
movement and direction of the eyes are seen tracking the factors and aspects that have a significant impact on the
direction an eye is looking, the physically mapped cursor accuracy and efficiency of eye gazing processes. In order to
will be used disable individuals will be able to move the conduct additional research and analysis, a variety of eye
cursor. From the past, hand disabled person are deprived to gaze-based HCI models and algorithms have been examined.
use computer or laptop for this reason and they took Below is a discussion of the advantages and disadvantages of
themselves too far from the new technologies. Though eye gaze-based models.
having enough knowledge on modern technology, they A Visual Analytic in Deep Learning Approach to Eye
cannot utilize their knowledge because of their disability. We Movement for Human-Machine Interaction Based on Inertia
have thought about intelligent disabled persons and proposed Measurement, 2020 [2]. This method uses the user’s eye

979-8-3503-9676-8/22/$31.00 ©2022 IEEE

Authorized licensed use limited to: Lakehead University. Downloaded on January 12,2024 at 17:22:32 UTC from IEEE Xplore. Restrictions apply.
gaze to control the computer cursor. A user can type, run Mohd Khalel published a work [10] which concentrates
computer software, use a computer mouse, access the on the usage of MATLAB to operate the pupil detection
internet, and create voice syntheses by staring at the control mechanism. Using a web camera, the head will be first
keys that are visible on the screen. The sole requirements for identified, followed by the eye, which is then extracted using
using the eye gaze are the ability to maintain a relatively the MATLAB library, leading towards the monitoring of
stable head position and control of at least one eye with gaze. The iris transition would then be determined by
decent vision. The technology employs an external speech calculating, and the transition would then be plotted using a
device and outputs fair results. graphical interface. The eye is then identified, and this is
Eye gaze estimation is a process that use human plotted with cursor, and thus the mouse pointer moves in
computer intersection (HCI). Gaze pointing method bring a accordance. In another promising work proposed by
new process which control cursor in screen with eyeball Abdullah S. Almansouri [16] mentioned MATLAB code
rotation instead of traditional mouse. Eye gaze pointing and analysis approach to monitor eye movement. This study
eye gaze direction both were estimated by Yong Huang and work on four eye direction: left/right, and up/down.
his team 2021 [4]. They constructed local sharing network Eye gaze estimation taking a great contribution on eye
for feature extraction and propose Multiview Multitasking blinking prediction. Nowadays eye blinking prediction
Learning (MML) for gaze direction. Information were taken method is used in various platforms like drowsiness
from eyes as input in real time. Neural light method is a kind detection, face recognition application, unlocking phones,
of method that use infrared light source to implement a gaze etc. In raspberry pi3 gazing eye blink is estimated in 2021.
estimation. Some limitation was faced like changing in the Dlib library was utilized to detect facial features. Eye aspect
visible light spectrum and low contrast images, but it is less ratio (EAR) was built to detect the blink. It provides an
sensitive to infrared light. accurate calculation using the ratio of distance among the
A strategy for human-computer interaction based on eye eye landmarks [11], [13], and [14].
movement was put forth by Ramsha and Usmani in an article Pupil center movement was being tracked dynamically.
published in 2016 [5]. Our paper is dedicated to outlining Gaze estimation expresses feelings, desires, emotions, and in
also implementing a human-machine interfaced framework tension. A person-dependent calibration-free system could
which traces the movements of the user’s eye while using deal with the illumination [12]. Examining and establishing
EAR equation to move the mouse cursor while using eye the variables that capture the changes caused by blinking in
pupil implementation, and Eye squints, an eye movement, this article study. Face detection is performed using the
are used while closing and opening a certain sign. The sixth Mediapipe technique, while eye extraction is performed via
sense technology, developed by a team at MIT [6], promises template matching. The eye part is removed using specific
to improve HCI by using palm and eye motions. The geometric dependencies. Eye tracking is carried out in its
complete system may be mounted on the person’s helmet in first stages via template matching. For cursor movement,
order to be operated anywhere around the globe and face classifiers were utilized (such as up, down, left, and
displayed onto flat surfaces. It does not offer a good solution right). The major steps of this system are: 1) it recognizes
that really can communicate with other suitable devices or faces in videos; 2) it extracts eyes; and 3) it extracts features;
ensure access and support for the impaired. Mr. Yushou and finally, it recognizes cursor movement and mouse
Tang and Jianhuan Su propose a model that predict human events. On the other hand, OpenCv and neural networks also
emotion based on eye movement. They thought that eye introduced and used for developing a system for eye
movement can express human emotion and so they use back- movement detection [15].
propagation (BP) algorithm to gaze eye movement [7]. In recent years, several machine learning(ML)
Sidrah liaqat and his co-author’s proposed a model on techniques, such as random forest, decision tree, k-nearest
predicting ASD diagnosis in children based on eye gaze neighbor, and support vector machine(SVM) are familiar for
process. They tried to predict autism spectrum disorder on achieving success in this field. S Akshay et al. [17] claimed
young children at a very early age. They proposed two ML algorithms for tracking eye movement using raw data of
machine learning method for implementation. The first eye movement. In addition to eye tracking classification also
method is synthetic saccade pattern which is used to focus by research scholar. In 2021, Thibhika Ravichandran
represent the baseline scan-path and the other method is et al. [18] proposed ML based approach to classify eye
image based which adopts a more holistic image based movement. Here, long-short term memory (LSTM) and
approach by feeding the input image and a sequence of convolutional neural network (CNN) applied and got 88.3%
fixation maps into a state-of-the-art convolutional neural and 90% accuracy, respectively. Yuehan Yin et al. [19] also
network [8]. Using gaze estimation, a model was proposed to proposed CNN for eye movement classification. After data
predict driver’s eye fixation to ensure safe driving in pre-processing and feature engineering they applied keras
autonomous vehicles. Convolution neural network predict framework on tensorflow for CNN model.
the potential saliency regions in the driving environment and After observing several research works proposed by
then use the probability of the driver gaze direction, given research community, in this study proposed a system for
head pose as a top-down factor [9]. Tracking the driver’s eye disable people to monitor personal computer using eye
movement, gaze direction and gaze movement with real time movement.
the driver’s next action is being predicted and can make a
judgment on safety and risk level.

Authorized licensed use limited to: Lakehead University. Downloaded on January 12,2024 at 17:22:32 UTC from IEEE Xplore. Restrictions apply.
 III. PROPOSED METHODOLOGY Alternately, automate keyboard and mouse clicks in
In this study, Mediapipe is used to track eye movement to order to establish interaction with another program using a
control computer mouse pointer movement. Eyeball Python script. This allows for cursor control. The user is
movement is identified by the camera and analyzed by required to stand in front of the personal computer screen,
MediaPipe. Real-time computer vision is the main emphasis and a dedicated video camera mounted above a computer or
of a suite of programming interfaces called MediaPipe. laptop’s screen to record the user’s gaze. The laptop
PyAutoGUI library is used here. The automation library continuously analyzes the attention’s video image to
PyAutogui for Python supports keyboard and mouse control. determine where the user is addressing the monitor. Figure 1
illustrates the overall working procedure of this study.

Figure 1. Overview of Proposed System.

MediaPipe Facemesh will detect the face and divide it lockstep with the pupil, so if the pupil moves to the
into se of landmarks to identify facial movement. From left, it will also move to the left if it moves to the
mediapipe detected face we will measure head movement to right. Sample input shown in Figure 2.
determine scroll event. When user will open and close their
mouth system will turn itself off/on. By extracting eye
pacement from MediaPipe we will determine where to place
the cursor. From the detected landmark for both eye we will
detect distance between upper eyelid and lower eyelid. If we
detect that the distance between the eyelids is very low, we
will execute following click event. For a user who is not
disabled our system will follow same set of execution order
Figure 2. Left Side Movement and Right Side Movement.
as we saw for disabled user but this user will also be able to
use his hands to turn the system off and on.
• A pupil scale can be used to move the eye’s pupils
A. Maintenance of the Mouse Pointer vertically. When looking downward, the eyes are
The center of the iris is where the pupil, which is the slightly half-closed. These phenomena can be
focal point of the eye, is located. Light enters the eye through exploited to direct the mouse pointer’s movement
the pupil; hence the main focus of the suggested technique is from top to bottom, presents in Figure 3.
locating the pupil. The pupil of the subject serves as the
source of eye movement information. When the user looks at
the mouse cursor in the middle, the cursor starts to travel in
the direction of the user’s gaze movement and stops when
the user’s eye lands on its starting point. This serves as the
input point for gaze tracking for cursor movement.
• Circular artifacts can be used to move the eye’s
pupil horizontally. The mouse pointer moves in Figure 3. Pupil Upwards and Pupil Downwards.

Authorized licensed use limited to: Lakehead University. Downloaded on January 12,2024 at 17:22:32 UTC from IEEE Xplore. Restrictions apply.
• The functionalities of the right click and left click
will be accessed through right eye winking and left
eye winking. For the left eye wink detection
landmarks 145159 was used in Figure 4 (b), and
for the right eye wink detection landmarks 386374
was used in Figure 4 (a).

a). Mouth Landmarks

a). Right Eye Landmarks.

b). System Off/On

Figure 5. (a) Mouth Landmarks and (b) System Off/On.

IV. RESULT ANALYSIS AND DISCUSSION

The accuracy of the system should have been 100
percent, but external factors like light intensity, an average
camera, and others caused it to be a little low. After much
b). Left Eye Landmarks trial and error, the accuracy is now between 80 and 95
percent, with the maximum average accuracy occurring in
Figure 4. (a) Right Eye Landmarks, and (b) Left Eye Landmarks.
bright sunlight and a decrease to 85 percent at night. If the
• Most of the time it will be a must to scroll a page, illumination is good, the device will provide a decent
without scrolling we can’t actually provide a good percentage of accurate reading.
usage. For scrolling, users need to move their head. TABLE I. EYE ASPECT RATIO VALUES.
Move up and down will scroll up and downwards. In
the same manner, moving the head right and left will Eye Features Ratio
scroll right and leftwards. Open fully 0.36-0.44
• User will be able to turn the system on when turned
off by opening their mouth. When the system is Open partially 0.31-0.37
turned on users will be able to turn it off by simply Closed fully <0.15
opening their mouth. Here we calculated the distance
of landmarks 1314 to determine the distance Right wink 0.12-0.18
between lips. That whole processes are shown in Left wink 0.12-0.18
Figure 5 (a) and (b).
• Users will be able to turn the system on by opening Table I prepared for presenting the eye movement taking
their hand. When the system is turned on users will ration. To understand the outcome of this, work several
be able to turn it off by closing their hand. results are visualized in Figure 6. After considering various
input forms, such as eye detection situation, system on-off

Authorized licensed use limited to: Lakehead University. Downloaded on January 12,2024 at 17:22:32 UTC from IEEE Xplore. Restrictions apply.
process using mouth, left click and right click using eyes. We Our system is now totally dependent on a laptop but it
prepared the visual results to observe the proposed system’s will be great if you could work with that without
performances. implementing it in the computer rather than running it using
raspberry pi3. Making cursor movement faster and slower
will also give a lot of application. For gaming, we may
introduce multiple control systems which will revolutionize
the gaming experience. Implementing the whole thing
alongside virtual reality (VR) will provide an experience
never experienced before. In near future we will try to
implement this system on personal computers, mobile
phones and other digital devices. We hope that most of the
device which controls by a mouse and screen touch system
will replace into eye movement controlling technology in
upcoming new generation.

REFERENCES
[1] Robert J. K. Jacob, “The use of eye movements in human-computer
interaction techniques: what you look at is what you get”. ACM
Trans. Inf. Syst. 9, 2, pp. 152–169, 1991. doi:
10.1145/123078.128728
[2] Fahim, Shahriar Rahman, et al., “A visual analytic in deep learning
approach to eye movement for human-machine interaction based on
inertia measurement.” IEEE Access 8: 45924-45937, 2020.
[3] V. Khare, S. G. Krishna and S. K. Sanisetty, “Cursor Control Using
Eye Ball Movement”. 5th International Conference on Science
Technology Engineering and Mathematics (ICONSTEM), 2019, pp.
232-235, doi: 10.1109/ICONSTEM.2019.8918780.
[4] Huang, Yong, Ben Chen, and Daiming Qu, “LNSMM: Eye gaze
estimation with local network share multiview multitask”. arXiv
preprint arXiv:2101.07116 (2021).
[5] R. Fatima, A. Usmani and Z. Zaheer, “Eye movement based human
computer interaction”. 3rd International Conference on Recent
Advances in Information Technology (RAIT), pp. 489-494, 2016,
doi: 10.1109/RAIT.2016.7507950.
[6] Mistry, Pranav and Maes, Pattie, “SixthSense: a wearable gestural
interface”. ACM SIGGRAPH ASIA 2009 Sketches. 85. doi:
10.1145/1667146.1667160.
[7] Tang, Yushou, and Jianhuan Su, “Eye movement prediction based on
adaptive BP neural network”. Scientific Programming (2021).
[8] Liaqat, Sidrah, et al., “Predicting ASD diagnosis in children with
Figure. 6. Device Monitoring using Eye Gaze. synthetic and image-based eye gaze data”. Signal Processing: Image
Communication 94 (2021): 116198.
[9] Shirpour, Mohsen, Steven S. Beauchemin, and Michael A. Bauer,
V. CONCLUSION AND FUTURE DIRECTION “Driver’s Eye Fixation Prediction by Deep Neural Network”.
VISIGRAPP (4: VISAPP), 2021.
This research proposed a novel HMI system for [10] Mohd Khalel, M. N., Mujeeb Rahman, K. K., Zubair, M., Ansari, H.
manipulating a pointer on the computer screen while & Mohamed, F., “Eye-controlled mouse cursor for physically
tracking the user’s eyes. It uses computer vision and pattern disabled individual”. Advances in Science and Engineering
recognition techniques. The following solution, which uses Technology International Conferences, ASET 2018: 1–4. doi:
10.1109/ICASET.2018.8376907.
PyAutoGUI, will give physically disabled users an
alternative method of interacting with real-world programs in [11] Ibrahim, Bishar R., et al., “Embedded system for eye blink detection
using machine learning technique”. 1st Babylon International
a better and more effective way. This particular technology Conference on Information Technology and Science (BICITS). IEEE,
allows physically challenged individuals greater adaptability 2021.
by allowing them to simulate mouse pointer action just using [12] Cazzato, Dario, et al., “Real-time gaze estimation via pupil center
facial recognition. Additionally, even non-disabled users will tracking”. Paladyn, Journal of Behavioral Robotics 9.1 (2018): 6-18.
be able to use this system and thus it can be concluded that [13] Chandra, B., M. Rohit, and R. Sriram Vignesh, “Eyeball Movement
further utilization of this system while updating the Cursor Control Using OpenCV”. ECS Transactions 107.1 (2022):
framework and graphical user interface will revolutionize the 10005.
IT world for the better and take gesture base computing to a [14] A. Sivasangari., D. Deepa., T. Anandhi, A. Ponraj and M. S. Roobini,
“Eyeball based Cursor Movement Control”. International Conference
new milestone. on Communication and Signal Processing (ICCSP), 2020, pp. 1116-
1119, doi: 10.1109/ICCSP48568.2020.9182296.

Authorized licensed use limited to: Lakehead University. Downloaded on January 12,2024 at 17:22:32 UTC from IEEE Xplore. Restrictions apply.
 [15] Sharanyaa, S. and K, Yazhini, R. P, Madhumitha and Rani.B, Inventive Technology (ICSSIT), 2020, pp. 949-955, doi:
Yamuna, “Eyeball Cursor Movement Detection Using Deep 10.1109/ICSSIT48917.2020.9214290.
Learning” Proceedings of the International Conference on Innovative [18] T. Ravichandran, N. Kamel, A. A. Al-Ezzi, K. Alsaih and N. Yahya,
Computing & Communication (ICICC) 2021, doi: “Electrooculography-based Eye Movement Classification using Deep
10.2139/ssrn.3884881 Learning Models”. IEEE-EMBS Conference on Biomedical
[16] A. S. Almansouri, “Tracking Eye Movement Using a Composite Engineering and Sciences (IECBES), 2021, pp. 57-61, doi:
Magnet”. IEEE Transactions on Magnetics, vol. 58, no. 4, pp. 1-5, 10.1109/IECBES48179.2021.9398730.
2022, Art no. 5200305, doi: 10.1109/TMAG.2022.3152085. [19] Yin, Yuehan, Chunghao Juan, Joyram Chakraborty, and Michael P.
[17] S. Akshay, Y. J. Megha and C. B. Shetty, “Machine Learning McGuire, “Classification of eye tracking data using a convolutional
Algorithm to Identify Eye Movement Metrics using Raw Eye neural network”. 17th IEEE International Conference on Machine
Tracking Data”. 3rd International Conference on Smart Systems and Learning and Applications (ICMLA), pp. 530-535. IEEE, 2018.

Authorized licensed use limited to: Lakehead University. Downloaded on January 12,2024 at 17:22:32 UTC from IEEE Xplore. Restrictions apply.

View publication stats