Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)

IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

Virtual Mouse with hand gestures using AI

2023 4th International Conference on Electronics and Sustainable Communication Systems (ICESC) | 979-8-3503-0009-3/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICESC57686.2023.10193709

Mr.Sk. Jilani Basha L.S.L. Sowmya Sk. Rumiya Sk.Afroz

Asst Prof, Dept of CSE Dept of CSE Dept of CSE Dept of CSE
VFSTR, Vadlamudi VFSTR, Vadlamudi VFSTR, Vadlamudi VFSTR, Vadlamudi
[email protected] [email protected] [email protected] [email protected]

Abstract This research presents a novel system to 3. Efficiency: Using a virtual mouse decreases the
control a mouse using hand gestures. Traditional
mouse controls require the user to use a physical amount of time needed to complete tasks by
device, such as a trackpad or a mouse. By using hand eliminating the need for physical movement across
gestures, the user can interact with the virtual mouse
in a more natural manner. The proposed system uses the desktop.
hand tracking techniques to capture and track hand
gestures, and uses a set of customizable rules to The development of virtual mouse AI technology
interpret them into actions. Without using a hardware
mouse, the computer can be operated remotely based
has had a significant impact on fields such as health
on hand gestures and can perform left-click and right- care, engineering, productivity, and education,
click operations. It is based on artificial intelligence for
among others. With continued advancements in AI
detecting the hands. So, the usage of this virtual mouse
will reduce the rapid spread of corona virus by technology, virtual mouse applications are expected
reducing the human-computer interaction. to become more prevalent and widely used in our
daily lives. The hand gestures are simple form of
Keywords— Finger tracking module, Artificial
communication. The moto is to perform various
intelligence hand gesture recognition, Virtual mouse.
operations that can be performed by a physical
mouse. Without using expensive sensors, web cam
is used to identify the gesture and perform the
I. INTRODUCTION
required operation. This enables the client to operate
Virtual mouse refers to a computer program that the system without involving the usage of any
enables users to control their computer cursor or physical device to perform mouse operations. The
other devices without physically touching them. It is first step is to identify the fingertip of hand a camera
particularly beneficial for people with motor and perform the mouse cursor operations. The
disabilities, physical injuries or disabilities, or structure of this research paper is as below. 2nd
anyone who finds it hard to use a traditional physical section describes the Methodology in detail. 3rd
mouse. The virtual mouse AI uses Media pipe, section describes purpose of the project. 4th section
OpenCV and autopy to track the movements of describes Related work on this theory. 5th Section
fingers and move the computer cursor accordingly. describes the proposed system. 6th shows the testing
The creation of virtual mouse technology has and evaluation of the project. 7th sections show the
brought numerous benefits to users, including: result. 8th, 9th, 10th describes the conclusion, future
scope and some of the References of the project.
1. Accessibility: Virtual mouse AI technology
makes computer devices more accessible to people II. METHODOLOGY
with disabilities or who have difficulty using a
This study has proposed a novel system to recognize
traditional mouse.
motions, identify fingers tips, and examine the
2. Improving working conditions: Virtual mouse AI mouse functions. For algorithm implementation,
technology can reduce the amount of repetitive Python programming is used. To process images,
motion injury in computer-intensive work operate the mouse, and create multimodal
environments.

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 1432

Authorized licensed use limited to: SAINT LOUIS UNIVERSITY LIBRARIES. Downloaded on July 01,2024 at 16:51:15 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)
IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

applications, OpenCV, AutoPy and media pipe 40 px, the PC response with play out Using the
library are used. pynput Python library, click the right mouse button.
Assuming both the pointer having tip Id = 1 and the
A. System Design
centre tip is raised with Id = 2 raised and length is
The proposed Hand Tracking and Gesture structure below 40 px and both tips are climbed the screen, the
relies upon the housings gotten using the cam in the PC used for play out the look into mouse work with
PC or computer. Utilizing Python PC visibility Autopy. If none of the fingers are raised, the PC does
module OpenCV, the clips get thing is made and the not do anything.
cam will start recording getting clip. The cam gets
B. System Description in General
and goes through the housings to Hand Tracking and
Gesture. This flowchart shown in the picture of virtual mouse
using hand gestures explains the many
1.Taking the Video and Fining
circumstances and functions employed in the
Hand Tracking and Gesture framework utilizes the system. The video interface will start after the
cam in every bundling is gotten up to the completion camera has been detected. From a video interface,
of process. The clip outlines are taken care of the camera can extract and identify human hand
converting BGR into RGB covering gaps to see the motions. Media Pipe is utilized for hand tracking
fingers in the clips outline with outline by displayed functionality. Once the cursor has been identified,
in the going with program. travel in the appropriate direction to carry out tasks
like click twice, zoom in and out, and zoom in. A
2. Region for functioning through the Window well-functioning web camera is the first crucial
piece of hardware. to take control of the mouse's
The Hand Tracking and Gesture structure uses the
training tool and replace the mouse's operation with
astounding evaluation, and it changes over the.
the tool. In order to make functioning easier, the item
Unequivocally up to fingers are seen and till we see
can be employed as our mouse once it has been
what tip is ready for playing out the device work, a
captured in our frames utilizing OpenCV. The
box in rectangle is shown in regards with PC screen
Webcam's function is to record human hand gestures
in the cam locale how the moment all through the
and motions so that its image can be stored in
screen.
memory. The system's overall diagram is as follow
3. Detecting finger tips

This process is identifying what tip is up utilizing

the finger data of the individual tip which is
observed utilizing Media Pipe and separate points of
the tips that raised, and as per that, the specific
mouse work is performed.

4. For the Movement on Computer Screen

Assuming that the pointer is raised with Id = 1 or

two fore tip having Id = 1 and centre tip having
Id = 2 are raised, the gesture is designed to move
around the PC screen utilizing the Autopy.

5. To detect some gestures

Accepting both the pointer having tip Id = 1 and

centre tip having Id = 2 raised and length is below

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 1433

Authorized licensed use limited to: SAINT LOUIS UNIVERSITY LIBRARIES. Downloaded on July 01,2024 at 16:51:15 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)
IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

separated from the current models and likewise the

model setbacks an enormous piece of the
obstructions of the current structures. Since the
proposed model has more key exactness, the Gesture
affirmation and following can be used for authentic
applications, and correspondingly, it will by and
large be used to decrease the spread of human
contact disorders, since the proposed mouse system
can be made basically using hand movements
without using the standard real mouse.

IV.RELATED ON THIS THEORY

There are numerous applications that use hand

gestures to control the cursor. Instead of relying on
crude communication, it will provide more
advanced human-computer connection.

In 2020 Gubbala Durga Prasanth and P. Srinivas

Reddy proposed a paper on “Virtual mouse
implementation using OpenCV”. They used
OpenCV, Deep learning, NumPy and anaconda
technology. Its major drawback is that it is very
complex to use. [1]

In 2019 Kabid Hassan Shibly, Samrat Kumar

Dey, Md. Aminul Islam, Shahriar Iftekhar Showrav
proposed a paper on “Design and Development of
Hand gesture based virtual mouse”. They used HCI
technology and its major drawback is that it needs
good lightning conditions. [2]

In 2020 Vantukala VishnuTeja Reddy, Thumma

Fig 1: Flow chart
Dhyanchand, Galla Vamsi Krishna, Satish
Maheshwaram proposed a paper on “Virtual mouse
using Coloured fingertips and hand gesture
III.PURPOSE OF THE PROJECT
recognition” They used Hand gesture recognition,
Movement affirmation and following plan is to neural networks and PyAutoGUI technologies for
control the mouse cursor limits by using the hand development.One of its major drawback is it uses
signals instead of using a certifiable mouse. The colored fingertips. [3]
proposed plan can be made by using a computer cam
In 2020 Monali Christina, A. Daniel Computer,
or a key cam which sees the hand signals and hand
Manthan K. Bhatkar, Ofrin P. Lopes proposed a
tip and cycles these edges to play out the particular
paper on “Virtual mouse using object tracing”. It
mouse capacities. From the unavoidable
uses HCV technique and OpenCV. Its main
consequences of the model, we can show up at an
drawback is that its accuracy varies according to the
objective that the proposed Gesture affirmation and
background it gives 95% accuracy in plain
worldwide situating system has performed well
background and 40% in non-plain background. [4]
overall and has a more basic accuracy stood

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 1434

Authorized licensed use limited to: SAINT LOUIS UNIVERSITY LIBRARIES. Downloaded on July 01,2024 at 16:51:15 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)
IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

In 2022 Nishi Jain, Sanjay Jadhav proposed a provided by Media Pipe Tasks. Import the audio,
research paper “AI Virtual mouse using OpenCV text, or visual library into your development project,
python” in JETIR conference 2022. Its main depending on the Media Pipe Task the app used. It
drawback is decrease in accuracy in right click and can be installed using “pip install media pipe”.
there are difficulties in clicking. [5]
AUTOPY:
In 2022 Hritik Joshi, Nithin waybhase, Ratnesh L,
A straightforward, cross-platform GUI automation
Dharmendra proposed a paper on “Design of virtual
library for Python is called AutoPy. It has tools for
mouse using gesture recognition and machine
managing the mouse and keyboard, detecting colors
learning” in Research square 2022. It main
and bitmaps on the screen, and showing alerts. It can
drawback is colour detection algorithm. It can only
be installed using "pip install autopy".
perform basic operations such as scrolling, selecting
and changing slides. [6] VI.TESTING AND EVALUATION
V. PROPOSED SYSTEM

The proposed Hand Tracking and Gesture Table1

framework can be utilized to conquer issues when in
doubt, for example, conditions where there is no
Hand tip Function Success Failure Accuracy
space to utilize a veritable mouse and likewise for
detection
people who have issues in their grasp and can't deal
with a few purposes. Also, in the midst of the human
contact sicknesses circumstance, it isn't defended to
utilize the contraptions by contacting them since it
could accomplish what is going on of spread of the Index Move 100 0 100
illness by arriving at the gadgets, so the proposed Finger up cursor
Hand Tracking and Gesture construction will be
utilized for conquer issues because arm motion and
finger ID is utilized for a couple of touchless
applications. Python code is utilized for empowering Index click 98 2 98
the Hand Tracking and Gesture framework, and in and
addition, OpenCV for PC visibility is utilized for middle
construction. In the proposed Hand Tracking and finger up
Gesture system, the model uses the Media Pipe pack (single
for hand tracking and monitoring of the tips of the tap)
fingers. The following are the libraries used in
implementation of this system. Index Double 98 2 98
and click
OPENCV:
middle
OpenCV is a open source computer vision library. It finger up
can be installed using "pip install OpenCV-python". (double
tap)
MEDIAPIPE:

An open-source system called Media Pipe can be Result 296 4 98.6

used to create computer vision inference over any
type of sensory data, including audio and video.
Three prebuilt libraries for vision, text, and audio are

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 1435

Authorized licensed use limited to: SAINT LOUIS UNIVERSITY LIBRARIES. Downloaded on July 01,2024 at 16:51:15 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)
IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

EVALUATION GRAPH

Evaluation graph
100
100
99.5
99
Accuracy

98.5
98 98
98
97.5
97
Move Single
Double
click click

Function Fig 5: Identifying landmarks

Fig 2: Graph

VII. RESULT

Fig 6: Identification of coordinates

Identifying the positions of the finger:

Fig 3: Starting video interface using webcam

Fig 7: All fingers down

Fig 4: Frame rate and display

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 1436

Authorized licensed use limited to: SAINT LOUIS UNIVERSITY LIBRARIES. Downloaded on July 01,2024 at 16:51:15 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)
IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

Fig 11: Click and double click

Fig 8: All fingers up

Tapping the index and middle finger for once leads

to click as shown in the above fig and twice leads to
double-click.

VIII. CONCLUSION

The fundamental objective of the hand gesture-

based virtual mouse is to eliminate the need for a
physical mouse in favour of hand gestures for cursor
control. The virtual Mouse with hand gestures make
use of a camera to direct the cursor of mouse and
carry out its function. We implemented mouse
selection and navigation. This includes icons, their
associated duties, and actions like left, right, double
Fig 9: Index and Middle finger up
click. The model has some drawbacks like dragging
and scrolling. In order to get over these restrictions
and deliver more accurate results. In the end, we
draw the conclusion that working on the projects
taught us a lot about how toa use open computer
vision, its library, and the python programming
language. In a word, it can be said that future goal is
to electronically give mouse to students who are
physically disadvantaged and students who desire to
do away with the physical mouse. The accuracy of
this project is 98.6%. The major drawback is that it
can perform all cursor operations except scrolling
function.

Fig 10: Identification of index finger IX. FUTURE SCOPE

This AI-driven device Virtual Mouse enables

users to move their hands or make gestures on their
computers to control the cursor. It recognises hand

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 1437

Authorized licensed use limited to: SAINT LOUIS UNIVERSITY LIBRARIES. Downloaded on July 01,2024 at 16:51:15 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)
IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

gestures and movements and converts them into [8] Lu, J., Li, H., & Chen, Y. (2020). An Intelligent Virtual Mouse
cursor movements and clicks using Artificial System Based on Face Recognition and Eye Tracking. IEEE
Intelligence (AI). This technology can be applied in International Conference on Artificial Intelligence and
a number of areas, including gaming, processing 3D Computer Applications (ICAICA) (pp. 164-168). IEEE.
video content, office work, web browsing, and more. [Link](https://ieeexplore.ieee.org/abstract/document/922417
Users benefit from its versatility and ease as they are 3)
able to complete tasks without a traditional mouse.
[9] Halder, S., Bhattacharjee, S., & Bhowmick, S. (2018). AI-
X. REFERENCES Based Virtual Mouse System Using Convolutional Neural
Network. International Conference on Advances in
[1]Prasad, G. D. (2020). Virtual mouse implementation using
Computing, Communications and Informatics (ICACCI) (pp.
OpenCV. JETIR, Volume 7, Issue 6, Page numbers - 5.
1945-1951).
[Link]( https://www.jetir.org/papers/JETIR2006455.pdf)
IEEE.[Link](https://ieeexplore.ieee.org/abstract/document/8

[2]Shibly, K. H. (2019). Design and Development of Hand 554684)

Gesture-Based Virtual Mouse. In Proceedings of the IEEE

[10] Chen, J., Du, B., Wang, J., & Xu, C. (2019). AI Virtual
Conference on ICASERT. [Link](
Mouse Control Using Convolutional Neural Networks.
https://ieeexplore.ieee.org/document/8934612)
International Conference on Machine Learning and

[3]Reddy, V. V., & Vantukala, T. (2020). Virtual Mouse Using Cybernetics (ICMLC) (pp. 932-937).IEEE.

Coloured Fingertips and Hand Gesture Recognition. In [Link](https://ieeexplore.ieee.org/abstract/document/894897

Proceedings of the IEEE- 3)

HYDCON.[Link](https://ieeexplore.ieee.org/document/924
2677)

[4]Christina, M., Daniel, A., Bhatkar, M. K., & Lopes, O. P.

(2020). Virtual Mouse Using Object Tracing. In Proceedings
of ICCES-2020. [Link](
https://www.researchgate.net/publication/342852902_Virtu
al_Mouse_Using_Object_Tracking)

[5]Nishi Jain, Sanjay Jadhav (2022). AI Virtual mouse using

OpenCV python. JETIR, Volume 9, Issue 7, Page numbers -
5. [Link]( https://www.jetir.org/papers/JETIR2207073.pdf)

[6]Hritik Joshi, Nithin waybhase, Ratnesh L, Dharmendra M

(2022). Design of virtual mouse using gesture recognition
and machine learning. Research square 2022.

[Link](https://assets.researchsquare.com/files/rs-
1616375/v2/9df1e90e-660f-4fdb-83a4-
446bc6e738f9.pdf?c=1654102964)

[7] Rajendra, P., Kaushik, P., Gautam, A., & Saha, S. (2020). AI-
Based Virtual Mouse Control Using Deep Learning
Techniques. IEEE 2nd International Conference on
Sustainable Energy, Electronics, and Computing Systems
(SECS) (pp. 647-652). IEEE.
[Link](https://ieeexplore.ieee.org/abstract/document/925107
4)

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 1438

Authorized licensed use limited to: SAINT LOUIS UNIVERSITY LIBRARIES. Downloaded on July 01,2024 at 16:51:15 UTC from IEEE Xplore. Restrictions apply.