VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)

ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

Virtual Mouse using Artificial Intelligence

Prathmesh Medhekar1, Anant Prajapati2, Tanushka Padelkar3, Monali Pimpale4
1
(Computer Engineering Department, Viva Institute of Technology, India)
2
(Computer Engineering Department, Viva Institute of Technology, India)
3
(Computer Engineering Department, Viva Institute of Technology, India)
4
((Asst. Professor,Computer Engineering Department, Viva Institute of Technology, India)

Abstract : This paper is an excellent example of Human computer interaction where the user can perform actions
of a mouse without using any external hardware mouse, we can achieve this by adapting various artificial
intelligence algorithms and python libraries. The working of this system starts when we first assign gestures into
the machine as input values eg; scroll up and down, click and double clicks and drag and drop etc, user can select
gestures as per his/her choices then these assigned gestures are recorded using webcam so by this the data is
filmed and actions are performed in the machine virtually. This saves the cost of the external mouse and also
provides users with privilege of operating the machine easily.
Keywords - Artificial Intelligence, Gesture assigning, Human computer interaction, Python libraries, Virtual
Mouse.

I. INTRODUCTION
This model and paper solve various previous problems of the external mouse where users had to buy
them and have to constantly click and move the external mouse which can sometimes hurt or damage the users’
hands and fingers. This drawback is avoided by using this system which as discussed operates the machine
virtually by using a virtual mouse and also avoids other issues of external mouse and thereby providing us the
efficient and cost-free technique, and still now the researches always make an effort to increase the qualities of
this virtual mouse systems as the organizations consider this system very useful.
The main aim of this system is to enhance the daily use of an external mouse for the daily computer
users which helps them to smoothly and quickly perform the operations and also give the users a gift of the new
AI techniques and rules. By just installing a basic webcam and some essential algorithms we can build this
superb setup of a virtual mouse for our machine which ensures that user is having a good time while operating
the machine using the system and is facing no issues, although a lot of researchers has found this system as an
amazing power of AI and spite, they suggest a user of this system must have basic knowledge of AI and other
essential libraries required for installing and running the system.
The other requirements are a proper machine because heavy programming languages and modules need
and take up big storage, compatible OS and inbuilt advanced machine properties, so before adapting this system
we must check for the basic and advanced modules of the machine to install this system.

II. LITERATURE SURVEY

The research shows the application of a virtual mouse which is an advanced form of an external mouse which
establishes a great comfort for operating the machine and thereby, the virtual mouse using Al tackles several other
issues regarding the operations of the mouse. It brings up a bit of easy and proper nice. implementation which is
much more efficient as compared to other techniques and methods in different researches, because as in other
researches there are complex and lengthy lists of requirements of materials other external stuff for operating the
machine, Whereas, in this research we provide an efficient and quite easy way to develop the system, by avoiding
the materials and adapting more logical modules and methods so as to use them in the correct form and correct
place. Here the system runs only with the help of fingers of user i.e. by hand detection, then user assigns gestures

1
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

as the inputs values for example: scroll up down, click, double clicks and drag and drops such gestures are stored
inside the system also the user can give its desired finger movements as its gestures as per comfort level of
operating so it also gives the choice to user for gesture formation and as for next the webcam plays a major role
as it records the finger and hand movements of the user by which the system reads and tries to recognize the
gesture played by user and matches them with stored gestures and after each successful match the system reads
and fetches the gestures for operating the machine and for every gesture the same cycle takes place. This allows
the user to operate at its own demand and comfort level also avoiding the need of an external hardware mouse
which also saves cost and time and thus we call it a virtual mouse system. The observations made during building
the system are clear and helpful as it teaches about different modules and applicative ways of exploring Al into a
machine. As discussed, it saves a lot of material, money and time which helps the user to invest time in adapting
such advantageous techniques. By means, the users learn about the new modules and algorithms of it and apply
their own logics and efforts to bring out such systems for making technologies easier to understand and operate.
The system is built indeed using AL, OpenCV. Hand tracking module and other well-known libraries of python
which lets the user to perform mouse actions virtually by just imitating gestures in front of a webcam and
recording them in system to run those gestures and by which we can handle the machine virtually i.e., by without
even touching the screen or needing any external mouse. The technique in our research ensures that the user
doesn't have to use any other equipment or other external stuff because a lot of different techniques and methods
are available in the field of Al, but they all need one or the other external tools or equipments for adapting virtual
mouse and has very complex structures because of their methods. Some methods need the stickers on fingers for
performing gestures, while others require facial movements and so on are the techniques which are very complex
for installing and running the machine. Deep learning based AI virtual mouse for Computer vision, The mouse is
one of the wonderful inventions of Human-Computer Interaction (HCI) technology. Currently, wireless mouse or
a Bluetooth mouse still uses devices and is not free of devices completely since it uses a battery for power and a
dongle to connect it to the PC. In the proposed AI virtual mouse system, this limitation can be overcome by
employing a webcam or a built-in camera for capturing of hand gestures and hand tip detection using computer
vision. The algorithm used in the system makes use of the machine learning algorithm. Based on the hand
gestures, the computer can be controlled virtually and can perform left click, right click, scrolling functions, and
computer cursor function without the use of the physical mouse. The algorithm is based on deep learning for
detecting the hands[1]. This paper proposes a way to control the position of the cursor with the bare hands without
using any electronic device. While the operations like clicking and dragging of objects will be performed with
different hand gestures. The proposed system will only require a webcam as an input device. The software that
will be required to implement the proposed system are OpenCV and python. The output of the camera will be
displayed on the system's screen so that it can be further calibrated by the user[2]. Real time virtual mouse using
RGB-D images, In this work, we propose a novel virtual-mouse method using RGB-D images and fingertip
detection. The hand region of interest and the center of the palm are first extracted using in-depth skeleton-joint
information images from a Microsoft Kinect Sensor version 2, and then converted into a binary image. Then, the
contours of the hands are extracted and described by a border-tracing algorithm. The K-cosine algorithm is used
to detect the fingertip location, based on the hand-contour coordinates. Finally, the fingertip location is mapped
to RGB images to control the mouse cursor based on a virtual screen. The system tracks fingertips in real-time at
30 FPS on a desktop computer using a single CPU and Kinect V2 [3]. Mouse cursor control based on hand
gestures
The apparition on market of the low-cost webcams with, at least, satisfactory qualities open up new directions
regarding the implementation of human computer interaction (HCI) interfaces. The paper presents a HCI interface
for mouse cursor control. The purpose of the implemented solution is to control the mouse cursor by user hand
gestures captured through a webcam. For improving the gesture recognition based on the fluctuation of
illuminance levels the finger strips color detection was used. The results reveal the good behavior of the system
in low light conditions[4]. In human-computer interaction, a virtual mouse implemented with finger tip
recognition and hand gesture tracking based on image in a live video is one of the studies. In this paper, virtual
mouse control using fingertip identification and hand gesture recognition is proposed. This study consists of two
methods for tracking the fingers, one is by using colored caps and other is by hand gesture detection. This includes
three main steps that are finger detection using color identification, hand gesture tracking and implementation on
on-screen cursor. In this study, hand gesture tracking is generated through the detection of the contour and
formation of a convex hull around it. Features of hands are extracted with the area ratio of contour and hull
formed[5]. An interactive model of virtual mouse by Researchers around the world is now focused on making
Our devices more interactive and try to make the devices operational with minimal physical contact. In this
research, we propose an interactive computer system which can operate without any physical keyboard and mouse.
This system can be beneficial to everyone, especially to theparalyzed people who face difficulties to operate

2
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

physical keyboard and mouse. We used computer vision so that the user can type on a virtual keyboard using a
yellow-colored cap on his fingertip, and can also navigate to the mouse controlling system. Once the user is in
mouse controlling mode, the user can perform all the mouse operations only by showing a different number of
fingers. We validated both modules of our system by a 52 years old paralyzed person and achieved around 80%
accuracy on average. Deep Learning-Based Real-Time AI Virtual Mouse System Using Computer Vision to
Avoid COVID-19 Spread, The AI virtual mouse system's main goal is to control mouse cursor functionalities
using hand gestures rather than a hardware mouse. The proposed system can be realized by using a webcam or a
built-in camera that detects and interprets hand motions and hand tips in order to perform specific mouse actions.
We can conclude from the model's results that the suggested AI virtual mouse system worked extremely well and
has a higher accuracy than existing models, and that the model also solves most of the constraints of existing
systems. The AI virtual mouse can be utilized in real-world applications because the proposed model is more
accurate, and it can also be employed in simulations[6]. Virtual Mouse using Hand Gestures, Our natural ways of
communication: body language and speech, should be reflected in our computers and technology. Intelligent
robots that can work alongside computers are now being developed, making Human-Computer Interaction (HCI)
more pleasant (HCI). On the other hand, our suggested project aims to create a hand gesture-based system that
allows users to control desktop mouse motions. A desktop webcam is used by our system to identify hand gesture
movements. The idea is to use a simple camera instead of a classic or standard mouse to control mouse cursor
functions. The Virtual Mouse provides an infrastructure between the user and the system using only a camera. It
allows a user to engage with a machine without the use of any mechanical or physical equipment, and it even lets
the user control the machine[7]. Virtual Mouse using Hand Gesture and Color Detection, Bluetooth hardware and
a Bluetooth dongle must be installed on the computer to utilise a wireless mouse. There will be no such constraints
with the proposed system, which will instead rely on gesture recognition. Object detection, picture processing,
and colour recognition using "Sixth sense technology" are the major technologies used in this research. Sixth
sense technology is a collection of wearable gadgets that serve as a gesture-based interface between the physical
and digital worlds. The goal is to move the mouse cursor on the screen without using any hardware, such as a
mouse, and just by using finger movements, which is known as gesture recognition. In this research, we offer a
novel Human Computer Interaction (HCI) approach in which cursor movement is controlled in real time[8].
Design and Development of Hand Gesture Based Virtual Mouse, Since the development of computer technology,
the technique for constructing a process of human-computer interaction has evolved. The mouse is a fantastic tool.
HCI (Human-Computer Interaction) technology is a new invention.Despite the fact that wireless or Bluetooth
mouse technology is still being developed,That technology isn't entirely devoid of devices. a wireless
mouseBattery power and a connecting dongle are required. The presence of additional electronics in a mouse
makes it more difficult to use. Make use of it. This limitation is not met by the proposed mouse technology. This
virtual mouse system based on HCI is proposed in this paper. Hand motions and computer vision With the use of
a camera, I was able to catch certain gestures and color-processed video from a built-in camera or a
webcamTechniques for segmentation and detection The user will have the ability to with which you can control
some of the computer's cursor functionalities[9]. Virtual Mouse Using Hand Gesture, In this modern era of
technology, Human Computer Interaction (HCI) has become one of the most essential undertakings. Many things
have changed in this world as a result of new technology, yet there are still a few things that remain the same. The
Computer System is one of the better examples one can give. The same computerized gadgets that were employed
previously are still utilized in today's computer systems. However, by utilizing HCI, we may replace these
electronic gadgets with the most up-to-date technology available. This study focuses on human-computer
interaction (HCI) with computerized devices such as the mouse. We shall be able to eliminate the use of external
and digital devices by using this gesture, movement [10] . A Review on Virtual Mouse Using Hand Gesture and
Color Detection, Intelligent machines that may be utilized in conjunction with computers are being developed
these days, which aid in friendly Human-Computer Interaction (HCI). For disabled people, using a mouse and
keyboard might be uncomfortable and inconvenient at times. The answer for impaired people to handle the
computer is a virtual mouse or keyboard system of computers In recent years, various technologies have been
applied to develop the virtual mouse. In this case,we attempted to offer a review of various virtual mouse
technologies[11]. Because of its wide range of applications and capacity to efficiently connect with machines via
human-computer interaction, the hand gesture recognition system has gotten a lot of attention in recent years. A
survey of modern hand gesture recognition systems is offered in this study. The most important aspects of Hand
gesture recognition systems are faced with a variety of obstacles. Examine the most modern methods. A technique

3
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

for recognising postures and gestures was also demonstrated. Summary of hand gesture research findings.
Methods, datasets, and a comparison of the various steps of gesture recognition are also provided[12]

III. METHODOLOGY
The research paper promotes a method whereby a real-time camera can be used to control cursor movements
using Human Computer Interaction (HCI). This is an alternative to the current methods, which involve manually
pressing buttons or changing the cursor's position. Of a physical computer mouse. Instead, it utilizes a camera
and computer vision technology to control various mouse events and is capable of performing every task that the
physical computer mouse can. The computer technologies continue to grow and the importance of Human
Interaction computers is also increasing. Creating a virtual human interactive device such as a mouse using a
webcam. The system takes various ways of providing for adoption of virtual mouse technique also researches
were made to apply the system using various ways and methods.
To simplify the process of the review we have divided the working and the methodology into four phases
which simplifies the understanding of the working and also the implementation:
1. Detection
2. Tracking
3. Pixel Positioning
4. Movement and clicks required.
The phase of detection includes object detection, hand detection, 21 landmarks detection and fingertip
detection. The virtual mouse gestures recognition program will constantly acquire real-time images where the
images will undergo a series of filtration and conversion. Upon completion of the conversion process, the program
applies image processing techniques to determine the coordinates of the target position which is called tracking.
After that, it will proceed to compare the existing positions within the frames with a list of hand
Landmarks taken from media pipe module, where different combinations consist of different mouse functions if
the current combination found a match, the program will execute the mouse function, which will be translated
into an actual mouse function to the user’s machine. Following the mapping of pixel position takes place, the
setting of cursor positions just happens after this phase which helps in the production of mouse movements and
the clicks which are required for the successful implementation.

4
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

Fig.1 Block Diagram

Above figure shows the block diagram of the proposed system. The block diagram has a
webcam which captures hand inputs, takes up image frames which are converted to fix frame values
and sets current location of the controller, using them mouse actions are performed.

IV. ALGORITHMS AND IMPLEMENTATION

ALGORITHMS:
Basic algorithms and libraries like OpenCV, NumPy, media pipe etc. are used and must be installed in
the implementation process, and for working an external webcam is installed for recording the touch, types and
scribbles of the user and according to that the mouse cursor controller will perform the actions.
Our project focuses on the implementation and designing and demonstration of the virtual mouse which
detects hand gestures patterns instead of physical mouse and it also demonstrates hand tracking and hand
detection to run operations in which the system used in our project detects 21 landmarks and converts
xyzzy coordinates in picture pixels to get the location of the specific landmark.

The algorithm for the proposed system is as follows:

Step 1. Import all necessary packages (OpenCV, Mediapipe, Time, numpy, autopy)
Step 2. Set the webcam resolutions
Step 3. Find hand landmarks and get Tip of fingers.
Step 4. Check which fingers are up and assign gestures.
Step 5. Assign those gestures to Moving or Clicking modes for the mouse controller.
Step 6: Convert coordinates into pixels for clear image framing.
Step 7: Move the mouse and check proper working and framing.
Step 8: Test gestures actions using given values and check working of clicking and moving modes.
Step 9: Find distance between fingers
Step 10: Set frame rate for proper frames and smoothening.
Step 11: Now, Display the screen, use gestures and operate the mouse controller.

IMPLEMENTATION:

5
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

This chapter provides the partial implementation and the screenshots of the results. In partial
implementation successfully installed webcam and other essential system modules and libraries. Also
designed and developed a Hand Tracking Module using python module which will be the highlight of this
project.

Fig. 2 shows the code used to open the webcam so that object detection following hand detection will take place.

Fig.3 code shows the parameters required for detecting hands using 21 landmarks.

Fig.4 shows the 21 landmarks used in the model.

Fig.5 shows the color intended to give specific landmarks according to the requirements.

6
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

Fig.6 code shows the Frame Rate Per second {FPS}on the webcam output

Fig.7. input shows the tracking of hands and the colored landmarks

V. CONCLUSION
A virtual gesture control mouse is a system that uses a real-time camera to guide the mouse cursor and do
its duty. Mouse navigation, icon selection, and features and duties such as left, right, double click, and scrolling
were all implemented. To do mouse indicator motions and icon selection, this system uses image comparison and
motion detection technology. Based on the results, we may assume that if we supply enough light and a good
camera, the algorithms will operate in any domain. In this proposed system will become more organized as a
result. More functionality, including interaction in multiple windows, increasing and shrinking windows, closing
windows, and so on, will be merged in the future using the palm and multiple fingers. The project's goal was to
make machines more interactive and sensitive to human behavior. The main purpose of this study was to create a
technology that is both inexpensive and portable, and that can run on any standard operating system. By detecting
the human hand and moving the pointer in the direction of the human hand, the suggested system is utilized to
control the mouse pointer. Simple mouse functions such as left-clicking, dragging, and cursor movement are
controlled by the system.

Acknowledgements
We would like to express a deep sense of gratitude towards our guide Prof. Monali Pimpale, Computer
Engineering Department for her constant encouragement and valuable suggestions. The work that we are able to
present is possible because of her timely guidance.
We would like to thank Project Head of the Computer Engineering Department, Prof. Monali Pimpale for her
support and coordination. We would like to thank Head of the Computer Engineering Department, Prof.
Ashwini Save for her support and coordination. We are also grateful to the teaching and non-teaching staff of
the Computer Engineering Department who lend their helping hands in providing continuous support.

7
www.viva-technology.org/New/IJRI
VIVA-Tech International Journal for Research and Innovation Volume 1, Issue 5 (2022)
ISSN(Online): 2581-7280
VIVA Institute of Technology
10th National Conference on Role of Engineers in Nation Building – 2022 (NCRENB-2022)

REFERENCES
[1] D.L. Quam, “Gesture recognition with a Data Glove,”10.1109/NAECON.1990.112862, vol. 2, pp. 755 - 760, 1990.
[2] C.-Chiung Hsieh, D.-Hua Liou, & D. Lee, , “A real time hand gesture recognition system using motion history image,” Proc. IEEE
Int'l Conf. Signal Processing Systems (ICSPS).
[3] C.-Y. Kao, C.-S. Fahn, "A Human-Machine Interaction Technique:Hand Gesture Recognition Based on Hidden Markov Models with
Trajectory of Hand Motion," in Procedia Engineering 15 pp. 3739 – 3743, 2011.
[4] P.S. Neethu,”Real Time Static & Dynamic Hand GestureRecognition,” International Journal of Scientific & Engineering
Research, Vol. 4, Issue3, March 2013
[5] A. Banerjee, A. Ghosh, K. Bharadwaj, “Mouse Control using a Web Camera based on Color Detection,”IJCTT,vol.9, March
2014.
[6] Shriram, NagarajJ. Jaya , 3 S. Shankar , 4 and P. Ajay 5., ‘Deep Learning-Based Real-Time AI Virtual Mouse System Using
Computer Vision to Avoid COVID-19 Spread’, Hindawi Journal of Healthcare Engineering Volume 2021, Article ID
8133076, 8 pages https://doi.org/10.1155/2021/8133076
[7] Saniya Khan, Dnyaneshwari Bagekar, Abhishek Chauhan, Prof.Devashree Wankhede, ‘Virtual Mouse Using Hand Gesture
‘, www.ijcrt.org, 5 May 2021.
[8] Binitha Chirakattu ‘Virtual Mouse using Hand Gesture and Color Detection’, www.ijcaonline.org,October 2015
[9] P. Kumari Kabid Hassan Shibly, Md. Aminul Islam, Samrat Kumar, ‘Design and Development of Hand Gesture Based Virtual
Mouse’, https://www.researchgate.net/publication/336304431, October 2019.
[10] Riza Sande, Neha Marathe, Neha Bhegade, Akanksha Lugade, Prof. S. S. Jogdand, ‘Virtual Mouse Using Hand
Gesture’,.www.ijarset.com, April 2021
[11] Nalini Jungare ‘A Review on Virtual Mouse Using Hand Gesture and Color Detection.’, 13 August, 2017.

[12] Zaman Khan and 2Noor Adnan Ibraheem, ‘HAND GESTURE RECOGNITION: A LITERATURE REVIEW’, International
Journal of Artificial Intelligence & Applications (IJAIA), July 2012.

8
www.viva-technology.org/New/IJRI