Volume 8, Issue 11, November 2023 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Virtual Mouse Using AI and Computer Vision

1
Rupali Shinganjude, 2Dipak Majrikar, 3Atharva Bawiskar, 4Sayali Bobhate, 5Jatin Chaurawar, 6Gaurav Ninawe
1
Assistant Professor
1
Department of Information Technology,
1
Priyadarshini Bhagwati College of Engineering, Nagpur, India.
2
Research Scholar, 3Research Scholar, 4Research Scholar, 5Research Scholar, 6Research Scholar

Abstract:- A paper proposing a virtual mouse system using gestures and movements, eliminating the need for
controlled by hand gestures that use AI algorithms to physical input devices.
recognize and translate them into mouse movements. In the
human-computer interaction module, hand gestures played This research paper is focused on various aspects of a
a significant role. The system is designed to provide an Virtual Mouse system, spanning the underlying technologies,
alternative interface for people who face difficulty using a the development process, and its practical applications. We
traditional mouse or keyboard. This virtual mouse system aimed to shed light on the possibilities and challenges
uses a camera to capture images of the user’s hand, which associated with this innovative technology, as well as its
are processed by an AI algorithm to recognize the gestures potential to redefine the way we interact with digital interfaces
made by the user. Once the gesture is recognized, it is in diverse domains, from gaming and healthcare to education
translated to a corresponding mouse movement, which is and industry.
then executed on the virtual screen. This model provides
potential applications like enabling the hand-free operation The foundation of the Virtual Mouse system lies in the
of devices in hazardous environments and providing an fusion of AI and Computer Vision. AI algorithms enable the
alternative interface for hardware mouse. This hand system to understand and interpret human gestures and
gesture-controlled virtual mouse system offers a promising movements, while Computer Vision technology provides the
approach to enhance user experience and improve means to capture and process visual data. By dissecting these
accessibility through human-computer interaction. underlying technologies, we aimed to provide a comprehensive
understanding of the principles that underpin this innovation.
Keywords:- Virtual Mouse, Hand Gesture Recognition, Creating a Virtual Mouse system involves a meticulous process.
Computer Vision, Media-Pipe. It encompasses the development of software algorithms for
gesture recognition, hardware components such as cameras or
I. INTRODUCTION sensors for data acquisition, and the integration of these
elements to ensure a seamless user experience. We understood
In an era characterized by the relentless advancement of the technical aspects of building a Virtual Mouse prototype, and
technology, the way we interact with computers and digital offered insights into potential challenges and solutions, from
devices has undergone a significant transformation. The calibration to real-time tracking.
traditional computer mouse, a resolute way of human-computer
interaction for decades, is now being challenged by innovative II. LITERATURE SURVEY
solutions that harness the power of Artificial Intelligence (AI)
and Computer Vision. This paper delves into a groundbreaking AI and Gesture Recognition have been fundamental to the
frontier of HCI (Human-Computer Interaction) - the development of Virtual Mouse systems. Gesture recognition is
development and application of a "Virtual Mouse" empowered also important for developing alternative human-computer
by AI and Computer Vision. interaction models. It enables humans to interface with
machines in a more natural way [1]. Furthermore, with
The computer mouse, a ubiquitous peripheral device, has additional voice assistant support, an AI virtual mouse using
long been the interface of choice for navigating the digital hand gestures can further enhance the experience of the user.
realm. However, it is limited in its capacity to provide a The voice assistant that is integrated with the virtual mouse
seamless and natural interaction between humans and system will provide users with even more control over their
machines. The emergence of AI and Computer Vision devices [3]. These works establish a foundational understanding
technologies has unlocked the potential for more intuitive, of the AI underpinning the Virtual Mouse system.
efficient, and versatile ways of interfacing with computers. The
concept of a Virtual Mouse represents a paradigm shift in HCI,
promising a future where users can interact with their devices

IJISRT23NOV265 www.ijisrt.com 327

Volume 8, Issue 11, November 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
The cornerstone of the Virtual Mouse technology is within Mediapipe detects and continuously track the user's
Computer Vision, particularly its ability to enable real-time hand throughout interactions.
tracking of hand movements. Computer vision concerns with  The module utilizes advanced techniques, including
the theories for building intelligent systems that could obtain Convolutional Neural Networks (CNNs) and graph-based
information from images. The image data can be taken in many models, to precisely locate landmarks on the hand. These
forms such as video sequences, views from multiple cameras, landmarks allow us to monitor the hand's position and
or multi-dimensional data from specific scanners [2]. This is a orientation accurately.
critical component in ensuring seamless user interactions and  Additionally, Mediapipe is pivotal in finger tracking,
control over the digital environment. estimating the positions and orientations of individual
fingers, which is vital for recognizing intricate hand
For humans, it is very easy to detect and recognize gestures and actions.
objects as humans have a great capability to distinguish objects
through their vision. But, for a machine object detection and C. Gesture Recognition using Python Libraries:
recognition is a great issue. To overcome this problem ‘Neural
networks’ have been introduced in the field of computer  Python libraries develop custom gesture recognition
science [4]. In the human-computer interaction module, hand algorithms. These algorithms utilize image analysis
gesture has an important role. Obtaining accurate values in techniques, including feature extraction and pattern
finger-pointing with 3-D space value is a difficult process [5]. recognition.
These practical applications illustrate the versatile and  By developing custom algorithms, we ensure that the
potentially transformative nature of the Virtual Mouse. system accurately classifies a wide range of hand gestures.
Convolutional layers of neural networks take a significant part Python libraries such as NumPy and SciPy support various
of computer resources even if the target object (hand) is absent numerical and statistical operations, enabling us to fine-tune
in the frame [6]. Furthermore, they delineate potential future the recognition model.
directions, underlining the ongoing evolution of this  "PyAutoGUI" plays a pivotal role in the system's response
technology and its growing impact in the realm of HCI. to recognize gestures, enabling the translation of gestures
into on-screen actions with precision.
III. PROPOSED METHODOLOGY

The proposed methodology outlines the steps involved in

implementing the Virtual Mouse system and its integration
with "Mediapipe," "OpenCV," and other Python libraries. This
methodology emphasizes the integration of these tools into the
project for hand tracking, gesture recognition, and real-time
image processing.

A. Data Collection and Preprocessing:

 The project commences with the creation of a diverse
dataset, crucial for training the Virtual Mouse system. This
dataset encompasses various hand movements and gestures
to ensure the system's versatility.
 OpenCV facilitates data collection through functions like
Video Capture, allowing us to capture high-quality image
frames from cameras. These frames serve as the foundation
for training and testing the system.
 Data preprocessing includes noise reduction and image
enhancement. OpenCV's image processing capabilities,
such as Gaussian Blur and Histogram Equalization, are
applied to ensure the collected data is clean and of high
quality.
 "PyAutoGUI" streamlines the data labeling process,
significantly expediting the annotation of the dataset, a
critical step in supervised machine learning.

B. Hand Tracking and Pose Estimation using Mediapipe:

 Mediapipe plays a central role in enabling real-time hand
tracking and pose estimation. The hand tracking module Fig. 1. Flow graph for Hand Gesture Recognition

IJISRT23NOV265 www.ijisrt.com 328

Volume 8, Issue 11, November 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
IV. ALGORITHMS AND TOOLS USED  Data Collection and Preprocessing:
 OpenCV is a foundational tool for data collection and
For the purpose of hand and finger detection, we used one preprocessing. It facilitates the capture of image frames
of the effective open-source libraries Mediapipe, it is one type from cameras, which is essential for building the dataset
of framework based on the cross-platform features which was used to train the Virtual Mouse system. The library provides
developed by Google and OpenCV to perform some Computer a straightforward and efficient way to gather visual data,
Vision related tasks. This algorithm uses machine learning ensuring that the training dataset is diverse and
related concepts for detecting hand gestures and tracking their representative of real-world scenarios.
movements.  Additionally, OpenCV offers a suite of functions for image
preprocessing, such as noise reduction, contrast adjustment,
A. Mediapipe and image enhancement. These preprocessing steps are
"Mediapipe" is a comprehensive open-source framework crucial to ensure the collected data is of high quality, with
developed by Google that specializes in various aspects of minimal noise or interference.
computer vision, including real-time hand tracking and pose
estimation. In the context of our project, "Mediapipe" played a  Hand Detection:
central role in achieving precise hand tracking, a critical  OpenCV's Haar Cascade Classifier is used for the initial
component of the Virtual Mouse system's functionality. hand detection phase. This classifier is an effective machine
learning object detection method, and it playes a pivotal role
 Hand Tracking Module: in identifying the presence of hands in image frames. By
 "Mediapipe" offers a specialized hand tracking module recognizing the hand's features, the Haar Cascade Classifier
capable of detecting and tracking hands in real-time from allows the system to extract the region of interest (ROI),
input images or video frames. This module leverages namely the hand, for further analysis.
machine learning models trained on vast datasets to  The Haar Cascade Classifier is an important building block
recognize and locate hands accurately. As a result, it for the Virtual Mouse system's data collection process. It
provides the system with the capability to continuously serves as the foundation for subsequent steps in hand
track the user's hand throughout the interaction. tracking and gesture recognition.
 "Mediapipe" hand tracking goes beyond merely detecting
the presence of a hand. It identifies key landmarks and key  Real-time Image Processing:
points on the hand, enabling the system to precisely map  OpenCV excels in real-time image processing, making it a
and follow hand movements. This is particularly valuable valuable tool for managing camera input and processing
for tracking gestures, as the system can monitor the relative video frames in real-time. This capability is vital for the
positions of fingers and the palm, ensuring a detailed Virtual Mouse system, as it requires seamless and
understanding of hand poses. responsive interactions.
 The library's real-time image processing functions ensure
 Finger Tracking and Pose Estimation: that the system could keep up with hand movements and
 One of the strengths of "Mediapipe" is its ability to perform gestures as they occur. By providing a robust foundation for
finger tracking and hand pose estimation. The framework image analysis and manipulation, OpenCV supports the
can accurately determine the positions and orientations of system in recognizing gestures and translating them into on-
individual fingers, allowing the Virtual Mouse system to screen actions with minimal latency.
recognize complex hand gestures and gestures involving
specific finger arrangements. V. RESULTS
 The framework's pose estimation capabilities are
particularly beneficial for understanding the orientation of In our project, the virtual mouse system exhibited a
the hand in 3D space. This information is vital for the moderate level of accuracy and precision in tracking hand
system to interpret hand movements accurately and gestures and translating them into cursor movements. While the
translate them into corresponding actions on the computer system performed adequately, there is potential for further
interface. refinement to enhance its precision. The response time of the
virtual mouse was found to be satisfactory, providing users with
B. OpenCV a responsive experience. However, there is room for
OpenCV, or Open Source Computer Vision Library, is a optimization to reduce response times and enhance real-time
powerful open-source framework that is widely employed in performance.
computer vision and image processing tasks. In our project,
OpenCV played a central role in various critical aspects of the Additionally, the gesture recognition rate was reasonable,
Virtual Mouse system's development. accurately identifying a majority of hand gestures used for
mouse control. Nonetheless, there is scope for improvement,
especially in cases involving complex or rapid gestures. These
results provide valuable insights into the performance of the

IJISRT23NOV265 www.ijisrt.com 329

Volume 8, Issue 11, November 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
virtual mouse system, acknowledging its strengths and [5]. Preetha V H and Sherin Mohammed Sali Shajideen,
highlighting areas for future enhancements. “Hand Gestures - Virtual Mouse for Human Computer
Interaction,” Proceedings of the International Conference
VI. CONCLUSION on Smart Systems and Inventive Technology (ICSSIT
2018) : Francis Xavier Engineering College, Tirunelveli,
We've seen how AI and computer vision can work India, date: December 13-14, 2018, 2018.
together to create a digital tool that's easy to use. This [6]. R. Golovanov, D. Vorotnev, and D. Kalina, “Combining
combination not only brings the concept of the Virtual Mouse Hand Detection and Gesture Recognition Algorithms for
to life but also opens the door to more exciting innovations that Minimizing
can make how we use computers even better. [7]. Computational Cost,” in 2020 22th International
Conference on Digital Signal Processing and its
This research has also shown us that creating a Virtual Applications, DSPA 2020, Institute of Electrical and
Mouse is a complex task. It involves writing smart computer Electronics Engineers Inc., Mar. 2020. doi:
programs, using specialized hardware, and making sure 10.1109/DSPA48919.2020.9213273.
everything happens in real time. Solving these challenges is [8]. N. Khade, M. S. Aswale, M. N. Kushwaha, M. Y. Bisen,
essential for turning the Virtual Mouse from an idea into a real M. R. Chaudhari, and M. T. Ramteke, “AI VIRTUAL
and common technology. MOUSE,” 2023.
[9]. A. Voulodimos, N. Doulamis, A. Doulamis, and E.
The Virtual Mouse has some really cool uses. It can help Protopapadakis, “Deep Learning for Computer Vision: A
doctors and nurses use computers without touching them, Brief Review,” Computational Intelligence and
which is important for keeping things clean and safe in Neuroscience, vol. 2018. Hindawi Limited, 2018.
healthcare. It can also make learning more fun and interactive [10]. W. Ouyang et al., “DeepID-Net: Object Detection with
in schools, and it can add more excitement to video games. Deformable Part Based Convolutional Neural Networks,”
Additionally, it can give people with physical challenges a way IEEE Trans Pattern Anal Mach Intell, vol. 39, no. 7, pp.
to use computers more easily, which is a big deal for 1320–1334, Jul. 2017.
inclusivity. [11]. X. Li and Y. Shi, “Computer vision imaging based on
artificial intelligence,” in Proceedings - 2018 International
Looking ahead, the Virtual Mouse is like the first step Conference on Virtual Reality and Intelligent Systems,
toward a new way of working with computers. It won't replace ICVRIS 2018, Institute of Electrical and Electronics
our regular computer mice, but it will give us another way to Engineers Inc., Nov. 2018.
get things done. This research has shown us the potential and [12]. E. Sankar, N. Bharadwaj, and A. Vignesh, “Virtual Mouse
challenges of this new technology. It's clear that the Virtual Using Hand Gesture,” International Journal of Scientific
Mouse isn't just a fancy idea; it's a real step towards a future Research in Engineering and Management, vol. 07, no. 05,
where we can work with computers in a smarter and more May 2023, doi: 10.55041/IJSREM21501.
natural way. [13]. D. Singh, A. Singh, A. K. Singh, S. Chaudhary, and J.
REFERENCES Kher, “Virtual Mouse using OpenCV,” Int J Res Appl Sci
Eng Technol, vol. 9, no. 12, pp. 1055–1058, Dec. 2021,
[1]. Shweta Yewale and Pankaj Bharne, “Hand Gesture doi: 10.22214/ijraset.2021.38160.
Recognition Using Different Algorithms Based on [14]. C. D. S. Nikhi, C. U. S. Rao, E. Brumancia, K. Indira, T.
Artificial Neural Network,” 2011 International Anandhi, and P. Ajitha, “Finger Recognition and Gesture
Conference on Emerging Trends in Networks and based Virtual Keyboard,” International Conference on
Computer Communications., 2011. Communication and Electronics Systems, 2020.
[2]. Kavitha R, Janasruthi U, Lokitha S, and Tharani G, [15]. F. Jabeen, L. Tao, and T. Linlin, “One Bit Mouse for
“Hand Gesture Controlled Virtual Mouse Using Artificial Virtual Reality,” in Proceedings - 2016 International
Intelligence,” 2023. Conference on Virtual Reality and Visualization, ICVRV
[3]. Chen-Chiung Hsieh, Dung-Hua Liou, and David Lee, “A 2016, Institute of Electrical and Electronics Engineers
Real Time Hand Gesture Recognition System Using Inc., Jun. 2017, pp. 442–446. doi:
Motion History Image,” 2010 2nd International 10.1109/ICVRV.2016.81.
Conference on Signal Processing Systems., 2010. [16]. D. S. Tran, N. H. Ho, H. J. Yang, S. H. Kim, and G. S.
[4]. Aishwarya Sarkale, Kaiwant shah, Anandji Chaudhary, Lee, “Real-time virtual mouse system using RGB-D
and Tatwadarshi Nagarhalli, “An Innovative Machine images and fingertip detection,” Multimed Tools Appl,
Learning Approach for Object Detection and vol. 80, no. 7, pp. 10473–10490, Mar. 2021, doi:
Recognition,” Proceedings of the International 10.1007/s11042-020-10156-5.
Conference on Inventive Communication and [17]. L. W. Heng, D. Chunjian, and L. Yi, “Implementation Of
Computational Technologies : ICICCT 2018 : 20-21, Virtual Mouse Based On Machine Vision,” International
April 2018, 2018. Conference on Apperceiving Computing and Intelligence
Analysis., 2010.

IJISRT23NOV265 www.ijisrt.com 330