VIRTUAL MOUSE

By

Nusrat Razzaq

Reg No: 2018-UMDB-004843

Arooj Riaz

Reg No: 2018-UMDB-004847

Session 2018-2022

Supervisor: Dr. Wajid Arshad Abbasi

DEPARTMENT OF COMPUTER SCIENCES &

INFORMATION TECHNOLOGY

FACULTY OF SCIENCES
UNIVERSITY AZAD JAMMU & KASHMIR
MUZAFFARABAD

i
 APPROVAL CERTIFICATE

It is certified that the project work presented in this report entitled “VIRTUAL
MOUSE” submitted by Nusrat Razzaq (Roll No. 16), Reg No. 2018-UMDB-004843
And Arooj Riaz (Roll No. 20), Reg No. 2018-UMDB-004847 of Session (2018-22)
supervised by Dr. Wajid Arshad Abbasi in our opinion is fully adequate in scope and
quality of Bachelors in computer sciences (BSCS).

(Supervisor) (External Examiner)

Dr. Wajid Arshad Abbasi
Assistant Professor
Dr.Muhammad Naveed Akhtar
Department of CS & IT, Assistant Professor
University of Azad Jammu & Kashmir Principal Scientist
Muzaffarabad PIEAS

(Chairman)
Dr. Syed Ali Abbas
Department of CS & IT
University of Azad Jammu & Kashmir
Muzaffarabad

ii
 ACKNOWELEDGEMENT

Allah alone deserves praise and thanks for the perfection of His favor upon us. May He

blesses and salutes His slave and Muhammad (SAW), and the teacher mankind ever had,

who led us out of the darkness of infidelity and hypocrisy.

We would also like to express our gratitude to our dignitary supervisor Dr. Wajid Arshad

Abbasi, for his appreciation and dedication to our project. Our level coordinator, Dr. syed

Ali Abbas, deserves special thanks for her cordial support, valuable suggestions, and

guidance throughout this process.

In addition, we are very grateful to our parents for their unconditional support. Hopefully,

Allah will spare their lives as they obey His commandments.

iii
 DECLARATION

We hereby declare that neither system we design nor any part of it is copied from any

other source. In addition, we are announcing that this project was completed entirely as a

result of our immense efforts and under the supervision of our supervisor. This thesis has

not been submitted to another institute or university for the award or degree or diploma

based on the findings in it. Our report has been written under the university’s guidelines.

In the text of thesis, whenever we have incorporated materials (data, theoretical analysis,

and text) from any other source, we have given them due credit and given their details in

the references.

Nusrat Razzaq

Reg No: 2018-UMDB-004843

Arooj Riaz

Reg No: 2018-UMDB-004847

iv
 ABSTRACT

The mouse is the wondrous development of people. In this time, remote mouse or a

contact less mouse actually utilizes gadgets and is not liberated from gadget or might be

from outside power sources like battery and gain space and electric power, likewise

during COVID pandemic it is encouraged to make social separating and keep away from

to contact this which gave by various people groups. In the proposed AI virtual mouse

system, this limitation can be overcome by employing webcam or a built-in camera for

capturing of hand gestures and hand tip detection using computer vision. The algorithm

used in the system makes use of the machine learning algorithm. Based on the hand

gestures, the computer can be controlled virtually and can perform left click, right click,

scrolling functions, and computer cursor function without the use of the physical mouse.

The algorithm is based on deep learning for detecting the hands. Hence, the proposed

system will avoid COVID-19 spread by eliminating the human intervention and

dependency of devices to control the computer.

KEYWORDS: Virtual Mouse (VM), OpenCV, AI.

v
 TABLE OF CONTENTS

Acknowledgement..........................................................................................iii
Declaration.....................................................................................................VI
Abstract............................................................................................................v
Chapter I: Introduction...........................................................................1
1.1 : problem statement.................................................................1
1.2 : Objective of project..............................................................2
1.3 : Limitations of project...........................................................2
1.4 : Future scope of project.........................................................2

Chapter 2: Background overview............................................................3

2.1 : Computer Vision.......................................................................3
2.1.1 : Camera system Integration.....................................................5
2.1.2 : Histogram of oriented gradient (HOG)..................................6
2.2 : Object Detection.......................................................................7
2.3 : Machine Learning and Deep Learning.....................................7
2.4 : Deep learning base Objects......................................................7
2.4.1 : Image Acquisition...................................................................8
2.5 : Object Localization...................................................................9
2.6 : Algorithm for Object Detection..............................................13
2.7 : Single Shot Detector..............................................................13
2.8 : YOLO....................................................................................14

Chapter 3: Tools and Implementation..................................................17

3: Algorithm Used for Hand tracking........................................17
3.1 : Media Pipe.............................................................................17
3.2 : Open CV................................................................................21
3.3 : NumPy...................................................................................22
3.4 : Flask.......................................................................................22

vi
Chapter 4: Project Working....................................................................26
4 Introduction............................................................................26
4.1 : Camera Used in AI virtual mouse system.............................28
4.2 : Virtual Screen Matching........................................................29
4.3 : Detecting finger UP...............................................................30
4.4 : Stop Cursor............................................................................31
4.5 : Double click...........................................................................32
4.6 : Right button click..................................................................33
4.7 : Scroll UP function.................................................................34
4.8 : Scroll Down function.............................................................35

Chapter 5: Accuracy and Results...........................................................36

5.1 : Accuracy and Future Scope.....................................................39
5.2 : Applications.............................................................................39

Chapter 6: conclusions.............................................................................42
6.1: conclusions.............................................................................42

REFERENCES.............................................................................................44

ABBREVIATIONS......................................................................................48

vii
 LIST OF FIGURES

Number Page

Fig 2.1 Computer Vision.........................................................................04

Fig 2.5 Object Localization.....................................................................09

Fig 2.5(a) Calculation of mean IOU........................................................10

Fig 2.5(b) IOU measurement...................................................................11

Fig 2.5(c) Object detection flow chart.....................................................12

Fig 2.8 Residual Block...........................................................................15

Fig 2.8(a) Bounding box regression...........................................................16

Fig 3.1 Media Pipe working...................................................................19

Fig 3.1(a) Media Pipe hand recognition.....................................................20

Fig 3.1(c) Flow chart of media pipe...........................................................21

Fig 3.2 Open CV pointing objects...........................................................22

Fig 3.5 Graph Accuracy...........................................................................25

Fig 4.1 Camera used in AI in VM...........................................................28

Fig 4.2 cursor moving through window...................................................29

Fig 4.3 Detecting finger UP......................................................................30

Fig 4.4 Cursor stop moving......................................................................31

Fig 4.5 Left button click...........................................................................32

viii
Fig 4.6 Right button click.........................................................................33

Fig 4.7 Scroll UP function of cursor..........................................................34

Fig 4.8 Scroll DOWN function of cursor..................................................35

Fig 5.1 Accuracy of VM project................................................................36

Fig 5.1(a) Result of full project.................................................................38

ix
 CHAPTER 1

INTRODUCTION

The most efficient and expressive way of human communication is through hand

gesture, which is a universally accepted language. In this project, real-time hand

gesture system is proposed. The setup of this system uses a web camera that detects

fingertips of our hand by using computer vision, to perform the mouse cursor

operations like right click, left click, scrolling and double clicking.

We have used python programming language for developing this virtual mouse also

OpenCV which is the library for computer vision is used. The models makes use of

Media Pipe package for the tracking of hands and for tracking of tip of hands.

The proposed model can work very well in real-world application.

1.1 Problem statement

This AI virtual mouse can be made use in places where using a physical mouse

is not feasible such as moist or wet conditions, to aid people who cannot handle a

device and where there is not a space for using a mouse and to eliminate the need of

mouse and cable altogether. In view of the pandemic, there arises a need to avoid

high-contact surfaces out of which the mouse is one. In order to eliminate the need

to avoid contact, the system provides an initiative way to interact with the computer

system with the use of hand gestures and finger tips to emulate mouse-like functions

such as scrolling, clicking, and pointing.

1
 1.2 Objective of project
Our aim is to make the mouse model that can be controlled by finger

gestures. We can make the mouse in which we can move the mouse cursor left

click, right click, scroll and pointing the object through hand gestures.

1.3 Limitations of project

o Detection of the hand becomes difficult when background

color matches our hand color.

o Not accurate as physical mouse.

1.4 Future Scope of project

Furthermore, this model can also be enhanced in which we can develop to

handle the keyboard functionalities along with the mouse functionalities virtually

which is another future scope of Artificial Intelligence of Human Interaction.

2
 CHAPTER 2

BACKGROUND OVERVIEW

2.1 Computer Vision (CV)

It is considered as the field of AI in which we can trains computers to

interpret and understand visual world. Computer vision is the process of teaching

computers to process the image at the pixel level and then comprehend them. In this

process machine retrieves visual information, handles it, and interprets the results

using special algorithms. In the same way that AI makes computer think, computer

vision gives them the ability to see, observe, and understand as shown in figure 2.1.

Basically it automates the task that human visual can do.

Human vision and computers vision work similarly, except that human have an

advantage in developing their vision. In computer vision, machines are taught to

perform these functions. However, they accomplish this much less time using

cameras, data and algorithms as opposed to retains, optic nerves and the visual cortex.

It is possible for the system that inspects products, or watches a production asset, to

identify overlooked issues or defects that are imperceptible to humans. Computer

vision requires a lot of data. Data is analyzed repeatedly until distinctions are

discerned. Deep learning and convolutional neural networks are two of the key

technologies used to accomplish this. An algorithmic models used in machine

learning to enable computers to learn from visual data. The computers will learn to

3
recognize one image from another if enough data is fed through the model. In contrast

to having someone program it to recognize an image, algorithm allow the machine to

learn on its own [1].

This picture shows the computer vision working and give detail that how objects are

detected in computer vision.

Fig 2.1: Computer Vision

4
 2.1.1 Camera System Interaction

Digital Camera System Controlled from an LCD Touch Panel presents the

design and implementation based on Digital Camera System for image capturing and

real-time image processing. Images captured with a web camera are initially stored

in the system’s memory and then they are displayed on an LCD Touch panel. The

main goal of this architecture is to be used as a platform to implement and test

advanced image processing algorithms. Apart of this, the system supports the control

of the image sensor, through the LCD Touch Panel. In addition, it has the ability to

communicate with a PC through an interface for storing the images on it. The Digital

Camera System is used due to the flexibility, mainly targeted as it used as an open

and low cost platform for implementing and testing real-time image processing

algorithms. In addition the exploitation of LCD touch Panel can effectively assist in

the control of more cameras’ parameters. Image processing algorithms can take place

before or after the data storing as system has the ability to be easily modified. Future

plans are to embed and test more advance image processing algorithms. In addition,

we intend to create an extended menu for the LCD touch panel, developing such a

menu the user can use in a friendly manner controlling cameras functionality.

Through this menu the user can also easily select the execution of the desirable image

processing algorithm [5].

5
2.1.2 Histogram of Oriented Gradients (HOG)

Histogram of oriented gradients (HOG) is a feature descriptor used to detect

objects in computer vision and image processing. The HOG descriptor technique

counts occurrences of gradient orientation in localized portions of an image -

detection window, or region of interest (ROI).

Implementation of the HOG descriptor algorithm is as follows:

1. Divide the image into small connected regions called cells, and for each cell

compute a histogram of gradient directions or edge orientations for the pixels

within the cell.

2. Discretize each cell into angular bins according to the gradient orientation.

3. Each cell's pixel contributes weighted gradient to its corresponding

angular bin.

4. Groups of adjacent cells are considered as spatial regions called blocks. The

grouping of cells into a block is the basis for grouping and normalization of

histograms.

5. Normalized group of histograms represents the block histogram. The set

of these block histograms represents the descriptor.

The following figure demonstrates the algorithm implementation scheme:

Computation of the HOG descriptor requires the following basic configuration

parameters:

1. Masks to compute derivatives and gradients

2. Geometry of splitting an image into cells and grouping cells into a block

6
 3. Block overlapping

4. Normalization parameters

2.2 Object Detection

Object detection is the process in which we can detect the object like

human, building, cars etc.it is the technology of deep learning.

Object detection recognizes the object with bounding box in the image, where in

image classification, we can simply categorize or classify that is an object in the

image or not in terms of probability[6].

2.3 Machine learning and Deep learning base Object detection

In machine learning the objects can be detected by extracting the features of

the image. These techniques are (HOG) Histogram of oriented gradients, color

histogram etc. [13].

2.4 Deep learning base Objects

It has the ability to extract the feature automatically with the help of deep

learning algorithms such as, CNN (convolution neural networks), Auto Encoder,

variance the feature from the image like edge, shape etc.

o We can face the problem of object detection that we have assumed

that multiple classes of objects may exist in an image at same time

[13].
7
 o We can also visualize this like two types of problem one is multi label

classification.

o Bounding box in which we have to predict the coordinates value of

the bounding box in terms of x, y, w, h.

2.4.1 Image Acquisition

Image acquisition is the first step in this system. Images are captured by an off-the

shelf HD webcam with a certain resolution. This webcam is interfaced with the

interface to detect the hand coordinates. Fourteen hand gesture coordinate points are

used in it which are obtained from one hand, and to show the gesture in front of the

camera to capture the image. Then, the coordinate points in the image are resized for

the convenience.

8
2.5 Object Localization
Object localization is the process in which we can predict the object in an

image as well as its boundaries as shown in figure 2.5.

Fig 2.5: Object Localization

Picture in the above figure shows the object and its boundaries in the box. Which is

called object localization [7].

A bounding box can be initialized using the parameters as shown in figure 2.5(a)

which are;

 bx , by :

Coordinates of the center of the bounding box.

 bw :

9
 Width of bounding box with respect the image width.

 bh :

Height of the bounding box with respect to image height.

Fig 2.5(a): calculation of IOU

We predict this for calculating mean IOU and predicting box which localize

an image.

IOU is area of overlap over area of union as the measurement given in figure

2.5(b).

1
 Fig 2.5(b): IOU Measurement
This figure above gives the detail about IOU or how it measure.

1
Object detection flow chart:

Fig 2.5(c): object detection flow chart

1
2.6 Algorithms for Object detection
There are many different algorithm that we use for the object detection. Some

of them are;

1. Fast R-CNN

2. Faster R-CNN

3. Histogram of Oriented Gradients (HOG)

4. Single shot detector

5. YOLO (you only look once)

2.7 Single Shot Detector (SSD)

The SSD network for image classification has a VGG-16 base layer followed

by several convolution layer. The CNN architecture for SDD is standard for high-

quality image classification, but it lacks the final classification layer. Features are

extracted using VGG-16. OUR VGG network is further enhanced by adding

convolution layer decreases. By decreasing the number of convolution layers, the

feature map size decreases and the depth decreases. They construct more abstract

more abstract representation and cover larger receptive fields. Detecting larger

objects is therefore easier [8].

1
2.8 YOLO
YOLO is an algorithm that utilize neural networks for the detection of real-

time objects. Due to its speed and accuracy, this algorithm is popular. In addition to

detecting traffic signals, parking meters and animals, it has also been used in

detecting people and animals.

The term YOLO means ‘you only look once’. Real time detection and recognition of

various objects within a picture is done with the help of this algorithm. In YOLO,

objects are detected as regression problems, and the class probabilities are computed.

Convolutional neural networks are the technology behind the YOLO algorithm [8].

1
 Residual Blocks

This picture shows that there are variety of grids. S x S grid is shown in it.
The residual block in computer vision shows that how the objects are pointed
and displays.
Fig 2.8: Residual Block
As in the figure 2.8 shows that when an object is detected firstly the residual
box are formed to separate the objects [12].

1
 Bounding box regression

This picture shows that the objects are in the boxes [12].
In the bounding box regression the object we want to point can be
differentiate from other objects by box around the objects.

Fig 2.8(a): Bounding box regression

1
 CHAPTER 3

TOOLS AND IMPLEMENTATIONS

Virtual mouse is the project that can be performed by using different software tools.

In this project we use different modules and libraries for performing the task and for

hand gestures recognitions. The tools and libraries that we use includes Open CV,

numpy, media pipe, flask and the software tool that we used is Anaconda.

3. Algorithm used for Hand Tracking

For the detection of hand gestures and hand tracking projects, the MediaPipe

framework is used, and OpenCV library is used for computer vision. The algorithm

makes use of the machine learning concepts to track and recognize the hand gestures

and hand tip. numPY and flask is also used in this process for making the virtual

mouse [9].

3.1. MediaPipe

MediaPipe is a framework which is used for applying in a machine learning pipeline,

and it is an opensource framework. The MediaPipe framework is useful for cross

platform development since the framework is built using the time series data. The

MediaPipe framework is multimodal, where this framework can be applied to various

audios and videos. The MediaPipe framework is used by the developer for building

1
and analyzing the systems through graphs, and it also been used for developing the

systems for the application purpose as shown in figure 3.1. The steps involved in the

system that uses MediaPipe are carried out in the pipeline configuration. The pipeline

created can run in various platforms allowing scalability in mobile and desktops. The

MediaPipe framework is based on three fundamental parts,they are performance

 evaluation

 framework for retrieving sensor data

 a collection of components which are called calculators,

They are reusable. A pipeline is a graph which consists of components called

calculators, where each calculator is connected by streams in which the packets of

data flow through. Developers are able to replace or define custom calculators

anywhere in the graph creating their own application. The calculators and streams

combined create a data-flow diagram is created with MediaPipe where each node is a

calculator and the nodes are connected by streams.

1
This graph shows the Media Pipe hand recognition.

Fig 3.1: media pipe working

1
Now Single-shot detector model is used for detecting and recognizing a hand or palm

in real time. The single-shot detector model is used by the MediaPipe as shown in

figure 3.1(a). First, in the hand detection module, it is first trained for a palm

detection model because it is easier to train palms [10]. Fig 3.1(a)

2
These coordinates are same in every algorithm for hand gestures recognitions are as
shown in figure below.

Flow chart showing the Media Pipe working.

Fig 3.1(b): flow chart of media pipe

3.2 OpenCV

Open CV is a computer vision library in which there is the image processing

algorithms for the detection an object. Open CV is the library of python language. It

is real time application. All the real time applications are formed by using this as

shown in figure 3.2. The Open CV library is used in image and video processing and

also analysis such as face detection and object detection [10].

2
 Fig 3.2: Open CV pointing objects

In this figure we see that the people are counting with Open CV and Python.

3.3 NumPy

It is basically called Numerical Python which is a library for data

manipulation and scientific computing.

3.4 Flask

Flask is a framework which is used for development of Web

applications using Python.

2
3.5 Anaconda Software

We can use anaconda software for making this project called Virtual Mouse.

How we can install that software:

Anaconda is a package manager, an environment manager, and Python distribution

that contains a collection of many open source packages. This is advantageous as

when you are working on a data science project, we find that we need many different

packages (numpy, scikit-learn), which an installation of Anaconda comes preinstalled

with. If we need additional packages after installing Anaconda, we can use

Anaconda's package manager, conda, or pip to install those packages. This is highly

advantageous as we don't have to manage dependencies between multiple packages..

Steps of installing the software are;

1. First step is to go to the Web site on chrome downloading an anaconda

choose the version of the software and press the downloading button.

2. Second step is to open the exe. File in to your download files.

3. When the screen demands next then click on the next button.

4. Read the license agreement and click on I Agree.

5. Click on next.

6. Note your installation location and press next.

2
7. The important part of the installation process is that when it recommends the

approach is to not check the box to add Anaconda to your path. This means

you will have to use Anaconda Navigator or the Anaconda Command Prompt

(located in the Start Menu under "Anaconda") when you wish to use

Anaconda (you can always add Anaconda to your PATH later if you don't

check the box). If you want to be able to use Anaconda in your command

prompt.

2
 Fig 3.5: Graph accuracy

Real-time virtual mouse system using RGB-D images and fingertip

detection

2
 Chapter 4

PROJECT WORKING

4. Introduction

A virtual mouse is a concept similar to a virtual keyboard, allowing usage of a

computer with a minimal amount of interactive physical peripherals. For the various

methods of implementing a virtual mouse design, this particular project focuses on

moving a computer mouse cursor by implementing an accelerometer-based wireless

glove. This glove, which is powered by a standard battery pack, will be attached with

an accelerometer and microcontroller that has the capability to wireless send data to a

nearby wireless receiver that is interfaced directly with a computer PC. Software will

then be incorporated to collect and analyze the user movement data for precise cursor

readjustment [3].

An example would be a customer who has limited physical mobility in a part of their

arm and wishes to minimize the work effort. An application example would be

changing the interaction experience between a user and CAD software modeling,

where the user can obtain a virtual experience of positioning an object without the use

of a peripheral mouse[4]. The intent of this project serves for various applications or

customers with specific requirements, making this an attractive and potentially

2
marketable product in a wide variety of markets (such as TV, mobile, gaming, health

care, and so forth) in the near future when virtual experience becomes more prevalent.

Our project Virtual Mouse can work by using web camera.

4.1 Camera used in AI virtual mouse system

In this project the mouse is based on the frames that is captured by the Web-

Camera of the laptop or PC. By using the python computer vision library OpenCV the

video capture object is created and the Web Camera start capturing the video of hand

as shown in the figure 4.1. Camera captures the frames and passes it to AI virtual

system [2].

Fig 4.1: Camera used in AI VM

This image shows that hand is being detected using a web camera.

2
Our project work in the following steps.

4.2 (Virtual Screen Matching) Rectangular Region for Moving

through the Window

The AI virtual mouse system makes use of the transformational algorithm, and it

converts the co-ordinates of fingertip from the webcam screen to the computer

window full screen for controlling the mouse [14]. When the hands are detected and

when we find which finger is up for performing the specific mouse function, a box is

drawn around the hand as shown in Figure 4.2.

Fig 4.2: cursor moving through window

2
4.3 Detecting Which Finger Is Up and Performing the Particular
Mouse Function

When the Index finger is UP and all the fingers like Thumb, Middle finger, Ring

finger, and Little finger are down then we say that the mouse cursor is ready to

move left and right on PC window as shown in the figure 4.3.

Fig 4.3: Detecting finger UP

2
4.4 For stop the movement of cursor

When the both index and middle fingers are up and the distance between

them is greater than 40px as shown in the figure than cursor stops moving as shown

in figure 4.4.

Fig 4.4: Cursor stop moving

3
4.5 For the Mouse to Perform Double Click function

When the Index finger and the Middle finger are UP and at the distance is

less than 40px, the double click function is performed as shown in figure 4.5.

Fig 4.5: Double button clicking

3
4.6 For the Mouse to Perform Right Button Click

When the Index finger, Middle finger, and a Little finger are UP and the Ring

finger is down then the cursor of the virtual mouse perform the right button click

function as shown in figure 4.6.

Fig 4.6: Right button click

3
4.7 For the Mouse to Perform Scroll up Function
When the Little finger is UP and the remaining fingers are closed, as shown

in figure, then the cursor will perform scroll up function as shown in figure 4.7.

Fig 4.7: Scroll UP function

3
4.8 For the Mouse to Perform Scroll down Function

When the Index finger and the Little finger is UP and all the remaining

fingers of the hand are down than the cursor of the mouse will perform the scroll

down function as shown in the figure.

Fig 4.8: Scroll down function

3
 CHAPTER 5

ACCURACY & RESULTS

In AI virtual mouse system, the concept of advancing the human-computer interaction

using computer vision is given. Fig 5.1: Accuracy of VM

3
This figure shows the accuracy of our project. According to which our project is 99%

accurate or show 99% accuracy.

Cross comparison of the testing of the AI virtual mouse system is difficult because

only limited numbers of datasets are available. The hand gestures and fingertip

detection have been tested in various illumination conditions and also been tested

with different distances from the webcam for tracking of the hand gesture and hand

tip detection.

AI virtual mouse has performed very well in terms of accuracy when compared to

the other virtual mouse models. The novelty of the proposed model is that it can

perform most of the mouse functions such as left click, right click, scroll up, scroll

down, and mouse cursor movement using fingertip detection, and also, the model is

helpful in controlling the PC like a physical mouse but in the virtual mode.

3
The result shown after checking and experiments are good and accurate.

Fig 5.1(b): Result of full project

3
5.1 Future Scope

The proposed AI virtual mouse has some limitations such as small decrease

in accuracy of the right click mouse function and also the model has some difficulties

in executing clicking and dragging to select the text. These are some of the limitations

of the proposed AI virtual mouse system, and these limitations will be overcome in

our future work.

Furthermore, the proposed method can be developed to handle the keyboard

functionalities along with the mouse functionalities virtually which is another future

scope of Human-Computer Interaction (HCI).

5.2 Applications

The AI virtual mouse system is useful for many applications; it can be

used to reduce the space for using the physical mouse, and it can be used in situations

where we cannot use the physical mouse. The system eliminates the usage of devices,

and it improves the human-computer interaction.

Major applications of AI virtual mouse includes:

3
1) The proposed model has a greater accuracy of 99% which is far greater than

that of other proposed models for virtual mouse, and it has many

applications.

2) In COVID-19 situation, it is not safe to use the devices by touching them

because it may result in a possible situation of spread of the virus by

touching the devices, so the proposed AI virtual mouse can be used to

control the PC mouse functions without using the physical mouse.

3) The system can be used to control robots and automation systems without the

usage of devices.

4) 2D and 3D images can be drawn using the AI virtual system using the hand

gestures.

5) AI virtual mouse can be used to play virtual reality- and augmented reality-

based games without the wireless or wired mouse devices.

6) Persons with problems in their hands can use this system to control the mouse

functions in the computer.

3
7) In the field of robotics, the proposed system like HCI can be used for

controlling robots.

8) In designing and architecture, the proposed system can be used for designing

virtually for prototyping.

4
 CHAPTER 6

CONCLUSIONS

6.1 Conclusions

The main objective of the AI virtual mouse system is to control the mouse

cursor functions by using the hand gestures instead of using a physical mouse. The

proposed system can be achieved by using a webcam or a built-in camera which

detects the hand gestures and hand tip and processes these frames to perform the

particular mouse functions.

From the results of the model, we can come to a conclusion that the proposed AI

virtual mouse system has performed very well and has a greater accuracy compared

to the existing models and also the model overcomes most of the limitations of the

existing systems. Since the proposed model has greater accuracy, the AI virtual

mouse can be used for real-world applications, and also, it can be used to reduce the

spread of COVID-19, since the proposed mouse system can be used virtually using

hand gestures without using the traditional physical mouse.

4
The model has some limitations such as small decrease in accuracy in right click

mouse function and some difficulties in clicking and dragging to select the text.

Hence, we will work next to overcome these limitations by improving the fingertip

detection algorithm to produce more accurate results.

4
 REFERENCES

[1] https://xd.adobe.com/ideas/principles/emerging-technology/what-is-

computer-vision-how-does-it-

work/#:~:text=Computer%20vision%20is%20the%20field,pixel%20level

%20and%20understand%20it.

[2] Abhik Banerjee, Abhirup Ghosh, Koustuvmoni Bharadwaj,” Mouse

Control using a Web Camera based on Color Detection”,IJCTT,vol.9,

Mar 2014.

[3] Angel, Neethu.P.S,”Real Time Static & Dynamic Hand Gesture

Recognition”, International Journal of Scientific & Engineering

Research Volume 4, Issue3, March-2013.

[4] Q. Y. Zhang, F. Chen and X. W. Liu, “Hand Gesture Detection

and Segmentation Based on Difference Background Image with

Complex

4
Background,” Proceedings of the 2008 International Conference on

Embedded Software and Systems, Sichuan, 29-31 July 2008, pp. 338-343

[5] https://www.alliedvision.com/en/ai-starter-guide/camera-technology/

[6] Dasiopoulou, Stamatia, et al. "Knowledge-assisted semantic video

object detection." IEEE Transactions on Circuits and Systems for Video
Technology 15.10 (2005): 1210–1224.

[7] https://www.frontiersin.org/research-topics/28766/object-
localization#:~:text=Object%20localization%20is%20a%20fundamental,
driving%2C%20and%20automatic%20monitoring%20systems.

[8] https://www.geeksforgeeks.org/difference-between-yolo-and-ssd/

[9] https://www.section.io/engineering-education/creating-a-hand-
tracking-module/

[10] https://en.wikipedia.org/wiki/OpenCV

[11] https://pyimagesearch.com/2020/10/05/object-detection-bounding-
box-regression-with-keras-tensorflow-and-deep-learning/

[12] https://towardsdatascience.com/hog-histogram-of-oriented-
gradients-67ecd887675f?gi=3145908a21df

[13] https://www.javatpoint.com/machine-learning-vs-deep-learning

[14] S.Shriram” Deep Learning-Based Real-Time AI Virtual Mouse

System Using Computer Vision to Avoid COVID-19 Spread”, Applied

Sciences, Volume 2021, Article ID 8133076, 31 Jul 2021, View at:

Publisher Site |Hindawi.

4
[15] Akshay L Chandra” Mouse Cursor Control Using Facial

Movements — An HCI Application, View at: Publisher Site| data science

4
 ABBREVIATIONS

CNN: Convolutional Neural Network

VM: Virtual Mouse

YOLO: You Only Look Once

IOU: Intersection over Union

Open CV: Open-Source Computer Vision

SSD: Single Shot Detector