Gesture and Voice controlled

Virtual Mouse
Supervisor: Prof. Girish Chandra

Ms. Deepa Verma

Submitted by: Manya Mishra (1805210026)

Roopam Mishra (1805210045)

Sonali Singh (1805210055)

Department of Computer Science and Engineering

Institute of Engineering and Technology, Lucknow

Gesture and Voice

Controlled Virtual
Mouse
 Abstract

● This project takes forward the approach of the Human Computer Interaction (HCI) by
controlling cursor movement through hand movement and voice commands using a
real-time camera and microphone.

● Virtual mouse acts as a contactless mouse, thus can be more useful and time saving.

● Specially abled people with some problem in their hands can use this virtual mouse to
control the mouse functions in the computer.

● Can be used used for distance controlling of systems in robots, classrooms, war
emunisions, AR and VR game, etc.

Gesture and Voice

Controlled Virtual
Mouse
 Objective

● Design for operation of a mouse with the help of a webcam which is responsible for
capturing the video in real-time.

● Convert hand gesture/motion into mouse function and the cursor is set to a particular
screen position. The Virtual Mouse application is programmed to detect the position of
fingertips and knuckles where it will be set as the position of the mouse pointers.

● Develop a multi user independent speech recognition system that captures voice in
real-time with the help of a microphone and is able to retrieve folders, sub-folders,
documents, copy, paste, left click, right click and double click by taking voice command
and checking its validity.

● Design a voice controlled mouse system that is integrated with the gesture controlled
system.

Gesture and Voice

Controlled Virtual
Mouse
 Related Work
● In 1990, an early hardware-based system was introduced by Quam; in this system, it was
mandatory for the user to wear a DataGlove.
● Zhengyou et al. (2001), proposed an interface system named Visual Panel that utilize arbitrary
quadrangle-shaped planar object as a panel to allow the user to use any tip-pointer tools.
● Kamran Niyazi et al. (2012), proposed color tracking mouse stimulation. . The said system
tracks colour tapes on the user fingers by utilizing the computer vision technology.
● Kazim Sekeroglu (2010), the system requires three fingers with three colour pointers to simulate
the click events.
● Chu-Feng Lien (2015), proposed method that requires only finger-tips to control the mouse
cursor and clIck events. The proposed system doesn't requires hand-gestures nor colour tracking
in order to interact with the system, instead it utilize a feature name Motion History
Images(MHI).
● S. Shriram(2021);the model makes use of the MediaPipe package for the tracking of the hands
and for tracking of the tip of the hands.

Gesture and Voice

Controlled Virtual
Mouse
 Features
Gesture Control

● Move Cursor
● Left Click Neural Gesture
● Right Click
● Double Click
● Scrolling
● Drag and Drop
● Multiple Item Selection
● Volume Control
● Brightness Control

Move Cursor
Gesture and Voice
Controlled Virtual
Mouse
 Left Click Right Click Double Click

Drag and Drop Volume/Brightness Control

Gesture and Voice
Controlled Virtual
Mouse
Methodology for Gesture Control
● Computer Vision technology is applied for Object identification and object video
capturing in BGR format which is done by OpenCV.
● Captured hand object processed by MediaPipe using two models-

a. Palm Detection Model

b. Hand Landmark Model

● MediaPipe renders 21 coordinates located on a detected hand in a 2.5D space as an

output.
 Methodology for Gesture Control
● Subsequently ids are assigned to each finger of the right hand in the rendered output
landmark.
● Corresponding coordinated of all the required ids are calculated and different operations
are performed based on the distances and angles between the points

In order for the mouse to perform a left click:

In the first frame, the index finger with Tip ID 1 and the middle finger with Tip ID 2 must be
up, followed by Index Finger (Tip Id 1) down and Middle finger (Tip id2) up in the second
frame, with both producing an angle greater than 33.5°.
Gesture and Voice
Controlled Virtual
Mouse
 Methodology
Voice Controller

● Launch Jerry on your monitor

(Jerry - name of our voice assistant)
● Give command to be followed.
● Speech_recognition library detects your
speech and jerry will perform the
operations as given in the next slide.

Gesture and Voice

Controlled Virtual
Mouse
 Features
Voice Control

● Launch/stop gesture recognition

● Google search
● Find a location on Google map
● File navigation
● Current date and time
● Copy and paste
● Sleep/Wakeup App
● Exit

Gesture and Voice

Controlled Virtual
Mouse
 Data Flow Diagram
Gesture Control Voice Control

Gesture and Voice

Controlled Virtual
Mouse
 Applications
Major applications:

● Can reduce physical stress on the body that is responsible for causing back pain, weak
eyesight, posture defect, etc.

● Can be used used for distance controlling of systems in robots, classrooms, war
emunisions, AR and VR game, etc.

● Amidst the COVID-19 situation when it is unsafe to use the devices by touching, the
proposed virtual mouse can be used to control the PC mouse functions hand free.

● Virtual mouse can be used to play virtual reality and augmented reality based games
without the wireless or wired mouse devices

● Persons with some problems in their hands can use this virtual mouse to control the
mouse functions in the computer
Gesture and Voice
Controlled Virtual
Mouse
 Future scope -
Virtual Mouse will soon be introduced to replace the physical computer mouse to promote
convenience while still being able to accurately interact and control the computer system.

● More features such as enlarging and shrinking window, closing windows, etc can be added
by using palm and multiple fingers.

● Better efficiency of gesture recognition can be achieved when it can detect hand gestures
in different types of lightings, different hand complexions and distance from camera.

● Virtual mouse can be designed to perform other complex tasks such as doing calculations
by verbal commands etc.

● Voice recognition can be more efficient if it can correctly process different accents and
can ignore background noise.

Gesture and Voice

Controlled Virtual
Mouse
 Hardware & Software requirements
(minimum)-
● Web Cam: 1.3 megapixel resolution
● Processor : Core2Duo (2nd generation)
● Main Memory : 2GB RAM
● Hard Disk : 320GB
● Microphone
● Python.
● OpenCV
● MediaPipe

Gesture and Voice

Controlled Virtual
Mouse
 Conclusion

The main objective of the virtual mouse system is to control the mouse cursor functions by using
hand gestures and voice commands instead of using a physical mouse.
The proposed virtual mouse system performs well and has greater accuracy compared to the
existing models and also the model overcomes most of the limitations of the existing systems.

Virtual mouse can be used for real-world applications. Since the proposed mouse system can be
used virtually using hand gestures and voice commands, it can be useful in preventing the
spread of COVID-19. Virtual mouse can be very useful to specially abled people
It can help in simplifying user interactions with personal computers and hardware systems. It
can create possibility of distance controlling of robots, AR and VR etc.

Gesture and Voice

Controlled Virtual
Mouse
 References
● N. Subhash Chandra, T. Venu, P. Srikanth, “A Real-Time Static & Dynamic Hand Gesture
Recognition System” International Journal of Engineering Inventions Volume 4, Issue 12,
August 2015

● S. Shiriam, B. Nagaraj, J. Jaya, “Machine Learning based real time AI Virtual Mouse system
using computer vision to avoid COVID-19 spread”, Hindawi Journal of Healthcare
Engineering, October 2021.

● Hritik Joshi, Nitin Waybhase, Ratnesh Litoria, “Towards controlling mouse through hand
gestures: A novel and efficient approach”, Medi-caps University, January 2022

● Mohhamad Rafi, Khan Sohail, Shaikh Huda, “Control mouse and computer system using
voice commands”, International Journal of Research in Engineering and Technology”,
Volume 5, Issue 3, March 2016

Gesture and Voice

Controlled Virtual
Mouse
 Thank You!

Gesture and Voice

Controlled Virtual
Mouse