Scribd
Upload
411 views10 pages

Project Report - AI Virtual Mouse

The document describes a virtual mouse system controlled by hand gestures. The system uses computer vision and machine learning techniques to track hand movements from a webcam image and use those movements to control the mouse pointer on screen. Key modules include hand tracking using mediapipe, finger pose recognition to determine click/move modes, and coordinate mapping to translate hand positions to screen positions for mouse control. The system is implemented with OpenCV, NumPy and other Python libraries. In conclusion, the virtual mouse allows accessible control of a computer system through hand gestures rather than requiring use of a physical mouse.

Uploaded by

Param Panwar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
411 views10 pages

Project Report - AI Virtual Mouse

The document describes a virtual mouse system controlled by hand gestures. The system uses computer vision and machine learning techniques to track hand movements from a webcam image and use those movements to control the mouse pointer on screen. Key modules include hand tracking using mediapipe, finger pose recognition to determine click/move modes, and coordinate mapping to translate hand positions to screen positions for mouse control. The system is implemented with OpenCV, NumPy and other Python libraries. In conclusion, the virtual mouse allows accessible control of a computer system through hand gestures rather than requiring use of a physical mouse.

Uploaded by

Param Panwar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 10

AI VIRTUAL MOUSE

Prepared By:
Param Panwar
2nd Year CSE

Objective:
Hand Gesture Recognition plays a key role in human-computer
interactions. As we can see that there are so many new Technological
advancements happening such as biometric authentication which we can
see frequently in our smart phones, similarly hand gesture recognition is a
modern way of human and computer interaction i.e., we can control our
system by showing our hands in front of webcam and hand gesture
recognition can be useful for all kinds of people. Based upon this idea this
paper is presented. This paper provides a detailed explanation to the
algorithms and methodologies for the virtual mouse.

Purpose:
As the technology increase everything becomes virtualized such as speech
recognition. Speech Recognition is used for recognition and translation of
the spoken language into text. Thus, Speech Recognition can replace
keyboards in the future, Similarly Eye Tracking which is used to control
the mouse pointer with the help of our eye. Eye Tracking can replace
mouse in the future. Gestures can be in any form like hand image or pixel
image or any human given pose that require less computational difficulty
or power for making the devices required for the recognitions to make
work. Different techniques are being proposed by the companies for
gaining necessary information/data for recognition handmade gestures
recognition models. Some models work with special devices such as data
glove devices and color caps to develop complex information about
gesture provided by the user/human.

OVERVIEW OF THE SYSTEM


A Computer Mouse is an input device that helps to point and to interact
with whatever that is being pointed. There are so many types of mouse in
the current trend, there’s the mechanical mouse that consists of a optical
sensor and the movement of the pointer is determined by the motion of
that mouse. Optical Mouse consists of a led sensor to detect the movement
of the pointer. Years Later the laser mouse was introduced to improve the
accuracy and to overcome the drawbacks of the Optical Mouse. Later as
the Technology has been increased drastically wireless mouse was
introduced so as to enable hassle free movement of the mouse and to
improve the accuracy. No Matter how much the accuracy of the mouse
increases but there will always be limitations of the mouse as the mouse is
a hardware input device and there can be some problems like mouse click
not functioning properly etc., as the mouse is a hardware device like any
other physical object even the mouse will have a durability time within
which is functional and after its durability time we have to change the
mouse.

Modules Implementation:
 Checking devices (like webcam) working properly or not,
 Collection tape or finger ribbon, which should be fit to the fingers,
 Import packages like NumPy, OpenCV, mediapipe
 Implement the Open Gesture Operation

Source Code:
File name – AiVirtualMouseProject.py
import cv2
import numpy as np
import HandTrackingModule as htm
import time
import autopy

# Camera
wCam, hCam = 640, 480
frameR = 100 # Frame Reduction
smoothening = 7

pTime = 0
plocX, plocY = 0, 0
clocX, clocY = 0, 0
cap = cv2.VideoCapture(0)
cap.set(10, wCam)
cap.set(11, hCam)
detector = htm.handDetector(maxHands=1)
wScr, hScr = autopy.screen.size()
# print(wScr, hScr)

while True:
# 1. Find hand Landmarks
success, img = cap.read()
img = detector.findHands(img)
lmList, bbox = detector.findPosition(img)
# 2. Get the tip of the index and middle fingers
if len(lmList) != 0:
x1, y1 = lmList[8][1:]
x2, y2 = lmList[12][1:]
# print(x1, y1, x2, y2)

# 3. Check which fingers are up


fingers = detector.fingersUp()
# print(fingers)
cv2.rectangle(img, (frameR, frameR), (wCam - frameR, hCam -
frameR),
(255, 0, 255), 2)
# 4. Only Index Finger : Moving Mode
if fingers[1] == 1 and fingers[2] == 0:
# 5. Convert Coordinates
x3 = np.interp(x1, (frameR, wCam - frameR), (0, wScr))
y3 = np.interp(y1, (frameR, hCam - frameR), (0, hScr))
# 6. Smoothen Values
clocX = plocX + (x3 - plocX) / smoothening
clocY = plocY + (y3 - plocY) / smoothening

# 7. Move Mouse
autopy.mouse.move(wScr - clocX, clocY)
cv2.circle(img, (x1, y1), 15, (255, 0, 255), cv2.FILLED)
plocX, plocY = clocX, clocY

# 8. Both Index and middle fingers are up : Clicking Mode


if fingers[1] == 1 and fingers[2] == 1:
# 9. Find distance between fingers
length, img, lineInfo = detector.findDistance(8, 12, img)
print(length)
# 10. Click mouse if distance short
if length < 40:
cv2.circle(img, (lineInfo[4], lineInfo[5]),
15, (0, 255, 0), cv2.FILLED)
autopy.mouse.click()

# 11. Frame Rate


cTime = time.time()
fps = 1 / (cTime - pTime)
pTime = cTime
cv2.putText(img, str(int(fps)), (20, 50), cv2.FONT_HERSHEY_PLAIN,
3,
(255, 0, 0), 3)
# 12. Display
cv2.imshow("Image", img)
cv2.waitKey(1)

File name – HandTrackingModule.py


import cv2
import mediapipe as mp
import time
import math
import numpy as np
class handDetector():
def __init__(self, mode=False, maxHands=2, detectionCon=0.5,
trackCon=0.5):
self.results = None
self.mode = mode
self.maxHands = maxHands
self.detectionCon = detectionCon
self.trackCon = trackCon

self.mpHands = mp.solutions.hands
self.hands = self.mpHands.Hands(self.mode, self.maxHands,
self.detectionCon, self.trackCon)
self.mpDraw = mp.solutions.drawing_utils
self.tipIds = [4, 8, 12, 16, 20]

def findHands(self, img, draw=True):


imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
self.results = self.hands.process(imgRGB)
# print(results.multi_hand_landmarks)

if self.results.multi_hand_landmarks:
for handLms in self.results.multi_hand_landmarks:
if draw:
self.mpDraw.draw_landmarks(img, handLms,
self.mpHands.HAND_CONNECTIONS)

return img

def findPosition(self, img, handNo=0, draw=True):


xList = []
yList = []
bbox = []
self.lmList = []
if self.results.multi_hand_landmarks:
myHand = self.results.multi_hand_landmarks[handNo]
for id, lm in enumerate(myHand.landmark):
# print(id, lm)
h, w, c = img.shape
cx, cy = int(lm.x * w), int(lm.y * h)
xList.append(cx)
yList.append(cy)
# print(id, cx, cy)
self.lmList.append([id, cx, cy])
if draw:
cv2.circle(img, (cx, cy), 5, (255, 0, 255), cv2.FILLED)

xmin, xmax = min(xList), max(xList)


ymin, ymax = min(yList), max(yList)
bbox = xmin, ymin, xmax, ymax

if draw:
cv2.rectangle(img, (xmin - 20, ymin - 20), (xmax + 20, ymax +
20),
(0, 255, 0), 2)

return self.lmList, bbox

def fingersUp(self):
fingers = []
# Thumb
if self.lmList[self.tipIds[0]][1] > self.lmList[self.tipIds[0] - 1][1]:
fingers.append(1)
else:
fingers.append(0)
# Fingers
for id in range(1, 5):
if self.lmList[self.tipIds[id]][2] < self.lmList[self.tipIds[id] - 2][2]:
fingers.append(1)
else:
fingers.append(0)

# totalFingers = fingers.count(1)

return fingers

def findDistance(self, p1, p2, img, draw=True, r=15, t=3):


x1, y1 = self.lmList[p1][1:]
x2, y2 = self.lmList[p2][1:]
cx, cy = (x1 + x2) // 2, (y1 + y2) // 2
if draw:
cv2.line(img, (x1, y1), (x2, y2), (255, 0, 255), t)
cv2.circle(img, (x1, y1), r, (255, 0, 255), cv2.FILLED)
cv2.circle(img, (x2, y2), r, (255, 0, 255), cv2.FILLED)
cv2.circle(img, (cx, cy), r, (0, 0, 255), cv2.FILLED)
length = math.hypot(x2 - x1, y2 - y1)

return length, img, [x1, y1, x2, y2, cx, cy]

def main():
pTime = 0
cTime = 0
cap = cv2.VideoCapture(1)
detector = handDetector()
while True:
success, img = cap.read()
img = detector.findHands(img)
lmList, bbox = detector.findPosition(img)
if len(lmList) != 0:
print(lmList[4])

cTime = time.time()
fps = 1 / (cTime - pTime)
pTime = cTime

cv2.putText(img, str(int(fps)), (10, 70),


cv2.FONT_HERSHEY_PLAIN, 3,
(255, 0, 255), 3)

cv2.imshow("Image", img)
cv2.waitKey(1)

if __name__ == "__main__":
main()

Output Screenshot:

CONCLUSION:
This model can conclude by using the topics of computer vision like open
CV, it can form masks that can variate colors by using color variation
techniques and also development of mouse movement by using certain
packages like ‘mouse’ which will be used for the movement of mouse by
using the coordinates that are linked to the detected color. This can provide
ease use of systems and many other applications. So, the open CV is
helping the users with different accessible forms of models that will make
ease life.

You might also like