Dynamic hand gesture detector using python and open CV

*Lakshya Gaur
**Kapil Tomar
***Dr. Naveen Tyagi

Abstract 1. Introduction
Hand Gesture Recognition and Image Overlay
Using OpenCV and MediaPipe Hand gesture recognition is a crucial technology in
This research paper presents a method for real- the field of human-computer interaction. It allows
time hand gesture recognition and image overlay users to interact with digital devices using natural
using OpenCV and MediaPipe. The system hand movements, enhancing the user experience.
captures live video feed from a webcam, detects This project focuses on creating a hand gesture
hand gestures, and overlays corresponding images recognition system that identifies specific gestures
based on the detected gestures. The and overlays images accordingly. The system uses
implementation leverages the capabilities of Python, OpenCV, and MediaPipe, leveraging their
MediaPipe for hand tracking and OpenCV for powerful image processing and machine learning
image processing and display. The proposed capabilities.
method is efficient and runs in real-time, providing
immediate feedback on detected gestures. This The journey towards gesture-based interaction
paper discusses the system model, the underlying represents a departure from the conventional
algorithm, and the results obtained from the notion of computer interfaces as passive tools
implementation. controlled solely through manual manipulation.
Instead, it seeks to imbue technology with a
deeper understanding of human intention and
expression, enabling seamless interaction that
mirrors the fluidity of human communication.
At its core, gesture-based interaction draws
inspiration from the rich tapestry of human
movement, encompassing gestures, postures,
facial expressions, and even subtle nuances
of body language. By decoding and
interpreting these non-verbal cues, computers
can discern user intent and respond in a
manner that feels more natural and intuitive.

The evolution of gesture-based interfaces has

been propelled by advancements in a myriad
of technologies, including computer vision,
machine learning, sensor technology, and
augmented reality.

*M.Tech Scholar, Department of Computer Science and Engineering, MIT, Bulandshahr

** Assistant Professor, Department of Computer Science and Engineering, MIT, Bulandshahr
***Professor, Department of Computer Science and Engineering, MIT, Bulandshahr
The main objectives of this project are: The algorithm identifies gestures by
analyzing the positions of finger tips and
 To develop a real-time hand gesture thumb:
recognition system.
 To overlay images based on detected gestures.  LIKE: Thumb up gesture.
 To provide a seamless and responsive user  DISLIKE: Thumb down gesture.
experience.  NEUTRAL: No specific gesture.
 DYBALA CELEBRATION: Specific finger
2. System Model arrangement.
2.1 The system comprises the following
components: 4. Code Implementation
4.1 Initialization
 Webcam: Captures live video feed. python
 OpenCV: Handles video capture, image Copy code
processing, and display.
 MediaPipe: Detects and tracks hand import cv2 import numpy as np import mediapipe as mp
landmarks. cap = cv2.VideoCapture(0) mpHands =
 Gesture Recognition: Identifies specific hand mp.solutions.hands hands = mpHands.Hands() draw =
gestures. mp.solutions.drawing_utils # Load images for overlay
 Image Overlay: Overlays corresponding like_img = cv2.imread('like.jpg') dislike_img =
images based on detected gestures. cv2.imread('dislike.jpg') neutral_img =
cv2.imread('neutral.jpg') dybala_img =
cv2.imread('dybala.jpg') Finger_tips = [8, 12, 16, 20]
2.2 The flow of the system is as follows: thumb_tip = 4
4.2 Frame Processing
1. Capture live video feed using OpenCV.
python
2. Process each frame to detect hand landmarks
using MediaPipe. Copy code
3. Identify gestures based on the positions of the while True: success, img = cap.read() img = cv2.flip(img,
landmarks. 1) h, w, c = img.shape rgb = cv2.cvtColor(img,
4. Overlay images on the video feed based on cv2.COLOR_BGR2RGB) results = hands.process(rgb) if
detected gestures. results.multi_hand_landmarks: for hand_landmarks in
5. Display the processed video feed in real-time. results.multi_hand_landmarks: lm_list = [] for lm in
hand_landmarks.landmark: lm_list.append(lm)
finger_fold_status = [] for tip in Finger_tips: x, y =
3. Algorithm int(lm_list[tip].x * w), int(lm_list[tip].y * h) cv2.circle(img,
3.1 Initialization (x, y), 13, (0, 255, 0), cv2.FILLED) if lm_list[tip].x <
lm_list[tip - 3].x: cv2.circle(img, (x, y), 13, (0, 0, 255),
 Import necessary libraries. cv2.FILLED) finger_fold_status.append(True) else:
 Initialize video capture using OpenCV. finger_fold_status.append(False) if
 Load images for overlay. all(finger_fold_status): if lm_list[thumb_tip].y <
 Initialize MediaPipe for hand detection. lm_list[thumb_tip - 1].y < lm_list[thumb_tip - 2].y: h, w, c
= like_img.shape img[0:h, 0:w] = like_img else: h, w, c =
dislike_img.shape img[0:h, 0:w] = dislike_img if
3.2 Frame Processing lm_list[Finger_tips[0]].x > lm_list[Finger_tips[0] - 1].x: h,
For each frame captured from the webcam: w, c = neutral_img.shape img[0:h, 0:w] = neutral_img if
lm_list[Finger_tips[0]].x < lm_list[Finger_tips[0] - 3].x: if
1. Flip the frame horizontally for a mirror effect. lm_list[thumb_tip].x < lm_list[thumb_tip - 2].x: h, w, c =
2. Convert the frame to RGB format. dybala_img.shape img[0:h, 0:w] = dybala_img
3. Detect hand landmarks using MediaPipe. draw.draw_landmarks(img, hand_landmarks,
4. Identify gestures by analyzing the positions of mpHands.HAND_CONNECTIONS) cv2.imshow('Hand
landmarks. Gesture Recognition', img) if cv2.waitKey(1) == ord('q'):
5. Overlay images based on detected gestures. break cap.release() cv2.destroyAllWindows()
6. Display the processed frame.

3.3 Gesture Detection

5. Conclusion
This project demonstrates an efficient and
real-time hand gesture recognition system
using OpenCV and MediaPipe. By leveraging
these libraries, we achieved accurate hand
tracking and gesture recognition, providing
immediate visual feedback through image
overlay. The system can be further enhanced
with additional gestures and applications,
making it a versatile tool for human-computer
interaction.
References

1. OpenCV: Open Source Computer Vision

Library. Available at: https://opencv.org/
2. MediaPipe: Cross-platform Framework for
Building Multimodal Applied ML Pipelines.
Available at: https://mediapipe.dev/
3. Python: The Python Programming Language.
Available at: https://www.python.org/

Images to Include

1. Development Environment:
 Screenshot of the code editor with the
project open.
 Photo of the setup with the webcam.
Hand Detection Process:
 Raw webcam feed before processing.
 Hand landmarks detection with
landmarks highlighted.
Gesture Recognition Output:
 Examples of each gesture ("LIKE",
"DISLIKE", "NEUTRAL", "DYBALA
CELEBRATION") with the overlay
image.

By incorporating these images and code

snippets, this thesis will effectively
illustrate the project's implementation and
results.