A Mini-Project Report on

COMPUTER CONTROL USING HAND GESTURE

Submitted in partial fulfillment of the requirements for the award of the degree of
BACHELOR OF TECHNOLOGY
IN
ELECTRICAL & ELECTRONICS ENGINEERING
By
A.V.SAI MANIKANTA 16A31A0217
B.SANKAR 16A31A0219
B.V.DURGA PRASAD 16A31A0220
N.SAI SURYA 16A31A0240

Under the esteemed guidance of

Sri.S.ASHOK REDDYM. Tech
Assistant Professor
Department of E.E.E.

DEPARTMENT OF ELECTRICAL & ELECTRONICS ENGINEERING

PRAGATI ENGINEERING COLLEGE
(AUTONOMUS)
(Approved by AICTE, Permanently Affiliated to JNTUK, KAKINADA & Accredited by NAAC with ‘A’ Grade)

1-378, A.D.B. Road, Surampalem, Near Peddapuram – 533437

2016-2020
 PRAGATI ENGINEERING COLLEGE
(AUTONOMUS)
(Approved by AICTE, Permanently Affiliated to JNTUK, KAKINADA & Accredited by NAAC with ‘A’ Grade)
1-378, A.D.B. Road, Surampalem, Near Peddapuram – 533437

CERTIFICATE
DEPARTMENT OF ELECTRICAL & ELECTRONICS ENGINEERING
(Approved by AICTE, Permanently Affiliated to JNTUK, KAKINADA & Accredited by NAAC with ‘A’ Grade)
1-378, A.D.B. Road, Surampalem, Near Peddapuram – 533437

This is to certify that the Mini Project Report entitled “COMPUTER CONTROL USING HAND
GESTURE” is being submitted by A.V.Sai Manikanta (16A31A0217), B.V.Durga Prasad (16A31A0220),
B.Sankar (16A31A0219), and N.Sai Surya (16A31A0240) in partial fulfillment for the award of the Degree
of Bachelor of Technology in Electrical & Electronics Engineering of Pragati Engineering College, for the
record of bonafide work carried out by them.

Sri.S.ASHOK REDDY
M. Tech
Assistant Professor

Dr. K. SATYANARAYANA
M. Tech., PhD, MIE, MIEEE, MISTE, C Eng.

Professor&H.O.D- E.E.E.
 ACKNOWLEDGEMENT

We express our thanks to our guide Sri.S.ASHOK REDDY, Assistant Professor in Electrical
and Electronics Engineering, who deserves a special note of thanks and gratitude, for having extended
their fullest co-operation and guidance, without this project would never have materialized.

We express our deep sense of gratitude to Dr. K. SATYANARAYANA, Vice Principal,

Professor and H.O.D. of Electrical and Electronics Engineering, for having shown keen interest at every
stage of development of our project work and for guiding us in every aspect.

We wish to express our special thanks to our beloved Dr. S. SAMBHU PRASAD, Principal for
giving us guidelines and encouragement.

We wish to express our sincere gratitude to our beloved and respected Dr.P.KRISHNA RAO,
Chairman and Sri. M. V. HARANATHA BABU, Director and Sri.M.SATISH, Vice President for their
encouragement and blessings.

We are thankful to all our faculty members of the department for their valuable
suggestions. Our sincere thanks also extended to all the teaching and non-teaching staff of Pragati
Engineering College.

A.V.SAI MANIKANTA 16A31A0217

B.SANKAR 16A31A0219
B.V.DURGA PRASAD 16A31A0220
N.SAI SURYA 16A31A0240
 COMPUTER CONTROL USING HAND GESTURE

CONTENTS

TITLE Page No.

LIST OF FIGURES 3

LIST OF TABLES 4

ABBREVATIONS 5

ABSTRACT 6

CHAPTER-1 INTRODUCTION 7-11

1.1 PROJECT OVERVIEW 8

1.2 What is a Hand Gesture? 8

1.3 DEFINITIONS 9

1.3.1 Power Supply 9

1.3.2 Arduino UNO 9

1.3.3 Ultrasonic sensors 10

1.3.4 Python 11

1.4 Features 11

1.5 Circuit Connections 11

1.5.1 Ultrasonic Sensor Connection 11

CHAPTER-2 LITERATURE SURVEY 13-17

2.1Portable ultrasonic range meter 14

2.2 Table of contents 14

2.2.1 Circuit diagram 14

2.2.2 Hardware required 14

2.2.3Circuit design ultrasonic range meter 15

PRAGATI ENGINEERING COLLEGE Page 1
 COMPUTER CONTROL USING HAND GESTURE
2.2.4 Working 16

Chapter-3 PROBLEM FORMULATION AND STATEMENT 18-22

3.1 PROPOSAL SYSTEM 19

3.1.1 Main Principle 19

3.1.2 Working Process 19

3.2 Block Diagram for Hand gesture control computer 20

3.3 Problem definition 21

3.4 Advantages of Proposed System 21

3.5 Applications 22

Chapter-4 HARDWARE IMPLEMENTATION 23-25

4.1 Components Required 24
4.2 The basic work of each element 24
4.2.1 Sensor 24
4.2.2 Software used 24
4.3 Hardware and Software requirements 24
4.3.1 Hardware requirements 24
4.3.2 Software requirements 24
4.4 Circuit Diagram 25
4.5 Software Implementation 25
4.5.1 Programming Guide 25

Chapter-5 RESULT 34-35

CONCLUSION 36
FUTURE SCOPE 36
REFERENCS 36
APPENDIX 37-41

PRAGATI ENGINEERING COLLEGE Page 2

 COMPUTER CONTROL USING HAND GESTURE
LIST OF FIGURES

Figure Description page No

Fig 1.1: Arduino Nano and Arduino UNO board 9

Fig 1.2: Ultrasonic Sensor 10

Fig 1.3: Measuring Distance from Ultrasonic Sensor to object 10

Fig 1.4: Ultrasonic Sensor Connection with Arduino UNO board 12

Fig 2.1: Real Time Portable Ultrasonic Range Metre 14

Fig 2.2: Real Time Portable Ultrasonic Range Metre for different values 15

Fig 3.1: Reflection of Ultrasonic Sensor To Object 19

Fig 3.2: Block Diagram of Hand Gesture Control Computer 20

Fig 4.1: Circuit Diagram for Hand gesture Control Computer 25

Fig 4.2: Scrolling Down of Web Page 26

Fig 4.3: Scrolling Up of Web Page 26

Fig 4.4: Moving to Next Tab 27

Fig 4.5: Moving To Previous Tab 27

Fig 4.6: Python GUI Output Screen 28

PRAGATI ENGINEERING COLLEGE Page 3

 COMPUTER CONTROL USING HAND GESTURE

LIST OF TABLES

Table Description Page No

Table3.1: Movement of Hand Gesture 20

PRAGATI ENGINEERING COLLEGE Page 4

 COMPUTER CONTROL USING HAND GESTURE
ABBREVIATIONS

US Transmitter - Ultrasonic Sensor Transmitter

US Receiver - Ultrasonic Sensor Receiver

Arduino GUI - Arduino Graphical User Interface

PRAGATI ENGINEERING COLLEGE Page 5

 COMPUTER CONTROL USING HAND GESTURE
ABSTRACT

The main objective of this project is to create a Hand gesture controlled computer .It is an innovative
technique of interaction between human and computer. As compared to existing techniques, hand gesture
technique has the advantage of being easy to use. By using this technique the traditional way of using
mouse and keyboard will be change because one can interact with the computer with hand gestures.
In this technique, ultrasonic sensor is used to classify the hand movement in real-time. Ultrasonic sensor
measures the distance of hand by using sound waves. The main idea of our approach is to speed up the
interaction with computer, using general purpose hardware like personal computer, Arduino UNO board and
low cost sensor like ultrasonic sensor. In this way any user can easily interact with the computer using hand
gestures.

PRAGATI ENGINEERING COLLEGE Page 6

 COMPUTER CONTROL USING HAND GESTURE

CHAPTER-1
INTRODUCTION

PRAGATI ENGINEERING COLLEGE Page 7

 COMPUTER CONTROL USING HAND GESTURE
1.1 PROJECT OVERVIEW:

The project was originally used for an interaction between human and computer in an easier
way. We all know that humans interact in the physical world by the means of the five senses. However,
gestures have been an important means of communication in the physical world from ancient times, even
before the invention of any language. In this era of machines taking control of every complex works,
interactions with machines have become more important than ever. Since this project deals with gesture
controlled laptop, the primary focus will be on the use of hand gestures for specific applications only.

There are several ways to capture a human gesture that a computer would be able to recognize. The gesture
can be captured using distance measurement, camera, or a data glove. Gestures can also be captured via
Bluetooth or infrared waves, Acoustic, Tactile, optical or with a motion technology. The embedded systems
designed for specific control functions can be optimized to reduce the size and cost of the device and
increase the reliability and performance.

Here this project consists of mainly three components –Arduino Uno, Ultrasonic sensors, and a laptop. The
ultrasonic sensors hooked to the Arduino are used to determine the gestures and the distance of the hand
from the ultrasonic sensors. The code loaded in Arduino finds the respective keyword for the distance found
and sends it to Windows OS. Python code that runs in the background recognizes the keywords and
generates the corresponding virtual keystrokes for Windows. The hotkeys then control particular function of
the application of intends to run VLC Media Player and web page.

1.2 What is a Hand Gesture?

A gesture is a motion of the body that conveys information. Waving the hands is a gesture that
suggests “good bye”. However, the movement of body parts that does not convey specific meaning is not
considered as gesture, such as pressing key in computer key board.
Gesture recognition is a topic in computer science and language technology with the goal of
interpreting human gestures via mathematical algorithms. Gestures can originate from any bodily motion or
state but commonly originate from the face or hand.

PRAGATI ENGINEERING COLLEGE Page 8

 COMPUTER CONTROL USING HAND GESTURE
1.3 DEFINITIONS:

1.3.1 Power Supply:

A power supply unit (or PSU) converts mains AC to low-voltage regulates DC power for
the internal components of a computer. They also provide a signal to the motherboard to indicate when the
DC voltages are in spec, so that the computer is able to safely power up and boot.The power supply to two
ultrasonic sensors is provided directly by computer through USB cable connected to Arduino.

1.3.2 Arduino UNO:

Arduino UNO is one of the most popular prototyping boards. It is used frequently in
robotic applications as it is small in size and packed with rich features. The board comes with built-in
Arduino boot loader. It is an At mega 328 based controller board which has 14 GPIO pins, 6 PWM pins, 6
Analog inputs and on board UART, SPI and TWI interfaces. In this project, 5 GPIO pins of the board are
utilized to connect the IR sensors and 6 GPIO pins are used to interface L293D motor driver IC.Arduino
Nano is the also a kind of Arduino family board with some limitations compared to UNO.

Fig (1.1): Arduino Nano and Arduino UNO Board

1.3.3 Ultrasonic Sensors:

The ultrasonic sensor is used to determine distance to an object. It emits an ultrasound and
this sound travels in the air. When it gets to an object it is reflected back to the ultrasonic sensor .The
ultrasonic sensor has four pins, those are: 1) VCC-used to powers the sensor 2) Trigger- to send the US
waves 3) Echo- the output pin .The US wave is returned back to sensor through eco pin and GND is
connected to the ground pin of Arduino.

PRAGATI ENGINEERING COLLEGE Page 9

 COMPUTER CONTROL USING HAND GESTURE

Fig (1.2): ultrasonic sensor

An optical sensor has a transmitter and

receiver, whereas an ultrasonic sensor uses a single ultrasonic element for both emission and reception. In a
reflective model ultrasonic sensor, a single oscillator emits and receives ultrasonic waves alternately. This
enables miniaturization of
the sensor head.

Fig (1.3): measuring distance from ultrasonic sensor to object

The ultrasonic sensors are the components that read the hand gesture and the distance of the hand from the
sensor. For the left sensor, bringing our hand closer to it will rewind the video and moving away from it will
fast forward the video. In the right sensor, the same actions would result in controlling the volume of the
player
Distance calculation:
The distance can be calculated with the following formula:
Distance L = 1/2 × T × C
Here L is the distance, T is the time between the emission and reception, and C is the sonic speed. (The
value is multiplied by 1/2 because it is the time for go-and-return distance.)

PRAGATI ENGINEERING COLLEGE Page 10

 COMPUTER CONTROL USING HAND GESTURE
1.3.4 Python:

Python is a high-level programming language designed to be easy to read and simple to implement.
It is open source, which means it is free to use, even for commercial applications.  Python is considered a
scripting language, like Ruby or Perl and is often used for creating web applications and dynamic Web
content.

1.4 Features:
The following list shows typical characteristics enabled by the detection system.
 Transparent object detectable
Ultrasonic waves can reflect off a glass or liquid surface and return to the sensor head, even
transparent targets can be detected.
 Resistant to mist and dirt
Detection is not affected by accumulation of dust or dirt.
 Complex shaped object detectable
Presence detection is stable even for targets such as mesh trays or springs.

1.5 Circuit Connections:

The two ultrasonic sensors are connected to Arduino Uno board and the board is connected to
Laptop through USB. The VCC pin of each sensor is provided by Arduino using the 5V output port.
Grounding of both sensors is attached to GND0. The Trigger and Echo pins are connected as per the code
written.

1.5.1 Ultra Sonic Sensor Connection:

Ultrasonic sensors measure distance by using ultrasonic waves. The sensor head emits
an ultrasonic wave and receives the wave reflected back from the target. Ultrasonic Sensors measure the
distance to the target by measuring the time between the emission and reception.

PRAGATI ENGINEERING COLLEGE Page 11

 COMPUTER CONTROL USING HAND GESTURE

Fig (1.4) ultrasonic sensor Connection with Arduino UNO board

PRAGATI ENGINEERING COLLEGE Page 12

 COMPUTER CONTROL USING HAND GESTURE

CHAPTER-2
LITERATURE SURVEY

PRAGATI ENGINEERING COLLEGE Page 13

 COMPUTER CONTROL USING HAND GESTURE
2.1 Portable Ultrasonic Range Meter:
For the transmission of this energy between transmitter and receiver, presence of a medium is necessary.
The medium can be solids, liquids or gases.

Fig (2.1): Real Time Portable ultrasonic range meter

The sound energy travels by causing disturbance in the medium it is travelling and this is called propagation
of sound waves. Under normal conditions, the velocity of the sound is 330m/s.
SONAR or Sound Navigation and Ranging is a non-contact distance measuring technique generally used in
submarines. In this technique, a high frequency sound wave is transmitted by a transmitter and the reflected
echo from a target is captured by a receiver.
As the velocity of the sound wave is known, by measuring the time of travel, the distance
between the source and the target can be calculated. Depending on the frequency of the sound wave, the
SONAR can be either Infrasonic or Ultrasonic.
Ultrasonic sensors produce sound waves with frequencies higher than the audible range
(20Hz to 20 KHz) i.e. greater than 20 KHz. In case of an infrasonic sensor, the frequency of sound wave is
less than 20Hz. In this project, a Portable Ultrasonic Range Meter is designed which can be used to measure
distance of a target in non-contact fashion. The project is based on Arduino, Ultrasonic Sensor and an LCD
display.

2.2Table of Contents:

 Circuit Diagram
 Hardware Required
 Circuit Design of Ultrasonic Range Meter
 Working

PRAGATI ENGINEERING COLLEGE Page 14

 COMPUTER CONTROL USING HAND GESTURE
2.2.1 Circuit Diagram:

Fig (2.2): circuit

diagram of portable ultrasonic range meter

2.2.2 Hardware Required:

 Arduino Uno
 Ultrasonic Sensor HC-SR04
 16X2 LCD Display
 10KΩ POT

2.2.3 Circuit Design of Ultrasonic Range Meter

The circuit consists of Arduino Uno, an Ultrasonic sensor and an LCD display to instantaneously
display the results. The design of the circuit is very simple. The available out of 14 I/O pins on Arduino, we
use 8 pins for this. 2 pins are used for Ultrasonic sensor and other 6 pins are used to control the LCD. The 4
pins of Ultrasonic sensor are Vcc, Gnd, Trig and Echo. Trig is connected to Pin 11 of Arduino and Echo is
connected to Pin 10. With respect to Arduino, Pins 10 and 11 are input and output respectively.

 Pins 15 and 16 (LED+ and LED-) of the LCD are backlight pins. They are connected to Vcc and Gnd respectively
.Four data pins of LCD are used to display the information.
 Pins 11, 12, 13 and 14 of LCD (D4 – D7) are connected to pins 5, 4, 3 and 2 of Arduino.
 Pins RS and E (pins 4 and 6) of LCD are connected to pins 7 and 6 of Arduino respectively while RW (pin 5) is
connected to ground. Pins 1 and 2 (Vss and Vdd) are connected to ground and Vcc respectively.
 In order to control the contrast of the LCD display, pin 3 (VE) of LCD is connected to the wiper of a 10 KΩ POT
with the other terminals of POT connected to Vcc and Gnd.

PRAGATI ENGINEERING COLLEGE Page 15

 COMPUTER CONTROL USING HAND GESTURE

2.2.4 Working:
Ultrasonic sensor is the main module in the range meter circuit. An ultrasonic sensor
consists of an ultrasound transmitter and a receiver. The transmitter sends a sonic burst of 8 pulses at 40
KHz frequency. This signal hits the target and the echo is received by the receiver module. By measuring
the time between the events of sending the pulse and receiving the echo, the distance can be calculated.

Fig (2.3): Real Time Portable Ultrasonic Range Meter

The ultrasonic sensor used here is HC-SR04. It can be used to measure distance in the range of 2cm to
400cm with accurate readings.

PRAGATI ENGINEERING COLLEGE Page 16

 COMPUTER CONTROL USING HAND GESTURE

The sensor module consists of 4 pins: Vcc, Gnd, Trig and Echo. When the Trig pin is high for a duration of
at least 10µs, the ultrasonic sensor sends the ultrasound signals. The Echo pin is high from the moment of
sending the signal and receiving it.

PRAGATI ENGINEERING COLLEGE Page 17

 COMPUTER CONTROL USING HAND GESTURE
Observed That How The Ultrasonic Sensor Measure The Distance By Sending The pulse And Then
Receiving The Echo As Output.

CHAPTER-3
PROBLEM FORMULATION AND STATEMENT

PRAGATI ENGINEERING COLLEGE Page 18

 COMPUTER CONTROL USING HAND GESTURE
3.1PROPOSAL SYSTEM:
3.1.1 Main Principle:
The principle behind the Arduino based Hand Gesture Control of Computer is actually
very simple. All we have to do is use two Ultrasonic Sensors with Arduino, place your hand in front of
the Ultrasonic Sensor and calculate the distance between the hand and the sensor. Using this
information, relevant actions in the computer can be performed.
The position of the Ultrasonic Sensors is very important. Place the two Ultrasonic Sensors
on the top of a laptop screen at either end. The distance information from Arduino is collected by a
Python Program and a special library called PyAutoGUI will convert the data into keyboard click
actions.

3.1.2 Working Process:

Gesture controlling is based on specifying hand position from the ultrasonic sensor.
For processing the raw data, a micro-controller is essential; for that we use Arduino UNO board. Via
USB connection the microcontroller Transfers the processed and calculated distance value which is
provided by the sensor.
The data which is send by the sensor is processed in the software in PC where all the calculations are
performed and the data is matched with the predefined conditions. In this model two ultrasonic sensors
are used to detect hand position and are connected to the Arduino.
As we know ultrasonic sensor continuously emits sound and it gets reflected back from user’s hand.
The distance between the sounds is send and detection of reflect back sound wave is calculated by the
micro-controller

Fig (3.1)reflection of ultrasonic sensor to object

PRAGATI ENGINEERING COLLEGE Page 19

 COMPUTER CONTROL USING HAND GESTURE
3.2 Block Diagram for Hand Gesture Control Computer:

ULTRASONIC ULTRASONIC
SENSOR SENSOR

AURDINO
UNO

LAP
TOP

Fig (3.2): Block Diagram of hand gesture control computer

From the above figure 3.2, we see that two ultrasonic sensors are connected to Arduino Uno board and the
board is connected to Laptop through USB. The ultrasonic sensors have a transmitter and a receiver where
the transmitter emits ultrasonic waves.

Table3.1: Movement of Hand Gesture

Hand Gesture Output produced by system

video Web Page

Right hand push in Increase volume Scroll up
Right hand pull out Decrease volume Scroll down
Swipe to right sensor Play Switch to next tab
Swipe to left sensor Pause Switch to previous tab

PRAGATI ENGINEERING COLLEGE Page 20

 COMPUTER CONTROL USING HAND GESTURE
3.3 Problem Definition:

Our main objective is to control the computer (or) laptop using a hand gesture. A gesture is a
motion of the body that conveys information This Gesture controlling is based on specifying hand position
from the ultrasonic sensor.
The hand gestures in front of the Ultrasonic sensors can be calibrated so that they can perform five different
tasks on your computer. Before taking a look at the gestures, let us first see the tasks that we can
accomplish.

 Switch to Next Tab in a Web Browser

 Scroll Down in a Web Page
 Scroll Up in a Web Page
 Switch between two Tasks (Chrome and VLC Player)
 Play/Pause Video in VLC Player
 Increase Volume
 Decrease Volume

The steps that are planned to complete the project are as follows:

1. Connect the Arduino which consists of 5v power supply to laptop through USB device

2. Connect the reflectance sensors to the Arduino board and test to ensure it is working properly.

3. Develop the program and dubbed into the Arduino Software

4. The code loaded in Arduino finds the respective keyword for the distance found and sends it to
Windows OS.
5. Python code that runs in the background recognizes the keywords and generates the corresponding
virtual keystrokes for Windows

3.4 Advantages of Proposed System:

 The main advantage of using hand gestures is to interact with computer as a non -contact human
computer input modality
 Reduce hardware cost by eliminating use of mouse
 Convenient for users those who are not comfortable with touch pad
 There is no need of sound to be created so no interruption of background noise
 A number of functions of computer can be operated by using ultrasonic sensor.
PRAGATI ENGINEERING COLLEGE Page 21
 COMPUTER CONTROL USING HAND GESTURE
 Using this technique it is easy to interact with the computer and there is no language barrier.
 By using this system we can control our laptop from a small distance and it can help to control laptop
in conference room presentation.

3.5 Applications:

 The hand gesture control computer can be used for VR (virtual reality), AR (augmented reality), and
3D Design.
 Gestures can be used in videogames to try and make the game players experience more interactive.
 Navigate in a virtual environment.
 Interact with 3D world, System control and image scaling.
 This type of hand gesture control of computers can be mostly used in Reading Sign Language.
 Communicating in a video conferencing.

PRAGATI ENGINEERING COLLEGE Page 22

 COMPUTER CONTROL USING HAND GESTURE

CHAPTER-4
HARDWARE IMPLEMENTATION

PRAGATI ENGINEERING COLLEGE Page 23

 COMPUTER CONTROL USING HAND GESTURE

4.1 Components Required:

 Arduino UNO x 1 
 Ultrasonic Sensors x 2 
 USB Cable (for Arduino) 
 Few Connecting Wires 
 A Laptop with internet connection 

4.2 The basic work of each element is as follow:-

4.2.1 Sensor:
 A device that measures or detects a real-world condition, such as motion, heat or light
and converts the condition into an analog or digital representation. An optical sensor detects the
intensity or brightness of light, or the intensity of red, green and blue for color systems.

4.2.2 Software Used:

To read hand gestures, will need to program an Arduino and design a Python code. As
mentioned earlier that Windows does not recognize commands given by Arduino, hence we use
Python to establish serial connection and turn the data received into virtual keystrokes.
These virtual keystrokes control the Hotkeys of VLC media player and web page
which in turn will control playback of video or audio being played and switch between two tabs.
The Arduino IDE software works well for programming the Arduino but for Python we need to
perform a few more steps after installing
Python 2.7
The steps are as follows:
(1) Upgrade pip function of Python

(2) Install the PyAutoGUI module using pip function

4.3Hardware and Software Requirements:

 4.3.1.HardwareRequirement :
1. HC-SR04 Ultrasonic Sensor
2. Arduino Uno
3. Laptop (Windows 7/8/10 OS)

PRAGATI ENGINEERING COLLEGE Page 24

 COMPUTER CONTROL USING HAND GESTURE

 4.3.2.Software Requirement:
1. Arduino IDE
2. Python 2.7.14
3. VLC Media Player

4.4 Circuit Diagram:

Fig (4.1): circuit diagram for hand gesture control computer

4.5 Software Implementation :

4.5.1Programming Guide:
The Arduino should be programmed to read the distance of hand from the US
sensor. The complete program is given at the end of this page; the explanation for the program is
given below. The ultrasonic sensor measure the distance between the sensor and the hand by
reading the value of distance we can arrive at certain actions to be controlled with gestures, for
example in this program I have programmed 5 actions as a demo.

Action 1: Place your hand in front of the Right Ultrasonic Sensor at a distance (between
15CM to 35CM) for a small duration and move your hand away from the sensor. From the
figure (4.5.2) .The gesture will Scroll down the Web Page or Decrease the Volume.

PRAGATI ENGINEERING COLLEGE Page 25

 COMPUTER CONTROL USING HAND GESTURE

Fig

Action 2: Place your hand in front of the Right Ultrasonic Sensor at a distance (between
15CM to 35CM) for a small duration and move your hand towards the sensor. From the
figure (4.5.3). The gesture will Scroll up the Web Page or Increase the Volume.

Fig
(4.3) shows scrolling up of web page

Action 3: Swipe your hand in front of the Right Ultrasonic Sensor. From figure (4.5.4) .The
gesture will move to the Next Tab.

PRAGATI ENGINEERING COLLEGE Page 26

 COMPUTER CONTROL USING HAND GESTURE

Fig

(4.4)shows moving to the next tab

Action 4: Swipe your hand in front of the Left Ultrasonic Sensor. From the figure
(4.5.4).The gesture will move to the Previous Tab or Play/Pause the Video.

Fig

Action 5: Swipe your hand across both the sensors (Left Sensor first). This action will
Switch between Tasks.

The Above Actions should be produced on the Python GUI output screen as shown below:

PRAGATI ENGINEERING COLLEGE Page 27

 COMPUTER CONTROL USING HAND GESTURE

Fig

Arduino code:
Let us see how the program is written to perform the above actions.  So, like all programs
We start with defining the I/O pins as shown below. The two US sensors are connected to
Digital pins 11, 10, 6 and 5 and are powered by +5V pin. The trigger pins are output pin
and
Echo pins are input pins.
The Serial communication between Arduino and python takes places at a baud rate of
9600

const int trigPin1 = 11; // the number of the trigger output pin ( sensor 1 )
const int echoPin1 = 10; // the number of the echo input pin ( sensor 1 )
const int trigPin2 = 6; // the number of the trigger output pin ( sensor 2 )
const int echoPin2 = 5; // the number of the echo input pin ( sensor 2 )
long duration;
int distance1, distance2;
float r;
unsigned long temp=0;
int temp1=0;
int l=0;

The function void find_distance (void); is used to returns the value in cm. we should not
trigger the both ultrasonic sensor at the same time. it might cause error result due to the
intraction of the both soundswaves. Here the pulseIn function won’t wait more than
5000us for the ultrasonic sound to come back.

PRAGATI ENGINEERING COLLEGE Page 28

 COMPUTER CONTROL USING HAND GESTURE

void find_distance (void)

{
digital Write(trigPin1, LOW);
delayMicroseconds(2);
digitalWrite(trigPin1, HIGH);
delayMicroseconds(10);
digitalWrite(trigPin1, LOW);
duration = pulseIn(echoPin1, HIGH, 5000);
r = 3.4 * duration / 2;
distance1 = r / 100.00;
digitalWrite(trigPin2, LOW);
delayMicroseconds(2);
digitalWrite(trigPin2, HIGH);
delayMicroseconds(10);
duration = pulseIn(echoPin2, HIGH, 5000);
r = 3.4 * duration / 2;
distance2 = r / 100.00;
delay(100);
}

We have to initialize the trigger and echo pins of both the sensor as input and output:
void setup()
{
Serial.begin(9600);
pinMode(trigPin1, OUTPUT);
pinMode(echoPin1, INPUT);
pinMode(trigPin2, OUTPUT);
pinMode(echoPin2, INPUT);
delay (1000);

}
Here the function find_distane () will stores the current distance measured by the
ultrasonic sensor in the global variable "distance1 and distance2". we have to place our
hands in front of the right sensor in the range between 15 to 35cm .

void loop()

PRAGATI ENGINEERING COLLEGE Page 29

 COMPUTER CONTROL USING HAND GESTURE

{
Find_distance();
if(distance2<=35 && distance2>=15)
{
temp=millis();
while(millis()<=(temp+300))
find_distance();

Here the below condition becomes true if we moves our hand away from the right sensor,
closer to the sensor, swipe in front of right sensor. Based on the action, here the word
“down”, ”up” or “next” will be sent out through serial port
if(distance2<=35 && distance2>=15)
{
temp=distance2;
while(distance2<=50 || distance2==0)
{
find_distance();
if((temp+6)<distance2)
{
Serial.println("down");
}
else if((temp-6)>distance2)
{
Serial.println("up");
}
}
}
else
{
Serial.println("next");
}
}

once if we placed our hands in front of the left sensor in the range between 15 to 35cm if the
condition becomes true.
else if(distance1<=35 && distance1>=15)
{
temp=millis();
while(millis()<=(temp+300))
{
find_distance();

PRAGATI ENGINEERING COLLEGE Page 30

 COMPUTER CONTROL USING HAND GESTURE

if our hand detects in the right sensor before 300 milli seconds this condition becomes true.
( usually it happens if we swipe our hand from left to right sensor ).it sends change serially
And it stores the variable 1 and break the loop.
if(distance2<=35 && distance2>=15)
{
Serial.println("change");
l=1;
break;
}
}

if we swipe our hand in front of left sensor then condition becomes true and enters into the loop then
the "previous" will send to serially.this loop will rotate untill we remove our hand infront of
The left sensor. This will avoid not to enter this if condition again.
if(l==0)
{
Serial.println("previous");
while(distance1<=35 && distance1>=15)
find_distance();
}
l=0;
}
}

Phython code:
Let us import all the three required modules for this project. They are pyautogui, serial
python and time.

The module PyAutoGUI will help us for programmatically control the mouse and the
keyboard.
With the help of PyAutoGUI, we can write a Python Program to mimic the actions of mouse
like
Left click, right click, scroll, etc. and keyboard like keypress, enter text, multiple key press,
etc.
Without physically doing them.

import serial #Serial imported for Serial communication

PRAGATI ENGINEERING COLLEGE Page 31

 COMPUTER CONTROL USING HAND GESTURE

import time #Required to use delay functions

import pyautogui
 Next we establish connection with the Arduino through COM port. In my computer the
Arduino

Is connected to COM 18. Use device manager to find to which COM port your Arduino is

Connected to and correct the following line accordingly.

Arduino Serial = serial.Serial ('com18', 9600) #Create Serial port object called Arduino Serial
Data

Time sleep(2) #wait for 2 seconds for the communication to get established

Arduino_Serial = serial.Serial('com12',9600)
while 1:
incoming_data = str (Arduino_Serial.readline())
print incoming_data
if 'next' in incoming_data:
pyautogui.hotkey('ctrl', 'pgdn')
if 'previous' in incoming_data:
pyautogui.hotkey('ctrl', 'pgup')
Here if incoming data is 'down' it performs "down arrow" operation which scrolls down the
page
if 'down' in incoming_data:
#pyautogui.press('down')
pyautogui.scroll(-100)
Here if incoming data is 'up' performs "up arrow" operation which scrolls up the page
if 'up' in incoming_data:
#pyautogui.press('up')
pyautogui.scroll(100)
Here if incoming data is 'change' performs "alt+tab" operation which switches the tab
if 'change' in incoming_data:
pyautogui.keyDown('alt')
pyautogui.press('tab')
pyautogui.keyUp('alt')
incoming_data = "";

PRAGATI ENGINEERING COLLEGE Page 32

 COMPUTER CONTROL USING HAND GESTURE

PRAGATI ENGINEERING COLLEGE Page 33

 COMPUTER CONTROL USING HAND GESTURE

Chapter-5
RESULT

Result:
The hand Gesture Control Computer kit was tested by our team members with 3 trials.
We tested the kit by playing a video in a VLC player and web Browser purpose. We Checked
for different commands like switching to a next tab, switching to a previous tab, scroll down
in a web page, scroll up in a web page, play/pause video in vlc player, increase volume and
decrease volume and switch between two tasks(chrome and VLC player).In The very first 1 st
trial, the commands: switching to a next tab, switching to a previous tab, scroll down in a web
page are Worked out perfectly.so we perform 2 more trials to workout the other commands
but it was not perfectly done. There is a need to update the code for such commands.

There is a great scope of updating the code to try with many functions. We can use
this basic code to surf in any file applications as opening a document, editing a text ,We can
use notepad. We can also implement automation techniques by updating the code as per
requirement.

PRAGATI ENGINEERING COLLEGE Page 34

 COMPUTER CONTROL USING HAND GESTURE

Conclusion:
 Gesture recognition is very challenging and interesting task in terms of accuracy and
usefulness in computer vision.
 Most important advantage is that we can efficiently interact with the application from a
distance without any physical restriction
 This prototype implements several applications that demonstrate the usefulness, validity
and flexibility of the system to interact with this information via natural hand gesture.
 The importance of gesture recognition lies in building efficient human machine interaction.
 Gesture recognition promises wide ranging applications in field from photo journalism
through medical technology to biometrics.

Future Scope:

PRAGATI ENGINEERING COLLEGE Page 35

 COMPUTER CONTROL USING HAND GESTURE

 Hand gesture technique is not only limited to gaming, using basic function of computer it can
be useful for medical applications.
 Hand gesture technique can work as input method between medical instruments and
human body as proposed.
 Rotation, background variations, illumination change and pose variation of hand makes the
problem more challenging.
 It can be used for operating each and every functions of computer

REFERENCES:

[1]Cost effective hand gesture computer control interface, Gergely Sziládi ; Tibor
Ujbányi; József Katona 2016 7th IEEE International Conference on Cognitive Info
communications

[2]Vision based computer mouse control using hand gestures , Sandeep Thakur; Rajesh
Mehra ; Buddhi Prakash, 2015 International Conference on Soft Computing Techniques and
Implementations (ICSCTI)

[3]A basic hand gesture control system for PC applications C. J. Cohen ;G. Beach
;G.Foulk Proceedings 30th Applied Imagery Pattern Recognition Workshop (AIPR 2001).
Analysis and Understanding of Time Varying Imagery

[4]A system for controlling personal computers by hand gestures using a wireless
sensor vice, Kaoru magishi; LeiJing; Zixue Cheng 2014 IEEE International Symposium
on Independent Computing (ISIC)

[5]CMSWVHG control MS Windows via hand gesture, Hafız Mohsin Abdul-Rashid;

Lehmia Kiran; M. Danish Mirrani; M. Noman Maraaj 2017 International Multi-topic
Conference (INMIC)

[6]Power point control using hand gesture recognition based on hog feature extraction
and k-nn classification, Tejashree P. Salunke; S. D. Bharkad,2017 International Conference
on Computing Methodologies and Communication (ICCMC)

[7]Distance Measurement using Ultrasonic Sensor and Arduino, CH. Neeraja Soni1,
CH.Sarita2, ShrikantMaheshwari3, Basantkumar Sahu4, Bhavishya Jain5, Garima
Shrivastava6 Student1, 2, 3, 4, 5, Assistant Profeesor 6 YITM, Rajnandgaon, India

PRAGATI ENGINEERING COLLEGE Page 36

 COMPUTER CONTROL USING HAND GESTURE

[8]Comfortable mouse control using 3Ddepth sensor, Szilvia Szeghalmy; Marianna Zichar;
Attila Fazekas, 2013 IEEE4th International Conference on Cognitive Info communications

[9]Hand gesture recognition Technology in Human-computer interaction, Xueyuan Li School

of Computer Science University of Birmingham

[10]Distance Sensing with Ultrasonic Sensor and Arduino, N. Anju Latha1 , B. Rama
Murthy2, K. Bharat Kumar 3 Department of Instrumentation, Sri Krishnadevaraya
University, Anantapur, A.P., India

APPENDIX:

ARDUINO CODE:
const int trigPin1 = 11; // the number of the trigger output pin ( sensor 1 )
const int echoPin1 = 10; // the number of the echo input pin ( sensor 1 )
const int trigPin2 = 6; // the number of the trigger output pin ( sensor 2 )
const int echoPin2 = 5; // the number of the echo input pin ( sensor 2 )
////////////////////////////////// variables used for distance calculation
long duration;
int distance1, distance2;
float r;
unsigned long temp=0;
int temp1=0;
int l=0;
////////////////////////////////
Void find_distance (void); // this function returns the value in cm.

/*we should not trigger the both ultrasonic sensor at the same time.
It might cause error result due to the interaction of the both sounds waves.*/
void find_distance (void)
{
digitalWrite(trigPin1, LOW);
delayMicroseconds(2);
digitalWrite(trigPin1, HIGH);
delayMicroseconds(10);
digitalWrite(trigPin1, LOW);
duration = pulseIn(echoPin1, HIGH, 5000); // here this pulsein function won’t wait more then
5000us for the ultrasonic sound to came back. (due to
this it won’t measure more than 60cm)
r = 3.4 * duration / 2; // calculation to get the measurement in cm using the time returned
by the pulsein function.
distance1 = r / 100.00;
/////////////////////////////////////////upper part for left sensor and lower part for right sensor

PRAGATI ENGINEERING COLLEGE Page 37

 COMPUTER CONTROL USING HAND GESTURE

digitalWrite(trigPin2, LOW);
delayMicroseconds(2);
digitalWrite(trigPin2, HIGH);
delayMicroseconds(10);
digitalWrite(trigPin2, LOW);
duration = pulseIn(echoPin2, HIGH, 5000);
r = 3.4 * duration / 2;
distance2 = r / 100.00;
delay(100);
}
void setup()
{
Serial.begin(9600);
pinMode(trigPin1, OUTPUT); // initialize the trigger and echo pins of both the sensor as
input and output:
pinMode(echoPin1, INPUT);
pinMode(trigPin2, OUTPUT);
pinMode(echoPin2, INPUT);
delay (1000);

}
void loop()
{
find_distance(); // this function will stores the current distance measured by the
Ultrasonic sensor in the global variable "distance1 and distance2".
if(distance2<=35 && distance2>=15) // if we placed our hands in front of the right sensor in
The range between 15 to 35cm this condition becomes true.

{
temp=millis(); // store the current time in the variable temp. (" millis " Returns
the no of milliseconds since the Arduino board began
running the current program )
while(millis()<=(temp+300)) // this loop measures the distance for another 300 milli
Seconds. ( it helps to find the difference between swipe
and stay of our hand in front of the right sensor )
find_distance();
if(distance2<=35 && distance2>=15) // this condition will true if we place our hand in front
Of the right sensor for more than 300 milli seconds.
{
Temp =distance2; // store current position of our hand in the variable temp.
while(distance2<=50 || distance2==0) // this loop will run untill we removes our hand in front
Of the right sensor.
{
find_distance (); // call this function continuously to get the live data.
if((temp+6)<distance2) // condition becomes true if we moves our hand away

PRAGATI ENGINEERING COLLEGE Page 38

 COMPUTER CONTROL USING HAND GESTURE

From the right sensor .here “temp+6 " is for calibration.

{
Serial.println ("down"); // send "down" serially.
}
else if((temp-6)>distance2) // this condition becomes true if we moves our hand
Closer to the right sensor.
{
Serial.println ("up"); // send "up" serially.
}
}
}
else // this condition becomes true, if we only swipe in
Front of the right sensor.
{
Serial.println ("next"); // send "next" serially.
}
}
else if(distance1<=35 && distance1>=15) // if we placed our hands in front of left sensor in the
Range b/w 15 to 35cm, condition becomes true.
{
temp=millis();
while(millis()<=(temp+300))
{
find_distance();
if(distance2<=35 && distance2>=15) // if our hand detects in the right sensor before 300 milli
seconds this condition becomes true.(usually happens if
we swipe our hand from left to right sensor)
{
Serial.println ("change"); // send "change" serially.
l=1; // store 1 in variable l.
break; // break the loop.
}
}
if(l==0) // this condition will become true, only if we swipe our
Hand in front of left sensor.
{
Serial.println ("previous"); // send "previous" serially.
while(distance1<=35 && distance1>=15) // this loop will rotate untill we removes our hand infront
Of the left sensor.
find_distance();
}
l=0; // make l=0 for the next round.
}
}
PYTHON CODE:
import serial # add Serial library for serial communication

PRAGATI ENGINEERING COLLEGE Page 39

 COMPUTER CONTROL USING HAND GESTURE

import pyautogui # add pyautogui library for programmatically controlling the

Mouse and keyboard.
Arduino_Serial = serial.Serial('com12',9600) # Initialize serial and Create Serial port object called
Arduino_Serial
while 1:
incoming_data = str (Arduino_Serial.readline()) # read the serial data and print it as line
print incoming_data # print the incoming Serial data
if 'next' in incoming_data: # if incoming data is 'next'
pyautogui.hotkey('ctrl', 'pgdn') # perform "ctrl+pgdn" operation which moves to the next tab
if 'previous' in incoming_data: # if incoming data is 'previous'
pyautogui.hotkey('ctrl', 'pgup') # perform "ctrl+pgup" operation which moves to the previous tab
if 'down' in incoming_data: # if incoming data is 'down'
#pyautogui.press('down') # performs "down arrow" operation which scrolls down the page
pyautogui.scroll(-100)
if 'up' in incoming_data: # if incoming data is 'up'
#pyautogui.press('up') # performs "up arrow" operation which scrolls up the page
pyautogui.scroll(100)
if 'change' in incoming_data: # if incoming data is 'change'
pyautogui.keyDown('alt') # performs "alt+tab" operation which switches the tab
pyautogui.press('tab')
pyautogui.keyUp('alt')
incoming_data = ""; # clears the data

PRAGATI ENGINEERING COLLEGE Page 40