International Journal of Research Publication and Reviews Vol (2) Issue (5) (2021) Page 18-22

International Journal of Research Publication and Reviews

Journal homepage: www.ijrpr.com ISSN 2582-7421

Cursor Movement by Object Detection Based on Image Processing

Nirvika Rajendra1, Aishwarya R1,Roopa Shree N K 1, Vibha C1, Bhaskar Rao2

1
Final year Students, Department of CSE, DSATM India
2
Assoc. Professor, Dept. of CSE, DSATM, India

AB STR ACT

Today, there have been an enormous growth in the field of computer vision technology and have been a lot of developments. The project will be an approach
for controlling the mouse movement using object detection. HCI is an interface where humans can virtually interact with computers easily. Here we use a
similar approach to interact with computer by implementing virtual mouse that performs the cursor movements such as a real mouse can. The software for
mouse functions is developed in Python Language, OpenCV and PyAutoGUI. We have used green colored object to perform actions such as cursor
movement of mouse. This method mainly focuses on developing a human computer interaction device virtually that uses a Web Ca mera in a cost effective
manner.

Keywords: Human Computer Interaction (HCI), Color Detection , PyAutoGUI, Webcam, Object Detection.

1. Introduction

The importance of computers is increasing constantly. Computers are being used for many purposes. Hardware devices such as mouse and keyboard are
often used to interact with the computers. Technologies are evolving day by day in today's world. One of such example is the Human- Computer Interface
(HCI). In a wired mouse , there is no scope to extend limit and should be carried everywhere where as in the wireless mouse, Bluetooth should be
installed with Bluetooth dongle in the system.
As touch screen is little expensive for average users, an alternative way for touch screen can be creation of a virtual human computer interactive device.
In recent years, a lot of interest has been growing towards providing more natural and human-centered means of interacting with computer. Computer
Vision-Based Mouse is a system that does not use any physical device or mouse for controlling the cursor of computer.
Our system basically uses image processing and object detection to control the mouse movement.The user will simply hold any green colored object,
place it in the viewing area of the webcam , and thus controlling the mouse by object detection .

2. Prior and Related work

Many of the researchers have tried to interact with computer through video devices and web camera and each of them used different approaches to make a
mouse cursor move without any hardware.

* Corresponding author.
E-mail address: [email protected]
 International Journal of Research Publication and Reviews Vol (2) Issue (5) (2021) Page 18-22 19

One of the method given by OnkarYadav[1] have control of mouse cursor using gesture recognition. In this paper hand gestures are used to control the
mouse cursor. The system uses a real time camera to control the mouse pointer and implement its operation. The goal of this project is to form a system
which will control the computer/laptop with hand gestures.
In this proposed work, various methods have been used and it can be classified as follows,

2.1 Skin Detection

Skin Detection is the process which detects the pixels which represents the skin color in an image.It is a fun damental step in gesture recognition, face
detection, and hand tracking.

2.2 Hand Contour Extraction

Once the skin segmented binary image is obtained, edge detection is performed to get the hand contour present in an image. Border finding, Canny edge
detection and Laplacian edge detection are some of the edge detection strategies.

2.3 Hand Tracking

Here in this step, the finger tip was used to control the movement of pointer .So to detect the finger tip the palm center should first be found.

2.4 Gesture Recognition

The proposed design uses the gesture recognition technique which is a combination of two methods, one method proposed by Yeo and another by Balazs.
Using the OpenCV inbuilt function “cvConvexityDefects”, the convexity defect for the hand contour was calculated and after obtaining the convexity
defects ,finger tip identification and number of fingers are the two main steps for gesture recognition.

2.5 Cursor Control

After recognition of the hand gestures, the specific mouse functions are mapped to different hand gestures. Nirmala Roy [2] proposed a paper which
controls the mouse cursor based on image segmentation and object detection. Here, in this paper, a real -time video using webcam is captured and
individual input frame is processed. The object which is to be tracked is selected and object boundary is detected using condensation algorithm. Thus, a
virtual mouse application has been developed by object tracking and implemented using a webcam. This has been implemented in Processing IDE using
Boof CV library. Here, object boundary is detected by contour tracking which uses condensation algorithm.

2.6 Object Tracking

It is a process where the moving objects are located using a web camera over time. There are many applications of object tracking, some of them are
traffic control, HCI, security, medical imaging, augmented reality. Target representation and localization, filtering and data association are the two
components of visual tracking system.

2.7 Condensation Algorithm

Condensation refers to Conditional Density Propagation. It is based on computer vision algorithm. Its application is to track the boundary of moving
objects by detecting them. It is the probabilistic algorithm that identifies the pixels in an image that makes the boundary of the object. Rafiqul Zaman
Khan [3] has proposed a paper where various methods for gesture recognition were discussed, they were neural network, fuzzy c- means clustering,
HMM, besides using orientation histogram for features representation. HMM tools are best for dynamic gestures and have been showing its efficiency in
robot control.

2.8 Feature Extraction

It is a process in which interesting parts of the image are obtained by dimensionally reducing and representing the large number of pixels in a image
efficiently. Kabid Hassan Shibly[4]. This system used technologies like motion detection, image comparison to perform selection of icons and mouse
indicator movements. Here a camera is used to develop a virtual mouse by gesture control. Mouse operations like selection of icons navigations and task
like right, left click, scroll up and down.

2.9 Color Detection and Masking

Color detection is a process which identifies the color pixels of the color caps on the fingertips. For masking, rectangular boundary such as boxes are made
around the color caps to detect and track them.

3. Problem Statement

Computer Vision-Based Mouse is a system which uses no physical device for controlling of mouse cursor of the computer. Here we will essential ly have a
green colored object in our hand. The video of the motion of green colored object has been captured by the web- camera which acts as a sensor. The green
colored object is tracked and using its motion, the cursor of the mouse is controlled. In order for the object to work, we will use green color within the
viewing area of the camera. The green color object should be placed in the Region of Interest(ROI). The video generated by the camera is detected and
analyzed using image processing and the computer cursor moves or displays its click events according to green color object movements.
20 International Journal of Research Publication and Reviews Vol (2) Issue (5) (2021) Page 18-22

4. Techniques Used

The proposed system is a computer vision based application which is an example of real time application system. It makes use of OpenCV for image
processing and image acquisition and PyAutoGUI for handling mouse control in order to replace the actual mouse with the green colored object.
A. Image Acquisition
Here an image sensor or web-cam can be used for detecting and capturing the information that is required to make an image. The digital representation of
the visual characteristics of the physical world is acquired by the image or the frame which was detected by the web-cam.
B. Image Processing
Image acquired are processed in this step. To remove the noise or any ir-relevent frequencies, the signals in the acquired images are filtered. If necessary,
images will be padded and transformed into a different space, to make them ready for the actual analysis.
C. Image Analysis
In this step the processed image is analyzed to extract useful information. It involves many important image properties like color recognition, pattern
identification, feature extraction, object recognition, image segmentation and motion tracking.
D. Morphologic Processing
Here, some set of simple basic operations are performed on the image shape. The operations are performed on binary images and it takes two inputs, one
is the original image, and another is the kernel or structuring element which decides the operations to be performed. Erosion and Dilation are the two basic
operators in the morphology process.
E. Image Segmentation
It is a process that partitions the digital image into group of pixels for multiple segments to represent the image in a simpler way so that it can be used to
analyze in an easier way.

Fig1: Block Diagram.

5. Methodology

A. Capture Video
In the proposed architecture, the input is the green colored object which is given to the web-camera. The camera reads the frame of the video. Here, the
web- camera which is used as a sensor will help to detect and track the object. Video is a collection of multiple visual images and is divided into Frame
Rate which ranges from 6 or 8 per second for the old mechanical camera and 120 or more frames per second for professional camera.
B. Read Frame from the video
Once the video is captured by the webcam, with the help of cam.read() function frames can be read from video ,where the cam is the output where the
video is captured. The image that we get is inverted after the frames are obtained by the OpenCV function. That means if we m ove the tracked object in
the right direction, the image of the pointer will move towards the left, so we need to flip the image frame.
C. Convertion to HSV Color space
HSV is a model which represents a color that remaps the rgb color into dimensions that makes humans easier to understand. The opencv inbuilt function
cv2.cvtColor(img,cv2.COLOR_BGR2HSV) is used for conversion of BGR to HSV color space.
D. Creating Mask
Masking may be also known as filtering, where we recalculate each pixel in an image based on given mask matrix.
E. Morphologic Operations
Morphological opening and closing operations are performed on the image. Opening is used for noise removal and is also known as erosion followed by
dilation. Closing is used in foreground objects for closing the holes present.
 International Journal of Research Publication and Reviews Vol (2) Issue (5) (2021) Page 18-22 21

F. Contour Extraction
It is simply a process where all the continuous points along the boundary which have same color or intensity are joined as a curve. It is helpful in detection
of contours, also helps to analyse the shapes.
G. Cursor Control
Here it is used to control the cursor movement by tracking the object without the use of physical mouse. The movement of the object that is the direction
of motion of object determines the movement of the cursor.

Fig 2: Flowchart

Comparison
TITLE OBJECTIVE TECHNOLOGIES ADVANTAGES DISADVANTAGES
USED

[1]Cursor Proposes a Virtual mouse Skin Detection, Hand contour Usage of pointing devices such as Gestures needs to be
Movement By application based on hand tracking Extraction, Hand tracking, Gesture mouse will be tougher in the case properly learnt and
Hand Gesture which is implemented by web cam. recognition. of mobile systems, so this project remembered by the user for
Mouse events left clicking benefits the mobile systems accuracy.
rightclicking and scrolling are without usage of physical mouse.
managed by gesture recognitions.

[2]Controlling Controlling the mouse and its Contour Tracking, Condensation Calculation of the center of object Detection of similar objects in
Mouse Cursor operations by detecting the object. algorithm adds for precision. It is useful in the background.
Based on Image drone technology and
processing and medical imaging.
Object
Detection
[3]Hand gesture Management of mouse movements Extraction method, Feature It helps in 3d modeling, numbers Application of neural
recognition :A and its tasks by gesture recognition extraction, gesture classification recognition, robot control. network classifier for
Literature review gesture classification is
time consuming and
increment in the number of
training data also increases
the time for classification.
[4]Design and Development of virtual mouse Gesture recognition, Color detection, Workspace is reduced and usage of Interaction with multiple
development of system that is based on HCI which Computer vision technology and extra devices such as mouse. window is not possible,
hand gesture uses hand gestures and computer Human Computer Interaction. This system helps for people shrinking and enlarging
based virtual vision, cursor movements and with no limb control. window is also not possible.
mouse functions are controlled by the Sign language for deaf and
hands with colored caps present on dumb people.
finger tips.

6. Conclusion

This project eliminated the need for a mouse or any physical device cursor control.The use of object detection, PyAutoGUI for the implementation of our
proposed work proved to be successful and the movement of the mouse is achieved with the high precision accuracy. This also led to better Humans
-Computer Interaction(HCI). It also has wide applications in the Augmented Reality ,modern Gaming and Computer Graphics. In the case of computer
graphics and gaming, this technology has been applied in modern gaming consoles to create interactive games where a person’s motions are tracked and
interpreted as commands. The main aim is to eliminate the use of any physical device that is mouse and make use of only web cam for controlling
computer which is readily available with the laptop. Though this project has many advantages, it has some drawbacks also. It can give an erroneous
response and may not work properly if the background image is clashed with the specified image. So it is advisable to use this technology where the
22 International Journal of Research Publication and Reviews Vol (2) Issue (5) (2021) Page 18-22

background light and the object color does not mix .System may run slower on certain computers of low resolution and computat ional capabilities. The
system might run slow if the camera is high on resolution, but this problem can be solved by reducing the resolution of the image.

REFERENCES

[1] A. Erdem, E. Yardimci, Y. Atalay, V. Cetin, A. E, “Computer Vision Based Mouse”, Acoustics, Speech, and Signal Processing, Proceedings. (ICASS).
IEEE International Conference, 2002.
[2] K. Pendke, P.Khuje, S. Narnaware, S. Thool, S. Nimje, “Computer Cursor Control Mechanism by Using Hand Gesture Recognitio n”, International
Journal of Computer Science and Mobile Computing, Vol.4 Issue.3, March- 2015,pg.293- 300.
[3] A Banerjee, A. Ghosh, K. Bharadwaj, H. Saikia, “Mouse Control Using a Web Camera Based on Colour Detection”, International Journal of Computer
Trends and Technology (IJCTT) – volume 9 number 1– Mar 2014.
[4] Yadav, A. Pandey, A. Singh, A. Kumar, “Computer Mouse Implementation Using Object Detection and Image Processing ”, International Journal of
Computer Applications. (IJCA), Volume 69-Number 21, 2013.
[5] D. Verma, et al., “Vision Based Computer Mouse Controlling Using Hand Gestures”, International Journal of Engineering Sciences & Research
Technology.(IJESRT), 7(6): June, 2018.
[6] Hojoon Park, “A Method for Controlling the Mouse Movement using a Real Time Camera”, Brown University, Providence, RI, USA, Department of
computer science, 2013.
[7] Chu-Feng Lien, “Portable Vision-Based HCI – A Realtime Hand Mouse System on Handheld Devices”, National Taiwan University, Computer Science
and Information Engineering Department.
[8] Kamran Niyazi, Vikram Kumar, Swapnil Mahe, Swapnil Vyawahare, “Mouse Simulation Using Two Coloured Tapes”, Department of Comp uter
Science, University of Pune, India, International Journal of Information Sciences and Techniques (IJIST) Vol.2, No.2, March 2012.
[9] Y. Zhou, G. Jiang & Y. Lin, “A novel finger and hand pose estimation technique for real-time hand gesture recognition,” in Pattern Recognition vol. 49,
pp. 102-114, January 2016.
[10] P. Kumari, S. Singh & V. Kr. Pasi, "Cursor Control using Hand Gestures," Recent Trends in Future Prospective in Engineering & Management
Technology 2016, International Journal of Computer Applications (0975 – 8887), 2016.
[11] S. S. Abhilash, L. Thomas, N. Wilson & C. Chaithanya, "VIRTUAL MOUSE USING HAND GESTURE," International Research Journal of
Engineering and Technology (IRJET), vol. 05 issue: 4, April 2018.
[12] A.Haria, A. Subramanian, N. Asokkumar, S. Poddar & J. S. Nayak, “Hand Gesture Recognition for Human Computer Interaction," 7th Internat ional
Conference on Advances in Computing & Communications, ICACC-2017, Cochin, India, Procedia Computer Science 115 (2017), pp.367–374, 22-24
August 2017.
[13] Rafiqul Z. Khan, Noor A. Ibraheem, (2012). “Survey on Gesture Recognition for Hand Image Postures”, International Journal of Computer And
Information Science, Vol. 5(3), Doi: 10.5539/cis.v5n3p110