HUMAN COMPUTER

INTERACTION USING A
HEAD CONTROLLED MOUSE
MINI PROJECT

Guide: Shravanthi Ma’am

19911A12C1 Aditya Varala

19911A12B5 Bhargav Vinjamuri

20915A1214 Yash Prakash Sharma

INTRODUCTION

Computers

• With the proceeding of computer science and technology, the usage of

computer has brought about significant facilitation in every aspect of the
society.

• However, the common computer input devices are usually designed for normal
capable users, instead of elderly and disabled ones.

• The use of computers requires a mouse, a touchpad, a keyboard or other external devices.
• Users with upper limbs disabilities are incapable of controlling the mouse or keyboard easily,
which makes it extremely difficult for them to use a computer.

• For common computer users, the long-term usage of conventional input devices causes chronic
sore in hands, shoulders or neck, and greatly increases the risk of getting cervical or vertebral
spondylosis.

• The solution is non-contact-type Interaction, which is to determine the user’s action by collecting
and analyzing the user’s facial images with image processing techniques.

• Compared with the contact-type method, non-contact-type method is easy to use, free of any
contacting devices, and with better experience of users.
• In this project we proposed an algorithm to detect the movements of head and mouth using image
processing technology and presented a complete head-trace mouse system via webcam.

• Head Tracking Virtual Mouse is an application that uses the feature classification method to map the
mouse pointer on the screen to the movements of head and eye in frames through a camera.

• Our aim is to use this application mainly for the upper limb disabled who are unable to use the
traditional mouse. 
OBJECTIVE

The aim of this study is to emphasis on intelligent interaction between Human and computer
using computer vision.

Out of all the communication channels through which information can travel, vision provides a lot of
information that can be used for the recognition of human actions and gestures which can be analyzed
and applied to interaction purposes.

Our aim is to use this application mainly for the upper limb disabled who are unable to use the traditional
mouse. 
EXISTING SYSTEM
A vision-based human--computer interface
• The interface detects voluntary eye-blinks and interprets them as control commands.

• They employed image processing methods include Haar-like features for automatic face detection, and
template matching based eye tracking and eye-blink detection.

• The interface is based on a notebook equipped with a typical web camera and requires no extra light
sources. The interface application is available on-line as open-source software.

• Five facial movement patterns were detected by four electromyography (EMG) sensors, and classified
using myoelectric pattern recognition algorithms.  
 DRAWBACKS OF EXISTING SYSTEM
 Mounting devices: These systems needed a mounted device like lasers or cameras on the user which became
tedious.

 Biometric identification: The system used biometric identification for which the users had to register
themselves before using the system. It wasn‟t open for all which has been rectified by the proposed
application.

 Complex algorithms: The previous systems used many complex algorithms that needed a lot of calculations
to be done depending on various markers.

• Hence, our aim is to devise an application that will be cost effective and not be dependent on the biometrics
but on the feature classifications of the user. It should use less hardware and simpler algorithms. The
objective is to use such a system that will help the upper limb disabled who cannot use the traditional mouse
or keyboard.
PROPOSED SYSTEM
A hands-free human-computer interface
• It helps patients manipulate computers using facial movements.

• Facial movement patterns were mapped to cursor actions including

 movements in different directions.
 click.
 Scroll.

• The Algorithm presented in this project performs operations deeply centered on predicting the EYE
landmarks of a given face.
• For face detection, a machine learning based approach is used, Object detection algorithm

• This technique employs a Haar-features based approach for object detection, which makes the
rapid and accurate object detection possible.

• We defined five motions as the basis of head movements, namely,

 head left
 head right
 head up
 head down

• and closing of the eye performs the selected option of

 Left click
 Right click
 Scroll mode.
 ADVANTAGES OF PROPOSED SYSTEM

1. Quick response time

2. Customized processing

3. Small memory factor

4. Really helpful for disabled people

SYSTEM REQUIREMENT SPECIFICATION
 2.1 HARDWARE REQUIREMENTS:
 Processor : intel i3+
 Hard Disk : 500 GB+
 Monitor : 14’ Colour Monitor.
 WebCamera : Internal/External
 RAM : 4 GB+

 2.2 SOFTWARE REQUIREMENTS:

 Operating system : Windows 7+, Linux or Mac.

 Coding Language : Python.

 Front-End : Python.
SYSTEM DESIGN

System Architecture
Activity Diagram:
UseCase Diagram:
Deployment Diagram:
Python Packages Used
 Imutils –python:
 Face_recognition (hog and cnn)
• A Convolutional Neural Network (ConvNet/CNN).
 Retina face model
 Haar cascade face detection.

 OpenCV2:
 open source computer vision
 machine learning software library.

 PyAutoGUI:
 Used to control mouse cursor.
 DLib:
 Fast Face Detection.
 Predicts 68 face landmarks.

 Numpy:
 To store pixel coordinates of computer vision in (x,y) form.

 Scipy.spatial:
 To calculate euclidean distances in the 2D plane.
MODULES
 (1) Main interface: it sets and manages the startup dialogue, provides user access to user
action detection module, halt (sleep) module, mouse function module, and keyboard
function module.

 (2) Mouse simulation engine: it creates dynamic tool window, automatically changes the
size of the tool window according to the screen resolution; when the function menu is
triggered, it sends the view point of user to the mouse function module.
 (3) User action detection module: the fixation function in this module is defined to calculate
the gaze time of the user and determines the mouse coordinates directed by head movement.
In addition, this module decides the function or key selected by the user and triggers the
corresponding mouse events.

 (4) Halt (sleep) module: it determines whether to stop head tracking and enter the sleep mode
and determines whether to jump out of the sleep mode and restart the head tracking.
 (5) Mouse function module: It receives the view point of user from simulation engine and
transfers the coordinate to the fixation function within user action detection module, which
allows the system to perform directly at the view point after the function is selected. If a
second view point is required to finish the action, the view coordinate is transferred to the
simulation engine to execute the event. Besides, the module provides various virtual
mouse functions for users to operate computers with head movement. There are totally six
mouse functions in this module, namely, left-click (LEFT), continuous left-click (LEFT),
double left-click (DOUBLE), right-click (RIGHT), drag (IDRAG), and scroll (SCROLL).
IMPLEMENTATION

 This project is deeply centered on predicting the EYElandmarks of a given face.

We can accomplish a lot of things using these landmarks.

 Using these predicted landmarks of the face, we can build appropriate features that
will further allow us to detect certain actions, like using the eye-aspect-ratio to
detect a blink or a wink, using the mouth-aspect-ratio to detect a yawn or maybe
even a pout.
Eye Aspect Ratio (EAR)
 that Eye-Aspect-Ratio is the simplest and the most
elegant feature that takes good advantage of the
EYElandmarks. EAR helps us in detecting blinks and
winks etc.
 You can see that the EAR value drops whenever the eye
closes.
 We can train a simple classifier to detect the drop.
However, a normal if condition works just fine.
Something like this:
 if EAR <= SOME_THRESHOLD: EYE_STATUS =
'CLOSE'
Eye Detection Using OpenCV
 Step 1: Considering our prerequisites, we will require an image, to begin with. Later we
need to create a cascade classifier which will eventually give us the features of the face.

 Step 2: This step involves making use of OpenCV which will read the image and the
features file. So at this point, there are NumPy arrays at the primary data points. All we need
to do is to search for the row and column values of the face NumPyN dimensional array.
This is the array with the face rectangle coordinates.

 Step 3: This final step involves displaying the image with the rectangular face box. Screen
shots
ALGORITHM
The algorithm for detection of head movements is described as below
 (1) Initialization:
 User sits up in front of the computer. Let the Head-Trace Mouse run.
 The python libraries OpenCV and Dlib and Imutils will detect :
 LeftEye, RightEye
 Nose

 Mouth

 (2) Mouth Aspect Ratio:

 We use MAR formula to detect whether the mouth is open.
 If the mouth is open for more than 10 frames then it toggles:
 INPUT_MODE.
 (3) Standard head relocation:
 If you want to set new standard head postion just open your mouth again to disable
input mode. Then position yourself as you wish and repeat step (1).
 (4) Input Mode:
 Analyze the images after initialization. The head central coordinates of one image is
noted as (Cx, Cy). We compare (Cx, Cy) with new coordinates (Sx, Sy) to get the
following conclusions:
 If Cx - Sx> Kx, the judgment is that head moves left, abbreviated as left.
 If Cx - Sx < -Kx, right.
 If Cy - Sy < -Ky, up.
 If Cy - Sy> Ky, down.
 If | Cx - Sx | < Kx and | Cy - Sy | < Ky, standard head.
 (4.1) Mouse Activation:
 According to the direction detected in input mode, we map that direction
to the mouse movement using the pyautogui commands:
• pyag.moveRel(src, dest);
 Up: pyag.moveRel(0, -18)
 Left: pyag.moveRel(-18, 0)
 Right: pyag.moveRel(18, 0)
 Down: pyag.moveRel(0, 18)

 (4.2) Scroll Mode:

 To activate scroll mode close both eyes simultaneously for a sec.
 Once in scroll mode we map the scroll operation to the mouse based on
the direction found in input mode.
 Up: pyag.scroll(40);
 Down: pyag.scroll(-40);
 (5) Mouse Click Event:
 Here we use Eye Aspect Ratio(EAR) formula for left and right eye.
 If the difference between the EARs of 2 eyes is above threshold then:
 If smaller EAR is less than EAR Threshold(eye is winked) then perform click
action respective to the eye.
 LeftEye: pyag.click(button='left');
 RightEye: pyag.click(button=‘right');
Testing Table
  TESTING

DATASET Dataset Errors

Local
  Dataset not
read
Yes DF Error
 
 

PREPROCESSIN Cleaning dataset, if not it gives null

values
G
TRAINING  
YES

MODELS CNN Algorithm and

Haar Cascasdes Algorithm
(Used Algorithms)
Cursor Movement Testing Table
         
       
LEFT RIGHT UP DOWN
         
USER 1 YES YES YES YES

         
USER 2 YES YES YES YES

         
USER 3 YES YES YES YES

         
USER 4 YES YES YES YES
Output Screens
Step 1: Initializing. Step 2: Open mouth for input mode

Step 3: Move your head outside the rectangle to move the cursor.
CONCLUSION
 This project introduced the principles of a computer-human interaction system based on real-time state-
detection of head and mouth.

 It was proved that this system was capable of performing the majority of an ordinary mouse’s operations.

 We have implemented a system to access the mouse pointer on the computer screen using only EYEfeatures.
With the use of a camera and python technology, the system architecture is prepared.

 Having the ability of manipulating electronc devices can give patients full access to the digital world,
thereby increasing their independence and confidence and enriching their lives.
Future Scope:
• AI and ML are at its peak now and are still advancing as we speak.

• The future seems rich for head tracking.

• Head tracking technology also needs to be improved to increase the validity and reliability of the
recorded data.

• There are eye trackers for detecting the gaze but even they are far from perfect.

• Slowly with further advancements this project can definitely be a more feasable option and could
possibly replace the traditional mouse.
Thank You