Assistive Technology for Enhanced Human-Computer Interaction
Assistive Technology for Enhanced Human-Computer Interaction
Interaction"
Dr. Salih sufian Munther
Chapter one
Introduction
1.1 Introduction
1
in real time to control a non-intrusive human–computer interface. To replace
the traditional mouse with the human face and eye movements to interact the
Computer. It is to assist the physically challenged persons without hands to
use the computer efficiently and also easy.
Our system is real time which captures a movement of mouse cursor through
face detection and facial features. It overcomes the existing system by avoiding
the use of external hardware that caused serious eye damages. It uses a
template matching method for eye extraction instead of using hardware, even
as in previous system the short blinks of eyes were avoided or neglected. In
this system the hard blink is only used for selecting particular file or folder.
With eye detection it is first aim is to captured face for the movement of mouse
cursor. Then it reacts as the mouse does .
The main objective of technology is to help people and facilitate their work,
and therefore within this context, many of the techniques and aids to disabled
people to use modern technology. The issue of communication with people
2
with disabilities is still fraught with weakness and inadequacy, although the
technologies continue to compensate for the incapacity of the disabled and to
meet their need to communicate with others. This research is a new
contribution in this area, where it is mainly directed to people with
quadriplegia who can control the mouse of the computer through facial
movements, and enables them to write on the computer. In our research, we
will discuss several chapters that show us how to design and implement a
software system that ensures the use of mouse and keyboard for people who
can not physically use them by recognizing the human face and moving the
mouse arrow with its movement with individual or double recognition through
a wink of the eyes, The screen in the form of a keyboard simulates the shape
and function of the physical keyboard but clicks through the mouse arrow.
The mere fact that we talk about disabilities and that people with disabilities
are deprived of the use of life technology has been very interesting to us. We
followed this new area and read articles about it, we liked its applications, and
we saw that the door is still open to any of our needs to help the disabled, so
we decided to be our research area. As we went along this path, we conducted
research and prepared theses, which necessitated attention to the subject of
control of the chair of the elderly through the movement of the pupil, and we
thought of a new contribution in this range.
Since the field of research is the use of the computer through facial
movements, we have followed the scientific and programmatic approach in the
application of the conflict between the face and the computer. Our plan of
3
action was the process of detecting the human face through the camera in the
computer in one of the algorithms as will be shown later, and then moved to
the second phase is the detection of the eyes, and then we have drawn the
features that distinguish the eye whether open or closed, we used these features
in the right click, left and double. As we shall see, we used the process of
processing the image and computer vision, and this was done using Visual
Studio to finally get an application that enables patients with quadriplegia to
use the computer. We followed in our work these standards and scientific
steps, and informed us before the previous studies and methods used to reach
the result and benefit from it, and detailed in the following chapters.
Chapter two: This chapter is talking about the help technique for people
with special needs, the operative software for typed text, font zoom
programs and the devices used in the project.
5
Chapter three: This chapter includes anatomical brief and spinal cord
infections, composite disability, quadriplegia and their causes, effects and
treatment.
6
Chapter two
2.1 Introduction:
Blindness: A person whose sharpness is less than 60/6 meters (200/20 feet)
or less than 20 degrees.
7
Poor sight: A person whose sharpness is between 26/6 - 60/6 m (20/80,
200/20 ft) with the strongest eyes after making possible corrections.
2.3 Software
8
and then reads the details of the page content of the links, images, and
punctuation. The program that provides this service can automatically scroll
the page while browsing.
9
2.4 Devices
The electronic line is a device with a row of special cells made of plastic or
metal dots. These points are controlled by the computer and move up or
down to display Braille characters on the computer screen. This type of
device is described as a "revolving device" because it changes as the user
moves and navigates the screen, usually placed next to the computer
keyboard, which works in conjunction with desktop or laptop computers to
act as an input unit and interact with the computer through A set of Perkins
input keys, linked to its interface with the computer-based screen reader
software, and also acts as a data output unit by translating data and
outputting them in matrix points that enable the blind to read the contents of
the computer in Braille.
11
Portable notebook with electronic line
This device is a laptop for the blind and visually impaired. It enables them to
perform a variety of tasks in order to meet their different needs in work,
study or reading environments. The most important tasks are writing, saving,
internet access and surfing. During reading it in a Braille way or listening to
it through the loudspeaker and then visually disabled can retrieval and
archiving study files.
11
Braille printing
Any text written and saved is printed in the depth of the txt or doc. Through
special printers that convert the text from normal language in to Braille.
This technique provides the visually disabled by the material after hearing it
in a way that can be retained and reviewed as he wishes.
12
2.5 Hearing Impairment
Deafness: a person who suffers from a loss of 70 decibel and above and
causes him disability of receive or transmit speech using or without the use
of hearing aid.
13
the use of multimedia techniques through the websites there is a set of
instructions for activating the multimedia for the disabled for example
activate the video clips in the sign language and write the audio explanation
in the user language for the video category, such as activating the audio files
in an assistant that includes the content of the audio file.
14
With the advancement of science and assistive medical equipment, patients
with physical disabilities participated in the design and perseverance of their
normal life and functioning. Among these patients, Stephen Hawking,
despite his 19 years of age, was one of the most prominent theoretical
physicists at the level worldwide. The following cases represent the majority
of physical disabilities:
Amputation.
Severe injuries
Muscular dystrophy
Cerebral palsy.
Multiple sclerosis
Spina bifida
There are suitable input devices and alternative to the keyboard that are used
with the normal or mouse which is connected to the computer to make it
more suitable for students with special needs, especially those who suffer
from problems in the coordination of musculoskeletal or cannot get the
mouse and control. And input devices Alternatives:
15
control. However, but controlling the cursor by rotating the ball does not
give it a chance to click.
• Touch screen: The idea of working of the touch screen based on the
development of transparent screen sensitive to the front of the computer and
the learner can use the computer and deal with files and lists easily.
16
A humidifier: can be boosted by a sensor of air and used as an infidel by
blowing the mouth of the disabled with quadriplegia. It is activated with
software for this purpose. A famous example is the free Breath Bird
program. My father works on the iPad in English and Japanese only. When
the program is run, the timeline contents of the person's Twitter page are
displayed automatically updated on the right side of the program. On the left
side a writable box is displayed and a virtual keyboard divided into five rows
and in order to write, the program lights the first character in each row. If the
character to be written is in the class, the user will blow the microphone to
lock the row, and then the program will shade each character in the row
respectively to select it. The process of interaction in this way may seem
tired and difficult, but from the point of view of the target group is certainly
a good step in order to integrate them into society.
17
• Bigkeys LX: is designed for large Big Key, Large-format keyboard with
large letters to facilitate the use of the computer for disabled and those with
mobility problems. Many technology companies have released computer
types for the help of quadriplegic patients, where the continent has been
dispensed with and replaced with an indicator on the camera and the
keyboard is customized so that it appears on the screen and is clicked by the
previous cursor or by voice commands. The technology that talked about
previously is rather expensive and therefore aimed at this project to improve
similar method to control the mouse through the digital camera, which we
will review it in the rest paragraphs.
18
Chapter three
Anatomical brief
1.3 Introduction
The spine, one of the most important and effectiveness in the human body
because it is the main axis, which the human depends on with the movement
and the situation, and has another importance lies in the protection of second
most important organ in the human body ''the spinal cord'', Everyone knows
that the backbone is one of the basic means of protecting the spinal cord
from shocks and external mechanisms.
The spine is containing spinal cord and holding the head and trunk. It is
connecting with rips and muscles. One of the most important point we must
know in our project is that the spinal cord ends in the level of first lumbar
vertebrae, therefore any damage below this site may result in hemiplegia.
19
Figure (3_1) spinal cord structure.
21
of injury and hands are not affected at all four quadruple means the
weakness of the four parties.
Anyone with spinal cord injuries is exposed to the young child - mothers -
parents - older adolescents, and the age of people exposed to such injuries is
15-29 years. The ratio of women to men is 4:1, most of which occur to
young people.
These injuries fall into two basic categories: those related to injuries or
distant from them, most of which fall within the first category and their
causes
- Falling: 19%.
- Violence: 3%.
21
3.5 Spinal Cord Injury effect
Any damage to the spinal cord is serious. The patient may have the
following symptoms:
- The inability to feel cold or heat in the area under pressure under the
affected area directly, and the arrival of blood to these places, which leads to
damage to skin cells.
- The inability to feel the desire to urinate when the bladder filled his leg
involuntarily. Muscle spasm down the affected area.
22
3.6 Treatment of spinal cord injuries
At first, a person cannot practice his normal life when he is aware of the
injury. He cannot go to school, university, for work, Exercise or shopping.
And after a short period of time he can return to his normal life, while
providing the right treatment and in the absence of any other repercussions
for the injury. A person suffering from hemorrhagic paralysis can return to
normal life and return to self-reliance within 4-12 months. Quadriplegia
takes a year or more. Unfortunately, at the present time, any spinal cord
injury is a permanent injury and there is no cure. These people do not
recognize any disability and do not rely on others for work. But there is
something important that no one can live away from The others do not resort
to them and rely on them in some things. The question is not what we can do
or do not do, but how to take care of what we do even if we are healthy.
3.7 Disability
Motor disability is not confined to the injury of the paralyzed person, there
are other injuries related to those that occur in the nerves, and the causes of
this type of disabilities: a defect in the electrical messages emitted from the
brain and lose the ability to reach the muscles correctly, as the muscles are
which move joints.
Where each muscle of fuel and brake on each side to be able to stretch and
shrink, and sometimes with brain injuries, these messages are inhibited and
cause the stimulation of one aspect of these muscles in excess, and this King
(exhausted) is not conductive electricity to overload it and thus affected
23
movement Muscles, if not reached the nerve message of these muscles have
a severe reaction.
3.7.1 Quadriplegia
The radical development of the treatments and their results was the result of
the realization that the fate of the injured does not depend on the integrity
and integrity of the spinal cord. Even without repairing the same
24
neurological damage, preventing complications and restoring functional
functioning allow for healing and congestion. The purpose of medical
treatment Occupational dysfunction is related to the extent, severity and
severity of the injury. Complete damage to the upper part of the cervical
spinal cord may result in dependence and dependence on the kidneys to help
others, and impede spontaneous self-breathing. In contrast, partial damage or
damage to the lower part of the cervical spinal cord can allow the patient to
perform his functions almost independently and be able to walk. The
involuntary contraction of the muscles, which accompanies paralysis and is
called spasm, mostly exacerbates the job disorder, but it may help in
functionality, sometimes.
among the causes of quadriplegia and the factors that cause damage to the
spinal cord: direct injuries, narrowing of the spinal canal and rupture of discs
between the paragraphs, multiple sclerosis, tumors, inflammation and
vascular disease. The quality of the agent causing the damage and the extent
of his response to the medical treatment have an impact on the extent and
duration of the damage. But as soon as we stir it, the damage will have dire
consequences, whatever the fall.
25
Chapter four
Theoretical foundations
Of the research
4.1 Computer vision
26
carried out in programs or devices. The cooperation between the fields of
physiological and computer vision study has developed a profound
understanding of both fields.
Motion Estimation.
Image Restoration.
Object recognition.
Computer vision systems vary widely and are distributed between large and
complex systems that perform general and comprehensive tasks, and
between small systems that perform dedicated and simple tasks. Most
computer vision systems, however, mainly include the following elements:
27
to many physical measurements such as distance, absorption, or
reflection of electromagnetic waves.
reduce noise to ensure that the sensor does not provide any
information.
3. Feature extraction.
Image parameters are obtained at different resolution levels from the image
data itself.
It is possible to get more complex features related to the colors and shapes in
the picture.
28
Select a set of distinct mark points.
divide one or more images containing the region containing the object
of interest.
B. High-level operations: At this stage, the input data is a small set of data,
for example a set of points or area of the image that is suspected to
contain the subject matter and the remaining processes, do the following:
Ensure that the data obtained agree with the assumptions of the
proposed application.
29
4. Knowledge – Based: This algorithm is based on the knowledge of the
facial components in the formation of the faceted templates and uses a
basic shape to locate the face.
Using the first face detection algorithm using the HAAR files inside the
EMGU library used in computer vision and HAAR files is a medical
platform for face shape, we will present the main steps in this algorithm,
noting that the K objects used for detection are vertical and have a front end
and are surrounded by the same lighting conditions.
31
4.2.1 Template matching algorithm
Start
Face tracking
no
Copare with
no The last
sample mass
s
yes
Delet the
detected
face
After you get the image from the camera, we do the following:
A. Skin Segmentation: determining the color of the skin in the images
is a well-known and useful process in the face detection. The input
material is in mode of RGB (Red - Green - Blue) image, and
because of this mode the picture is greatly affected by the lighting
31
conditions so face detection through it will cause problems, so
convert the image to one of the following modes:
HSV (Hue - Saturation - Value):
H: refers to pure color, which is a number between zero and one.
S: refers to the degree of white color, is a number between zero and one,
White takes zero value.
V: Indicates the degree of color darkness, is a number between zero and one,
black takes zero value.
The following figure which called Single_ Hexcone Model illustrates the
distribution of the values of the contest.
32
YIQ (Yellow - In -phase - Quadrature):
Y: Express the degree of illumination.
I, Q: Expressing the degree of coloring, their values between zero and one.
The convertion between RGB and this mode by using the following
equation:
( ) ( )( )
( ) ( )( )
33
For 164 faces with in seven images, each of Cb and Cr of each pixel of skin
color, their values are shown in Figure (4_3) and their statistical distribution
is shown in Figure (4_4) (a|) and (b)
34
Figure (4_4) (a) statistical distribution for Cb
35
Figure (4_5) The experimental image
36
37