A portable wireless eye movement-controlled Human-Computer

Interface for the Disabled

Xiaoxiang Zheng, Xin Li, Jun Liu, Weidong Chen, Yaoyao Hao

Abstract— Human-Computer Interface (HCI) has become have been developed to help these people for
an important area of research and development for the communication. Unlike traditional HCIs (a keyboard, or a
disabled. A portable wireless eye movement-controlled mouse, etc.), modern HCIs have played an important role in
Human-Computer Interface which can be used for the disabled the area of rehabilitation.
who have motor paralysis and who cannot speak in multiple
HCIs can be divided into cortical (all interfaces that
applications (such as communication aid and smart home
applications) is described here. This Interface consists of four exploit information collected from the human brain cortical
major parts: (1) surface electrodes, (2) a two-channel amplifier, relays) and non-cortical (all interfaces that do not access the
(3) a laptop (or a micro-processor), and (4) a ZigBee wireless signals generated by the human cortex directly) [1]. The
module. Horizontal and vertical Electro-Oculography (EOG) signals that drive the cortical interfaces can be single or
signals are measured using five surface electrodes placed on multiple unit recordings, local field potentials (LFPs),
the head .The vertical electrodes are placed about 1.0 cm above
electrocorticography (ECoG) and electroencephalography
the right eyebrow and 2.0 cm below the lower lid of the right
eye, the horizontal electrodes are placed 2.0 cm lateral to the (EEG) [2, 3]. The signals that drive the non-cortical
each side of outer canthi and the last electrode is placed on interfaces are measured in the peripheral nervous system, on
user’s forehead to serve as a ground. The two-channel the muscles, or are the result of muscular activity (change of
amplifier is comprised of instrumentation amplifiers, body posture or physical interaction of the body with the
band-pass filters and shift circuits. The EOG signals are interface) (See details in [1]). However, the disabled with
sampled at the rate of 250Hz and then sent to a laptop or a
severe paralysis and patients who need intensive care may
micro-processor for signal processing which is based on the
method of mathematical morphology to recognize the direction not be able to speak, and the eye muscles are the only
of eye movements and voluntary eye blink. The ZigBee wireless muscles they can control. For these people, methods based
communication technology, which is proved to be reliable, on eye movement or blinking can be selected.
low-power and cost-efficient, is used in the portable interface. Several methods have been developed for detecting eye
The subjects can control the wireless device or move a cursor movement and blinking, including scleral search coils [4, 5],
over a screen by using this interface. The delay of this interface
electrooculography (EOG) [6-15], infrared oculography
is less than 0.5s and errors are very limited. This interface
provides a flexible method for the disabled to improve the life [16-18], and image based methods [19-21]. The EOG based
quality. methods are relatively more convenience, low-cost and easy
to set up although the electrooculography sensors should
I. INTRODUCTION touch the user’s face and can be uncomfortable. And for

P ERSONS with severe diseases, such as amyotrophic many applications, the accuracy of EOG based methods is
lateral sclerosis (ALS), brainstem stroke, brain or spinal adequate. In this study, we focus on developing an EOG
cord injury, cerebral palsy, muscular dystrophies, multiple based HCI which is cheap, portable and non-invasive.
sclerosis, etc., have difficulty conveying their intentions and The cornea of the eye is electrically positive relative to the
communicating with other people in daily life. With the back of the eye, as observed from Emil du Bois-Reymond in
development of Human-Computer Interface (HCI), methods 1840s , whereas the application of EOG technology for the
disabled was not developed until 1990 [6]. In practice, the
signal is corrupted by various sources, resulting in a baseline
Manuscript received February 13, 2009. This work was supported by "drift" that obscures the eye movement signal, so researchers
National Natural Science Foundation of China (60703038, 60873125).
X. Zheng is with the Qiushi Academy for Advanced Studies, Zhejiang have tested different techniques for detecting and removing
University, Hangzhou 310027, PR China. She is also with the Department this drift from the EOG [7]. EagleEyes was developed in
of Biomedical Engineering, Zhejiang University, Hangzhou 310027, PR 1990s, which allowed a person to control the computer
China (phone: +86-571-87952865; fax: +86-571-87952838; e-mail:
[email protected]). simply by moving his or her eyes or head and was regularly
W. Chen and X. Li are with the Qiushi Academy for Advanced Studies, used by many people to control application programs to
Zhejiang University, Hangzhou 310027, PR China. They are also with the enter messages, create art, navigate the Internet, or play
College of Computer Science and Technology, Zhejiang University,
Hangzhou 310027, PR China (e-mail: [email protected]).
games [8, 9]. An EOG-based communication system
J. Liu and Y. Hao are with the Qiushi Academy for Advanced Studies, controlled by eight-directional eye movements was
Zhejiang University, Hangzhou 310027, PR China. They are also with the implemented only using three electrodes [10], and an
Department of Biomedical Engineering, Zhejiang University, Hangzhou electric wheelchair controlled by EOG has also been
310027, PR China (e-mail: [email protected]).

978-1-4244-3316-2/09/$25.00 ©2009 IEEE

Authorized licensed use limited to: Univ of Calif San Francico. Downloaded on March 18,2010 at 20:11:53 EDT from IEEE Xplore. Restrictions apply.
developed [11]. Recently, a wearable and light-weight electrodes placed around the eyes. The EOG value varies
system named EyeMote was reported, and EOG was used from 0.05-3.5 mV with a frequency range of about 0-100 Hz.
for tracking eye movements in stationary and pervasive In this paper, there are five electrodes in all which are
game scenarios in this system [12]. classified as horizontal, vertical and reference (ground)
In the present study, we describe a novel portable wireless electrodes. As showed in Fig. 1, the vertical electrodes are
eye movement-controlled HCI for the disabled. This placed about 1.0 cm above the right eyebrow and 2.0 cm
interface is a real-time communication control system based below the lower lid of the right eye, the horizontal electrodes
on EOG signals. There are two main differences between are placed 2.0 cm lateral to the each side of outer canthi. And
our system and others mentioned above: (1) designing and the last electrode is placed on user’s forehead to serve as a
implementing a mathematical morphology method to ground.
preprocess original EOG signals, (2) including a wireless If the eyes move left, horizontal EOG (HEOG) signal
module based on the ZigBee protocol to increase the scope which is the difference between signals collected by
of applications (communication aid, smart home electrode HEOL and HEOR acquires a positive voltage
applications, etc.) of this system. value. If the eyes turn right, HEOG signal changes into a
negative voltage value. Identically, if the eyes move from
II. METHODS the central position towards upside, vertical EOG (VEOG)
signal which is the difference between signal collected by
A. System Overview electrode VEOU and VEOL acquires a positive voltage
The system we have developed consists of four major value. If the eyes move downside, VEOG signal changes
parts: (1) five surface electrodes, (2) a two-channel amplifier, into a negative voltage value. An eye blinking can be
(3) a laptop (or a micro-processor), and (4) a ZigBee described by EOG signals as a peak in VEOG but a flat in
wireless module. Fig. 1 is the schematic diagram of this HEOG. We can distinguish the voluntary and involuntary
system and the whole system adopts the star topology which blinking by the value and duration of the peak mentioned
is also used in [22]. above. Fig. 2 shows EOG signals (after the amplifier) during
eye movement and blinking (voluntary and involuntary).

Fig. 2. EOG signals during eye movement and blanking. (a) HEOG signals.
(b) VEOG signals.

Fig. 1. Overview of the EOG-based wireless Human-Computer Interface.

C. Amplifier
In this system, horizontal and vertical EOG signals are
measured by five surface electrodes placed around eyes. The horizontal and vertical eye movement signals
After a two-channel amplifier, the EOG signals are sampled captured by the electrodes were then transmitted to a
at the rate of 250 Hz and then sent to a coordinator node two-channel amplifier which consists of (1) preamplifiers,
which is connected with a laptop or a micro-processor (2) band-pass filters, (3) shift circuits, (4) right-leg driven
through ZigBee wireless communication technology. The circuits and (5) power supply. The schematic of a single
software on the laptop or micro-processor recognizes the channel is shown in Fig. 3.
direction of eye movement and voluntary eye blinking. The preamplifier is a micro-power instrumentation
Programs (typewriter, patient assistant software, etc.) in amplifier (INA126, Texas Instruments Inc., Dallas, TX,
laptop or remote devices (TV, lamps, telephone, etc.) can be USA) for accurate and low noise differential signal
controlled by the recognized results. acquisition. The gain of the preamplifier is set to be 21 with
a single external resistor. The band-pass filter (0.01-41 Hz)
B. Electrodes and the Principle is provided with two Sallen-Key filters (One second-order
The cornea of the eye is electrically positive relative to the high-pass filter and one fourth-order low-pass filters). The
retina of the eye and the potential is slowly varying when following circuits are secondary amplifier with variable gain
eyes move. The standing potential can be measured by and shift circuit to transform the signal level into the range

Authorized licensed use limited to: Univ of Calif San Francico. Downloaded on March 18,2010 at 20:11:53 EDT from IEEE Xplore. Restrictions apply.
of 0 V to 3 V for adapting the following analog-to-digital 250Hz and transmitted. At the reception node, EOG signals
converter (ADC). Right-leg driven circuit connected with are transported to laptop with RS232-USB interface for
the reference electrode is used to reduce the common-mode signal processing. In the prototype software, the protocol is
components in the signal. Power for the board is supplied by based on a ZigBee stack called MSSTATE_LRWPAN
one common 6V battery, which is then transformed into ± which implements a ZigBee subset wireless stack. The
3.3 V with AMS1117 and MAX828 respectively. program in CC2430 is based on this protocol completely.
E. EOG Signal processing
Fig. 4 shows the flowchart of EOG signal processing. The
method is based on the mathematical morphology (MM),
differential and integral algorithms to recognize the
direction of eye movement and voluntary blinking. VEOG
signals are used to detect up/down movement and voluntary
eye blinking, while HEOG signals are used to detect
left/right movement.

Fig. 4. The flowchart of EOG signal processing.

MM Algorithm: The method of MM is widely used in

ECG signal processing and other fields [23]. It provides a
good way to remove drift and magnify feature of the signal.
Fig. 3. The schematic of a single channel. (a) Power supply. (b) The operators of MM are dilation, erosion, opening and
Preamplifiers. (c) Band-pass filters. (d) Shift circuits. (e) Right-leg driven closing (see details in [23]). Opening generally smoothes the
circuits.
contour of an object, breaks narrow isthmuses, and eliminate
thin protrusions. Closing also tends to smooth sections of
D. Wireless module contours and which generally fuses narrow breaks and long
The Wireless module takes responsibility for transmitting thin gulfs, eliminates small holes, and fills gaps in the
two-channel EOG signals from one node attached to the contour. In our work, we only used symmetrical sequences
user’s body to the coordinator node connected with the as the structuring element. The result of VEOG signals after
laptop. Meanwhile, the coordinator can send messages to MM filter is shown in Fig. 5b.
other remote controllers (TV, lamp, telephone, etc). The Differential Algorithm: The VEOG signals, after MM
ZigBee wireless communication technology, which is filter, are feed into the differential module implemented by
proved to be reliable, low-power and cost-efficient, is used (1). The result of this step is shown in Fig. 5c.
in this system. Compared with the popular Bluetooth and 9 9

Wi-Fi technologies, ZigBee has a wider range of

y (n) = ∑ x ( n + i + 10 ) − ∑ x ( n + i )
i=0 i =9 (1)
communication and supports more nodes. Most importantly,
Where x(n) is the VEOG signals after MM filter, and y(n)
the power consumption of ZigBee is very low. Therefore,
is the result after the differential module.
ZigBee is perfectly suitable in terms of data rate for the
Integral Algorithm: The difference of original VEOG
wireless transmission of physiological vital signs or even
signals (delay 2N points, N is the length of the structuring
continuous monitoring.
element in MM algorithm) and signals after MM filter can be
The module is established using CC2430 (Texas
used for eye blinking recognition. Because the peak value of
Instruments Inc., Dallas, TX, USA), which is a true
voluntary blinking is much larger than involuntary blinking,
System-on-Chip solution specifically tailored for IEEE
we can distinguish those two kinds of blinking by the
802.15.4 and ZigBee applications. The CC2430 combines
integral module using (2) and the threshold. The result of
RF transceiver with an industry-standard enhanced 8051
this step is shown in Fig. 5d.
MCU, 32/64/128 KB flash memory, 8 KB RAM and many 19
other powerful features. At the transmission node, analog y (n) = ∑ x ( n + i ) / 20
EOG signals from amplifiers are sampled at the rate of i=0 (2)

Authorized licensed use limited to: Univ of Calif San Francico. Downloaded on March 18,2010 at 20:11:53 EDT from IEEE Xplore. Restrictions apply.
 Where x(n) is the difference of original VEOG signals B. Application Software Test
(delay 2N points) and signals after MM filter, y(n) is the We have developed two application programs to test this
result after the integral module. system: the typewriter and the patient assistant software.
Decision Module: In Fig. 4, S1, S2 and S3 are the results
by the methods mentioned above. Threshold1 is the
voluntary eye blinking threshold, Threshold2 is the
involuntary eye blinking threshold, Threshold3 is the
movements (up/down) threshold, and Threshold4 is the
movements (left/right) threshold. We can distinguish eye
blinking (voluntary and involuntary) and eight-direction
movement through these thresholds.

Fig. 7. User interface of the two applications. (a) Typewriter application

test. (b) Patient assistant application test: the icon named “Nurse” was
selected by eye-movement (from the icon in the center), and the LED
(indicated the “Nurse” icon, controlled by the remote ZigBee module) was
lighted when voluntary eye blinking.

The typewriter user interface is showed in Fig. 7a. Users

make the cursor move up, down, left and right to select a
letter in the table. The letters selected (by voluntary eye
blinking) are showed above the table. The patient assistant
software is showed in Fig. 7b. In this application, users
move the cursor by eight-directional eye movements, and
the size of icon selected is enhanced. At the same time, the
LED which indicates the direction of eye movement is
Fig.5. Example of VEOG signal processing. (a) Original VEOG signals. (b)
Signals after MM filter. (c) The result after the differential module. (d) The lighted by the controlling of the remote ZigBee module.
result of integral algorithm. These two applications indicate that the portable wireless
eye movement-controlled we developed is doable. The
performance (bit rate and latency) of this system can be
III. RESULTS calculated as the following paragraphs.
A. Results of Signal Processing C. Bit Rate
The results of VEOG signal processing are shown in Fig. The bit rate of this system is calculated by using (3).
5. Here, Fig. 6 shows the results of HEOG signal processing. Where B is the bit rate per selection, N is the possible
Processing of HEOG signals is similar to VEOG except eye choices, and P is the accuracy. Bit rate per minute of the
blinking detection because the influence of VEOG is much system can be obtained from B multiplied by the number of
more obvious than HEOG. selections in a minute.
1− p
B = log 2 N + p log 2 P + (1 − p ) log 2
N −1 (3)
Several subjects were asked to test the patient assistant
software (N in this application is 8). They got accuracies all
above 80% and could make at least 10 selections per minute.
Therefore, the bit rate of this system is above 17.17 bits per
minute.
D. Latency
The Latency of this system is related with the method of
real-time signal processing. The delay time of mathematical
morphology, differential, and integral algorithms is T1, T2
and T3, respectively. The delay time of each algorithm is
Fig.6. The results of HEOG signal processing. (a) Original HEOG signals. calculated by using (4). Thus, T1 = 0.4 s, T2 = T3 = 0.08 s.
(b) Signals after MM filter. (c) The result after the differential module. T = N / Fs (4)
Where T is the delay time, N is the number of sampled
points delayed, and Fs is the sampling frequency.

Authorized licensed use limited to: Univ of Calif San Francico. Downloaded on March 18,2010 at 20:11:53 EDT from IEEE Xplore. Restrictions apply.
 Ttotal = T1 + max( T2 , T3 ) (5)
[6] J. R. LaCourse and F. C. Hludik, Jr., "An eye movement
communication-control system for the disabled," IEEE Trans. Biomed.
The sampling frequency is 250 Hz, and 100 sampled Eng., vol. 37, pp. 1215-1220, 1990.
points are delayed during the method of mathematical [7] D. W. Patmore and R. B. Knapp, "Towards an EOG-based eye tracker
for computer control," in Proceedings of the third international ACM
morphology. 20 sampled points are delayed both in the conference on Assistive technologies Marina del Rey, California,
processing of differential and integral algorithms. Thus, the United States 1998 pp. 197 - 203.
total time delay Ttotal calculated by using (5) is 0.48 s. [8] J. J. Tecce, J. Gips, C. P. Olivieri, L. J. Pok, and M. R. Consiglio, "Eye
movement control of computer functions," International Journal of
Psychophysiology, vol. 29, pp. 319-325, 1998.
IV. DISCUSSION [9] P. Di Mattia, F. X. Curran, and J. Gips, An Eye Control Teaching
Device or Students Without Language Expressive Capacity:
A. Factors Influencing Performance EagleEyes. Lewiston, NY: Edwin Mellen Press, 2001.
[10] Y. Kenji, H. Junichi, and M. Michio, "Development of EOG-Based
The EOG signals are mostly concentrated on the low Communication System Controlled by Eight-Directional Eye
frequency, especially near the DC component where lots of Movements," in Engineering in Medicine and Biology Society, 2006.
useful information locates. Therefore, the cut-off frequency EMBS '06. 28th Annual International Conference of the IEEE, 2006,
pp. 2574-2577.
of the high-pass filter should be set as low as possible (0.01 [11] R. Barea, L. Boquete, M. Mazo, and E. Lopez, "System for assisted
Hz in this system) otherwise the eye movement signals mobility using eye movements based on electrooculography," Neural
would decline rapidly rather than be hold for a long time. Systems and Rehabilitation Engineering, IEEE Transactions on, vol.
10, pp. 209-218, 2002.
Because of the MM method, the influences of the drift and [12] A. Bulling, D. Roggen, and G. Tröster, "EyeMote – Towards
other noise were reduced. However, slow involuntary Context-Aware Gaming Using Eye Movements Recorded from
blinking (duration above 0.8 s) would be recognized as eye Wearable Electrooculography," in Fun and Games, 2008, pp. 33-45.
[13] A. R. Kherlopian, J. P. Gerrein, M. Yue, K. E. Kim, J. W. Kim, M.
movement (Up), which is a mistake. So when using this Sukumaran, and P. Sajda, "Electrooculogram based system for
system, slow involuntary blinking should be performed as computer control using a multiple feature classification model," in
few as possible. Engineering in Medicine and Biology Society, 2006. EMBS '06. 28th
Annual International Conference of the IEEE, 2006, pp. 1295-1298.
B. Improvement of the System in Future [14] T. B. Krueger and T. Stieglitz, "A Naïve and Fast Human Computer
Interface Controllable for the Inexperienced - a Performance Study,"
Four thresholds which are measured in advance were in Engineering in Medicine and Biology Society, 2007. EMBS 2007.
set manually at the initialization stage of the software. 29th Annual International Conference of the IEEE, 2007, pp.
Clearly, it is time-consuming and may result in unnecessary 2508-2511.
[15] D. Borghetti, A. Bruni, M. Fabbrini, L. Murri, and F. Sartucci, "A
errors. Later development should make the thresholds set low-cost interface for control of computer functions by means of eye
automatically through a test program before use. Meanwhile, movements," Computers in biology and medicine, vol. 37, pp.
the thresholds can be updated during test period 1765-1770, 2007.
[16] H. Kimmig, M. W. Greenlee, F. Huethe, and T. Mergner,
auto-adaptively according to the user’s current state. For
"MR-Eyetracker: a new method for eye movement recording in
instance, the amplitude of the EOG signals would change functional magnetic resonance imaging," Experimental Brain
slowly in the latter stage because of fatigue. Research, vol. 126, pp. 443-449, 1999.
The current signal process program is implemented on a [17] F. Lo Castro, "Class I infrared eye blinking detector," Sensors and
Actuators A: Physical, vol. 148, pp. 388-394, 2008.
laptop which also provides user interface on the screen. If [18] S. Anders, N. Weiskopf, D. Lule, and N. Birbaumer, "Infrared
we do not need user interfaces (e.g. controlling remote oculography--validation of a new method to monitor startle eyeblink
devices), the process program can be carried out only in a amplitudes during fMRI," Neuroimage, vol. 22, pp. 767-770, 2004.
[19] A. H. Clarke, J. Ditterich, K. Druen, U. Schonfeld, and C. Steineke,
micro-processor which is integrated in the coordinator node. "Using high frame rate CMOS sensors for three-dimensional eye
Then the processed results were sent wirelessly to the remote tracking," Behav Res Methods Instrum Comput, vol. 34, pp. 549-60,
device which is attached with a ZigBee reception node. 2002.
[20] F. Karmali and M. Shelhamer, "Automatic detection of camera
translation in eye video recordings using multiple methods," in
REFERENCES Engineering in Medicine and Biology Society, 2004. IEMBS '04. 26th
[1] O. Tonet, M. Marinelli, L. Citi, P. M. Rossini, L. Rossini, G. Megali, Annual International Conference of the IEEE, 2004, pp. 1525-1528.
and P. Dario, "Defining brain-machine interface applications by [21] S. C. Kim, K. C. Nam, W. S. Lee, and D. W. Kim, "A new method for
matching interface performance with device requirements," Journal of accurate and fast measurement of 3D eye movements," Medical
Neuroscience Methods, vol. 167, pp. 91-104, 2008. Engineering & Physics, vol. 28, pp. 82-89, 2006.
[2] M. A. Lebedev and M. A. Nicolelis, "Brain-machine interfaces: past, [22] P. Qiyu and T. F. Budinger, "ZigBee-based Wireless Intra-oral
present and future," Trends Neurosci, vol. 29, pp. 536-46, Sep 2006. Control System for Quadriplegic Patients," in Engineering in
[3] J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. Medicine and Biology Society, 2007. EMBS 2007. 29th Annual
M. Vaughan, "Brain-computer interfaces for communication and International Conference of the IEEE, 2007, pp. 1647-1650.
control," Clinical Neurophysiology, vol. 113, pp. 767-791, 2002. [23] S. H. Oguz and M. H. Asyali, "A morphology based algorithm for
[4] D. A. Robinson, "A method of measuring eye movement using a baseline wander elimination in ECG records," in Biomedical
scleral search coil in a magnetic field," IEEE Trans Biomed Eng, vol. Engineering Days, 1992, Proceedings of the 1992 International, 1992,
10, pp. 137-45, 1963. pp. 157-160.
[5] R. S. Remmel, "An Inexpensive Eye Movement Monitor Using the
Scleral Search Coil Technique," Biomedical Engineering, IEEE
Transactions on, vol. BME-31, pp. 388-390, 1984.

Authorized licensed use limited to: Univ of Calif San Francico. Downloaded on March 18,2010 at 20:11:53 EDT from IEEE Xplore. Restrictions apply.