EyePhone: Activating Mobile Phones With Your Eyes

Emiliano Miluzzo, Tianyu Wang, Andrew T. Campbell

Computer Science Department
Dartmouth College
Hanover, NH, USA

ABSTRACT touchscreen technology introduced with the Apple iPhone [1]

As smartphones evolve researchers are studying new tech- and recently followed by all the other major vendors, such
niques to ease the human-mobile interaction. We propose as Nokia [2] and HTC [3]. The touchscreen has changed
EyePhone, a novel “hand-free” interfacing system capable of the way people interact with their mobile phones because it
driving mobile applications/functions using only the user’s provides an intuitive way to perform actions using the move-
eyes movement and actions (e.g., wink). EyePhone tracks ment of one or more fingers on the display (e.g., pinching a
the user’s eye movement across the phone’s display using the photo to zoom in and out, or panning to move a map).
camera mounted on the front of the phone; more specifically, Several recent research projects demonstrate new people-
machine learning algorithms are used to: i) track the eye and to-mobile phone interactions technologies [4, 5, 6, 7, 8, 9,
infer its position on the mobile phone display as a user views 10]. For example, to infer and detect gestures made by the
a particular application; and ii) detect eye blinks that em- user, phones use the on-board accelerometer [4, 7, 8], camera
ulate mouse clicks to activate the target application under [11, 5], specialized headsets [6], dedicated sensors [9] or radio
view. We present a prototype implementation of EyePhone features [10]. We take a different approach than that found
on a Nokia N810, which is capable of tracking the position in the literature and propose the EyePhone system which
of the eye on the display, mapping this positions to an ap- exploits the eye movement of the user captured using the
plication that is activated by a wink. At no time does the phone’s front-facing camera to trigger actions on the phone.
user have to physically touch the phone display. HCI research has made remarkable advances over the last
decade [12] facilitating the interaction of people with ma-
chines. We believe that human-phone interaction (HPI) ex-
Categories and Subject Descriptors tends the challenges not typically found in HCI research,
C.3 [Special-Purpose and Application-Based Systems]: more specially related to the phone and how we use it.
Real-time and embedded systems We term HPI as developing techniques aimed at advanc-
ing and facilitating the interaction of people with mobile
phones. HPI presents challenges that differ somewhat from
General Terms traditional HCI challenges. Most HCI technology addresses
Algorithms, Design, Experimentation, Human Factors, Mea- the interaction between people and computers in “ideal”
surement, Performance environments, i.e., where people sit in front of a desktop
machine with specialized sensors and cameras centered on
Keywords them. In contrast, mobile phones are mobile computers with
which people interact on the move under varying conditions
Human-Phone Interaction, Mobile Sensing Systems, Machine and context. Any phone’s sensors, e.g., accelerometer, gy-
Learning, Mobile Phones roscope, or camera, used in a HPI technology must take
into account the constraints that mobility brings into play.
1. INTRODUCTION For example, a person walking produces a certain signature
Human-Computer Interaction (HCI) researchers and phone in the accelerometer readings that must be filtered out be-
vendors are continuously searching for new approaches to fore being able to use the accelerometer for gesture recog-
reduce the effort users exert when accessing applications on nition (e.g., double tapping the phone to stop an incoming
limited form factor devices such as mobile phones. The most phone call). Similarly, if the phone’s camera is adopted in a
significant innovation of the past few years is the adoption of HPI application [11, 5, 9] the different light conditions and
blurred video frames due to mobility make the use of the
camera to infer events very challenging. For these reasons
HCI technologies need to be extended to be applicable to
Permission to make digital or hard copies of all or part of this work for HPI environments.
personal or classroom use is granted without fee provided that copies are In order to address these goals HPI technology should be
not made or distributed for profit or commercial advantage and that copies less intrusive; that is, i) it should not rely on any external
bear this notice and the full citation on the first page. To copy otherwise, to devices other than the mobile phone itself; ii) it should be
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
readily usable with minimum user dependency as possible;
MobiHeld 2010, August 30, 2010, New Delhi, India. iii) it should be fast in the inference phase; iv) it should be
Copyright 2010 ACM 978-1-4503-0197-8/10/08 ...$10.00.
lightweight in terms of computation; and v) it should pre- ples: two HPI applications, one using the accelerometer, the
serve the phone user experience, e.g., it should not deplete other relying on the phone’s camera. Imagine exploiting the
the phone battery over normal operations. accelerometer to infer some simple gestures a person can per-
We believe that HPI research advances will produce a leap form with the phone in their hands, e.g., shake the phone
forward in the way people use their mobile phones by im- to initiate a phone call, or tap the phone to reject a phone
proving people safety, e.g., HPI techniques should aim to re- call [7]. What is challenging is being able to distinguish
duce the distraction and consequently the risk of accidents if between the gesture itself and any other action the person
driving for example, or facilitating the use of mobile phones might be performing. For example, if a person is running or
for impaired people (e.g., people with disabilities). if a user tosses their phone down on a sofa, a sudden shake
We propose EyePhone, the first system capable of track- of the phone could produce signatures that could be easily
ing a user’s eye and mapping its current position on the confused with a gesture. There are many examples where
display to a function/application on the phone using the a classifier could be easily confused. In response, erroneous
phone’s front-facing camera. EyePhone allows the user to actions could be triggered on the phone. Similarly, if the
activate an application by simply “blinking at the app”, em- phone’s camera is used to infer a user action [5][9], it be-
ulating a mouse click. While other interfaces could be used comes important to make the inference algorithm operating
in a hand-free manner, such as voice recognition, we focus on the video captured by the camera robust against lighting
on exploiting the eye as a driver of the HPI. We believe conditions, which can vary from place to place. In addition,
EyePhone technology is an important alternative to, for ex- video frames blur due to the phone movement. Because HPI
ample, voice activation systems based on voice recognition, application developers cannot assume any optimal operating
since the performance of a voice recognition system tends to conditions (i.e., users operating in some idealized manner)
degrade in noisy environments. before detecting gestures in this example, (e.g., requiring a
The front camera is the only requirement in EyePhone. user to stop walking or running before initiating a phone call
Most of the smartphones today are equipped with a front by a shaking movement), then the effects of mobility must
camera and we expect that many more will be introduced be taken into account in order for the HPI application to be
in the future (e.g., Apple iPhone 4G [1]) in support of video reliable and scalable.
conferencing on the phone. The EyePhone system uses ma- Hardware Challenges. As opposed to HCI applica-
chine learning techniques that after detecting the eye create tions, any HPI implementation should not rely on any ex-
a template of the open eye and use template matching for ternal hardware. Asking people to carry or wear additional
eye tracking. Correlation matching is exploited for eye wink hardware in order to use their phone [6] might reduce the
detection [13]. We implement EyePhone on the Nokia N810 penetration of the technology. Moreover, state-of-the art
tablet and present experimental results in different settings. HCI hardware, such as glass mounted cameras, or dedicated
These initial results demonstrate that EyePhone is capable helmets are not yet small enough to be conformably worn for
of driving the mobile phone. An EyePhone demo can be long periods of time by people. Any HPI application should
found at [15]. rely as much as possible on just the phone’s on-board sen-
The paper is organized as follows. In Section 2, we dis- sors.
cuss the challenges encountered in the development of HPI Although modern smartphones are becoming more com-
technology. Section 3 presents the design of the EyePhone putationally capable [16], they are still limited when run-
system followed by its evaluation in Section 4. The future ning complex machine learning algorithms [14]. HPI solu-
research direction are reported in Section 5. Section 6 dis- tions should adopt lightweight machine learning techniques
cusses related work and Section 7 finishes with some con- to run properly and energy efficiently on mobile phones.
cluding remarks.
3. EYEPHONE DESIGN
2. HUMAN-PHONE INTERACTION One question we address in this paper is how useful is a
Human-Phone Interaction represents an extension of the cheap, ubiquitous sensor, such as the camera, in building
field of HCI since HPI presents new challenges that need HPI applications. We develop eye tracking and blink de-
to be addressed specifically driven by issues of mobility, the tection mechanisms based algorithms [13, 17] originally de-
form factor of the phone, and its resource limitations (e.g., signed for desktop machines using USB cameras. We show
energy and computation). More specifically, the distinguish- the limitations of an off-the-shelf HCI technique [13] when
ing factors of the mobile phone environment are mobility and used to realize a HPI application on a resource limited mo-
the lack of sophisticated hardware support, i.e., specialized bile device such as the Nokia N810. The EyePhone algorith-
headsets, overhead cameras, and dedicated sensors, that are mic design breaks down into the following pipeline phases:
often required to realize HCI applications. In what follows, 1) an eye detection phase; 2) an open eye template creation
we discuss these issues. phase; 3) an eye tracking phase; 4) a blink detection phase.
Mobility Challenges. One of the immediate products In what follows, we discuss each of the phases in turn.
of mobility is that a mobile phone is moved around through Eye Detection. By applying a motion analysis tech-
unpredicted context, i.e., situations and scenarios that are nique which operates on consecutive frames, this phase con-
hard to see or predict during the design phase of a HPI ap- sists on finding the contour of the eyes. The eye pair is iden-
plication. A mobile phone is subject to uncontrolled move- tified by the left and right eye contours. While the original
ment, i.e., people interact with their mobile phones while algorithm [17] identifies the eye pair with almost no error
stationary, on the move, etc. It is almost impossible to pre- when running on a desktop computer with a fixed camera
dict how and where people are going to use their mobile (see the left image in Figure 1), we obtain errors when the
phones. A HPI application should be able to operate reliably algorithm is implemented on the phone due to the quality of
in any encountered condition. Consider the following exam- the N810 camera compared to the one on the desktop and
   

Figure 1: Left figure: example of eye contour pair re-

turned by the original algorithm running on a desk-   
top with a USB camera. The two white clusters
identify the eye pair. Right figure: example of num-
ber of contours returned by EyePhone on the Nokia
N810. The smaller dots are erroneously interpreted
as eye contours.
  
the unavoidable movement of the phone while in a person’s
hand (refer to the right image in Figure 1). Based on these
Figure 2: Eye capture using the Nokia N810 front
experimental observations, we modify the original algorithm
camera running the EyePhone system. The in-
by: i) reducing the image resolution, which according to the
ner white box surrounding the right eye is used to
authors in [13] reduces the eye detection error rate, and ii)
discriminate the nine positions of the eye on the
adding two more criteria to the original heuristics that filter
phone’s display. The outer box encloses the tem-
out the false eye contours. In particular, we filter out all
plate matching region.
the contours for which their width and height in pixels are
such that widthmin ≤ width ≤ widthmax and heightmin ≤
height ≤ heightmax . The widthmin , widthmax , heightmin ,
and heightmax thresholds, which identify the possible sizes template matching. The template matching function cal-
for a true eye contour, are determined under various exper- culates a correlation score between the open eye template,
imental conditions (e.g., bright, dark, moving, not moving) created the first time the application is used, and a search
and with different people. This design approach boosts the window. In order to reduce the computation time of the
eye tracking accuracy considerably, as discussed in Section template matching function and save resources, the search
4. window is limited to a region which is twice the size of a
Open Eye Template Creation. While the authors box enclosing the eye. These regions are shown in Figure 2,
in [13] adopt an online open eye template creation by ex- where the outer box around the left eye encloses the region
tracting the template every time the eye pair is lost (this where the correlation score is calculated. The correlation co-
could happen because of lighting condition changes or move- efficient we rely on, which is often used in template matching
ment in the case of a mobile device), EyePhone does not problems, is the normalized correlation coefficient defined in
rely on the same strategy. The reduced computation speed [18]. This coefficient ranges between -1 and 1. From our
compared to a desktop machine and the restricted battery experiments this coefficient guarantees better performance
requirements imposed by the N810 dictate a different ap- than the one used in [13]. If the normalized correlation co-
proach. EyePhone creates a template of a user’s open eye efficient equals 0.4 we conclude that there is an eye in the
once at the beginning when a person uses the system for search window. This threshold has been verified accurate
the first time using the eye detection algorithm described by means of multiple experiments under different conditions
above1 . The template is saved in the persistent memory of (e.g., bright, dark, moving, not moving).
the device and fetched when EyePhone is invoked. By tak- Blink Detection. To detect blinks we apply a thresh-
ing this simple approach, we drastically reduce the runtime olding technique for the normalized correlation coefficient
inference delay of EyePhone, the application memory foot- returned by the template matching function as suggested in
print, and the battery drain. The downside of this off-line [13]. However, our algorithm differs from the one proposed
template creation approach is that a template created in cer- in [13]. In [13] the authors introduce a single threshold T
tain lighting conditions might not be perfectly suitable for and the eye is deemed to be open if the correlation score
other environments. We intend to address this problem as is greater than T , and closed vice versa. In the EyePhone
part of our future work. system, we have two situations to deal with: the quality of
In the current implementation the system is trained in- the camera is not the same as a good USB camera, and the
dividually, i.e., the eye template is created by each user phone’s camera is generally closer to the person’s face than
when the application is used for the first time. In the fu- is the case of using a desktop and USB camera. Because of
ture, we will investigate eye template training by relying on this latter situation the camera can pick up iris movements,
pre-collected data from multiple individuals. With this su- i.e., the interior of the eye, due to eyeball rotation. In partic-
pervised learning approach users can readily use EyePhone ular, when the iris is turned towards the corner of the eye,
without going through the initial eye template creation phase. upwards or downwards, a blink is inferred even if the eye
Eye Tracking. The eye tracking algorithm is based on remains open. This occurs because in this case the majority
of the eye ball surface turns white which is confused with
1
The eye template is created by putting the phone at a dis- the color of the skin. We derive four thresholds: T1min =
tance of about 20 cm from the eyes. 0.64, T1max = 0.75, T2min = -0.53, and T2max = -0.45. These
 100
Table 1: EyePhone average eye tracking accuracy steady
for different positions of the eye in different lighting 90 walking

Eye Tracking Accuracy (%)

and movement conditions and blink detection aver- 80
age accuracy. Legend: DS = eye tracking accuracy 70
measured in daylight exposure and being steady; AS
60
= eye tracking accuracy measured in artificial light
exposure and being steady; DM = eye tracking ac- 50
curacy measured in daylight exposure and walking; 40
BD = blink detection accuracy in daylight exposure. 30
Eye position DS AS DM BD 20
Top left 76.73% 74.50% 82.81% 84.14%
10
Top center 79.74% 97.78% 79.16% 78.47%
Top right 80.35% 95.06% 60% 82.17% 15 20 25 30 35 40 45
Middle left 98.46% 97.19% 70.99% 74.72% Distance (cm)
Middle center 99.31% 84.09% 76.52% 79.55%
Middle right 99.42% 75.79% 65.15% 80.1%
Bottom left 98.36% 93.22% 78.83% 74.53% Figure 3: Eye tracking accuracy for the middle-
Bottom center 90.76% 71.46% 85.26% 67.41% center position as a function of different distances
Bottom right 84.91% 93.56% 78.25% 72.89% between the N810 and the eyes when the person is
steady and walking.

thresholds are determined experimentally and again under

different experimental conditions as discussed previously. If ing the device in their hand, represents the eye position on
we indicate with cmin and cmax , respectively, the minimum the phone display. It is evident that nine different positions
and maximum normalized correlation coefficient values re- for the eye are identified. These nine positions of the eye
turned by the template matching function, the eye is inferred can be mapped to nine different functions and applications
to be closed if T1min ≤ cmax ≤ T1max and T2min ≤ cmin ≤ as shown in Figure 4. Once the eye locks onto a position
T2max . It is inferred to be open otherwise. The results of the (i.e., the person is looking at one of the nine buttons on the
blink detection algorithm are discussed in Section 4. display), a blink, acting as a mouse click, launches the ap-
plication corresponding to the button. The accuracy of the
4. EVALUATION eye tracking and blink detection algorithms are reported in
In this section, we discuss initial results from the evalua- Table 1. The results show we obtain good tracking accuracy
tion of the EyePhone prototype. We implement EyePhone of the user’s eye. However, the blink detection algorithms
on the Nokia N810 [19]. The N810 is equipped with a 400 accuracy oscillates between ∼67 and 84%. We are studying
MHz processor and 128 MB of RAM2 . The N810 operat- further improvements in the blink detection as part of future
ing system is Maemo 4.1, a Unix based platform on which work.
we can install both the C OpenCV (Open Source Computer Artificial Light Exposure for a Stationary Subject.
Vision) library [20] and our EyePhone algorithms which are In this experiment, the person is again not moving but in
cross compiled on the Maemo scratchbox. To intercept the an artificially lit environment (i.e., a room with very low
video frames from the camera we rely on GStreamer [21], the daylight penetration from the windows). We want to verify
main multimedia framework on Maemo platforms. In what if different lighting conditions impact the system’s perfor-
follows, we first present results relating to average accuracy mance. The results, shown in Table 1, are comparable to
for eye tracking and blink detection for different lighting and the daylight scenario in a number of cases. However, the
user movement conditions to show the performance of Eye- accuracy drops. Given the poorer lighting conditions, the
Phone under different experimental conditions. We also re- eye tracking algorithm fails to locate the eyes with higher
port system measurements, such as CPU and memory usage, frequency.
battery consumption and computation time when running Daylight Exposure for Person Walking. We carried
EyePhone on the N810. All experiments are repeated five out an experiment where a person walks outdoors in a bright
times and average results are shown. environment to quantify the impact of the phone’s natural
Daylight Exposure Analysis for a Stationary Sub- movement; that is, shaking of the phone in the hand induced
ject. The first experiment shows the performance of Eye- by the person’s gait. We anticipate a drop in the accuracy of
Phone when the person is exposed to bright daylight, i.e., in the eye tracking algorithm because of the phone movement.
a bright environment, and the person is stationary. The eye This is confirmed by the results shown in Table 1, column
tracking results are shown in Figure 2. The inner white box 4. Further research is required to make the eye tracking
in each picture, which is a frame taken from the front camera algorithm more robust when a person is using the system
when the person is looking at the N810 display while hold- on the move.
Impact of Distance Between Eye and Tablet. Since
2
The reason we use the N810 instead of the Nokia N900 in the current implementation the open eye template is cre-
smartphone is because we obtain better quality video frames ated once at a fixed distance, we evaluate the eye track-
from the front camera. However, with its 600 MHz processor ing performance when the distance between the eye and the
and up to 1 GB of application memory the N900 presents a tablet is varied while using EyePhone. We carry out the
more powerful hardware than the N810. Hence, we expect
better performance (computation and inference delay) when measurements for the middle-center position in the display
running EyePhone on the N900. This is part of our on-going (similar results are obtained for the remaining eight posi-
work. tions) when the person is steady and walking. The results
Table 2: Average CPU usage, RAM usage, and com-
putation time for one video frame. The front cam-
era supports up to 15 frames per second. The last 
column reports the percentage of used battery by
  
EyePhone after a three hour run of the system.
CPU RAM Computation time Battery used after 3h
65.4% 56.51% ∼100 msec 40%   

  

are shown in Figure 3. As expected, the accuracy degrades
for distances larger than 18-20 cm (which is the distance be-
tween the eye and the N810 we currently use during the eye
template training phase). The accuracy drop becomes se-
vere when the distance is made larger (e.g., ∼45 cm). These
results indicate that research is needed in order to design Figure 4: EyeMenu on the Nokia N900. While look-
eye template training techniques which are robust against ing the display, a button is highlighted if it matches
distance variations between the eyes and the phone. the eye position on the display. The highlighted but-
System Measurements. In Table 2 we report the av- ton is ready to be “clicked” with a blink of the eye.
erage CPU usage, RAM usage, battery consumption, and In this example, the user is looking at the SMS but-
computation time of the EyePhone system when processing ton and the SMS keypad is launched by blinking the
one video frame – the N810 camera is able to produce up eye.
to 15 frames per second. EyePhone is quite lightweight in
terms of CPU and RAM needs. The computation takes 100
msec/frame, which is the delay between two consecutive in- particular, variations of lighting conditions or movement of
ference results. In addition, the EyePhone runs only when the phone in a person’s hand might make the one-time tem-
the eye pair is detected implying that the phone resources plate inaccurate by not matching the current conditions of
are used only when a person is looking at the phone’s display the user. A template created in a bright environment might
and remain free otherwise. The battery drain of the N810 work poorly to match the eye in a darker setting. Similarly,
when running the EyePhone continuously for three hours is an eye template created when the person is stationary does
shown in the 4th column of Table 2. Although this is not not match the eye when the person is walking. We observe
a realistic use case, since a person does not usually continu- the implications of the one-time template strategy in the re-
ously interact with their phone for three continuous hours, sults presented in Section 4. It is important to modify the
the result indicates that the EyePhone algorithms need to template generation policy in order for the system to be able
be further optimized to extend the battery life as much as to either evolve the template according to the encountered
possible. contexts if the template is generated in a context different
from the current one, or, create new templates on the fly for
4.1 Applications each of the encountered settings (e.g., bright, dark, moving,
EyeMenu. An example of an EyePhone application is etc.). In both cases the template routine should be fast to
EyeMenu as shown in Figure 4. EyeMenu is a way to short- compute and minimize the resources used.
cut the access to some of the phone’s functions. The set A second important issue that we are working on is a filter-
of applications in the menu can be customized by the user. ing algorithm that minimizes false positives, (i.e., false eye
The idea is the following: the position of a person’s eye is contours). One way to solve this problem is by using a learn-
mapped to one of the nine buttons. A button is highlighted ing approach instead of a fixed thresholding policy. With a
when EyePhone detects the eye in the position mapped to learning strategy the system could adapt the filter over time
the button. If a user blinks their eye, the application associ- according to the context it operates in. For example, a semi-
ated with the button is lunched. Driving the mobile phone supervised learning approach could be adopted, having the
user interface with the eyes can be used as a way to facilitate system evolve by itself according to a re-calibration process
the interaction with mobile phones or in support of people every time a completely new environment is encountered. In
with disabilities. order to be sure the filter is evolving in the right direction
Car Driver Safety. EyePhone could also be used to the user could be brought into the loop by asking if the result
detect drivers drowsiness and distraction in cars. While car of the inference is correct. If so, the new filter parameters
manufactures are developing technology to improve drivers are accepted, otherwise, discarded. Clearly, proper and ef-
safety by detecting drowsiness and distraction using dedi- fective user involvement policies are required, so prompting
cated sensors and cameras [22], EyePhone could be readily is not annoying to the user.
usable for the same purpose even on low-end cars by just
clipping the phone on the car dashboard. 6. RELATED WORK
There are a number of developments in HCI related to
5. FUTURE WORK mobile phones over the last several years [4, 5, 23, 6, 7, 8, 9,
We are currently working on improving the creation of the 10]. Some of this work exploits accelerometers on the phone
open eye template and the filtering algorithm for wrong eye in order to infer gestures. The PhonePoint Pen project [4]
contours. The open eye template quality affects the accu- is the first example of a phone being transformed into a
racy of the eye tracking and blink detection algorithms. In pen to write in the air in order to quickly take small notes
 without needing to either write on paper or type on a com- [4] S. Agrawal, I. Constandache, S. Gaonkar, and R.R.
puter. A similar project evaluated on the iPhone has re- Choudhury. PhonePoint Pen: Using Mobile Phones to
cently been proposed [8]. The uWave project [7] exploits a Write in Air. In MobiHeld ‘09, pages 1–6. ACM, 2009.
3-D accelerometer on a Wii remote-based prototype for per- [5] eyeSight. http://www.eyesight-tech.com/.
sonalized gesture recognition. Phones can be used as radars [6] DoCoMo eye control. http://tinyurl.com/ycza2jd.
as in [10] where proximity based sensing is used to infer [7] J. Liu, L. Zhong, J. Wickramasuriya, and
speed of surrounding subjects. V. Vasudevan. uWave: Accelerometer-Based
Custom-built sensors interfaced with a mobile phone are Personalized Gesture Recognition and its
used for driving the phone user interface by picking up eye Applications. In Pervasive and Mobile Computing, vol.
movement [6] or converting muscular movement into speech 5, issue 6, pp. 657-675, December 2009.
[9]. The drawback of these technologies is that they require [8] Accelerometer-Based Gesture Recognition with the
the support of external sensors attached to the mobile phone iPhone. http://tinyurl.com/yjddr8q.
which might be an obstacle towards the large scale adoption
[9] The Mobile Phone that Could “Read Lips”.
of the technology.
http://www.cellular-news.com/story/42211.php.
Eye movement has recently been used for activity recog-
nition [24]. By tracking the eye researchers show that it is [10] Nokia Mobile Radar. http://tinyurl.com/yfo2dtd.
possible to infer the actions of watching a video, browsing [11] The Eye Mouse project.
the web or reading notes. http://www.arts.ac.uk/research/eyemouse/index.htm.
The work in [5, 23, 11, 13] is more closely related to the [12] F. Karray, M. Alemzadeh, J.A. Saleh, and M.N. Arab.
EyePhone system. The eyeSight [5] technology exploits the Human-Computer Interaction: Overview on State of
phone camera to control the mobile phone by simple hand the Art. In International Journal on Smart Sensing
gestures over the camera. The openEye project [23] relies and Intelligent Systems, Vol. 1, No. 1, March 2008.
on an eyeglasses mounted camera to track a person’s eyes to [13] M. Chau and M. Betke. Real Time Eye Tracking and
realize HCI applications. The Eye Mouse project [11] and Blink Detection with USB Cameras. In Boston
[13] are designed for desktop machines with fixed cameras University Computer Science Technical Report No.
pointed at the user to pick up eye movement and use the 2005-12, 2005.
eye as a mouse in [11], or to enable general HCI applications [14] E. Miluzzo, C.T. Cornelius, A. Ramaswamy,
[13]. However, these systems are for static machines, using T. Choudhury, Z. Liu, A.T. Campbell. Darwin Phones:
a specific fixed infrastructure of external sensors and cannot The Evolution of Sensing and Inference on Mobile
be easily replicated on mobile phones. Phones. In Eighth International ACM Conference on
Mobile Systems, Applications, and Services (MobiSys
7. CONCLUSION ’10), San Francisco, CA, USA, June 15-18, 2010.
In this paper, we have focused on developing a HPI tech- [15] EyePhone demo video.
nology solely using one of the phone’s growing number of on- http://sensorlab.cs.dartmouth.edu/eyephone/demo.m4v.
board sensors, i.e., the front-facing camera. We presented [16] Nokia n900. http://maemo.nokia.com/n900.
the implementation and evaluation of the EyePhone proto- [17] Eye Tracking and Blink Detection Library.
type. The EyePhone relies on eye tracking and blink de- http://tinyurl.com/yb9v4f2.
tection to drive a mobile phone user interface and activate [18] JP Lewis. Fast Normalized Cross-Correlation. In
different applications or functions on the phone. Although Vision Interface, volume 10, pages 120–123. Citeseer,
preliminary, our results indicate that EyePhone is a promis- 1995.
ing approach to driving mobile applications in a hand-free [19] Nokia n810 tablet. http://www.nokiausa.com/find-
manner. A video of the EyePhone demo can be found at products/phones/nokia-n810.
[15]. [20] OpenCV. http://opencv.willowgarage.com/wiki.
[21] GStreamer. http://www.gstreamer.net/.
8. ACKNOWLEDGMENTS [22] Mercedes-Benz Drowsiness Detection System.
This work is supported in part by Nokia, Intel Corp., http://tinyurl.com/353u8h6.
Microsoft Research, NSF NCS-0631289, and the Institute [23] D. Li, J. Babcock, and D.J. Parkhurst. openEyes: a
for Security Technology Studies (ISTS) at Dartmouth Col- Low-Cost Head-Mounted Eye-Tracking Solution. In
lege. ISTS support is provided by the U.S. Department of 2006 Symposium on Eye Tracking Research &
Homeland Security under award 2006-CS-001-000001, and Applications, page 100. ACM, 2006.
by award 60NANB6D6130 from the U.S. Department of [24] A. Bulling, J.A. Ward, H. Gellersen, and G. Troster.
Commerce. The views and conclusions contained in this Eye Movement Analysis for Activity Recognition. In
document are those of the authors and should not be inter- 11th International Conference on Ubiquitous
preted as necessarily representing the official policies, either Computing, pages 41–50. ACM, 2009.
expressed or implied, of any funding body. A special thanks
goes to Prof. Lorenzo Torresani for his valuable input on
this work.

9. REFERENCES
[1] Apple iPhone. http://www.apple.com/iphone.
[2] Nokia Series. http://europe.nokia.com/nseries/.
[3] HTC. http://www.htc.com/www/product.

View publication stats