Abstract

The Pebbles research project (http://www.cs.cmu.edu/~pebbles) has been studying the use of hand-held personal digital assistants (PDAs) along
with other kinds of hand-held computers, at the same time as other computing devices. A key focus of our research is that the hand-held comput-
ers are used both as output devices and as input devices to control the activities on the other computers. Our previous articles have described
parts of the project in detail. This article presents four scenarios that illustrate some of the capabilities we are already investigating.

Using Multiple Devices Simultaneously for

Display and Control
Brad A. Myers, Carnegie Mellon University

T he Pebbles research project

(http://www.cs.cmu.edu/~pebbles) has been studying the use
of hand-held computers simultaneously with other computing
Rather than disrupting the main activities and the main dis-
play, the PDA can be pulled out, and a special unobtrusive
cursor will appear on the main display, so the user can point
devices. A key focus of our research is that the hand-held to the item of interest. Then the user can privately “drill
computers are used both as output devices and as input down” to get the additional specialized information displayed
devices to control the activities on the other computers. Our on the PDA. The display of the information is appropriately
previous articles [1-3] have described parts of the project in adjusted to the limited size of the PDA screen.
detail. This article presents four scenarios that illustrate some We are currently exploring various aspects of this scenario
of the capabilities we are already investigating. as part of the “Command Post of the Future” project (see
http://www.cs.cmu.edu/~cpof). In cooperation with MayaViz
Scenario 1: Presentations (www.mayaviz.com), we have created a PDA-based visualiza-
tion and control program that runs on Windows CE and
The presenter of a briefing or talk has a laptop, and the dis- Palm. On the PDA, you can see a view of a map on which you
play is projected onto a large screen. The laptop’s powerful can scribble and select objects, and a table view of the detailed
processor is needed to control the animations and external information. The user can operate either connected (so oper-
applications that are part of the presentation. In the presen- ations on the PDA are immediately reflected on the main
ter’s hand is a PDA on which the current slide’s notes are dis- screen) or disconnected. Figure 2 shows some example
played. The PDA can be used to cause the presentation to go screens.
forward, backward, or skip to a specific slide under discussion. We have a number of other applications that support
Also on the PDA are custom controls to switch among various meetings where the participants are co-located. All partici-
other applications on the laptop that the presenter will be pants’ PDAs are in continuous two-way communication with
demonstrating and discussing. Each member of the audience the main computer, which is often projected on a screen to
sees on their personal hand-held the current slide, which is serve as the focal point of the discussion. Some of our initial
kept synchronized with the talk. Audience members can also applications use the PDAs as remote mice and keyboards so
make private notes and annotations on their PDAs. When that everyone in the meeting can control the main computer.
enabled by the presenter, an audience member’s marks on This might be used as a shared whiteboard that supports mul-
their PDA can be displayed on the main screen for general tiple inputs simultaneously, for private side messages via a
viewing and discussion. “chat” program, and to display multiple cursors for pointing
This scenario is partially implemented in our “Slide Show and scribbling on arbitrary applications. More details on our
Commander” application [3]. Figure 1 shows some screen- groupware applications are presented in another paper [1].
shots. This application has been released commercially by
Synergy Solutions (http://www.synsolutions.com/software/ Scenario 3: Augmenting the Desktop
slideshowcommander). In the future, we will be investigating
more on supporting the audience members and note-taking. When the user is sitting and working, various devices are
placed on the desk: a laptop, a PDA, a cell-phone, etc. They
immediately communicate with each other to establish each
Scenario 2: device’s capabilities and specifications. As the user works, var-
Public and Private Spaces ious controls appear on the screens of the other devices rather
than on the laptop’s screen. For example, scroll bars might be
In a military command center, several large displays show drawn on the PDA so the user can operate them with the left
maps, schedules, and other visualizations of the current situa- hand while using the mouse with the right hand, which has
tion that will be useful to the group. Individuals carry a per- been shown to be fast and effective. The user’s custom short-
sonal PDA. While in the command center, someone might cuts for the laptop applications also appear on the PDA, and
want more details on an item displayed on a large display. the user has memorized their location and can operate them

62 1070-9916/00/$10.00 © 2000 IEEE IEEE Personal Communications • October 2000

 (a) (b) (c)

(d) (e) (f)

■ Figure 1. The Pebbles Slideshow Commander program. (a) and (b) show a HP Jornada Windows CE machine. (a) shows the full
device, on which you can see a thumbnail of the slide on the top, and the notes for that slide on the bottom. Drawing on the thumbnail
causes the same drawing to appear on the main screen. (b) shows a close-up of the screen viewing the list of titles for the presentation.
Tapping on a title causes PowerPoint to change to that slide. (c)–(f) show the user Slideshow Commander on the Palm. (c) shows the
Palm, viewing the thumbnail of the slide. (d)–(f) are close-ups of the Palm screen. (d) is the Notes pane, (e) is the list of titles, and (f) is a
timer. Meanwhile, a PC is running PowerPoint and a PDA is in continuous two-way communication with the PC.

quickly without looking. Information can be easily moved Scenario 4:

among the devices, and other information is automatically dis-
tributed based on predefined user preferences. Universal Personal Controller
We have started to investigate some aspects of this sce-
nario. The PDA can be used as a scrolling device, as a gen- When the user points a PDA at a light switch, at a photocopi-
eral-purpose button panel (to create screens of “shortcuts”), er in an office, at a machine tool in a factory, at a VCR at
as an index page or table of contents for web surfing, and to home, at a piece of test equipment in the field, or at almost
cut and paste information back and forth from the PDA to any other kind of device, the device sends to the PDA a
the PC. Initial studies show that scrolling with the PDA in description of its input and output requirements. The PDA
the left hand while using the right hand to select items in converts this description into a custom control panel, taking
the window with the mouse can be faster than using the into account the properties of the controls that are needed,
mouse with conventional scroll bars [2]. A related study the properties of the PDA (the display type and input tech-
shows that moving both hands off the keyboard to the PDA niques available), and the properties of the user (which lan-
on the left and the mouse on the right (or back to the key- guage is preferred, whether left or right handed, how big the
board from the devices) is only about 15 percent slower buttons should be based on whether the user prefers using a
than moving one hand to the mouse. Thus, there is little finger or a stylus). The user can then control the device using
penalty to using both devices. Figure 3 shows some example the PDA. The device will not need to dedicate much process-
screens we have created with our “Shortcutter” application. ing power, hardware, or cost to the user interface, since it will
Other related applications are described in another paper only need to contain a description of its capabilities and stor-
[3]. Currently, we are only using PDAs, and the communica- age for the current settings, along with hardware for wireless
tion uses the PDA’s cradle and serial cable, but in the communication. The PDA programs will use intelligent
future we will expand to other devices and wireless commu- “model-based” techniques to create useful and appropriate
nication such as Bluetooth [4]. interfaces that are customized for each user.

IEEE Personal Communications • October 2000 63

 (b)

(c)

(a)

■ Figure 2..(a) A public view displayed on the wall from a PC, and the views (b)–(c) on a palm-size Windows CE machine for private
viewing and editing. (b) is the map view, and (c) is the table for drill-down information.

(a) (b) (c)

(d) (e) (f)

■ Figure 3. Panels created with Shortcutter: (a) a numeric keypad; (b) a collection of scrollers and a knob; (c) buttons for controlling
any of a set of compilers; (d) a gesture pad and two rows of small buttons; (e) a controller for the WinAmp MP3 player for PCs; and
(f) a panel for browsing in Internet Explorer.

64 IEEE Personal Communications • October 2000

 We are just beginning to explore this scenario, and are [3] B. A. Myers et al., “Extending the Windows Desktop Interface With Con-
nected Handheld Computers,” 4th USENIX Windows Systems Symp.,
investigating related technologies such as XML, the WML 2000, Seattle, WA, pp. 79–88.
language for WAP (Wireless Access Protocol (www.wapfo- [4] J. Haartsen et al., “Bluetooth: Vision, Goals, and Architecture.” ACM
rum.org)), Salutation (www.salutation.org), Universal Plug- Mobile Computing and Communications Review, 1998. vol. 2, no. 4,
And-Play (www.upnp.org), etc. Oct., www.bluetooth.com, pp. 38–45.
[5] R. Want et al., “An Overview of the ParcTab Ubiquitous Computing
Experiment,” IEEE Pers. Commun., 1995, vol. no. 28–43, Dec.; also
Research Issues appears as Xerox PARC Technical Report CSL-95-1, Mar. 1995.
[6] J. Rekimoto and M. Saitoh, “Augmented Surfaces: A Spatially Continu-
Many significant research issues are involved in bringing these ous Work Space for Hybrid Computing Environments,” Proc. SIGCHI’99:
Human Factors in Computing Systems, 1999, Pittsburgh, PA, pp.
visions to fruition, which we are committed to investigating as 378–85.
part of the Pebbles project. None of the other research pro- [7] S. Greenberg, M. Boyle, and J. Laberg, “PDAs and Shared Public Dis-
jects investigating novel ways to use PDAs (e.g., [5–8]) have plays: Making Personal Information Public, and Public Information Per-
addressed these issues. We are particularly interested in the sonal,” Personal Technologies, Mar. 1999.
[8] R.C. Davis et al., “NotePals: Lightweight Note Sharing by the Group, for
appropriate ways to distribute the user interfaces across multi- the Group,” Proc. CHI’99: Human Factors in Computing Systems, 1999,
ple devices, how to support multiple people interacting with Pittsburgh, PA, ACM. pp. 338–45.
the same screen using their various devices as auxiliary input [9] J. E. Stewart, “Single Display Groupware,” SIGCHI’97 Adjunct Proc.:
and output devices (which is sometimes called “single-display Human Factors in Computer Systems, Extended Abstracts, 1997,
Atlanta, GA, pp. 71–72.
groupware” [9]), the automatic creation of appropriate and
usable control panels from high-level specifications, and
usability issues with multi-computer interaction techniques. Biography
The Pebbles research project has made substantial progress by B RAD A. M YERS ([email protected]) is a senior research scientist in the
building example applications, releasing them for general use, Human-Computer Interaction Institute in the School of Computer Science
at Carnegie Mellon University, where he is the principal investigator for var-
and formally testing them in usability experiments. Many of ious research projects including: Silver Multi-Media Authoring, Natural Pro-
these applications are available from our web site gramming, the Pebbles Hand-Held Computer Project, User Interface
(http://www.cs.cmu.edu/~pebbles) and have been downloaded Software, and Demonstrational Interfaces. He is the author or editor of
more than 25,000 times in the last 2 1/2 years. We invite you more than 200 publications, including the books “Creating User Interfaces
by Demonstration” and “Languages for Developing User Interfaces,” and he
to try these out and let us know about other ideas for using is on the editorial board of five journals. He has been a consultant on user
multiple devices at the same time. interface design and implementation to about 40 companies, and regularly
teaches courses on user interface software. Myers received a Ph.D. in com-
References puter science at the University of Toronto, where he developed the Peridot
UIMS. He received the M.S. and B.Sc. degrees from the Massachusetts Insti-
[1] B. A. Myers, H. Stiel, and R. Gargiulo, “Collaboration Using Multiple tute of Technology, during which time he was a research intern at Xerox
PDAs Connected to a PC,” Proc. CSCW’98: ACM Conf. Computer-Sup- PARC. From 1980 until 1983 he worked at PERQ Systems Corporation. His
ported Cooperative Work, 1998, Seattle, WA, pp. 285–94. research interests include user interface development systems, user inter-
[2] B. A. Myers, K. P. L. Lie, and B.-C. J. Yang, “Two-Handed Input Using a faces, hand-held computers, programming by example, programming lan-
PDA And a Mouse,” Proc. CHI’2000: Human Factors in Computing Sys- guages for kids, visual programming, interaction techniques, window
tems, 2000, The Hague, The Netherlands, pp. 41–48. management, and programming environments.

IEEE Personal Communications • October 2000 65