APPENDIX

Boeing Wire Bundles, An

Augmented Reality Application
n the area of manufacturing, the Virtual Systems Research and Technology group

I at Boeing Computer Services is exploring augmented reality (AR) as an aid in

assembling wiring bundles for aircraft. Making wire bundles is a difficult, tedious
task and is different for each individual aircraft. Boeing performed this research in
collaboration with CMU, Honeywell, and Virtual Vision, Inc. under a 3-year DARPATRP
project titled "Wearable Computer Systems with Transparent, Head-Mounted
Displays," begun in August 1994. In August of 1997, the project culminated in a field
study testing the new paradigm with wire shop workers (FIGURE C-1).

FIGUREC-1 Adam Janin

demonstrating the 1993
laboratory prototype, with
stereo CRTs and a magnetic
tracking system (Image
courtesy of David Mizell,
Boeing, Inc.).
514 [ BOEING WIRE BUNDLES, AN AUGMENTED REALITY APPLICATION

Application Description
The traditional way of assembling aircraft wire bundles begins with an engineer
designing the bundle on a CAD workstation using the CATIA CAD package. Next the
3D CAD model of the bundle gets converted into a 2D schematic-currently a manual
process-plus a list of wires at specific lengths. Large plotters then draw the wiring
scheme onto mylar printouts. These are then glued to the physical formboards, which
are approximately 8 ft. by 3 ft. Before putting the wires on the formboards, connec-
tors are added at one end. The serial number printed on the wire is used to deter-
mine which wire goes in a particular slot. Next, they position pegs on the board to
support the wires while they are on the formboard. In the final step of the formboard
stage, the assembly person procures the correct wires, stands at the board, lays the
wires through the proper routes, and sleeves and ties the wire groupings (FIG-
URE C-2). Following this, the second ends of the connectors are added, the bundles
are tested, and then installed.

FIGURE C-2 The traditional

formboard wire bundle
assembly process. Hundreds
of boards are stored for all
the particular wiring harnesses
(Image courtesy of David
Mizell, Boeing, Inc.).
515 I APPLICATION IMPLEMENTATION

One of the issues that makes this task so critical is that there are so many
different configurations for these bundles. Each type of aircraft uses different bundles,
and each requires many different bundles. For example, a Boeing 747 requires about
a thousand bundles. Also, an individual 747 may require bundles that are different
from other 747s because of different cabin configurations (the cabin design for large
aircraft is sometimes customized for the route it will fly).
While the primary concern is to improve performance of the hundreds of
workers "boarding" wire bundles, an additional benefit is that the generic pegboard
style formboards used in the augmented reality (AR) system can help reduce the
enormous storage space required for the existing racks and racks of boards, each of
which only works for a single wire bundle. Boeing is looking to improve this situa-
tion. The AR system is one experiment for finding a better solution.

Application Implementation
The AR system they developed provides the wearer with lines superimposed on the
real world using a see-through HMD. In the initial, prefunded stage of research, an
inexpensive see-around HMD called the Private Eye was used; this is a monocular
display that takes up only a small fraction of the field of view [Caudeli and Mizell
1992].
AR provides an effective means of "annotating" reality. In this case, a single
blank pegboard can be annotated for each specific layout. The assembler wears a
portable computer around their waist that supplies the computer-generated virtual
world superimposed onto the real world.
The net result is that a blank board can be used for assembling a wire
bundle. Traditional boards had the pegs prepositioned, with individual boards for
each wire bundle. The boards for the augmented reality project have an array of holes
in them like a pegboard, so pegs can be placed appropriately for the bundle being
constructed. After the pegs are placed as instructed by the system, the wires are run
and then bundled. The task is much simpler, because the AR system displays only
one wire or a few wires that all route to the same destination at one time. The tra-
ditional method of putting the schematic on the board itself produces the negative
consequence of the diagram becoming obscured as wires are put in place.
No audio output, voice input, or other modalities are part of this pilot
project. However, Boeing is investigating many of these technologies in separate
projects, and as applications prove to be effective they will consider adding interface
516 I BOEING WIRE BUNDLES, AN A U G M E N T E D REALITY APPLICATION

technologies as they mature. The next interface for the formboard application will
likely include speech recognition.
The formal study of this pilot project was conducted in August 1997 at the
Boeing widebody (747, 767, 777) factory in Everett, Washington. This study was
designed to systematically compare three prototype AR systems with the traditional
method. Both time and accuracy of bundle assembly were measured. The systems
used were built by TriSen Systems, Inc. of Minneapolis, a spin-off of Honeywell. Each
system was composed of a wearable PC, a see-through HMD, and a videometric
tracking system, all developed by TriSen (FIGUREC-3). Other systems were delivered,
but the tracking did not meet the minimal specifications for the work.
Although the formal study demonstrated that a workable augmented reality,
formboard system was possible, the particular implementation did not result in
increased performance in time to construct a bundle. However, there was no loss of
performance, and the workers were able to completely construct wire bundles on the
AR formboard. These bundles were then transferred to traditional boards and passed
quality-assurance inspection.

FIGURE C-3 Here, the packaged

TriSen system is used to
create a wire bundle. The
system consists o[ the light-
weight head-based display
and a computer worn around
the waist (A). In the augmented
view o[ the board (B), com-
puter graphic lines indicate
where to place the next wire
(Image courtesy o[ TriSen
Systems, Inc.) See color
plate 29.
517 I APPLICATION IMPLEMENTATION

David Mizell, manager of Boeing's Virtual Systems Group, states that they
expected the new system to result in faster construction times. In particular, they
expected the speed increase to stem from the fact that the old method did not
contain all the information directly on the board in the assembler's line of sight. It
required the assembler to refer to separate paperwork (the "Shop Aid") for infor-
mation on how to route each set of wires. The new system was targeted to eliminate
this bottleneck by keeping all the information overlaid on the formboard, so the
worker would never have to look away from their work.
In analyzing the study, Mizell and his collaborators believe that the lack of
improved performance resulted from a poor user interface. The interface used in the
study required the worker to use a mouse click paradigm to proceed through the
task. Boeing is confident that as they gain experience in working with this new AR
display paradigm, they will be able to construct significantly better user interfaces.
One specific improvement that they are already integrating is speech recognition.
With a better interface, they expect to reduce formboarding time by 1o-2o%. Mizell
added that after exposing factory personnel to augmented reality, the workers have
suggested additional ideas of how AR might be useful in a variety of factory tasks.
The DARPA TRP program (under which this project was funded) is designed
to integrate military technology into civilian use, which is intended to bring down the
cost of the technology by increasing the market size. In this case, Boeing is working
with see-through HMD technology created by Honeywell for helicopter pilots. This col-
laboration helps Honeywell by providing input for the requirements of an AR system.
Augmented reality systems have design constraints that differ from standard
virtual reality systems. Whereas in virtual reality, the challenge is to render enough
polygons to make a worthwhile world, the world already exists in AR, and only a
small amount of information needs to be added. For augmented reality, the challenge
comes in producing a self-contained, portable system that still accurately registers
the virtual world with the physical world. Environments displayed in VR systems typ-
ically allow the user to virtually move through the space. In AR, physical locomotion
by the user or a robot (i.e., for telepresence) is generally the only acceptable method
of travel.
TriSen Systems, Inc. put together augmented reality systems that met these
constraints for Boeing. TriSen themselves manufactured the HMD and tracking system
and integrated them with wearable computers from ViA Inc. The wearable computer
for the prototype system was a 4 lb., 33 MHz Intel 486. During the 1997 study, the
518 I BOEING WIRE BUNDLES, AN AUGMENTED REALITY APPLICATION

computers were upgraded to a 12o MHz 586 processor. More interesting, though, is
the tracking solution provided by TriSen. Their package uses videometric tracking.
Videometric tracking is accomplished by using a camera mounted on the
object to be tracked-in this case the HMD--and using the video input to recognize
the 6-DOF position of the camera. The incoming video image is analyzed for known
landmarks. By finding a sufficient number of landmarks (at least four), the algorithm
can determine the camera's location. Typically, artificial landmarks, known as [iducial
markers (or ~'ducial landmarks) are added to the environment in which the system
is to be used. For this system, the fiduciai landmarks look like strange polka dots
and are placed directly on the formboards in a specific pattern that allows the system
to rapidly determine absolute position (FIGURE C'4).
Because the fiducial landmarks are only on the formboards, the system
cannot track the user when they look away. However, this poses no problem since
the user only needs the augmented information during the time they are facing the
formboards. It is important, though, that tracking resume in short order when the
user returns to working on the formboards. In this case, the system took less than
one second to reacquire position tracking once enough fiducials came into view.

FIGURE C-4 This bundle,

assembled during the 1997
pilot project, shows the
special formboards created
by TriSen Systems, Inc.,
with fiducial markers (the
light-colored dots) directly
integrated into the boards
for videometric tracking
(Image courtesy of David
Mizell, Boeing, Inc.).
519 I APPENDIX SUMMARY

Because the entire videometric tracking system is carried on the user-the

camera and the wearable PC--there are no tethers from the user to an external
system. This is a major benefit of using this entirely self-contained system. This par-
ticular solution is not possible in all augmented reality scenarios. It works perfectly
in this scenario because the fiducials can be easily added to the environment and
because the processing power to handle the complete task can be contained on a
wearable PC.

AppendixSummary
Boeing VR research is conducted within the Virtual Systems group under David
Mizell. The AR project was conceived by Tom Caudell (now at the University of New
Mexico in the EECS department). Initial development was done by Adam Janin (now
a Ph.D. student at the University of California at Berkeley). Current work is now being
done by Dan Curtis and Peter Gruenbaum, with Chris Esposito handling much of the
prototyping of the wearable PC systems.
Boeing is also involved in non-AR virtual reality work. Due in part to the
$1oo million cost of building physical, interactive mockups of planes, they are inves-
tigating virtual prototyping of aircraft. Visualization of the Boeing 777-the first large
aircraft built without the creation of a physical mockup-was done with a package
named FlyThru, written by Bob Abarbanel and Eric Brechner (now at Microsoft).
F/yThru was a desktop visualization system and was not designed to work with head-
based displays nor at the rendering speeds required by VR.
Following this work, a real-time rendering package, RealEyes, was devel-
oped. Designed and implemented by Henry Sowizral (now at Sun Microsystems) and
Karel Zikan (now at Masaryk University in the Czech Republic), RealEyesuses paral-
lel processes to cull hidden polygons, as well as other computer graphics techniques
to render large numbers of polygons at real-time rates. RealEyeshas been interfaced
to a Fakespace BOOM and demonstrated using partial models of a 777 as a virtual
reality application. This work is being carried out at Boeing by Bill Brown and Charkar
Swamy.
In addition to the cost savings of not building physical prototypes, virtual
prototypes are effective as a tool for maintenance analysis. Boeing research has also
produced a series of experiments to analyze the benefits of virtual reality, including
its use for maintainability analysis. These experiments have led them to conclude
that HBDs are preferable over the BOOM-style, handheld method, because they free
520 I BOEING WIRE BUNDLES, AN AUGMENTED REALITY APPLICATION

the hands to reach into spaces. Maintenance aspects of a design can be analyzed
by performing collision detection on the hands and arms of the user to determine
reachability. TriSen Systems, Inc. now markets a hardware package that combines a
wearable PC with head-based display, video camera, and videometric tracking for use
in assembling wire bundles and other tasks.
For Mizell, a general point of doing this particular research on augmented
reality is that he believes "there are applications all over the factory, for example
maintenance and maintenance training." Also, he sees AR as having great benefit for
manufacturing, training, and maintenance in both military and civilian aircraft pro-
duction. The technology is also advancing rapidly to make augmented reality an easy
to use interface for workers. As Mizell puts it: "It would be great if one could grab a
wearable PC, clip it on, and be tracked in airplane coordinates."

A NOTE ON INFORMATIONACQUISITION

We did not have the opportunity to directly experience the Boeing formboard wire
bundting application firsthand. The information in this description comes from a talk
by Henry Sowizral, the paper by Caudell and Mizell [1992], and an article by Sara
Reese Hedberg [1996] in Virtual Reality Special Report. A considerable amount of
information and clarifications have been provided through conversations, comments,
and interviews with David Mizel[, who also reviewed a late draft of this appendix.