Editor: Lawrence 3.

Rosenblum
[email protected]

riving Simulation: Challenges for VR Technology

James Cremer, irtual driving environments represent a challeng- overview of our work on the problems of scenario and
Joseph Kearney, ing test for virtual reality technology. They require scene modeling for virtual environments in the context
and Yiannis a combination of of the Iowa Driving Simulator (IDS).
Papelis
ra high-resolutionvisual, auditory, and haptic feedback, The Iowa Driving Simulator
Universiiy of ra numerically accurate, real-time simulation of com- The University of Iowa's Center for Computer-Aided
Iowa plex physical systems, Design (CCAD) is home to the Iowa Driving Simulator,'
ra modeling and control of believable agents and sce- a high-fidelity, operator-in-the-loop, ground-vehicle
narios, simulator.The simulator consists of a vehicle cab mount-
ra modeling of large-scale virtual environment data- ed inside a dome that rests on a 6-degree-of-freedom
bases, and motion base (see Figure 1).The base provides acceler-
ra a software architecture providing real-time database ation cues with an instantaneous maximum of 1 G and
access and runtime coordination of multiple threads. a frequency response of 3 to 6 Hz. At present, the IDS
supports three fully instrumented cabs: an HMMWV
The graphics and VR communities have made sub- (High Mobility Multi-Wheeled Vehicle), a Saturn, and
stantial progress on the first item, while engineering a Ford Taurus.
researchers have made much progress on the second The vehicle cabs have operational gauges and instru-
item for specific domains. Our view is that truly excit- ments. The driver receives haptic feedback through the
ing VR applications require development and integra- steering wheel, accelerator, and brake pedal. Within the
tion of all these elements. dome, high-resolution, textured graphics generated by
Modeling and control of believable agents has an Evans & Sutherland ESIG 2000 are projected onto a
received significant attention recently, but often in a panoramic screen in front of the cab (190-degree hori-
context free from the constraints imposed by real-time zontal field ofview) and a smaller screen behind the cab
interactive virtual environments. A variety of VE repre- (65-degree horizontal FOV) visible through rear-view
sentations and architectures have also been proposed mirrors. Directional audio cueing is provided by a mul-
and demonstrated, but typically again in rather uncon- tichannel audio system.
strained contexts. In this column, we present an The IDS uses high-fidelity,multibody, vehicle-dynam-
ics models running at high frequency to capture engi-
neering-level detail of vehicle operation. The scenario
control subsystem models semi-autonomous vehicle
behavior as well as the behavior of other dynamic enti-
ties. It also generates and controls traffic within the vir-
tual environment.
1 The Iowa
Driving Applications of virtual driving
Simulator environments
facility for real- The IDS provides a safe, virtual environment in which
time operator- to study driving behavior. Experiments can be conduct-
in-the-loop ed that are either impossible or unethical to undertake
driving in real environments because of the risks to human
simulation. subjects.
Since beginning operation in early 1992, the IDS has
been used primarily in three areas: transportation, vir-
tual prototyping, and medical research. In addition,
because of its fidelity, the IDS has served as a valuable
testbed for simulator technology research.

16 September 1996 0272-1 7-1 6196185.00 0 1996 IEEE

 In transportation, IDS studies
have focused on the use of in-vehi-
cle equipment such as intelligent
cruise controls, advanced collision
warning and roadway-departure
alerting devices, traveler informa- 2 Artist’s
tion systems, and heads-up dis- rendering of the
plays. The IDS hosted experiments National
on novel highway design, including Advanced
alternative pavement markings and Driving
new types of signs. Under Federal Simulator
Highway Administration funding, (scheduled to
the IDS has run a series of studies to be operational
gain insight into issues related to in 1999).
Automated Highway Systems
(AHS) and Intelligent Transporta-
tion Systems (ITS) technologies.
Experiments have focused on enter-
ing and exiting high-speed auto-
mated lanes along with various
failures of the automated systems.
High-fidelitydynamics models make IDS a valuable vehicles within a backdrop of ambient traffic. The essen-
tool for virtual prototyping-a process to design vehi- tial aspects of experimental situations must be replicat-
cles by using the simulator in lieu of a physical proto- ed from trial to trial in order to compare the
type. Actual hardware components can be used as part performance of subject groups.
of the dynamic model, providing hardware-in-the-loop We have developed a single computational frame-
simulations that can help test devices such as anti-lock work that can be used to model basic vehicle driving
brakes in a realistic environment. behavior, as well as to create interesting scenarios from
The IDS has been used in a variety of medical exper- ambient traffic composed of microscopicallysimulated
iments, for example, studying the effect of drugs on dri- autonomous vehicles. Our modeling framework, HCSM,
ving. Recently, experimental results on the IDS were is based on communicating hierarchical state machines.
used to partially fulfill requirements for obtaining FDA This extended state-machine model facilitates creation
approval for multifocal lenses. Upcoming studies will of scenarios and complex autonomous agents by pro-
compare the effects of various medical treatments on viding abstraction, communication, and arbitration
driving performance. mechanisms.
The University of Iowa has been selected as the site As described e l ~ e w h e r eHCSM
,~ draws on several
of the National Highway and Transportation Safety related efforts in behavior modeling for reactive sys-
Administration’s (NHTSA’s) National Advanced tems, including Harel’s state-charts and Reynolds’work
Driving Simulator (NADs).’ Development of the sys- on flocking. An HCSM state machine can be defined
tem was awarded to TRW of Redondo Beach, recursively; it is either a leafmachine or contains a num-
California. When completed in 1999, the NADs will be ber of child HCSM state machines. In asequential HCSM
the most advanced driving simulator in the world (see state machine, exactly one child HCSM is active at a
Figure 2). It will significantly extend the fidelity of all time. The active child can change based on the firing of
cueing subsystems and, due to its higher range of transitions. In a concurrent HCSM state machine, all chil-
motion, will be able to provide sustained accelerations. dren are active at once and there are no transitions.
The NADs will serve as a national shared facility avail- Concurrency allows behaviors to be decomposed into a
able for use by academic institutions, medical set of possibly, but not necessarily, orthogonal compo-
researchers, and industry. nents. When combined with hierarchical abstraction,
this significantly reduces the complexity of program-
Behavior and scenario modeling ming behaviors.
Virtual driving environments need roadway traffic HCSM state machines can communicate through
simulated to the microscopic level. That is, vehicle message passing. Conceptually, a state machine has a
motions must be smooth and continuous; cars cannot control panel consisting of “buttons” and “dials” that
discretelyjump from lane to lane as they typically do in provide a communication interface between the state
macroscopic traffic simulations. Furthermore, the machine and the world outside it. The state machine’s
behavior of the synthetic vehicles must be consistent behavior may be influenced by sending messages to the
with the human driver’s expectations. machine, “pushing”buttons, or “setting” dials. Buttons
It is also important to control global properties of traf- and dials address the critical need for a means to create
fic, such as density and flow. These properties should agents that are not purely reactive, but instead are part-
fall out naturally from the behaviors of individuals. ly reactive and also controllable or directable.
Lastly, we must be able to create specific situations HCSM state machines explicitlyincorporate a notion
and events requiring coordinated actions of groups of of the activities they control. Whenever an HCSM

IEEE Computer Graphics and Applications 17

 ChangeLane Speedfactor IgnoreLight Turn

-
Vehicle behavior model
Follow Vehicle

Cruise
lStopvehicle1
Pass -
3 HCSM

LJ
modeling basic
Inactive
driving
behavior. I Decelerate I Turn
Active turn

I Tracklane I Pass leader

Traffic light violation director

1 Clear lane director I

Wait for arrival
of subject vehicle I Conscript violator director I
Synchronize light director

4 Multiple I I
I
/
director HCSMs
can be used to
orchestrate
situations such Trigger
as a vehicle
driving through
a red light in
front of the
I
simulator driver. ubject vehicle

u
Clear lane director
Patented
conscript light

iachine is executed, it outputs avalue (or set ofvalues) a way to modify, filter, or otherwise change values
etermined by the activity function associated with the passed up from lower level behaviors. HCSM provides a
iachine. Specifically,an activity function computes an uniform framework for modeling the behavior of indi-
ICSM’s output as a function of the values of local vari- vidual entities and for modeling scenarios that involve
bles, input parameters, messages received by buttons coordination of multiple entities. We call HCSM state
nd dials, and values output by child HCSMs. The val- machines used for the latter purpose directors.
ies output by top-level HCSM machines are used as con- Figure 3 shows a portion of one of our basic driving
rol inputs for entities modeled in the simulation behavior HCSMs. Figure 4 illustrates how three concur-
nvironment. rent directors can be used to direct ambient traffic to cre-
Activity functions provide a means for arbitrating ate a situation where a vehicle unexpectedly drives
letween or resolvingthe possibly conflicting values out- through a red light into the path of the oncoming simu-
hut by concurrent child state machines competing to lator driver’svehicle. Situationssuch as this one highlight
ontrol the same resources. Activity functions are also the difficulties of providingboth replicable circumstances
iseful in sequential state machines, since they provide (required to compare subject performance) and an envi-

18 eptember 1996
ronment that feels natural, reactive, and uncontrived.
Because of the intrinsic variability and unpre-
dictabilityof subject driving behavior, we cannot create
realistic scenarios by preselecting the vehicles that will
play roles and scripting their actions. Instead, the par-
ticipatingvehicles are selected and directed on line. As
the subject approaches the intersection, a director is
used to conscript an appropriate scenario vehicle to run
the light. Other directors control nearby traffic and the
behavior of the traffic light. In this case, all of the direc-
tors are activated by a trigger in the road that senses the
approach of the subject vehicle. The scenario requires
coordination of the ambient traffic to create consistent
circumstances for the event.

Scene authoring
Driving simulation requires scene databases of large
geographic regions with properly constructed roads, 5 Intersection
smooth terrain, natural foliage, and appropriate cultural scene as
features. Because of the close surroundings, databases ated from the logical database. rendered on the
for driving simulation must be modeled at a much high- The design of road networks consistent with civil image
er resolution than flight terrain databases. To enable engineering standards, including banked curves with generator for
simulation of the physical interactions between the transition zones, is a difficult and time-consuming display to
ground andvehicle, it is particularlyimportant that dri- process that requires substantial expertise. It takes sig- simulator
vable surfaces be modeled with high fidelity. nificant effort (often months ofwork) by scene database drivers.
In IDS, the virtual environment is represented with specialists to go from an initial plan for a scene to an
three separate but correlated database components: operational model of a moderately complex scene with
visual, logical, and terrain. The visual component is tai- a variety of road types and intersections. It is impracti-
lored for image generation. It represents a scene in mul- cal to train experimenters to use complex modeling sys-
tiple levels of detail using texture-mapped polygons. The tems, and it is unreasonable to ask them to pay costly
logical component is used for scenariocontrol.It includes development fees for customized scenes, then wait
high-levelrepresentationsof the road network, vehicles, months for their completion.The compromiseapproach
buildings, signs, trees, and so forth. Roads are repre- usually taken is to adapt existing scenes as best as pos-
sented using oriented space curves along with associat- sible to meet experimental requirements.
ed lane and intersection topology. Roads are annotated We are investigating ways to support rapid con-
with attributes for speed limits, passing zones, and other struction, modification, refinement, and debugging of
information. The terrain database is used for vehicle scene databases. Part of our effort focuses on the devel-
dynamics computations. It contains a high-resolution opment of software and techniques to assist database
piecewise-uniformgrid of terrain elevations.The geom- modelers, such as methods for automatic terrain gen-
etry of drivable surfaces must be knownwith high accu- eration and modeling toolkits adapted for the special
racy to minimize errors in the vehicle dynamics needs of driving environments. For example, software
computations. The terrain database also includes infor- developed at CCAD formed the basis of the road tools
mation about surface type, roughness, friction, and soil currently provided in the popular MultiGen VE model-
properties. The terrain model supports overlapping ter- ing software.
rain, enabling bridges to be modeled. Figures 5 and 6 A second thrust of our work is to develop scene
present differentviews of an example intersection scene. authoring software aimed at nonspecialists.A prototype
The first figure was rendered by the image generator scene editor (see Figure 7, next page) allows an experi-
using the visual database, while the second was gener- menter to define the VE by laying out prefabricated

IEEE Computer Graphics and Applications 19

 istic real-time software system that
integrates componentsfor user inter-
action, simulation, and scenario and
scene modeling-make it avaluable
proving ground for virtual environ-
ment technologies. The goal of our
7 A prototype research is not simplyto improvedri-
tile-based ving simulation, but to developtech-
virtual driving nology that benefits a wide variety
environment of VE applications. Our work on
scene editor. authoring high-fidelityVE databas-
es and on directable scenarios pop-
ulated with believable agents, for
example, also targets applications
involving interaction with simulat-
ed, walking humans and training in
the operation of complexmachinery.
Thiswork has benefited greatly from
scene tiles on a grid. Tiles contain roads, scenery, and the experience of developingcomponents for a full-scale
cultural features. Tile edge profiles are restricted to a operationalVE systemlike IDS, and we believe that many
small set of standard shapes so that it is easy to match other proposed VE technologies would similarlybenefit
tiles to their neighbors. The editor automatically checks from such real-world testing. W
the consistency of placements to ensure that surface
geometry is continuous across tile boundaries and that Acknowledgments
roads are consistently joined. Many people from the Center for Computer Aided
Design and the Computer Science Department con-
Real-time software architecture tributed to this work in scenario control, Omar Ahmad,
Our scenario control software manages the execution Shaheen Bahauddin, Michael Booth, and Peter Willem-
of behavior and scenario models, and maintains a data- sen; in authoring tools, Tzy-Yu Chow, Douglas Evans,
base for efficientlyresponding to queries about driving- Michael Morrison, Michael Noth, and Matthew Schiko-
related aspects of the virtual environment. The primary re; in scene modeling, Michael Bartelme, Brian
architectural constraints on the scenario control soft- Mannlein, Steven Zellers, Shawn Allen, and Rachel
ware originate in its current use in IDS and its future use Nador. This work was sponsored in part by grants from
in NADs and other high-fidelity real-time simulators. the National Highway Transportation Safety Adminis-
The main constraint is deterministic, real-time exe- tration, the Department of Defense, the National Sci-
cution. The scenario system runs as part of a large set of ence Foundation, and by a number of corporate and
cooperating modules. Module execution rate and order government contractors.
are controlledby a frequency-basedscheduler. The actu-
al time allocated to scenario control and other subsys-
tems varies with the simulator configuration, but
real-time execution is always the primary constraint. References
To create scenario events, the HCSM framework is 1. J. Kuhl et al.,“The Iowa Driving Simulator-An Immersive
used to modelvehicle behaviors, traffic light behaviors, ResearchEnvironment,” Computer, July 1995,Vol.28, No.
and directors responsible for coordinating entitybehav- 7, pp. 3541.
iors. The outputs of HCSM state machines are used as 2. D. Stoll and S. Bourne, “The National Advanced Driving
control inputs to lower level physical simulations that Simulator: Potential Applications to ITS and A H S
implement, for example, simple dynamic or kinematic Research,” Proc. 6th ITS (Intelligent TransportationSOC.)
models of scenario vehicles. Am. An. Meeting.,Apr. 1996,ITS America,Washington, D.C.
Thevirtual Environment Database maintains and effi- 3. J. Cremer, J. Kearney, and Y. Papelis, “HCSM: A Frame-
ciently answers queries about the road network, moving work for Behavior and Scenario Control invirtual Envi-
vehicles, traffic control devices and signs, and other ronments,” ACM Trans. Modeling and Computer
objectsthat affectdrivingbehavior. The database is struc- Simulation,Vol. 5, No. 3, July 1995,pp. 242-267.
tured around two basic kinds of queries: static and
dynamic. Static queries involve information that does
not change during the simulation, including basic road- Contact Cremer at Computer Science Dept., University
way and intersectiontopology, sign placement and mark- ofIowa, Iowa C i a L4.52242, [email protected],
ings, and so on. Dynamicqueries encompassrequests for or on the Web at http://www.cs.uiowa.edu/-cremer.
nearby vehicle information, traffic light status, and other To submit a column or aproposal for one to Projects in
information dependent on the state of the simulation. W, contact department editor Lawrence J. Rosenblum,
Advanced Information Technology Branch, Code 5580,
Conclusion Naval Research Laboratoy, Washington,DC20375-5320,
The requirements of driving simulation-a determin- e-mail [email protected]

20 September 1996