Auton Robot (2007) 22:101–132

DOI 10.1007/s10514-006-9013-8

Development environments for autonomous mobile robots: A

survey
James Kramer · Matthias Scheutz

Received: 2 July 2006 / Revised: 16 October 2006 / Accepted: 20 October 2006 / Published online: 1 December 2006


C Springer Science + Business Media, LLC 2006

Abstract Robotic Development Environments (RDEs) have of this survey and a brief discussion of future trends in RDE
come to play an increasingly important role in robotics re- design.
search in general, and for the development of architectures
for mobile robots in particular. Yet, no systematic evalua- Keywords Robotics . Programming environment .
tion of available RDEs has been performed; establishing a Comparison . Software
comprehensive list of evaluation criteria targeted at robotics
applications is desirable that can subsequently be used to
compare their strengths and weaknesses. Moreover, there 1 Introduction
are no practical evaluations of the usability and impact of a
large selection of RDEs that provides researchers with the Robots, unlike many software agents, operate under real-
information necessary to select an RDE most suited to their world, real-time constraints where sensors and effectors with
needs, nor identifies trends in RDE research that suggest specific physical characteristics need to be controlled. To fa-
directions for future RDE development. cilitate research in autonomous robotics and help architecture
This survey addresses the above by selecting and de- designers in managing the complexity of embodied agents,
scribing nine open source, freely available RDEs for mobile several robot development environments (RDEs) have been
robots, evaluating and comparing them from various points developed that support various aspects of the agent develop-
of view. First, based on previous work concerning agent sys- ment process, ranging from the design of an agent architec-
tems, a conceptual framework of four broad categories is ture, to its implementation on robot hardware, to executing
established, encompassing the characteristics and capabili- it on the robot.
ties that an RDE supports. Then, a practical evaluation of While several frameworks for comparing agent systems
RDE usability in designing, implementing, and executing have been proposed, some of them specifically for RDEs
robot architectures is presented. Finally, the impact of spe- (see Section 3), there is currently no survey available that (1)
cific RDEs on the field of robotics is addressed by providing provides a set of conceptual RDE features comprehensive
a list of published applications and research projects that give enough to serve as a basis for a conceptual evaluation that
concrete examples of areas in which systems have been used. does justice to the specific aims with which most RDEs have
The comprehensive evaluation and comparison of the nine been developed, (2) applies the conceptual criteria system-
RDEs concludes with suggestions of how to use the results atically to a large selection of RDEs, (3) augments the the-
oretical evaluation with a practical usability evaluation that
includes architecture design, implementation, and execution
J. Kramer (
) · M. Scheutz
Artificial Intelligence and Robotics Laboratory, Department of within each RDE on a robot, with special emphasis on ease of
Computer Science and Engineering, University of Notre Dame, use and performance, (4) includes the impact of the RDE in
Notre Dame, IN 46556, USA terms of categorized published work using it, an indicator of
e-mail: [email protected] an RDE’s prevalence in and influence on the robotics field,
M. Scheutz and (5) provides a principled way of combining the three
e-mail: [email protected] evaluations (conceptual, practical, and impact) to obtain an

Springer
102 Auton Robot (2007) 22:101–132

overall measure of how well an RDE can be adapted to the sive application developed in the system (i.e., a repository of
particular needs of researchers choosing among available components is not considered for inclusion). To our knowl-
systems or RDE developers considering future directions of edge, this requirement is not met by Orocos (Bruyninckx,
system development. 2001; OROCOS, 2005), The Rossum Project (Lucas, 2004),
This paper addresses all five points. Starting with a set of Nomadic (Sprouse, 2005), Dave’s Robotic Operating Sys-
constraints used for the selection of RDEs to be examined tem (Austin, 2004), the Open Automation Project (Walters,
(including a rationale for excluding certain RDEs), Section 2 2003), or YARP (Metta et al., 2006). Similarly, this excludes
introduces nine general purpose, freely available RDEs. some projects that, at the time this research was begun, were
Section 3 reviews previous work concerning agent system either just being developed (e.g., Orca Brooks et al., 2005;
and RDE comparisons, establishing four categories of Orca, 2005) or in a pre-release stage (e.g., the RObotics and
criteria corresponding to typical stages of application Learning Environment (ROLE) (Heckel, 2005)).1
development for autonomous mobile robots. The RDEs are Given the above constraints, nine RDEs have been se-
then systematically evaluated according to the criteria in lected,2 listed in Table 1 The following synopses give an
Section 4. Section 5 contains a practical evaluation based on overview of the systems’ use and operation, including a
designing, implementing, and running a simple architecture broad system description, the stated purpose of the system,
and some more complex architectural components in each the platforms on which it runs, the release version, and a
RDE. The subsequent discussion in Section 6 ties together summary of notable features. To characterize the strengths
the conceptual and practical evaluations and augments them of the systems more completely, the end of each subsection
with one possible evaluation of the impact of each RDE, lists publications from particular robotics research subareas,
also suggesting a principled method for using the results determined by the presentation groupings established in the
of this survey by both researchers and RDE developers. 2001–2005 Proceedings of the IEEE International Confer-
Section 7 summarizes the results and extrapolates to identify ence on Robotics and Automation (ICRA), which have been
some future trends in RDE development. divided into three categories:
r Single robot: SLAM, Planning/Navigation, Learning,
Hierarchical Behavior, and Education
2 Autonomous mobile robot systems r Human-Robot Interaction: Task Allocation, Learning, and
Assistive Robotics
A complete accounting and systematic comparison of all r Multi-robot: Sensing, Exploration, Mapping, Localization,
RDEs is clearly impossible within the confines of a survey
Planning, Coordination, Formation, and Task Allocation
paper, not only because of the number of RDEs available
and the release of new systems, but also due to the scope Only a single publication represents a sub-area; citations
of robotics as a discipline. To make the task manageable, are also used in evaluating an RDE’s impact in Section 6.
a group of qualifying constraints is used to limit the selec-
tion to a specific subset of representative RDEs. First, we 2.1 TeamBots
consider only open source packages unencumbered by li-
censing costs and available for free download. CyberBotics TeamBots (Balch, 2004; Balch and Ram, 1998) (which su-
Webots (Michel, 2004; Webots, 2005), White Box Robotics persedes JavaBots) is a Java-based collection of application
(WhiteBoxRobotics, 2005), and Evolution Robotics’ ERSP programs and Java packages for multi-agent mobile robotics
(ERSP, 2004) are excluded as commercial packages. Also ex- research. Although it is no longer under active development
cluded are BERRA (Lindstrom et al., 2000) and CLARAty (the most recent version available, 2.0e, was released in April
(Volpe et al., 2001; Nesnas et al., 2003, 2006) due to down- 2000), it is included due to its appearance in both Jia et al.
load unavailability. Systems are also required to generalize (2004) and Orebäck and Christensen (2003) and because it
beyond specific hardware platforms, but provide more speci- has found wide use in both research and education. The main
ficity than a general framework. So, while Lego Mindstorms author of TeamBots is now affiliated with the laboratory that
(LEGO, 2005) is a widely-used robotics platform with many develops MissionLab (see Section missionlab-desc); for the
related packages available, we do not consider it (or projects above reasons, this description will be brief.
such as CotsBots (Bergbreiter and Pister, 2003; CotsBots, A highly touted feature of TeamBots, stemming from a
2005) or Modular Controller Architecture (MCA2, 2005)) strict separation of hardware interfaces and control code, is
due to specificity in relation to a single platform or custom
hardware construction. Conversely, LAAS’s GenoM (Fleury 1
Exclusion of the listed systems is only indicative of not meeting the
et al., 1997; Mallet et al., 2002; Fleury and Mallet, 2004) specified constraints; further examination is encouraged.
is excluded as a framework for the generic definition of 2
Almost all of the selected RDEs are under constant revision and more
robot components. Finally, there must be at least one cohe- recent versions might be available.

Springer
Auton Robot (2007) 22:101–132 103

Table 1 RDEs selected for this survey

RDE Originating institution More information

TeamBots, v.2.0e Carnegie-Mellon University http://www.teambots.org/

ARIA, v.2.4.1 MobileRobots, Inc http://robots.mobilerobots.com/
Player/Stage, v.1.6.5, 1.6.2 University of Southern California http://playerstage.sourceforge.net/
Pyro, v.4.6.0 Bryn Mawr College http://emergent.brynmawr.edu/pyro/?page = Pyro
Swarthmore College
University of Massachusetts
SRI International
CARMEN, v.1.1.1 Carnegie-Mellon University http://carmen.sourceforge.net/
MissionLab, v.6.0 Georgia Institute of Technology http://www.cc.gatech.edu/aimosaic/robot-
lab/research/MissionLab/
ADE, v.1.0beta University of Notre Dame http://ade.sourceforge.net/
Miro, v.CVS-March 17, 2006 University of Ulm http://smart.informatik.uni-ulm.de/MIRO/
MARIE, v.0.4.0 Université de Sherbrooke http://marie.sourceforge.net/
FlowDesigner, v.0.9.0 http://flowdesigner.sourceforge.net/
RobotFlow, v.0.2.6 http://robotflow.sourceforge.net/

the use of the same control code both in simulation and for the number and position of sonar), and the effectors (e.g.,
an actual robot. While the only robot platforms supported the maximum velocity). The parameters are used by ARIA
are Probotic’s Cye and Nomad 150 robots, there are many for various calculations (e.g., the “RobotRadius” parame-
example simulation environments and control systems avail- ter is used to determine the robot’s turn limits). In support
able. The simulator was developed to be extremely flexible, of distributed computing, ARIA provides the ArNetwork-
supporting multiple, heterogeneous robot platforms and con- ing package as a wrapper around socket communications.
trol systems simultaneously. In addition, the Clay package In addition, the Simplified Wrapper and Interface Generator
allows hierarchical behavior specification, specifically tar- (SWIG, 2004) development tool is used to provide Java and
geting schema-based control. Inter-robot communication is Python support.
supported via sockets and serial ports only. A notable inclu- Supporting software is available, but is limited in some
sion is the Java CMU-Vision package, which supports frame cases by licensing or purchase requirements. The Mo-
captures and blob-tracking. bileSim 2-dimensional simulator, a modified version of
TeamBots publications include those from the hierarchi- Player/Stage’s Stage simulator (see Section 2.3), is freely
cal behavior (Balch, 2000) and education (Balch, 2002) sub- available. A demo version of the ActivMedia Color Track-
areas. ing Software (ACTS) is available for free download, but
has restricted functionality that disallows integration with
2.2 Advanced robotics interface for applications (ARIA) ARIA (not true of the licensed version). Additional open
source software includes the ArSpeech components that pro-
ARIA (MobileRobots, Inc., 2005; LaFary and Newton, 2005), vide interfaces to Sphinx speech recognition and both Festi-
the base software that comes packaged with the purchase of val and Cepstral speech production packages, SonARNL for
MobileRobots (neé ActivMedia) robots, is a set of C++ sonar-based localization, Mapper3 Basic for map creation
classes available for free download. At the lowest level, and editing, and VisLib for single camera object tracking.
ARIA provides system architecture capacities; that is, soft- Also available, but restricted to license and/or purchase are
ware that describes the structure of a robot (including its MobileEyes (which provides a remote robot display and con-
sensors, effectors, and physical specifications) and imple- trol GUI), ARNL (which provides laser-based mapping and
ments the low-level interaction between software and hard- localization), and Mapper3 (which augments the basic map-
ware components. At a higher level of abstraction, it also per package with laser support and automated map creation
includes some sensory interpretation functionality, basic ac- from sensor logs).
tions (analogous to behaviors), and an elementary action There are no publications available from projects that have
resolver. used ARIA.3
Although freely available, ARIA is a product of MobileR-
obots, Inc. and thus only supports MobileRobots platforms, 3
While publications exist for ARIA’s ancestral software, the authors
using robot parameter files as the means of defining the were explicitly requested to not refer to it.
characteristics of a robot. This includes information about
the robot body (e.g., the robot’s radius), the sensors (e.g.,

Springer
104 Auton Robot (2007) 22:101–132

2.3 Player/stage tion for ARIA’s MobileSim simulator. In addition to Stage,

a high-fidelity, 3-dimensional simulator called Gazebo is
The Player/Stage (Gerkey et al., 2001, 2003, 2005) project is available. In both cases, client code uses the same inter-
designed to be a programming interface, specifically avoid- face on real robots as in the simulator. The authors mention
ing being a development environment. Rather than treating that the device model makes it easy to simulate non-existent
a robot as the primary unit of agency, it instead focusses on devices (for instance, a type of sonar that penetrates walls
devices, or the various sensors and effectors. A “collection” to some extent) for further research in device design and
of devices is typically, although not necessarily, located on use.
a single robot. Supported platforms include MobileRobots, Player/Stage publications include those from the SLAM
RWI/iRobot’s RFLEX-based, Segway, Acroname, Botrics, (Wolf and Sukhatme, 2005), learning (Provost et al., 2004),
Evolution Robotics, and K-Team robots, while components education (Matarić, 2004), HRI task allocation (Tews
that are packaged along with the download include vector et al., 2003), multi-robot sensing (Jung and Sukhatme,
field histogram goal-seeking/obstacle avoidance, adaptive 2002), multi-robot exploration (Howard et al., 2002), multi-
Monte-Carlo localization, and a wavefront propagation path robot mapping (Howard et al., 2004), multi-robot localiza-
planner and interfaces for ACTS (see Section 2.2), CMVi- tion (Howard et al., 2003), multi-robot planning (Howard
sion, Festival, a service discovery mechanism, and others. et al., 2004), multi-robot coordination (Jones and Matarić,
In this software, Player refers specifically to the device 2004), multi-robot formation (Fredslund and Matarić, 2002),
and server interface. Devices are independent of one another and multi-robot task allocation (Gerkey and Matarić,
and “register” with a Player server to become accessible to 2002).
clients. Each client uses a separate socket connection to a
server for data transfer, thus preserving concurrent operation 2.4 Python robotics (Pyro)
of devices and the servicing of multiple requests. Minimal
constraints are placed on the use of devices; in a very real Pyro (Blank et al., 2003, to appear; Pyro, 2005) is a robot
sense, usage is only a communication protocol, leaving the programming environment aimed at, but not limited to, ed-
client the freedom (and by extension, the work) of design- ucational purposes, leading to specific choices in its design.
ing and implementing a control architecture. Stage, which One goal is to provide a top-down approach to the design
is the second part of the software, is a device simulator. of controllers, insulating students from low-level details of
Client control code uses the same programming interface implementation while preserving access to the low-level if
when used in conjunction with either simulated or physical it is desired. Some of the abstractions include: range sen-
devices. sors, robot units, sensor groups, motion control, and devices,
A stated design goal of the device model used in which encapsulate lower levels. This includes “wrapping”
Player/Stage is the separation of interface and function. The Player/Stage (see Section 2.3) and ARIA (see Section 2.2)
fact that servers communicate via a socket interface means functionality, so that any component written for those sys-
that client programs can be written using any language with tems are also available to Pyro users. A large selection of
socket support. According to Gerkey et al. (2003), currently platforms are supported, including K-Team Kheperas and
available libraries include those written in C, C++, Tcl, Hemission, MobileRobots Pioneer, Handyboard, Sony Aibo,
Python, Java, and Common LISP. Due to the prevalent use and all robots supported by Player/Stage (see Section 2.3).
of socket communication, the Player system inherently fits Educational modules exist to demonstrate control
the distributed computing paradigm. Client code is able to paradigms (e.g., neural networks, evolutionary algorithms,
operate on any host that has network connectivity, enabling vision processing, and reactive, behavior-based, finite state
location independence. A side effect of the device model and machines, etc.). Python, an interpreted language, was chosen
its networked basis is that combinations of devices can be as the basis of the system due to its ease of use for beginning
formed to create novel types of agents (e.g., one composed students, while permitting more knowledgeable designers to
of only sonar devices from many different robots). An ad- write more advanced code. While it is acknowledged that
ditional feature of the device model is that the frequency using an interpreted language leads to slower operation, the
of sensor and effector updates are independent, providing trade-off between usability and performance is consciously
clients the ability to make full use of the data generated by made. Construction of graphical visualization of robot oper-
devices that operate at a high frequency, while not being ation are explicitly supported through the use of pre-defined
hindered by those that are slower. facilities and Python’s OpenGL interface. Another goal is to
Stage is a graphical, 2-dimensional simulator that models design control code that operates on many different robots
devices in a user defined environment. Specifically designed with no modification. An example of this is the use of
to support research in multi-robot systems through its use “robot units” that replace traditional measurements such as
of socket-based communication, it also forms the founda- meters.

Springer
Auton Robot (2007) 22:101–132 105

Pyro publications include those from the learning (Blank width, sonar offsets, maximum velocities, etc. Parameters
et al., 2002), education (Blank et al., to appear), and HRI for each component are stored in a human readable text file
task allocation (Desai and Yanco, 2005) subareas. repository, but a graphical editor can be used to modify pa-
rameters at run-time. In addition, each component relies on
2.5 Carnegie mellon robot navigation toolkit (CARMEN) a set of IPC message definitions to which other components
can subscribe, allowing component distribution through an
CARMEN (Montemerlo et al., 2003a, b) is an open source IPC server.
collection of (mobile) robot control software written in the C CARMEN publications include those from the SLAM
programming language that is meant to provide a “consistent (Thrun et al., 2000), learning (Osentoski et al., 2004), HRI
interface and a basic set of primitives for robotic research”. assistive robotics (Pineau et al., 2002), and multi-robot co-
Oriented towards single robot control, it uses a three layer ordination (Simmons et al., 2000) subareas.
agent “architecture,” in which the first layer is the hard-
ware interface, providing low-level control and integration 2.6 MissionLab
of sensor and motion data, the second layer is concerned
with basic robot tasks such as navigation, localization, ob- MissionLab (MacKenzie et al., 1997; MissionLab, 2003) is a
ject tracking, and motion planning, and the third layer is the set of software tools for executing military-style plans using
user-defined application, which relies on the primitives of individual or teams of real and simulated robots. Developed
lower layers. Modularity is a primary concern, supported by as part of the DARPA Mobile Autonomous Robot Software
the Inter-Process Communication System (IPC) communi- (MARS) project, the main stated goal is to control the
cation protocol/software (discussed in more detail below). motion of robots in highly dynamic, unpredictable, and
Besides supporting a number of robot platforms (including possibly hostile environments. Collaboration and coordina-
MobileRobots, Nomadic Technologies Scout and XR4000, tion of robot teams is based on the Societal Agent theory,
Segway, iRobot ATRV, ATRVjr, and B21R) and navigation which views abstract “assemblages” of agents as agents
primitives (map-making, Monte-Carlo particle filter local- themselves and whose behavior, in turn, is the aggregate
ization (Thrun et al., 2000), and Markov decision process of coordinated “primitive” behaviors of “atomic” agents.
path planning (Konolige, 2000)), CARMEN also provides Assemblages are hierarchical, while behavior coordination
configuration tools, a simulator, and graphical displays and is achieved through finite state automata (either competitive
editors. or temporally sequenced) or vector summation cooperation.
IPC (Simmons, 1994, 2004) provides high-level support The Configuration Description Language (CDL) is used
for connecting processes using TCP/IP sockets and send- to recursively define abstract societal agents (called config-
ing data between processes, including opening and clos- urations), usually accomplished using the graphical CfgEdit
ing sockets, registering, sending, and receiving messages, tool. A configuration can be bound to a specific set of robots
which may be anonymous publish/subscribe or client/server and devices; robot choices include MobileRobots Pioneer,
type communications. The IPC library contains functions iRobot ATRVjr and Urban, Evolution Robotics ER-1, and
to marshall (serialize) and unmarshall (de-serialize) data, Nomad 150/200. CDL is compiled to Configuration Network
handles data transfer between machines with different En- Language (CNL) code, which is then compiled to C++ code
dian conventions, invoke user-defined handlers when a mes- and finally compiled to machine code, resulting in a robot
sage is received, and invoke user-defined callbacks at set executable. The executable contains a communication mod-
intervals. In essence, IPC performs a function similar to ule (called HClient) to interface with an HServer, an abstract
a naming service for components; besides providing the control interface used for all robot hardware via IPT commu-
means to define message abstractions used for commu- nication software. (IPT supports distributed computing and
nication over a network, it also encourages extensibility is related to the IPC communication software, described in
(in that components are self-contained) and fault-tolerance Section 2.5; both are derived from the Task Control Architec-
(in that failure of a component ceases communication, ture (TCA, Simmons, 1994) project.) Developers also have
but does not actively interrupt other components in the the option of using the higher level Command Description
system). Language (CMDL) to describe robot missions, which is a
CARMEN components generally take the form of a sin- mechanism for providing high-level input to robot behaviors.
gle executable, such as pioneer (for a MobileRobots Pio- As a primary concern of MissionLab is usability, the
neer robot), laser (for a SICK laser range finder), or localize graphical interface is quite extensive, allowing non-experts
(for robot localization using a pre-made map). Particular to write control code without any programming. Logging
platform definitions are contained in “base” specifications, consists of writing a robot’s position, velocity, heading, and
which are then abstracted to a generic “robot” configura- the current state of the robot with respect to time to a disk file,
tion that includes basic parameters such as body length and while debugging toggles are used to display program output

Springer
106 Auton Robot (2007) 22:101–132

to a console. A unique feature relative to other systems is the general-purpose rule interpreter, a Prolog interface, and
inclusion of “Motivational Variables” (anger, fear, hunger, “wrappers” to incorporate external software. It also includes
curiosity) that simulate emotionality. Developers can also a Java implementation of a Player (see Section 2.3) client
assign user-defined “Personalities” to robots. Finally, there that interfaces with the Stage 2-dimensional robot simulator
is an extensive set of components available with Mission- and other Player/Stage components, in addition to an
lab, including a case based reasoner, Q-learning, graphical interface to the simulator packaged with the now defunct
behavior building tool, D∗ Lite planner, and human/robot (Saphira Konolige et al., 1997; Konolige, 2002) system.
interaction interfaces. ADE publications include those from the plan-
MissionLab publications include those from the learning ning/navigation (Kramer and Scheutz, 2003), hierarchical
(Arkin et al., 2003), hierarchical behavior (MacKenzie and behavior (Scheutz and Andronache, 2004), HRI task al-
Arkin, 1993), HRI task allocation (MacKenzie and Arkin, location (Scheutz et al., 2004), assistive robotics (Scheutz
1998), multi-robot planning (Endo et al., 2004), multi-robot et al., 2006), multi-robot sensing (Andronache and Scheutz,
coordination (MacKenzie et al., 1997), multi-robot formation 2004a), and multi-robot coordination (Scheutz, 2006).
(Balch and Arkin, 1999), and multi-robot task allocation
(Arkin et al., 1999) subareas. 2.8 Middleware for robots (Miro)

2.7 APOC development environment (ADE) Miro (Utz et al., 2002; Miro, 2005) is a distributed, ob-
ject oriented framework for mobile robot control that is
ADE (Andronache and Scheutz, 2004a, b; Scheutz, 2006) meant to facilitate heterogeneous software integration and
is a programming environment that combines (1) support foster portability and maintainability of robot software. Core
for developing and implementing agent architectures with components have been developed in C++ for Linux based
(2) the infrastructure necessary for distributing architectural on Common Object Request Broker Architecture (CORBA)
components. An explicit goal is to combine features of technology using the adaptive communication environment
multi-agent systems (by treating architectural components (ACE, Schmidt, 1994) as its communication framework. Due
as “agents” in a MAS-sense) with those of a programming to the programming language independence of CORBA, fur-
environment and toolkit for complex agent design and im- ther components can be written in any language and on any
plementation. ADE is a Java implementation of the APOC platform that provides CORBA implementations.
(Activating, Processing, Observing Components) (Scheutz Miro currently supports three platforms: iRobot B21,
and Andronache, 2003; Scheutz, 2004) universal agent MobileRobots Pioneer, and the custom-built Sparrow. Ab-
architecture framework, which provides arbitrary levels of straction interfaces include odometry, motion, rangesensor
(possibly hierarchical) component abstraction and intercon- (sonar, infrared, bumper, laser), stall, video, pantilt, GUI but-
nection. Communication among ADE components relies tons, and speech. Components exchange data based on sub-
on Java’s Remote Method Invocation (RMI) facilities. ADE scriptions, which allow for event driven notification. Defined
provides infrastructural components for an enhanced naming messages include those for odometry, rangesensor (scan-
service, connection mediation and monitoring, security event, groupevent, bunchevent), sonar, infrared, bumper,
features (access control and authentication), and the ability stall, and GUI buttons. Miro includes a “behavior engine”
to store the run-time state of the system, which in turn allows for reactive behavior specification, which allows hierarchi-
for the detection and recovery from component failures. cal decomposition of timed, event and task behavior sets
While ADE is limited to MobileRobots robots and Arrick into “policies”. There are two types of policy transitions,
Robotics’ Trilobot, a set of abstractions for typical robotic local and global, that can be edited via a graphical inter-
sensors and effectors provide the means for extending face; global policies preempt behaviors, local do not. The
support to other platforms. Configuration files can take the configuration of hardware, data subscriptions, and logging
form of either text or XML files and include both an abstract specification is stored in XML files. Two types of logging
architecture description and/or the run-time specification are defined, “log levels” and “log categories”, that allow de-
of component distribution. Graphical representations of velopers to vary the granularity of log data, while a graphical
individual components exist, accessible via a distributed, LogPlayer allows the replay of logged data.
multi-user GUI, which provides a view of the complete agent Miro publications include those from the SLAM (Kraet-
architecture and the means to control individual components. zschmar et al., 2004), planning/navigation (Kraetzschmar
Logging facilities allow any component in an ADE system et al., 2000), learning (Fay et al., 2004), hierarchical
to write to multiple files. ADE provides several predefined behavior (Utz et al., 2005), HRI assistive robotics (Gassull,
components including facilities for behavior definition, 2001), multi-robot sensing (Utz et al., 2004), and multi-robot
vision processing, speech recognition and production, a coordination (Utz et al., 2004) subareas.

Springer
Auton Robot (2007) 22:101–132 107

2.9 Mobile and autonomous robotics integration various aspects of multi-agent systems (MAS), including
environment (MARIE) comparing agent platforms (Altmann et al., 2001; Nguyen
et al., 2002; Laukkanen, 1999; Nowostawski et al., 2000;
MARIE (Côté et al., 2004, 2006; Côté, 2005) is a pro- Ricordel and Demazeau, 2000), agent development kits (Bit-
gramming environment that is specifically designed with ting et al., 2003), mobile agent systems (Silva et al., 2001),
the integration and distribution of robot applications, com- or agent environments (Eiter and Mascardi, 2002). There are
ponents, and tools in mind. For brevity, “MARIE” is used also comparisons of general agent systems and agent archi-
throughout this description and the rest of the survey to tectures per se (Sloman, 1998; Logan, 1998; Sloman and
signify the MARIE software and the related FlowDesigner Scheutz, 2002). Comparisons that concern robotic agents
(Valin and Létourneau, 2004) and RobotFlow (Michaud in particular have addressed mobile robotic architectures
and Létourneau, 2004) packages (described below), unless (Orebäck and Christensen, 2003) and robot programming
clarification is necessary. MARIE is implemented in C++ environments (MacDonald et al., 2003; Jia et al., 2004; Biggs
and the integration aspect of MARIE proper uses (but is and MacDonald, 2003). Common to all is the need to estab-
not dependent upon) the Adaptive Communication Environ- lish an appropriate set of criteria that serves as a basis for the
ment (ACE, Schmidt, 1994) communication framework. Fol- comparison. Clearly, the choice of criteria is critical, for, as
lowing the mediator design pattern (Gamma et al., 1994), pointed out in Ricordel and Demazeau (2000), “any criteria
MARIE provides a centralized component that connects a is relevant to a specific outside need.”
variety of (possibly different) software. There are four func- We briefly review some of this prior work to situate our
tional components: application adapters, communication proposed evaluation criteria, giving a general overview of
adapters, communication managers, and application man- the conceptual breakdown in each and why each proves in-
agers. Application adapters act as proxies between the central sufficient for the purposes of this paper. To avoid ambiguities
component and applications. Communication adapters trans- and equivocations among the different terms used, we will
late data between application adapters, while communication adhere to the following terminology for the rest of this paper:
managers create and manage the links. Finally, application r Platform: the hardware on which an application will be
managers coordinate system states and configure and control
executing; this includes the sensors, actuators, comput-
system components on any one processing node. In keeping
ers, operating system(s), and other hardware or software
with the aim of integrating software, components have been
intimately tied to hardware.
developed for Player/Stage (see Section 2.3), CARMEN (see r Component: a functionally independent part of an agent
Section 2.2), and ARIA (see Section 2.2).
or system.
FlowDesigner is a data-flow processing library coupled r Architecture: the structure and interaction of components;
with a graphical display that allows developers to create
if necessary, a distinction will be made between system
reusable “software blocks” linked together in a (possibly
and agent architectures.
hierarchical) network. Available libraries include support r Agent: the sum of the software and hardware required for
for signal processing, audio processing (DSP), vector quan-
an individual robot to perform its task. In particular, we
tization, neural networks, fuzzy logic, an Octave plug-in,
will not consider infrastructure or strictly software agents
and RobotFlow. RobotFlow is a mobile robotics toolkit for
(e.g., a naming service or communication agent) as agents
FlowDesigner that includes support for MobileRobots Pio-
per se, as is done in the field of multi-agent systems. These
neer2 robots and other hardware devices, behaviors, finite
are instead considered functional components that are part
state machines, vision processing (color training, tracking,
of the broader environment or application.
etc.) and the interfaces for use with MARIE. r Programming environment: the tools, infrastructure, and
MARIE publications include those from the plan-
components that are not left for implementation by the
ning/navigation (Beaudry et al., 2005), education (Michaud,
developer. The term system will be used interchangeably
2005), HRI assistive robotics (Labonté et al., 2005), multi-
in this context.
robot localization (Rivard, 2005), multi-robot coordination
(Guilbert et al., 2003), and multi-robot formation (Lemay The most general and, for our purposes, pertinent, frame-
et al., 2004) subareas. work is Eiter and Mascardi (2002). Although founded in
MAS research, the classification is intended to be compre-
hensive, establishing a framework for all types of agent sys-
3 A conceptual framework for comparing RDEs tems. Additionally, the authors provide a practical method
for choosing an appropriate system for a task selected by
Several comparisons of agent systems and agent develop- an application designer. Criteria are divided into five cate-
ment environments have been proposed in the recent litera- gories: (1) agent attitudes, (2) software engineering support,
ture. For software agents, they are typically concerned with (3) implementation concerns, (4) technical issues, and (5)

Springer
108 Auton Robot (2007) 22:101–132

economical aspects.4 While the software engineering, im- aspects of RDEs. For one, the boundaries of the categories
plementation, and technical issues categories usually have overlap to such an extent as to be unclear. For instance, infras-
a prominent role in discussions of RDEs, the agent atti- tructure conflates the facilities provided by the environment
tudes aspect is often omitted–not because it is unimportant with both the programming and the agent architecture cate-
or ignored, but because features therein often form the task, gories. Similarly, the broad scope of the HRI (human/robot
or object, of investigation. Yet, according to Eiter, the at- interaction) category largely overlaps the robot programming
titudes category is comprised of features that discriminate category, yet contains individual features that are too specific
between agent and non-agent software: they are either basic for a general system comparison (i.e., excluding systems that
(i.e., “close to the very core of agenthood”) or advanced (i.e., are not especially intended nor designed for HRI). Moreover,
“desirable but not of central interest”). Hence, an agent devel- the proposed categorization is not structured in a way that
opment environment (and by extension an RDE) should, at is easily amenable to a systematic comparison (e.g., concep-
least in part, be evaluated with respect to the degree to which tually different items are subsumed under the rubric “robot
it supports these attitudes. While Eiter and Mascardi’s cat- programming”).
egorization is comprehensive, it lacks some details of con- The study closest in intent to this survey is Orebäck and
siderable importance for evaluating RDEs. In fact, this is Christensen (2003), which attempts to establish the char-
explicitly acknowledged with the disclaimer, “other features acteristics of a “common software architecture” for mo-
and criteria should be taken into account” for the unique is- bile robot systems. In particular, seven categories (hardware
sues that arise in the development of physical agents (e.g., abstraction, scalability, overhead, control model, software,
support for devices, real-time operation, etc.). tools and methods, and documentation) are proposed as a
In their framework proposal, Jia et al. (2004) isolate three basis for comparing RDEs, covering an extensive range of
high-level categories for analysis of an RDE: (1) openness, features. However, while the proposed framework is gener-
(2) abstraction, and (3) modularity. Openness refers to ex- ally suitable, the actual comparison is limited to only three
tensibility: a programming environment should support the RDEs (Balch, 2004; Konolige et al., 1997), and BERRA
addition and evolution of hardware and software. Abstrac- (Lindstrom et al., 2000)) and does not adhere strictly to the
tion forms the basis on which openness is built, providing conceptual framework. Rather, criteria are grouped into six
a well-defined application programming interface (API) that areas that mostly, but not always, correspond to the categories
allows a developer to work at a level beyond the hardware as defined, in some instances leaving out or introducing new
(see also Vaughan et al., 2003). Different from abstraction, criteria.
which is focussed on hardware, modularity concerns soft- While all of the above studies agree that the main purpose
ware, promoting good design and reusability. While these of an RDE is to provide appropriate tools and abstractions
three categories address the design and implementation of that help the agent designer, they fail to provide a compre-
autonomous mobile robotic applications (as demonstrated hensive, yet succinct conceptual framework that allows for a
by their in-house development of the Frontier-1 robot), they systematic comparison of RDEs. Based on the three typical
are too general to address specific concerns of RDEs (e.g., stages in the development process of a robotic agent archi-
real-time support, hardware-dependence of a robotic plat- tecture5 (design, implementation, and execution), we propose
form, debugging tools, etc.). four categories of criteria for RDE comparison, categorized
MacDonald et al. (2003) give a detailed and comprehen- in terms of:
sive description of RDE features in three categories: (1) robot
F1: Specification, which includes formalisms, methodolo-
programming (both at the system and task level, which en-
gies, and design tools,
able programmers to describe robot behavior), (2) infrastruc-
F2: Platform support, which is related to the hardware and
ture (which supports the execution of behavior descriptions),
its low-level interface (e.g., the operating system),
and (3) human-robot interaction (HRI, which allows inter-
F3: Infrastructure, which refers to components and capa-
action with the robot programming area; see also Biggs and
bilities that are part of the RDE, but not the “agent
MacDonald, 2003). The proposed features will be largely
architecture proper,” and
included in our comparison, but there are some issues con-
F4: Implementation, which includes aspects of application
cerning the analysis, organization, and application to various
development (including predefined components used in
an agent architecture).
4
Eiter’s economical aspects category will not be considered here, ex-
cept for the documentation criterion, as the selected RDEs are both Of the four categories, three reflect features relevant to
open source and research-oriented. Related considerations, such as the specific development stages (e.g., specification features are
cost of application development, RDE maintenance or modification,
training, etc. are, however, addressed by the usability evaluation in 5
Section 5. Our stages are similar to Ricordel and Demazeau (2000), although we
subsume the analysis category as part of the design stage.

Springer
Auton Robot (2007) 22:101–132 109

central to design, implementation features pertain to imple- F2: Platform support

mentation, as the name suggests, and platform features play
a role in the execution). The fourth category, infrastructure, Robotic applications necessarily incorporate real-world sen-
is added to explicitly distinguish aspects of an RDE that are sors and effectors; thus, they require a more diverse set of
separate and distinct from the agent architecture (e.g., distri- hardware than software-only systems. The principles of ab-
bution mechanisms that are integral to system operation, yet straction, modularity, and openness, as put forth in Jia et al.
usually transparent to the agent designer). (2004), are of particular importance to this category, promot-
We note in advance that the four categories are comprised ing application use across varying platforms.
of features that an RDE objectively has or does not have, with
F2.1: Operating system. An RDE may be compatible with
an emphasis on the software engineering aspects of its func-
one or many operating systems, but must be compat-
tional characteristics and capabilities. These criteria alone
ible with the designer’s choice. This can become a
are not sufficient for a full evaluation of an RDE and are
major obstacle when certain libraries or components
supplemented with additional criteria in Sections 5 and 6.
are implemented only for a particular operating sys-
The different types of evaluation can be distinguished by
tem.
an identifying prefix; criteria in this section are denoted
F2.2: Hardware support. “Hardware support” refers to the
by F, followed by a category and item number. It is cru-
variety of sensors and effectors that are available in
cial to note that even though the expanded criteria list pro-
an RDE, such as cameras, sonar, and laser devices.
vides a comprehensive foundation for RDE evaluation, it is
Since the number of standard (that is, common and
impossible in principle to address every concern a devel-
non-custom) devices is limited and widely used on
oper might have. A remedy for the situation is discussed in
different platforms, we will refer instead to particular
Section 6.
robot manufacturers. In support of increased modular-
ity, ease of device modification, and addition of cus-
F1: Specification
tom devices, a hierarchy of device abstraction is often
specified, allowing control code to be easily ported and
The specification of a robotic agent or application occurs
executed on different robots.
in the design stage and concerns issues such as the appli-
F2.3: Simulator. Simulation of the physical world allows de-
cation domain(s), software engineering, and determination
velopers to test applications, model currently unavail-
of an appropriate agent architecture. To preserve the focus
able hardware, and replay actual application execution.
on RDEs, the criteria presented are somewhat broad, but are
Simulators can be low- or high-fidelity, approximating
sufficient to address the prevailing concerns.
an environment to some lesser or greater degree, and
F1.1: Architectural Primitives. An RDE provides various can also be two- or three-dimensional. Some simula-
types of predefined functional component and/or tors have the ability to include multiple robots in a
knowledge primitives useful in robotic applications single simulation or to mix real and simulated robots
(e.g., behaviors, methods of control, tasks, objects, in an environment.
etc.), or the means to create, organize, and manipulate F2.4: Configuration method. The configuration of a robot
them. is often changed to meet the demands of various
F1.2: Software engineering. Software engineering support applications. This information may be incorporated
promotes the creation of high-quality software. En- into the source code (requiring compilation to effect
abling modularization and code reuse, it can be ac- changes) or in configuration files that can be eas-
complished through the use of stated design princi- ily modified, either with a text editor or a graphical
ples, explicit frameworks or tools, methodologies, or interface.
formalisms, and includes application verification, pro-
totyping, and the abstractions mentioned in Jia et al. F3: Infrastructure
(2004), Orebäck and Christensen (2003) and Vaughan
et al. (2003). Infrastructure refers to RDE functionality that affects
F1.3: Architecture neutrality. An RDE may be associated multiple components (or the system as a whole) and is not
with a particular theoretical foundation that promotes tailored to individual architectural components, application
a specific agent/application architecture, separate from domains, or particular stages of application/agent develop-
implementational concerns. Alternatively, it may be ment. For example, logging facilities can be used with any
architecture neutral, leaving the choice to the designer or all components, are often invaluable as debugging tools
or even providing the means to compare application during the implementation stage, and provide records of an
implementations using different agent architectures. execution instance for later performance analysis. In some

Springer
110 Auton Robot (2007) 22:101–132

cases, however, it may be impossible to determine whether zation, while distribution mechanisms (F3.4) encom-
a feature is due to a specific component or part of the passes mechanisms used to add computational hosts.
infrastructure by function alone. For instance, the graphical Additional concerns might include the overhead in-
representation of components might be implemented on an volved with message passing, both within a single
ad hoc basis, removing it from consideration as infrastruc- host and among connected hosts, task allocation for
ture. A system must provide generic mechanisms that supply multi-robot applications, or other concerns. Scalabil-
these capabilities for them to be considered as infrastructure. ity is used here in a broad sense as a general system
property, inclusive of the above.
F3.1: Low-level communication. Inter-process communica- F3.6: Component mobility. “Mobility” refers to the potential
tion (such as memory mapping, pipes, or sockets), ba- to relocate components at run-time. In robotic applica-
sic networking protocols (such as UDP, TCP/IP, etc.), tions, however, it is somewhat constrained by possible
and mid-level protocols (such as IIOP or RMI) are part dependence on the location of the requisite hardware.
of the system infrastructure. These capabilities are of- When an application is distributed across many hosts,
ten dictated by the platform being used, as their avail- component mobility can be used for dynamic resource
ability is contingent on the operating system and/or allocation or run-time system reconfiguration, assum-
programming language. ing there are mechanisms that allow reconnection to
F3.2: Logging facilities. Log files of application operation data sources. Ultimately, these capabilities would be
can be used for debugging, repetition of an application automatic, adjusting operation with a changing com-
execution, or gathering performance statistics. Log- puting environment.
ging mechanisms can have various levels of flexibility, F3.7: System monitor/management. A system monitor dis-
including fixed (which generally captures all data pro- plays the status of multiple application components,
duced by components) vs. configurable data content, often in graphical form. An extension of simple mon-
local vs. remote logging, file name selection, single itoring can allow for the management of the compo-
vs. multiple data streams and/or files, or the ability to nents’ operation, ranging from starting and stopping
start and stop logging at run-time. to adjustment of parameters. Such extensions are often
F3.3: Debugging facilities. While logging facilities can suf- implemented as part of individual components, which
fice for basic debugging, robust debugging tools can be are treated separately as an implementation character-
invaluable during application implementation. Such istic in Subsection crit-impl-char and do not qualify
tools can range from low-level code editors, to mid- as part of the infrastructure.
level graphical representations of sensors and effec- F3.8: Security. An application executing on a single robot
tors, to high-level graphical behavior or task modifi- may not need any security mechanism, but distribution
cation, possibly allowing run-time suspension, modi- across many hosts raises such concerns. Predefined
fication, and restarting. components for encryption, authentication, and access
F3.4: Distribution mechanisms. Distribution mechanisms, control can be available for ready integration into ap-
as part of the infrastructure, are required for multi- plications. (A related discussion of security concerns
host applications. Typically, distribution capabilities in the multi-agent system RETSINA can be found in
are enabled by middleware (e.g., Poggi et al., 2002), Singh and Sycara (2004).)
either as a generic component implementation frame- F3.9: Fault-tolerance. Repeated failures of both hardware
work (such as CORBA, 2005, SOAP, 2003, etc.) and software are common in robotic applications. The
or in the form of particular components (such as system infrastructure may incorporate generic mech-
an agent naming service, directory facilitator, bro- anisms for failure detection, or be structured such
ker agents, or other components that provide similar that disruptions due to failed components do not halt
functionality). the entire application. Extending this concept, mecha-
F3.5: Scalability. As robotic applications grow in scope nisms for failure recovery may exist that enable com-
and capabilities, a developer must be concerned with ponents to automatically recover from failures with no
how an RDE handles increasing complexity. The term outside intervention (for instance, see Melchior and
“scalability” can refer to many different aspects of a Smart, 2004).
system, some of which are addressed more specifically
by other criteria. For instance, architectural primitives F4: Implementation
(criteria F1.1) and a high-level language (F4.1.2) in-
cludes facilities for managing complex actions and In practice, an important reason for selecting a particular
behaviors, software engineering (F1.2) takes into ac- RDE is to facilitate the implementation of an agent architec-
count modularization that promotes system organi- ture. We subdivide implementation features into two areas:

Springer
Auton Robot (2007) 22:101–132 111

(1) implementation characteristics, which are somewhat ab- F4.2: Predefined components
stract and refer to implementation concerns that are not pre-
defined components, and (2) predefined components, which Predefined components are analogous to software libraries;
perform some specific function that can be directly incorpo- since the list is open-ended and will most assuredly expand
rated into an architecture. in the future, we deviate from the format used thus far and
give a necessarily incomplete list of common components
F4.1: Implementation characteristics with corresponding citations. Furthermore, the list assumes
a fairly high-level viewpoint, necessary to maintain an ac-
F4.1.1: Programming language. Architecture implementa- ceptable level of commonality among systems.
tion necessitates the use of programming languages, Currently, most RDEs include predefined components
such as C or Java. An RDE that is itself imple- for map-making (F4.2.1), localization (F4.2.2, e.g., Thrun,
mented in the particular language used for the appli- 2003), route planning (F4.2.3, e.g., Konolige, 2000), speech
cation guarantees compatibility; however, an RDE recognition (F4.2.4), speech production (F4.2.5), and vi-
may also supply interfaces or wrappers that inter- sion processing (F4.2.6, with various capabilities such as
face easily with other languages. blob tracking, edge detection, motion tracking, etc.). Some
F4.1.2: High-level language. Some programming environ- less common components are rule interpreters (F4.2.7, e.g.,
ments integrate higher-level languages, such as JESS, 2003 or Sloman, 2002), planners (F4.2.8, e.g., Maes,
The Behavior Language (Brooks, 1990), COLBERT 1990; Jensen and Veloso, 1998; Stentz, 2002), neural net-
(Konolige, 1997), or GRL (Horswill, 2000) for be- works ( F4.2.9, e.g., Koker et al., 2004), and machine learn-
havior description or ACL (FIPA-ACL, 2002) or ing (F4.2.10, e.g., Vijayakumar et al., 2002; Russell, 2004).
KQML (Mayfield et al., 1996) for agent commu- Even less common, and therefore not included in the evalua-
nication. These high level languages can be used tion criteria, are support for instruction/teaching (e.g., Sku-
within an agent architecture (e.g., to facilitate data bic and Volz, 1998; Bentivegna and Atkeson, 2002), human
transfer between components) or in multi-robot ap- robot interaction facilities (e.g., Fong et al., 2003), affect
plications. (e.g., Pfeifer, 1988; Moshkina and Arkin, 2003; Scheutz
F4.1.3: Documentation. The usability of an RDE is greatly et al., 2006), and coordination mechanisms (e.g., Hoff and
enhanced by the inclusion of well-documented code Bekey, 1995; Chaimowicz et al., 2003; Dias and Stentz,
and user manuals that may include the system’s 2003).
API specification, answers to frequently asked ques-
tions, trouble-shooting guides, instructions concern-
ing custom extensions, etc. 4 RDE feature criteria evaluations
F4.1.4: Real-time operation. Real-time constraints are often
critical in designing and operating robot architec- For each of the RDEs in Section 2, a value has been as-
tures. Real-time capabilities of an RDE are generally signed for the criteria from Section 3, determined using the
dependent on the operating system and/or program- system’s documentation and verified based on usage expe-
ming language. rience (a synopsis of experimental implementations and the
F4.1.5: Graphical interface. An RDE may supply pre- usability evaluation is provided in Section 5). Three types
implemented graphical interfaces that enhance in- of assignments
√ are made: (1) binary, signified by a blank
dividual component visualization during applica- for no and for yes, (2) ternary, signified by
for not
tion execution, including displays related to var- supported,  for partially supported, and  for well sup-
ious sensors, effectors, behaviors, robot control, ported, and (3) listings, which are text descriptions. Table 2
navigational plans, etc. Additionally, RDEs may shows the values assigned to each system for each criteria,
define a standardized method of adding such while further explanation is given in the text. The following
displays. shorthand column headings are used to designate particular
F4.1.6: Software integration. RDEs may provide tools that systems: TB–TeamBots, AR–ARIA, P/S–Player/Stage, Py–
facilitate the integration of external software, either Pyro, C–CARMEN, ML–MissionLab, AD–ADE, Mi–Miro,
at the component level (e.g., a localization routine) or and MA–MARIE.
a complete application-as-component (e.g., speech
production), greatly enhancing development time F1: Specification
and effort. A notable development in this area is
“wrappers” for components of other robotic systems F1.1 Architectural primitives: To attain a somewhat sup-
that promote the integration, sharing, and reuse of ported value, a system must provide at least one
components. form of robot control. Systems that provide additional,

Springer
112 Auton Robot (2007) 22:101–132

Table 2 Feature criteria evaluation by RDE

Category Criteria TB AR P/S Py C ML AD Mi MA

Specification F1 F1.1 Architectural primitives 
 
   

F1.2 Software engineering




 


√ √ √ √ √ √ √ √ √
F1.3 Architecture neutrality

Platform F2 F2.1 Operating system J U,W U,W U,W U U J U,W U

F2.2 Hardware support     



F2.3 Simulator

 




F2.4 Configuration method 


  

Infrastructure F3 F3.1 Low-level communication S S S S I I R C S

F3.2 Logging facilities   

 

F3.3 Debugging facilities 
      

F3.4 Distribution mechanisms 



  
F3.5 Scalability 
 
 

F3.6 Component mobility  




F3.7 Monitoring/Management         
F3.8 Security 
      

F3.9 Fault-tolerance  




Implementation F4 F4.1.1 Programming language Java C++ C++ Pyth C C++ Java C++ C++
√ √ √ √ √ √
F4.1.2 High-level language
F4.1.3 Documentation      


F4.1.4 Real-time operation

F4.1.5 Graphical interface    




F4.1.6 Software integration  




√ √ √ √ √ √
F4.2.1 Map-making
√ √ √ √ √ √ √ √
F4.2.2 Localization
√ √ √ √ √ √ √
F4.2.3 Route planning
√ √ √ √ √
F4.2.4 Speech recognition
√ √ √ √ √ √
F4.2.5 Speech production
√ √ √ √ √ √ √ √
F4.2.6 Vision processing
√ √ √
F4.2.7 Rule interpreters
√ √
F4.2.8 Planners
√ √ √
F4.2.9 Neural networks
√ √ √
F4.2.10 Learning

likely more complex, methods of robot control receive menting robot control and so receives a not supported
a well supported value. Player/Stage does not pro- value. ARIA provides a set of basic actions and an
vide any predefined control methods, following their elementary priority-based action resolver. CARMEN
policy of providing only the framework for imple- provides a Markov decision process planner as part of

Springer
Auton Robot (2007) 22:101–132 113

its navigation component. Each receives a somewhat run on UNIX systems. Letter codes in Table 2 are as
supported value. The rest of the systems are considered follows: J = Java, U = UNIX, W = Windows.
well supported. TeamBots provides schema-based mo- F2.2 Hardware support: Hardware support, as used here,
tor control, finite state machine (FSM) sequencing, and refers to specific robot manufacturers/platforms. We
hierarchical behaviors via the Clay behavior configu- assume a relatively limited pool of sensors and ef-
ration system. Pyro provides both subsumption and fectors are used across platforms (such as SICK LMS
fuzzy blending of behaviors, while MissionLab pro- lasers), although all systems allow specification of cus-
vides schema-based control, behavior sequencing and tom sensors and/or effectors. To attain a somewhat sup-
artifacts. ADE provides access to a general-purpose ported value, a system must support at least three dif-
rule interpreter, schema-based and subsumption-based ferent platforms; more than five earns a well supported
behavior primitives, a Prolog interface, and a dis- value. ARIA supports only MobileRobots robots, ADE
tributed neural-network style component model based supports MobileRobots and Arrick Trilobot, Team-
on APOC (Scheutz and Andronache, 2003). Miro in- Bots supports Cye and Nomad 150 robots, Miro sup-
cludes a custom “behavior engine”, based on that in- ports MobileRobots, iRobot B21, and their in-house
troduced in Brooks (1991). MARIE, via the Robot- Sparrow platforms, MissionLab supports MobileR-
Flow and FlowDesigner packages, provides hidden obots, iRobot, Evolution Robotics ER-1, and Nomad
Markov models, fuzzy blending, FSMs, an interface robots, MARIE supports MobileRobots platforms na-
to Octave software, and other primitives. tively, in addition to all platforms available through
F1.2 Software engineering: To attain a somewhat supported ARIA, Player/Stage, and CARMEN via its adapters,
mark, an RDE must explicitly state design principles, Player/Stage supports MobileRobots, iRobot, Segway,
be implemented using an object oriented programming Acroname, Botrics, Evolution Robotics, and K-Team
language (e.g., C++ or Java), or make use of a high- platforms, CARMEN supports MobileRobots, Aibo,
level object language (e.g., CORBA). An explicit theo- Nomadics, iRobot, and Segway platforms, and Pyro
retical foundation yields a well supported mark. Team- supports MobileRobots, Aibo, Cye, iRobot, Khepera,
Bots, Player/Stage, ARIA, CARMEN, Pyro, MARIE, Nomad, and Segway platforms.
and Miro are of the former type, while the use of F2.3 Simulator: To attain a somewhat supported value, an
Societal Agent theory in MissionLab and the APOC RDE must at least provide a low-fidelity, 2-dimensional
formalism in ADE provides the basis for receiving a simulator (which may or may not support multi-robot
well supported value. simulations). To attain a well supported value, an RDE
F1.3 Architecture neutrality: All the systems under consid- must provide a high-fidelity, 3-dimensional simulator
eration are neutral with regard to agent architectures, that supports multi-robot simulations and may be used
although MissionLab has a strong association with the to model robotic mechanisms. CARMEN includes
AuRA architecture (Arkin and Balch, 1997) and CAR- a 2-dimensional simulator that supports low-fidelity
MEN has been described by its authors as an exam- single-robot simulation that can, with some manipu-
ple of the 3T hybrid architecture (Montemerlo et al., lation of the IPC communications, be used for multi-
2003b). However, neither enforces the use of the as- robot simulations. TeamBots, ARIA, MissionLab and
sociated architecture, and can therefore be considered ADE provide low-fidelity, multi-robot simulators. Mis-
agent architecture neutral. sionLab also supplies a low-fidelity 3-dimensional sim-
ulator (although the manual states that its use will
F2: Platform support halt the system). A major component of Player/Stage,
as indicated by its name, is the Stage 2-dimensional
F2.1 Operating system: Compatible operating systems have simulator, which is low-fidelity and supports multiple
been determined according to information from system robots. Also available is the Gazebo high-fidelity 3-
documentation and do not necessarily discount those dimensional simulator, which elevates Player/Stage to
not listed. If a system, such as TeamBots or ADE, is fully supported status, along with Pyro and MARIE,
implemented in Java, the assumption is made that it which provide interfaces to Stage, Gazebo, and the
will execute on any computer platform for which a ARIA and CARMEN simulators.
Java Virtual Machine of the required type is imple- F2.4 Configuration method: A system, such as TeamBots,
mented. Similarly, no differentiation is made among that embeds configuration in source code has a not
the various “flavors” of UNIX, although each sys- supported status. If a system stores configuration in
tem has at least been tested in a Linux environment. a text file (possibly XML), it receives a somewhat
Player/Stage, ARIA, and Pyro run on both UNIX and supported value. Player/Stage, ARIA, Pyro, and Miro
Windows. CARMEN, MissionLab, Miro, and MARIE all use text files, of which Miro supports XML. A

Springer
114 Auton Robot (2007) 22:101–132

system that provides a graphical means of accessing F3.4 Distribution mechanisms: To be elevated from a not
and modifying configuration settings gets a well sup- supported to somewhat supported value, a system must
ported value, which includes CARMEN, MissionLab, include a component that functions as middleware. To
and ADE. MARIE also receives a well supported value qualify as well supported, an agent framework that
due to the graphical interfaces included with FlowDe- treats components as independent agents is required.
signer and RobotFlow; MARIE itself uses XML con- Neither TeamBots nor Pyro provide middleware mech-
figuration files. anisms and receive a not supported. The IPC and IPT
software used by CARMEN and MissionLab and the
F3: Infrastructure Player server in Player/Stage act as a centralized nam-
ing service, while the ArNetworking package provided
F3.1 Low-level communication: All systems considered pro- in ARIA fills a similar function. ADE, MARIE, and
vide socket support and common networking protocols. Miro each specifically incorporate enhanced middle-
TeamBots, Player/Stage, ARIA, and Pyro use direct ware functionality as part of their infrastructure, earn-
socket connections as their primary method of com- ing a well supported value.
munication. CARMEN and MissionLab use IPC and F3.5 Scalability: To use “scalability” as a general reflection
IPT, respectively, which adds a level of abstraction to of an RDE’s properties, a combination of criteria from
general TCP/IP sockets. ADE uses Java’s RMI, while categories F1, F3, and F4 (specification, infrastructure,
Miro relies on CORBA’s IIOP. MARIE makes use of and implementation) is used. To earn a well supported
shared memory or sockets, relying on ACE for the lat- value, a system must provide scalability support in all
ter. Letter codes in Table 2 are as follows: S = Socket, categories (as defined below); a somewhat supported
I = IPC/IPT, R = RMI, C = CORBA IIOP. value indicates support in any two categories, while
F3.2 Logging facilities: All systems provide some means support for a single category or none at all receives
of monitoring component operation as console output a not supported value. In the specification category,
or graphical display, which forms the baseline for the an RDE must have a well supported value for either
value assignment (i.e., a not supported value). To gain architectural primitives or software engineering (cri-
a somewhat supported value, at the very least a sys- teria F1.1 or F1.2). For support in the infrastructure
tem must supply a predefined logging facility; to gain category, a system must earn at least a somewhat sup-
a well supported value, a system must allow for remote ported value for distribution mechanisms (F3.4), while
data capture, run-time starting and stopping of logging, the implementation category is comprised of satis-
and dynamically configurable data capture that can be faction of at least one of high-level language, rule
recorded in one or more files in one or more locations. interpreters, or planners (F4.1.2, F4.2.7, and F4.2.8,
TeamBots provides only simple console/graphical out- respectively).
put. Logging in ARIA, Player/Stage, CARMEN, ADE, F3.6 Component mobility: To receive a somewhat supported
and Miro is well supported, while logging in Pyro, Mis- value, an RDE must provide architectural components
sionLab, and MARIE is somewhat supported. to operate independently of one another in addition
F3.3 Debugging facilities: To attain a somewhat supported to continuing system operation when a component is
status, a system must allow non-simulated application removed, restarted, and reconnects. To attain a well
interruption and restart in conjunction with the ability supported value, mechanisms must be in place that can
to obtain information about component data. To qual- perform this task automatically. TeamBots, ARIA, and
ify as well supported, a system must allow run-time Pyro all use a fixed run-time system architecture that
suspension, modification, and replacement of arbitrary does not allow mobility and so receive a not supported
components. TeamBots is the only RDE receiving a value. The portability of devices in Player/Stage allows
not supported value. ARIA receives a somewhat sup- manual component relocation at run-time, as do the
ported value. Player/Stage, Pyro, CARMEN, Mission- modules in CARMEN and the object structure found in
Lab, ADE, MARIE, and Miro all qualify as well sup- MARIE and Miro, while MissionLab provides mech-
ported, but MissionLab and MARIE excel due to their anisms to upload robot executables to remote hosts.
integrated and extensive graphical interfaces. Mission- Each of these systems receives a somewhat supported
Lab is unique in that it also uses the included case- status. ADE provides mechanisms for saving state, au-
based reasoner to analyze a “mission” after comple- tomatic component start-up, and automatic component
tion to identify the source of operational errors. Also re-location due to detected failures, earning a well sup-
notable is ADE’s ability to dynamically compile and ported value.
replace components at run-time using the ADE class F3.7 System monitoring/management: To gain somewhat
loader. supported status, an RDE must provide an interface

Springer
Auton Robot (2007) 22:101–132 115

that gives access to all components in the system F4: Implementation

architecture. To gain well supported status, an
RDE must also provide mechanisms to manage all F4.1: System implementation characteristics
components. (Note that graphical representations
of a robot’s sensors, effectors, or other individual F4.1.1 Programming language: Both TeamBots and ADE
components do not qualify as infrastructure; see the are written in Java, while CARMEN and Pyro
Graphical Interface in Section 4). None of TeamBots, are implemented in C and Python, respectively.
Miro, nor CARMEN provide coherent system-wide Player/Stage, ARIA, MissionLab, MARIE, and Miro
facilities. ARIA supplies the MobileEyes GUI for are implemented in C++.
robot display and control, but source code is not F4.1.2 High-level language: To qualify as supporting a
freely available and thus earns a not supported high-level language, an RDE must supply a struc-
status. Pyro, Player/Stage, MissionLab, ADE, and tured method for controlling a robot (e.g., a behavior
MARIE all have graphical interfaces that not only or agent communication language). TeamBots sup-
display component status, but also allow component plies the Clay behavior hierarchy, ARIA provides ac-
control. tion specification via the ArAction class, MissionLab
F3.8 Security: To gain a somewhat supported value, a sys- provides both CDL and CMDL, Pyro and MARIE
tem must provide a way to securely authenticate com- supply both a set of foundational behavior classes
ponents. To gain a well supported value, an RDE and finite state automata, and Miro provides a “be-
must also provide access control and encryption. None havior engine” for behavior specification. None of
of TeamBots, Player/Stage,6 Pyro, CARMEN, Mis- Player/Stage, CARMEN, nor ADE\ provide a high-
sionLab, Miro, or MARIE use security mechanisms. level language.
MARIE and Miro both might inherit security fea- F4.1.3 Documentation: To attain a somewhat supported
tures from their use of ACE for component com- value, an RDE must have well documented source
munication, but do not exploit its availability. ARIA code and a publication outlining its use. If an RDE
provides authentication services as part of the Ar- also supplies a manual that describes how to use
Networking package, earning a somewhat supported the system (including installation instructions, guide-
value. ADE explicitly addresses all three aspects lines for developing applications and extending ca-
of security (encryption, authentication, and access pabilities into new areas, solutions to common prob-
control). lems, and example code), it receives a well supported
F3.9 Fault-tolerance: To achieve somewhat supported sta- value. While TeamBots, ADE, Miro, and MARIE
tus, a system must isolate components such that failure all provide some level of documentation, both web-
of a single component does not cause the entire appli- based and in source code, it is either incomplete or
cation to fail. To receive well supported status, an RDE they do not provide finished manuals that detail their
must also provide mechanisms in support of failure re- use. Player/Stage, ARIA, CARMEN, and Mission-
covery. None of the TeamBots, ARIA, or Pyro RDEs Lab all have complete and detailed manuals avail-
provide component isolation. Player/Stage, CARMEN, able, while Pyro provides the equivalent through its
and MissionLab, through their reliance on IPC soft- extensive online documentation.
ware, each isolates components, while MARIE and F4.1.4 Real-time operation: None of the systems directly
Miro’s use of ACE objects serve the same purpose. It provide real-time support, although MissionLab has
is worth noting that MissionLab also incorporates a the mechanisms in place for use with a purchased
case-based reasoning wizard for the purpose of repair- license of proprietary software from Honeywell.
ing a mission post-execution (Moshkina et al., 2006) F4.1.5 Graphical interface: To obtain a somewhat sup-
and that ACE implements the Fault Tolerant CORBA ported value, an RDE must supply graphical inter-
specification, although neither MARIE nor Miro have faces for visualizing component operation or design-
yet incorporated it. ADE\ provides both fault detec- ing control code without actual programming. To re-
tion and fault recovery at the component level through ceive a well supported value, an RDE must provide
its use of heartbeats between ADEServers and both items just mentioned, in addition to a standard
clients. method for creating new displays. Only TeamBots
does not provide a graphical display for a robot at
6
While the Player server in Player/Stage can optionally be set to require run-time (although it does supply a graphical simu-
authentication, it is explicitly acknowledged that the authentication is lator facility), and so receives a not supported value.
not for security, as keys are passed in plain text. While the MobileEyes GUI is available with ARIA,

Springer
116 Auton Robot (2007) 22:101–132

source code is not freely available and thus has to MARIE all include map-making facilities, which
be classified as not supported. CARMEN provides are combined with localization.
ad hoc, component specific graphical interfaces, but F4.2.2 Localization: TeamBots provides a landmark-based
does not provide standard methods for adding vi- localization component, while Miro provides par-
sualization nor graphical control code tools, and so ticle filter localization. ARIA provides sonar-based
receives a somewhat supported value. MissionLab localization, but full localization facilities must be
and Miro provide interfaces that allow developers purchased. Player/Stage, Pyro, CARMEN, ADE,
to design control code without actual programming, and MARIE all include localization facilities, which
but do not provide a standard method for defining are combined with map-making.
new displays, also earning them a somewhat sup- F4.2.3 Route planning: TeamBots and ADE provide
ported value. Player/Stage, ADE, Pyro, and MARIE schema-based navigation, ARIA supplies a nav-
all provide both implemented displays and a stan- igator integrated with its localization package,
dardized method of creating new displays, earning Player/Stage provides a wavefront propagation
well supported values. route planner, CARMEN uses a Markov decision
F4.1.6 Software integration: To attain somewhat supported process planner, MissionLab relies on geometric
status, an RDE must provide a standard API or mech- map analysis and an A∗ graph search, and MARIE
anism for incorporating “outside” software, gener- integrates Player/Stage and CARMEN navigation
ally using socket connections (with the recognition components.
that translation code will always have to be written). F4.2.4 Speech recognition: Player/Stage, ARIA, ADE,
Providing additional codified facilities that interface Pyro, and Miro all provide speech recognition
with other RDEs, thereby allowing their software to support through integration of outside software
be “dropped into” the environment, elevates the sta- such as Sphinx (Sphinx, 2004) or Sonic (Pellom
tus to well supported. Neither TeamBots, ARIA, nor and Hacioglu, 2003).
MissionLab provide such standard APIs or mech- F4.2.5 Speech production: Player/Stage, ARIA, ADE,
anisms. Player/Stage and CARMEN provide such Pyro, MARIE, and Miro all provide speech produc-
APIs (for their devices and modules, respectively), tion support through integration of outside software
Pyro and ADE explicitly include steps to “wrap” ex- such as Festival (Festival, 2004).
ternal software, MARIE supplies a variety of APIs F4.2.6 Vision processing: MissionLab and Miro have ba-
and mechanisms for integration, while Miro relies on sic image/video capture capabilities, but Miro also
writing TAO interfaces in Interface Device Language provides stereo image capture and many video fil-
(IDL), used to produce C++ code. Each receives at ters. TeamBots includes CMVision software, which
least a somewhat supported value. MARIE and Pyro can capture images and perform blob detection.
also provide translation facilities such that compo- ARIA has two vision packages available, the Ac-
nents written for CARMEN, Player/Stage, or ARIA tivMedia Color Tracking Software (ACTS) and
can be used and earn a well supported value. VisLib. ACTS is a blob detection package, while
VisLib includes image filters, blob detection, and
object tracking. Player/Stage supports both ACTS
F4.2 Predefined components
and CMVision. Pyro includes image/video capture,
blob, edge, and motion detection, assorted filters,
As mentioned earlier, any list of predefined components is
and stereoscopic tools, implemented in C++ for
open-ended and therefore necessarily incomplete. We limit
speed reasons. ADE\ includes both an ACTS inter-
this list to components that are commonly available and only
face and custom blob detection, object tracking, and
list the RDEs that include them. Furthermore, no quantita-
face/emotion detection. MARIE, via the RobotFlow
tive evaluation is given; the intent is not to establish a full
software, provides custom image capture, blob de-
taxonomy, but to provide a high-level indication of system
tection, movement detection, text/symbol extrac-
functionality. It should also be noted that both ARIA and
tion routines, and supports OpenCV.
MissionLab provide some of the following components so
F4.2.7 Rule interpreters: ADE includes an interface to
long as they are licensed; due to the limitation of this sur-
POP-Rulebase (Sloman, 2002) and Prolog, while
vey to open source software, such components have been
MARIE and MissionLab both include custom rule
excluded.
interpreters.
F4.2.1 Map-making: ARIA provides the Basic Mapper F4.2.8 Planners: While item F4.2.3 specifically covers
software, which can be used to manually construct navigation planners, these are considered too
maps. Player/Stage, Pyro, CARMEN, ADE, and task-specific to qualify under the general rubric of

Springer
Auton Robot (2007) 22:101–132 117

“planner”. MissionLab and MARIE both supply to adhere to a ternary value assignment, a value of na is
generic planners. used to indicate that a specific item was not examined in
F4.2.9 Neural networks: Pyro, Miro, and MARIE all in- sufficient depth to assign a value for some reason (e.g.,
clude neural network software. incompatible hardware, difficulties with prerequisite soft-
F4.2.10 Learning: TeamBots provides both reinforcement ware, etc.). na values will not be included in an RDE’s
and Q-learning components, Pyro has a reinforce- score. Results are shown in Table 3, where the following
ment learning module, while MissionLab includes shorthand column headings are used to designate particular
integrated case based reasoning and Q-learning systems: TB–TeamBots, AR–ARIA, P/S–Player/Stage, Py–
components. We do not include neural network soft- Pyro, C–CARMEN, ML–MissionLab, AD–ADE, Mi–Miro,
ware under the learning heading, as it appears as a and MA–MARIE.
separate category above. All systems were installed on at least two computers out
of a selection of five: two laptops, two desktops, and the on-
board PC of an ActivMedia PeopleBot P2DXe robot (shown
5 RDE usability evaluations on the right in Fig. 1). None of the computers were the same
make and model, with varying CPUs (850 MHz Pentium
The systematic comparison of RDEs with respect to their III, 1.3 GHz and 2.0 GHz Pentium M, 2.3 GHz Pentium 4,
supported features based on a conceptual framework is one and a 1.8 GHz AMD Athlon) and memory capacities (from
important part of an RDE evaluation. Another important part 128MB-1GB), although all used Linux (either Debian or Fe-
is RDE usability, for the extent to which an RDE can be easily dora distributions) running a 2.6.x kernel. Various supporting
installed and used in research is ultimately a decisive factor hardware included microphones, speakers, a Firewire cam-
for its adoption. Yet, surprisingly, there is only one previous era, and both wired and wireless Ethernet networking. All
study (Orebäck and Christensen, 2003) that provides a prac- non-simulated experiments were conducted on the robot,
tical RDE evaluation. And while a robotic architecture was which also has a pan-tilt unit, sonar, bumpers, and a SICK
actually implemented and executed on a robot in Orebäck LMS200 laser range finder.
and Christensen (2003), their study is very limited in scope
(only three RDEs were evaluated according to a small set of U1: Installation
criteria based on a single application) and does not provide
a methodology for systematic comparisons and subsequent Prior to actually using an RDE, it must be properly installed.
evaluations that ties together conceptual, practical, and im- Since we believe that installation difficulties might often be
pact factors. Consequently, the conclusions (Orebäck and a deterrent for potential users, we give “Installation” its own
Christensen, 2003) arrived at have limited applicability. category and criteria.
We believe that a comprehensive evaluation needs to en-
compass at least the three categories of usability criteria: U1.1 Documentation: Installation documentation refers
specifically to how well the documentation described
U1: Installation. Basic steps required to obtain a usable sys- the installation process, including required prepara-
tem, evaluated in terms of the required time and effort. tory steps and supporting software, minimum system
U2: Basic usability. Implementation and execution of a specification, a clearly laid-out sequence of instruc-
simple “lowest common denominator” architecture for tions, a list of known or potential issues, inclusion
RDE comparison, focussing on “low-level” sensor and of mailing list or contact addresses, and references to
effector access and allows for an investigation of archi- further information. Satisfying more than five of the
tectures that reside on a single host. above requirements receives a  value, three or four
U3: Advanced usability. Usage of individual, predefined, receives a  value, while less than three receives a
.
“high-level” components that would commonly form U1.2 Non-RDE Installation: In all cases, installation re-
sub-architectures of a complex, distributed architecture; quired additional supporting software. In some cases,
an effort is made to explore uncommon (and possibly this is limited to a single package (e.g., an adequate
unique) “high-level” system features. Java system), while in others, a large set of additional
The following subsections describe the three categories software is required to enable all available features. A
in more detail. As was done with features in Sections 3 value of  indicates that installing non-RDE software
and 4, a set of criteria is defined and subsequently evalu- (including, if necessary, determining what supporting
ated. Due to the variability of each criteria’s subject, value software was required, actual compilation and instal-
meanings are specified per item; in general they can be lation, and any needed debugging) took less than three
interpreted as:
for below average,  for average, and hours, a  indicates less than two days, while a
in-
 for above average. While an attempt has been made dicates more than two days. Due to the level of detail

Springer
118 Auton Robot (2007) 22:101–132

Table 3 Usability Criteria

Evaluation by RDE Category Criteria TB AR P/S Py C ML AD Mi MA

Install U1.1 Documentation         

U1.2 Non-RDE installation     

 

U1.3 RDE installation    
  

U1.4 installation usability 
 

 

Low-level U2.1 documentation 
  

U2.2 architecture implementation 
  
  

U2.3 architecture execution  
   


U2.4 graphical tools 


 

U2.5 overhead (memory, CPU) †




† 
† 

U2.6 “low-level” usability 
  

 

High-level U3.1 documentation na
  



U3.2 predefined components na
   



U3.3 task implementation na
 
 

U3.4 distribution na


 

U3.5 graphical tools na 

 
 

a † indicates that execution of U3.6 system integration na 

 


the “low-level” architecture was
done in simulation due to U3.7 “high-level” usability na   



difficulties running it on the
robot.

Fig. 1 Left: The “simple” architecture algorithm implementing a wan- distance rdistance to account for obstacles, multiplied by some system-
der behavior with obstacle avoidance. At each time step, a set of polar dependent scalar α. The translational velocity forward is calculated
range readings R = (r1 , r2 , . . . , rn ) is obtained, where −π ≤ rangle ≤ similarly, using the y component sin(rangle ), subtracted from the to-
π (rangle = 0 is straight ahead) and rdistance is relative to the center tal to make it repulsive, then adding a constant β for default forward
of the robot. The rotational velocity turning is calculated by summing movement. Right: The robot on which experiments were performed
the x component cos(rangle ) of polar readings, divided by square of the

Springer
Auton Robot (2007) 22:101–132 119

necessary to cover the variety and scope of additional to test a minimal set of capabilities supplied by each RDE.
software packages, we do not address many of the The implemented architecture consists of a basic wander
related issues encountered, although some additional behavior that incorporates obstacle avoidance. Only motor
information is given as part of criteria U1.4. control and range finder sensors were accessed, in as direct a
U1.3 RDE Installation: RDE installation refers to the steps manner as possible, providing a “lowest common denomina-
necessary to have a usable system, assuming all sup- tor” for RDE comparison. The basic algorithm, which uses
porting software has been installed. Values are the a potential field method, is shown on the left side of Fig. 1.
same as non-RDE installation (U1.2): a  value indi- Note that because the robots available to the authors are not
cates that installation took less than three hours, a  supported by TeamBots, the architecture was implemented
indicates less than two days, and a
indicates two or but execution could only be done in simulation. Similarly,
more days were required to attain a usable system. MissionLab and Miro were also run in simulation due to re-
U1.4 Installation Usability: Usability, as related to the in- peated failures. Each time an installation was not successful,
stallation procedure, is the overall (and ultimately sub- the error was located, fixed, and another attempt was made.
jective) impression of the experience. Values are as- Once simulated architecture execution was possible, a few
signed relative to the other systems, thus three systems additional attempts were made to run the architecture on the
each receive
, , and  values. The following notes robot; when these also failed, simulated results were deemed
provide selected information (presented in no particu- acceptable. Systems that were only evaluated in simulation
lar order) gathered during the implementation process are marked with a † in Table 3.
that help explain the evaluations:
U2.1 Documentation: In relation to the “low-level” archi-
r Pyro has a bootable “LiveCD” available, which tecture, documentation refers to information that en-
should avoid installation issues altogether. However, hances basic usability (e.g., APIs, example code, a fre-
actual robots rarely have a CD drive, making this ir- quently asked question list, etc.). A
value indicates
relevant for non-simulated use. The packages in the that it was difficult to find information concerning ei-
Pyro yum repository conflicted in some cases, but ther basic functionality (e.g., how to send a command
manual installation was done without issue. to the robot) or a solution to a relatively simple prob-
r The version of MissionLab available required the use lem (e.g., how to start architecture execution). A 
of gcc version 3.2 or below and related libraries, value indicates that information was available but re-
which, due to its age and incompatibility with current quired some effort to locate, while a  value indicates
versions, was the cause of time-consuming installa- that very little effort was required to find installation
tion issues. and implementation information.
r Installation of ACE/TAO software, required for Miro U2.2 Architecture implementation: Architecture implemen-
and MARIE, was particularly time-consuming, par- tation refers to the process of writing the program that
ticularly due to an initial misconfiguration that re- performs the wander behavior with obstacle avoid-
quired multiple manual de- and re-installation. Miro ance. A
value indicates that implementing the sim-
requires a particular ACE/TAO configuration instal- ple architecture was a major undertaking (requiring
lation, which is not documented. more than two days), a  value indicates that substan-
r Installation of supporting packages for ADE includes tial effort was involved (requiring more than 5 hours),
a hardware interface for Java, Player/Stage for simu- and a  value indicates that implementation required
lation, a secure shell client and server for distribution, an expected amount of effort (less than 5 hours). It is
and assorted other packages to attain the full comple- important to note that because the simple architecture
ment of system functionality. was a low-level implementation that circumvented or
r Few, if any, installation issues were experienced with avoided the abstractions and/or tools supplied by some
TeamBots, ARIA, Player/Stage, and CARMEN. The RDEs (e.g., TeamBots’ Clay system, ARIA’s prede-
issues that were encountered mostly concerned plat- fined actions, or MissionLab’s behavior libraries), the
form configuration (e.g., appropriate privileges and value is not necessarily indicative of what might be
permissions, default hardware settings that differed considered “normal” usage, which is considered in-
from the particular configuration in use, etc.). stead as part of “high-level” usability.
U2.3 Architecture execution: Architecture execution refers
U2: Basic usability to the effort required to start and stop a fully imple-
mented architecture (including both the control code
Basic usability in this context means to be able to imple- and supporting software, if they are separate). A

ment and execute a simple robotic architecture to be able value indicates a sequence of more than four steps,

Springer
120 Auton Robot (2007) 22:101–132

perhaps requiring multiple terminal connections that notes provide selected information (presented in no
each require separate initialization. A  value indi- particular order) gathered during the implementation
cates that two to four steps are required, sometimes process that help explain the evaluations:
mitigated by the GUI. A  value indicates that startup r Execution for TeamBots, MissionLab, and Miro were
and shutdown requires a single step.7
performed in simulation (denoted by a † in Table 3)
U2.4 Graphical tools: For the “low-level” architecture,
due to unsupported laser hardware in TeamBots and
graphical tools refer to the non-command line inter-
difficulties in hardware communication for MissionLab
face presented by an RDE. A value of  is given if the
and Miro.
system provides both a display of basic operational r Although RobotFlow supplies components for inter-
information and a graphical manner of starting and
facing with a Pioneer and SICK lasers, execution for
stopping architecture execution. A value of  is given
MARIE used a Player server as the hardware interface
if an RDE provides either, while a value of
indicates
so that MARIE functionality was included in the per-
that neither are provided.
formance measurements.
U2.5 Operational overhead: Operational overhead refers r Inclusion of an easily accessible simulator was ex-
to the memory and CPU (M and C, respectively)
tremely useful in implementing and debugging an ar-
resources used during architecture execution.8 Initial
chitecture; Player/Stage and CARMEN are particularly
conditions were kept constant by rebooting the
strong in relation to simulator integration, while Miro
computer for each system, turning the swap file off,
was relatively difficult to access.
then executing the architecture three times, recording r The structure of TeamBots is such that implementations
measurements at one second intervals. Each execution
that deviate from the included software (e.g., non-Clay
run is divided into two phases: (1) startup (denoted
behaviors or unsupported hardware) are very difficult
by a subscript S), demarcated by the time just prior to
to program.
system startup until robot movement is first detected r CARMEN and MissionLab require some knowledge
and (2) execution (denoted by a subscript E), which
of IPC messages to implement custom components or
begins when robot movement is detected, continuing
write routines that access the available components.
for 90 seconds. r Player/Stage, Pyro, and ADE provide a good balance
Figure 2 shows the average (with standard deviation
of high-level component abstractions while also pre-
bars) and maximum values across all three runs. The
serving accessibility to low-level sensor and effector
average values form the basis for calculating evalu-
interfaces.
ation scores by dividing the range in thirds that are r Player/Stage and Pyro provide a varied base of example
assigned values of 0 for the top third, 1 for the middle
code combined with relatively easy configuration and
third, and 2 for the lower third. More specifically, M S
relevant documentation.
and M E receive: < 20 MB = 2, < 40 MB = 1, and r Both MissionLab and CARMEN provide excellent doc-
> 40 MB = 0. For C S and C E , <33% = 2, < 66%
umentation; MissionLab has a clear and complete user
= 1, and > 66% = 0. The values shown in Table 3
manual, while CARMEN’s documentation is structured
are the sum of M S , M E , C S , and C E , where a total of
and organized in a manner that made it easy to find de-
6 or better receives a , a 4 or 5 receives a , and
sired information.
3 or less receives a
. It is interesting to note that, in r Although ARIA provides very good documentation, the
most cases, the operational overhead displays the clas-
“low-level” nature of sensor and effector access used in
sic memory vs. CPU usage tradeoff in both the startup
this architecture deviates from the standard methods
and execution phases.
addressed therein, leading to a more difficult imple-
U2.6 “Low-level” usability: Usability, as related to the
mentation.
“low-level” architecture, is the overall (and ultimately r While both the FlowDesigner and MissionLab GUIs are
subjective) impression of the experience. Values are
extensive and polished, some low-level tasks are easier
assigned relative to the other systems, thus three sys-
to perform using a text editor (in other words, the GUI
tems each receive
, , and  values. The following
was actually a hindrance in some respects).

7
We do not consider placing a sequence of commands in a shell script U3: Advanced usability
for execution as a single step.
8
While disk space usage and bandwidth are important, neither is con- While the “low-level” architecture provides a lowest
sidered. We exclude disk space due to the variability of packages re-
quired, while bandwidth is not addressed due to the single-host nature
common denominator for subjective RDE evaluation, the
of the “low-level” architecture. implementation of what we will refer to as “high-level

Springer
Auton Robot (2007) 22:101–132 121

Fig. 2 Operational overhead of each RDE executing the “low-level” Bottom Row: CPU usage (as a % of system load), again showing aver-
architecture. Top Row: Memory usage (in KB), showing average (with age and maximum values. As with the memory overhead, startup is on
standard deviation bars) and maximum values. The left-hand figure the left and execution is on the right
shows the startup phase, while the right shows the execution phase.

subarchitectures” provides a basis for evaluating an RDE components and because the robots available to the authors
under expected normal usage (although with a focus on are not supported.
distributed computing which becomes necessary due to ar-
U3.1 Documentation: In relation to the varied tasks, doc-
chitecture complexity and real-time processing constraints).
umentation refers to information concerning the ad-
Because there are few areas in which all RDEs provide
vanced RDE functionality (e.g., explanation of avail-
the same capabilities in the same way, the underlying idea
able components and their use, example code, etc.). In
is to obtain an estimate of the effort required to distribute
particular, the focus was on the components required
a single complex architecture over at least two hosts. A
for the varied tasks (vision, speech recognition, speech
model example of such an architecture is DIARC (Scheutz
production, and robot control) and their distribution.
et al., 2006), which provides a foundation for experiments
A
value indicates either missing or incomplete in-
in human-robot interaction.
formation for at least two basic functionalities. A 
The “fundamental” components examined are: (1) vi-
value indicates that information was either missing or
sion processing (monocular, that uses blob-detection), (2)
incomplete for only one basic functionality or that the
speech recognition, (3) speech production, and (4) the pro-
supplied documentation for any basic component was
vided robot control primitives, although attempts were made
minimal or unclear. A  value indicates that docu-
to use of some (possibly unique) components and capabil-
mentation was complete, easy to locate, and highly
ities (e.g., MissionLab’s selection of behaviors, ADE’s au-
readable.
tonomous distribution, etc.). The implementation consisted
U3.2 Predefined components: Predefined components refers
of installing, configuring, and testing individual components,
to the integrated capabilities included with an RDE. A
then connecting at least two of them across a network as an

value indicates that an RDE was either (1) missing
indication of ease of distribution, with the assumption that
at least one component required to implement all “fun-
connections among the other components will require sim-
damental” functionality for the envisioned architecture
ilar effort. No attempt was made to implement these tasks
(robot control, vision, speech production, and speech
in TeamBots, due to both the limited number of predefined

Springer
122 Auton Robot (2007) 22:101–132

recognition) or (2) did not provide at least two ad- a user does not have to write low-level programs. A
ditional components for each missing “fundamental” value of  is given if an RDE provides both, a value of
component. A  value indicates that the basic compo-  if only one is provided, and
if neither is provided.
nents were available or at least two other components U3.6 System integration: System integration refers to the
were available for each missing basic component. A separate issues of (1) easily connecting and control-
 value indicates that the RDE comes packaged with ling components in a complex architecture and (2) co-
more than four predefined components beyond what herent system usage in terms of providing a consistent
is required for a  value, in addition to other tools interface to the complete system that allows access to
or functionality. Of particular note are Player/Stage, and control of individual components. A  value is
Pyro, MissionLab and MARIE: Player/Stage supports assigned if both objectives are met, a  if only one is
the largest number of devices, Pyro and MissionLab satisfied, and a
if neither.
include various additional functionalities (e.g., Pyro U3.7 “High-level” usability: As with the “low-level” archi-
includes working examples from Russell and Norvig tecture, usability is the overall (and subjective) im-
(2002), MissionLab includes a case-based reasoner pression of the experience of using each RDE. Values
for post-execution analysis), while MARIE provides are assigned relative to the other systems, thus three
many signal processing and vision tools via the Robot- systems each receive  and  values, while two re-
Flow and FlowDesigner packages. ceive a
value (because TeamBots was not evaluated
U3.3 Task implementation: Task implementation refers to in terms of the high-level tasks). The following notes
the effort required to implement the subarchitectures, provide selected information (presented in no particu-
both in terms of accessing individual components and lar order) gathered during the implementation process
their distribution. As with the “low-level” architec- that help explain the evaluations:
ture, a
value indicates that implementation was a
r Both ARIA and MissionLab have additional compo-
major undertaking (requiring more than two days), a
 value indicates that substantial effort was involved nents that were not considered here as they require
(more than five hours), and a  value indicates that licensing and were not included in the downloadable
implementation required an expected amount of effort package.
r While ARIA’s documentation is in general very good,
(of less that five hours). It is imperative to note that
the “high-level usability” case made use of the ab- the distribution package, ArNetworking, lacks com-
stractions and/or tools supplied by some RDEs (which plete documentation. For some basic tasks, MARIE’s
sometimes proved detrimental), which may account documentation is sparse, limited to “node” listings,
for differences from the value given for the “low-level” while ADE and Miro are comparably spotty.
r Pyro’s interface is well integrated, allowing access to
architecture.
U3.4 Distribution: Distribution refers to the ease of locat- various conceptually separate parts of an application
ing components across hosts once the distribution (i.e., the server, robot, devices, and “brain”), while also
mechanisms have been implemented, accounting for providing the ability to enter Python commands at run-
both startup/shutdown procedures and relocation of time. However, non-graphical usage is not quite as pol-
components. A  value indicates that the system ished (for instance, hanging when exiting the system).
r The user interface provided by MissionLab is espe-
provided facilities for automatic login and component
startup/shutdown, in addition to providing the means cially suited to task specification implemented by non-
to reconfigure component location. A  value programmers, matching its objective of providing a
indicates that only one of those specifications was high-level view of robot control.
r Systems that require the strict use of abstractions in
met, and a
value indicates that neither is supported.
Of note is that RDEs in which components must support of their component model (e.g., the ACE/TAO
be implemented with network capabilities (e.g., in Miro and FlowDesigner networks in MARIE) or, to
components in CARMEN or Miro and servers in a lesser extent, require some level of abstraction re-
Player/Stage or ADE) all have a relative advantage. moved from actual source code (e.g., IPC/IPT commu-
U3.5 Graphical tools: In relation to the “high-level nications in CARMEN and MissionLab and RMI in
subarchitectures,” graphical tools refers to both the ADE) can be either beneficial or detrimental to some
presentation of an integrated display and the means to degree. For instance, Miro’s requirement that all com-
graphically implement robot control procedures. This ponents are CORBA objects makes component inte-
differs from the “low-level” architecture in that (1) an gration and distribution extremely simple, but the im-
emphasis is placed on integration, such that architec- plementation of an arbitrary component is made more
ture display and system control is consolidated and (2) difficult.

Springer
Auton Robot (2007) 22:101–132 123

6 Discussion To arrive at a quantitative evaluation, an assignment of

values to criteria must be made, which can then be applied
The evaluations presented in Sections 4 and 5 provide the to a selection (or all) of the criteria. To arrive at numerical
foundation for comparing RDEs, both at a conceptual level comparison scores, the three categories of criteria mentioned
and from a practical perspective. Augmented with an impact above are retained, yielding a feature score F, usability score
evaluation (to be described below), we envision the results U, and impact score I, which can be summed to give a total
of this survey being useful to the robotics community in at score T. Within each category, values of 0 or 2 are assigned
least two ways: (1) by providing researchers with a practical to binary-valued criteria and values of 0, 1, or 2 to ternary-
means of selecting an RDE that will most closely match valued criteria (listing features are not scored).9
their requirements and (2) by giving RDE developers an In formal terms, the selected RDEs form the set S =
overview of the innovations being made in other systems, {TB, AR, P/S, Py, C, ML, AD, Mi, MA} and are assigned a
possibly suggesting improvements and extensions to their score that is the sum of the category scores F, U , and I. Given
own system. a number of individual criteria within each category (e.g.,
F1.1 denotes Architectural Primitives, while U1.1 denotes In-
6.1 Researchers stallation Documentation), category scores are comprised of
the sum of individual criteria values Fi , U j , and Ik , where
To perform a comprehensive comparison of the RDEs pre- m, n, and o are the total number of feature, usability, and im-
sented, three separate measures are given: (1) a summary pact criteria and 1 ≤ i ≤ m, 1 ≤ j ≤ n, and 1 ≤ k ≤ o. In
of the evaluations from Section 4 concerning an RDE’s the simplest case, where all criteria are given equal weight,
features, (2) a summary of the evaluations from Section 5 the RDE comparison scores are given by the formula:
concerning an RDE’s usability, and (3) an estimate on the in-
fluence an RDE has had on the robotics field (i.e., its impact),

m

n

o

which is gauged by the breadth of publications from different Ts∈S ← Fis + Uis + Iis
i=0 i=0 i=0
research areas (as listed at the end of each description from
Section 2) and the number of other RDEs that provide inter- Should a quantitative evaluation that weights some categories
operability interfaces with a system. A researcher examining or criteria more or less than others be desired, weights can
a group of RDEs to find one that best fits their needs might be assigned at both a coarse-grained level (for each category,
conduct evaluations using either qualitative or quantitative α, β, and γ ) and a fine-grained level (for each criterion within
measures. a category, W Fi , WU j , and W Ik ). The resultant formula for
A qualitative evaluation is highly contingent on the user’s establishing the total comparison scores is:
purpose; on the one hand, very specific capabilities may be
required, while on the the other, needs may be highly abstract

m

n

o

or only loosely defined. For instance, an application designer Ts∈S ← α W Fi Fis + β WUi Uis + γ W Ii Iis
who has a large body of already written Octave software and i=0 i=0 i=0

desires to use it with a robot might look at the system descrip-  

where α + β + γ = 3, W Fi = m, WUi = n, and
tions in Section 2 and find that MARIE already has an Octave 
plug-in. Conversely, educators establishing an “Introduction W Ii = o. This method will give an objective comparison
to Robotics” class might consider the items from the usabil- in that scores are not biased for or against any particular
ity evaluation in Section 5 to be of overriding importance; an RDE, although the choice of features and their assigned
examination of the values given to the documentation criteria weights are based on the particular application requirements.
(U1.1, U2.1, and U3.1) might lead them to limit attention to The quantitative evaluation that does not weight any cri-
Player/Stage, Pyro, and CARMEN. teria or category more than another follows, supplemented
More likely, however, is that a mixture of characteristics by a discussion of each category score and the totals. The
is desired. For example, a developer might require a sys- tabulated scores are shown in Tables 4, 5, and 6, respectively,
tem that provides a simulated environment and a fair level and summarized in Table 7.
of distribution facilities, with a preference for a system that MARIE and ADE have the highest score (44 and 43,
supplies extensive GUI capabilities and usability oriented to- respectively) in terms of the Feature score F. The Implemen-
wards non-programmers. An examination of Section 4 would tation category contributes more than half of the F score,
lead to considering criteria F2.3 (Simulator), F3.4 (Distribu- exerting the largest influence. This is quite acceptable, as it
tion Mechanisms), F4.1.2 (High-level Language), and F4.1.5 corresponds to the expectation that the purpose of an RDE
(Graphical Interface), while Section 5 would U2.4 (Graph-
ical Tools) and U3.1–U3.7 (High-level Usability), making 9
Binary and ternary values range from 0 to 2 so as to not introduce a
MissionLab the most likely choice. bias towards ternary criteria.

Springer
124 Auton Robot (2007) 22:101–132

Table 4 The raw Feature score F and %, broken down by categories from Section 4 for each RDE

Feature category
Specification (6) Platform (6) Infrastructure (16) Implementation (30) Total (58)
RDE Score % Score % Score % Score % Score %

TB 5 83 1 17 0 0 10 33 16 28
AR 4 67 2 33 6 38 14 47 26 45
P/S 3 50 5 83 9 56 17 57 34 59
Py 5 83 5 83 5 31 22 73 37 64
C 4 67 5 83 8 50 10 33 27 47
ML 6 100 4 67 10 63 15 50 35 60
AD 6 100 3 50 16 100 18 60 43 74
Mi 5 83 3 50 9 56 14 47 31 53
MA 5 83 6 100 10 63 23 77 44 76

is to provide appropriate tools and abstractions that facilitate Table 5 The raw Usability score U and %, broken down by categories
application development. MARIE and Pyro have the highest from Section 5 for each RDE (a † indicates some criteria were not
included)
implementation scores (23 and 22), indicative of their wide
selection of components (partially due to its interoperabil- Usability category
ity with other RDEs) and advanced GUI capabilities. Pyro Installation “Low-level” “High-level” Total
(8) (12) (14) (34)
is assigned the third highest F score (37) due to the Infras-
RDE Score % Score % Score % Score %
tructure category, which accounts for over 25% of the final
score. ADE, which has a score of 18 in the implementation TB† 8 100 4 33 na na 12 35
category, ends up with the highest F score (43) due to the AR 7 88 7 58 4 29 18 53
infrastructure category, as it the only RDE that earns a well P/S 8 100 9 75 11 79 28 82
supported value for each criterion therein. Py 8 100 11 92 11 79 30 88
C 6 75 8 67 6 43 20 59
Of particular note is the fact that the RDEs with the highest
ML 3 38 8 67 13 93 24 71
scores (Pyro, ADE, and MARIE) all have explicit interop- AD 6 75 10 83 10 71 26 76
erability interfaces with other systems. While providing a Mi 2 25 4 33 4 29 10 29
boost in total feature score, we also note that this makes MA 3 38 4 33 10 71 17 50
them reliant, to some degree, on the availability of the other
systems for certain features, in addition to potentially affect-
ing their stability (in that changes to the other RDEs may opment with a MAS infrastructure, while MissionLab pro-
impact their operation). We also note again that comparing vides military personnel with non-programming methods of
RDEs in terms of F alone does not provide a full picture controlling robots. On the other hand, the low score given to
of evaluation, leaving out aspects such as system usability, Miro can be attributed to its reliance on the ACE/TAO com-
discussed next. munication framework10 and incomplete documentation. In
As noted in Section 3, an appropriate set of criteria must addition, it is necessary to point out that both ARIA and
be considered to serve as the basis for comparing RDEs. Not MissionLab’s scores would be higher if the restriction to
only do applications have markedly different characteristics open-source components was lifted.
that may impact the designer’s choice of RDE, but users It is interesting to note that MissionLab and MARIE have
tend to have different needs and working styles. The the widest discrepancy in score between usability categories,
Usability score U attempts to address the practical aspects each scoring relatively highly for “high-level” usability but
of RDE usage by adding criteria relevant for the actual low on the “low-level” architecture. Personal experience
implementation and execution of robotic architectures (in determined that much of the difference can be attributed
particular, the two classes of tasks described in the previous to predefined components (both their number and usage)
section), the results of which are summarized in Table 5. and their integration into a cohesive user interface. Both
Pyro, Player/Stage, ADE, and MissionLab have the high- provide a comprehensive graphical method for connecting
est usability scores (30, 28, 26, and 24, respectively). From
a usability point of view, this indicates that each has ful- 10
The impact of ACE/TAO is acknowledged in the user manual thusly:
filled their stated purpose: Pyro is aimed at novice users for “The CORBA environment and the Miro framework seem to raise the
educational purposes, Player/Stage is a flexible and adapt- bar for an easy entry into robot programming. While this can hardly
be denied they facilitate tremendously the task of writing distributed
able programming interface, ADE combines robotic devel- programs.”

Springer
Auton Robot (2007) 22:101–132 125

Table 6 The raw Impact score I and % for each RDE, the sum of “Application Area References” citations from Section 2 and the other RDEs
that provide interoperability interfaces

Application area TB AR† P/S Py C ML AD Mi MA

√ √ √
SLAM √ √ √
Planning/Navigation √ √ √ √ √
Learning √ √ √ √
Hierarchical behavior √ √ √ √
Education √ √ √ √
HRI — task allocation
HRI — learning √ √ √ √
HRI — Assistive Robotics √ √ √
Multi-robot Sensing √
Multi-robot Exploration √
Multi-robot Mapping √ √
Multi-robot Localization √ √
Multi-robot Planning √ √ √ √ √ √
Multi-robot Coordination √ √ √
Multi-robot Formation √ √
Multi-robot Task Allocation
Research areas (out of 16) 2 0† 12 3 4 7 6 6 7
Interoperability facilities 0 2 4 0 1 0 0 0 0
Total score (out of 20) 2 2† 16 3 5 7 6 6 7
Total % 10 10† 80 15 25 35 30 30 35

a † indicates some criteria were not included.

Table 7 The Total comparison score T and % for each RDE, com- criteria unevaluated for any RDE, while columns under the All Criteria
prised of the sum of feature F, usability U, and impact I scores. The include all criteria, using a value of zero for unevaluated items
left-hand columns under the Removed † Criteria heading do not include
Score
Removed † criteria All criteria
F (58) U (20) I (4) Total (82) F (58) U (34) I (20) Total (112)
RDE Raw % Raw % Raw % Raw % Raw % Raw % Raw % Raw %

TB† 16 28 12 60 0 0 28 34 16 28 12 35 2 10 30 27
AR† 26 45 14 70 2 50 42 51 26 45 18 53 2 10 46 41
P/S 34 59 17 85 4 100 55 67 34 59 28 82 16 80 78 70
Py 37 64 19 95 0 0 56 68 37 64 30 88 3 15 70 63
C 27 47 14 70 1 25 42 51 27 47 20 59 5 25 52 46
ML 35 60 11 55 0 0 46 56 35 60 24 71 7 35 66 59
AD 43 74 16 80 0 0 59 72 43 74 26 76 6 30 75 67
Mi 31 53 6 30 0 0 37 45 31 53 10 29 6 30 47 42
MA 44 76 7 35 0 0 51 62 44 76 17 50 7 35 68 61

components, but suffer from either not providing a graphical systems are most likely to have other RDEs provide interop-
interface to all parts of the system (e.g., MARIE requires erability interfaces. Table 6 summarizes the robotics research
shell scripts for startup/shutdown and the definition of subareas and citations given in Section 2 for each RDE, in
communication channels) or by their orientation towards addition to giving a count of the number of RDEs that inter-
very high-level tasks. operate with it. We reiterate that a single publication is used
Finally, an oblique way to determine the strengths of an to satisfy research in any subarea, simply to provide an idea
RDE is to establish an Impact score I that reflects the influ- of the breadth of research areas in which it has been used;
ence it has had on the robotics field. The number of research we also note the likelihood that some relevant publications
areas in which there are publications serve as an indicator of were not included, as the particular RDE used is sometimes
successful usage, as does the recognition that widely used not mentioned in a publication. Because criteria are all either

Springer
126 Auton Robot (2007) 22:101–132

binary in nature or a simple count, total scores are a simple are made for TeamBots because it is no longer under active
sum of items, deviating slightly√from the previous conven- development.
tion of assigning 2 points to a . Player/Stage clearly has To begin, we examine the Feature scores shown in
had the largest impact, reflected by the fact that its score Table 4 by discussing each category. In the Specification
(16) is more than double that of the next highest system. It category, all RDEs score at least 50%, while 8 of the 9 score
is also necessary to point out that ARIA’s impact score is 67% or higher, indicating that all supply adequate support
deceptively low (indicated by the † symbol), due to the fact for application design. Considering the Platform category,
that the authors were asked not to include references to its only two systems score less than 50%: TeamBots, which
ancestral software. is no longer being actively developed, and ARIA, which
Two overall comparison scores T† and Tall are shown in has been developed in support of a proprietary platform
Table 7, where T† does not include criteria that were unevalu- and thus has unique objectives. Increasing the hardware
ated for any RDE and Tall does. Overall scores are the sum of support in MissionLab, ADE, and Miro would yield scores
the feature F, usability U, and impact I scores. Player/Stage of 67% or higher for all of the remaining systems, such
has the highest total score Tall (78 out of a possible 112), that all could be considered to have adequate platform
partially due to having the highest I score, even though it had support. In terms of the Implementation category, only
the fifth highest F score and second highest U score. The CARMEN and MissionLab score below 50%. Recalling
next three highest scoring RDEs (ADE with 75, which had the information found in Table 2, it is evident that the
the highest F score; Pyro with 70, which had the highest U score is substantially due to supporting a limited number
score; and MARIE with 68, which had the second highest F of predefined components (although it is important to note
score) are all relatively new systems; in addition to provid- that some components are available with MissionLab if
ing some level of interoperability interfaces with other RDEs licensed). ARIA’s score is similarly affected by licensing
(thus capitalizing on prior innovations), we believe that part issues, in that an integrated GUI is available; inclusion
of their score can be attributed to identifying areas of ap- would put its score at about 60%. ADE, Miro, and MARIE
plication development that can be improved, partially based all have inadequate documentation, which would improve
on the examples of already established RDEs (discussed in their Feature score, while also increasing their Usability
more depth in the next section). Of note is that when the scores. The last category factoring into the feature score
unevaluated criteria are removed, the top four RDEs (ADE, is Infrastructure, discussed very briefly here due to its
Pyro, Player/Stage, and MARIE, respectively) remain the inclusion in Section 7. ARIA, Pyro, and CARMEN all score
same. 50% or below; again, ARIA’s licensing requirements affect
We reiterate that the total scores T† and Tall may not reflect its score to some degree, leading to a not supported value for
the particular requirements of a particular person or group the Monitoring and Management criterion. The most preva-
and that while the evaluations here are comprehensive, they lent unsupported criterion is Security, which only ARIA and
necessarily miss some criteria that may be important for a ADE support at all. As noted earlier, Player/Stage, Miro, and
specific designer or application. Such items can be added at MARIE all have the potential to easily incorporate security
will to further refine and customize the evaluations, adjusting (Player/Stage by utilizing its authentication mechanism and
the evaluation formula given earlier. Miro/MARIE by leveraging the ACE/TAO framework).
The next least supported criteria are Component Mobility
6.2 RDE maintainers and developers and Fault-tolerance, related to Distribution Mechanisms.
Suggestions for improvements in these areas is beyond the
The selected RDEs in this survey are, as defined by the scope of this survey, and we again refer to Section 7 for more.
constraints of system selection, open source projects. While Three categories make up the Usability score, In-
their availability is of obvious benefit to users, individual stallation, the “low-level” architecture, and “high-level”
RDE maintainers can also potentially reap some benefit by subarchitectures. Three RDEs are at or below 50% in
examining other systems. Hopefully, this will facilitate the installation (MissionLab, Miro, and MARIE), three in the
transfer of techniques and tools (e.g., Vaughan et al., 2003; “low-level” category (TeamBots, Miro and MARIE), and
Montemerlo et al., 2003b; Hattig et al., 2003; Howard and three in the “high-level” category (ARIA, CARMEN, and
Roy, 2004) across environments and promote progress in the Miro). Discounting unavailable software, ARIA’s individual
field of robotics as a whole. Using the Feature and Usabil- scores are well distributed among criteria, indicating that
ity comparison scores from the previous section as a basis while each could be improved, a good overall mix is
(the Impact score is not considered, as it is not directly con- established. MissionLab’s installation score is low due to its
trolled by RDE maintainers), it becomes possible to not only reliance on older versions of gcc; a new release would most
make specific improvement suggestions, but also to make likely greatly improve its score. As with the Implementation
some high-level points. We note here that no suggestions category mentioned above, CARMEN would benefit greatly

Springer
Auton Robot (2007) 22:101–132 127

from additional predefined components. Miro’s framework, types of score (Feature, Usability, and Impact), providing
relying as it does on CORBA, has great potential in terms robotic architecture designers with information useful in
of usability; however, additional tools and documentation picking an RDE for themselves. Finally, the comparisons
are required to “lower the bar” that, as they say in their provided the foundation for suggesting potential areas of im-
user manual, has been placed quite high. In MARIE’s case, provement to RDE maintainers based on features currently
the lowest individual criteria scores are concentrated in the found in other systems. In conclusion, we extrapolate from
“low-level” architecture. In particular, the necessity of man- the results and attempt to identify some likely future trends.
ually specifying component connections lowers usability, as The comparison of different RDEs suggests that com-
does what might be considered a high learning curve. mon features will increasingly be expected in all systems,
From the above, one broad point that should be clear strengthened by the interoperability mechanisms found in
is that interoperability tools are of great utility, allowing some recent systems (e.g., Pyro and MARIE). In addition to
one RDE to incorporate any component functionality devel- creating a set of (possibly de facto) standards, this will lead
oped in another RDE. Interoperability has been discussed to an increasing number of predefined components that can
in the literature (e.g., Baum et al., 2002; Nesnas et al., be expected in any given RDE. Furthermore, we expect the
2003; Côté et al., 2005), and while no single technique has list of predefined components given in Section 3 to continue
been accepted, MARIE’s foundations suggest a direction to expand, both in relation to high-level functionality (e.g.,
for future standards. While Pyro provides a large base of various types of robot control) and more specific low-level
interoperability tools, MARIE’s stated intention is to pro- functionality (e.g., “vision processing” will split into sepa-
vide a well-grounded framework that is not only compatible rate categories such as monocular vs. stereo vision process-
with existing systems, but also provides the conceptual ba- ing). We feel that a similar trend will develop in relation to
sis for adding interoperability in the future. The benefits RDE infrastructure (see Section 3), such that users expect
of this approach are apparent when considering the avail- inclusion of a suite of tools that implement various non-
able pre-defined component list; components available in architectural functions. This suspicion is borne out by a cur-
Player/Stage, ARIA, and CARMEN are also available in sory examination of the origination of RDEs. Early systems
Pyro and MARIE. (e.g., TeamBots, 1998) provide little in the way of infras-
Finally, to gain acceptance, an RDE must pay attention tructure: an application in TeamBots is the sum of the Java
to usability and the quality of its user and developer inter- classes that implement it. Player/Stage (2001) incorporates a
faces (see Steinfeld, 2004 for a general treatment). An area minimal amount of infrastructure; the authors acknowledge
that is receiving increasing attention is robotics education. and deliberately reject this trend, making the system “free
In addition to opening up the field to new people, an RDE from the computational and programmatic overhead that is
that caters to novices should, almost by definition, promote generally associated with the practical application” of such
usability. Pyro is particularly strong in this respect. Another mechanisms (Gerkey et al., 2003). More recent RDEs, such
aspect of usability concerns those who are not interested in as MARIE (2004) and ADE (2004), explicitly incorporate
going beyond the functionality already provided by an RDE, substantial infrastructure into their design and use, with the
but rather use the already established components to imple- stated aims, respectively, of improving interoperability and
ment their own applications without programming. In this distribution.
sense, the FlowDesigner package (related to MARIE) pro- The necessity of providing infrastructural interoperability
vides a graphical method for defining data flow throughout and distribution is illustrated by a quote from the authors
an application. Taking this a step further, MissionLab pro- of the GRACE project (Simmons et al., 2003): “One of the
vides graphical tools not concerned with robot particulars at more difficult parts of the Challenge for us was determining
all, but only with their actions. Additionally, the Mission- how to integrate a vast amount of software that had been de-
Lab developers have conducted many usability studies (e.g., veloped by the participating institutions, mostly on different
MacKenzie and Arkin, 1998; Collins et al., 2000; Endo et al., hardware platforms.” Such mechanisms should immediately
2004; Moshkina et al., 2006) in conjunction with system de- bring to mind multi-agent systems (MAS) research, which
velopment. has found particular traction in the robotics field in the form
of multi-robot applications (such as the citations listed at the
bottom of Table 6; (see also Sycara and Zeng, 1996; Altmann
7 Conclusion and outlook et al., 2001; Dias and Stentz, 2003; Gerkey and Matarić,
2004). We suggest that it will be critical for future RDEs to
This survey has evaluated nine RDEs with respect to an ex- incorporate other aspects of MAS research, including, but
tensive set of relatively common criteria supporting the de- not limited to security (e.g., Singh and Sycara, 2004) and
velopment of robotic applications. Results were then com- system-wide management facilities (such as those discussed
piled and used to compare the systems according to three in Bellifemine et al. (1999) and Sycara et al. (2003)).

Springer
128 Auton Robot (2007) 22:101–132

A final trend we expect to take shape in RDEs in the Arkin, R. and Balch, T. 1997. AuRA: principles and practice in review.
future is the prominent promotion of autonomic computing JETAI, 9(2–3):175–189.
Arkin, R., Collins, T., and Endo, T. 1999, Tactical mobile robot mission
functionality (e.g., Bantz et al., 2003). On the one hand, specification and execution. Mobile Robots XIV, pp. 150–163.
we expect improvements in low-level system characteristics Arkin, R., Endo, Y., Lee, B., MacKenzie, D., and Martinson, E.
that are transparent to users such as fault tolerance (e.g., 2003, Multistrategy learning methods for multirobot systems. In
Varakantham et al., 2002; Long et al., 2003; Melchior Proceedings of the 2nd International Workshop on Multi-robot
Systems, Washington, DC, pp. 137–150.
and Smart, 2004). ADE, for example, already explicitly Austin, D. 2004. Dave’s Robotic Operating System. http://dros.org/.
incorporates features for monitoring, relocating, and Balch, T. 2000. Hierarchic social entropy: An information theo-
restarting of components integrated into its infrastructure. retic measure of robot group diversity. Autonomous Robots,
Moreover, MARIE and Miro can potentially take advantage 8(3):209–238.
Balch, T. 2002. Teambots Proposal. http://www.cs.cmu.edu/ trb
of recent advances in middleware, e.g., the Fault Tolerant
/robocupjr/.
CORBA specification (see Chapter 23 in CORBA, 2005), Balch, T. 2004. Teambots. http://www.teambots.org/.
which incorporates mechanisms that promote robust system Balch, T. and Arkin, R. 1999. Behavior-based formation control
operation. On the other hand, we expect that development for multi-robot teams. IEEE Transactions on Robotics and
Automation, 20(5).
of high-level AI techniques that enhance a robot’s apparent Balch, T. and Ram, A. 1998. Integrating robotics research with
intelligence will increasingly find inclusion in RDEs, like JavaBots. In Working Notes of the AAAI 1998 Spring Symposium.
the tools found in MissionLab. We expect that robot learning Bantz, D., Bisdikian, C., Challener, D., Karidis, J., Mastrianni, S.,
(e.g., Russell, 2004; Blank et al., 2005), findings from and Mohindra, A. 2003. Autonomic personal computing. IBM
Systems Journal, 42(1):165–176.
human-robot interaction (HRI) research (e.g., Salter et al., Baum, W., Bredenfeld, A., Hans, M., Hertzberg, J., Ritter, A., and
2005; Fong et al., 2006; Moshkina et al., 2006; Scheutz Schönherr, F. 2002. Integrating heterogeneous robot and software
et al., 2006), and the study of social robotics (e.g., Bruce components by agent technology. Robotik 2002 Leistungsstand -
et al., 2002; Breazeal, 2003) will become commonplace. Anwendungen - Visionen - Trends, pp. 655–660.
Beaudry, E., Brosseau, Y., Côté, C., Raı̈evsky, C., Létourneau, D., and
In sum, we believe that the increase in capability of robotic Kabanza, F. 2005. Reactive planning in a motivated behavioural
applications will soon require extensive infrastructure sup- architecture. In Proceedings American Association for Artificial
port, with expanding development of support for autonomic Intelligence Conference, pp. 1242–1247.
computing in the future. Such tools and techniques will Bellifemine, F., Poggi, A., and Rimassa, G. 1999. JADE—a
FIPA-compliant agent framework. In Proceedings of the 4th Inter-
be used not only for the development and debugging of national Conference and Exhibition on the Practical Application
robotic architectures, but also for the execution and main- of Intelligent Agents and Multi-agents. London, pp. 97–108.
tenance of robotic architectures as part of application de- Bentivegna, D. and Atkeson, C. 2002. Learning How to Behave from
ployment. If true, the choice of one RDE over another will Observing Others (SAB02 Workshop on Motor Control in Humans
and Robots: on the interplay of real brains and artificial devices).
be based on more than just the development support of- Bergbreiter, S. and Pister, K. 2003. CotsBots: An off-the-shelf platform
fered, but increasingly on the features it provides for the for distributed robotics. In Proceedings of 2003 IEEE/RSJ
long-term operation of robotic applications. Furthermore, International Conference on Intelligent Robots and Systems
and perhaps more significantly, the integration of system in- (IROS 2003), Vol. 3, pp. 1632–1637.
Bergbreiter, S. and Pister, K. 2005. CotsBots: An Off-the-
frastructure with the development of intelligent robotic archi- shelf Distributed Robot Platform. http://www-bsac.eecs.
tectures will lead to robots that display ever greater levels of berkeley.edu/projects/cotsbots/.
autonomy. Biggs, G. and MacDonald, B. 2003. A survey of robot programming
systems. In Proceedings of the Australasian Conference on
Robotics and Automation. Brisbane, Australia.
References Bitting, E., Carter, J., and Ghorbani, A. 2003. Multiagent systems
development kits: An evaluation. In Proceedings of the 1st Annual
Conference on Communication Networks and Services Research
Activmedia robotics mobilerobots developer support. 2005. (CNSR 2003). Moncton, Canada, pp. 101–107.
http://robots.mobilerobots.com/. Blank, D., Kumar, D., and Meeden, L. 2002. A developmental
Altmann, J., Gruber, F., Klug, L., Stockner, W., and Weippl, E. 2001. approach to intelligence. In S. Conlon (Ed.), Proceedings of the
Using mobile agents in real world: A survey and evaluation of Thirteenth Annual Midwest Artificial Intelligence and Cognitive
agent platforms. In T. Wagner (Ed.), Proceedings of the Second Science Society Conference.
International Workshop on Infrastructure for Agents, MAS, and Blank, D., Kumar, D., Meeden, L., and Marshall, J. 2005. Bringing
Scalable MAS at the 5th International Conference on Autonomous up robot: Fundamental mechanisms for creating a self-motivated,
Agents. ACM Press, Montreal, Canada, pp. 33–39. self-organizing architecture. Cybernetics and Systems, 36(2).
Andronache, V. and Scheutz, M. 2004a. ADE—a tool for the devel- Blank, D., Kumar, D., Meeden, L., and Yanco, H. 2003. Pyro: A python-
opment of distributed architectures for virtual and robotic agents. based versatile programming environment for teaching robotics.
In Proceedings of the 4th International Symposium “From Agent Journal on Educational Resources in Computing, 3(4):1–15.
Theory to Agent Implementation.” Blank, D., Kumar, D., Meeden, L., and Yanco, H. to appear. Pyro: A
Andronache, V. and Scheutz, M. 2004b. Integrating theory and practice: python-based versatile programming environment for teaching
The agent architecture framework APOC and its development robotics. ACM Journal on Educational Resources in Computing
environment ADE. In Proceedings of AAMAS 2004. (JERIC).

Springer
Auton Robot (2007) 22:101–132 129

Breazeal, C. 2003. Towards sociable robots. Robotics and Autonomous Physical Agents.
Systems, 42(3–4):167–175. Fleury, S., Herrb, M., and Chatila, R. 1997. Genom: A tool for the
Brooks, A., Kaupp, T., Makarenko, A., Oreback, A., and Williams, specification and the implementation of operating modules in
S. 2005. Towards component-based robotics. In IEEE/RSJ a distributed robot architecture. In International Conference
International Conference on Intelligent Robots and Systems on Intelligent Robots and Systems, IEEE, Vol. 2, pp. 842–
(IROS 2005), pp. 163–168. 848.
Brooks, R. 1990. The Behavior Language: User’s Guide (Tech. Rep. Fleury, S. and Mallet, A. 2004. LAAS Open Software for Autonomous
No. AIM-1227). Massachusettes Institute of Technology. Systems. http://softs.laas.fr/openrobots/tools/genom.php.
Brooks, R. 1991. Intelligence without representation. Artificial Fong, T., Kunz, C., Hiatt, L., and Bugajska, M. 2006. The human-
Intelligence Journal, 47:139–159. robot interaction operating system. In Proceedings of the ACM
Bruce, A., Nourbakhsh, I., and Simmons, R. 2002. The role of Conference on Human-robot Interaction (HRI2006), ACM.
expressiveness and attention in human-robot interaction. In Fong, T., Nourbakhsh, I., and Dautenhahn, K. 2003. A survey of
Proceedings of the IEEE International Conference on Robotics socially interactive robots. Robotics and Autonomous Systems,
and Automation. 42:143–166.
Bruyninckx, H. 2001. Open robot control software: The OROCOS Fredslund, J. and Matarić, M. 2002, A general, local algorithm for robot
project. In Proceedings of IEEE International Conference on formations. IEEE Transactions on Robotics and Automation,
Robotics and Automation (ICRA) 2001, Vol. 3, pp. 2523–2528. Special Issue on Multi-Robot Systems, 18(5):837–846.
Bruyninckx, H. 2005. The OROCOS project. http://www.orocos.org/. Gamma, E., Helm, R., Johnson, R., and Vlissides, J. 1994. Design
Chaimowicz, L., Cowley, A., Sabella, V., and Taylor, C. 2003, ROCI: Patterns: Elements of Reusable Object-Oriented Software.
A distributed framework for multi-robot perception and control. Addison-Wesley.
In Proceedings of 2003 IEEE/RSJ International Conference on Gassull, G. 2001. Communication Services and User Interfaces for
Intelligent Robots and Systems (IROS 2003), Vol. 3, pp. 266–271. Tele-operating Mobile Robots via the Internet. Master’s thesis,
The CMU sphinx group open source speech recognition engines. University of Barcelona and University of Ulm, Neuroinformatics.
2004. http://cmusphinx.sourceforge.net/html/cmusphinx.php. Gerkey, B., Howard, A., and Vaughan, R. 2005. Player/Stage.
The Sphinx Group at Carnegie Mellon University. http://playerstage.sourceforge.net/.
Collins, T., Arkin, R., Cramer, M., and Endo, Y. 2000. Field results Gerkey, B. and Matarić, M. 2002, Sold!: Auction methods for
for tactical mobile robot missions. In Unmanned Systems 2000, multi-robot coordination. IEEE Transactions on Robotics and Au-
Orlando, FL. tomation, Special Issue on Multi-Robot Systems, 18(5):758–768.
Common object request broker architecture (CORBA/IIOP). 2005. (Also Technical Report IRIS-01-399).
http://www.omg.org/technology/documents/corba spec catalog. Gerkey, B. and Matarić, M. 2004. Are (explicit) multi-robot coor-
htm. Object Management Group. dination and multi-agent coordination really so different? In
Côté, C. 2005. Mobile and Autonomous Robotics Integration Proceedings of the AAAI Spring Symposium on Bridging the
Environment (MARIE). http://marie.sourceforge.net/. Multi-agent and Multi-robotic Research Gap, pp. 1–3.
Côté, C., Brosseau, Y., Létourneau, D., Raievsky, C., and Michaud, F. Gerkey, B., Vaughan, R., and Howard, A. 2003. The Player/Stage
2006. Robotic software integration using MARIE. International project: Tools for multi-robot and distributed sensor systems. In
Journal on Advanced Robotics Systems, 3(1):55–60. Proceedings of the 11th International Conference on Advanced
Côté, C., Létourneau, D., Michaud, F., and Brosseau, Y. 2005. Software Robotics. Coimbra, Portugal, pp. 317–323.
Design Patterns for Robotics: Solving Integration Problems with Gerkey, B., Vaughan, R., Støy, K., Howard, A., Sukhatme, G., and
MARIE. Submitted for workshop to ICRA2005. Matarić, M. 2001, Most valuable player: A robot device server for
Côté, C., Létourneau, D., Michaud, F., Valin, J., Brosseau, Y., and distributed control. In Proceedings of the IEEE/RSJ International
Raievsky, C. 2004. Programming mobile robots using RobotFlow Conference on Intelligent Robots and Systems. Wailea, Hawaii,
and MARIE. In Proceedings IEEE/RSJ International Conference pp. 1226–1231.
on Robots and Intelligent Systems. Guilbert, N., Beauregard, M., Michaud, F., and de Lafontaine, J.
Desai, M. and Yanco, H. 2005, Blending human and robot inputs 2003. Emulation of collaborative driving systems using mobile
for sliding scale autonomy. In Proceedings of the 14th IEEE robots. In Proceedings IEEE Conference on Systems, Man, and
International Workshop on Robot and Human Interactive Cybernetics, pp. 856–861.
Communication. Nashville, TN. Hattig, M., Horswill, I., and Butler, J. 2003, Roadmap for mobile robot
Dias, M. and Stentz, A. 2003, A comparative study between centralized, specifications. In Proceedings of 2003 IEEE/RSJ International
market-based, and behavioral multirobot coordination approaches. Conference on Intelligent Robots and Systems (IROS 2003), Vol.
In Proceedings of 2003 IEEE/RSJ International Conference on In- 3, pp. 2410–2414.
telligent Robots and Systems (IROS 2003), Vol. 3, pp. 2279–2284. Heckel, F. 2005. ROLE Robotics Development Environment.
Eiter, T. and Mascardi, V. 2002. Comparing environments for http://www.cse.wustl.edu/ fwph/role/.
developing software agents. AI Communications, 15(4):169–197. Hoff, J. and Bekey, G. 1995. An architecture for behavior coordination
Endo, Y., MacKenzie, D., and Arkin, R. 2004, Usability evaluation of learning. In IEEE International Conference on Neural Networks.
high-level user assistance for robot mission specification. IEEE Horswill, I. 2000. Functional programming of behavior-based systems.
Transactions on Systems, Man, and Cybernetics, 34(2):168–180. Autonomous Robots, 9(1):83–93.
ERSP 3.0 Robotic Development Platform. 2004. http:// Howard, A., Matarić, M., and Sukhatme, G. 2002. An incremental self-
www.evolution.com/products/ersp/. Evolution Robotics. deployment algorithm for mobile sensor networks. Autonomous
Fay, R., Kaufmann, U., Schwenker, F., and Palm, G. 2004. Learning Robots Special Issue on Intelligent Embedded Systems, 13(2):
object recognition in a NeuroBotic system. In H. Groß, K. Debes, 113–126.
and H. Böhme (Eds.), 3rd Workshop on Selforganization of Howard, A., Matarić, M., and Sukhatme, G. 2003. Putting the ‘I’ in
Adaptive Behavior SOAVE 2004. Dusseldorf, VDI, pp. 198–209. ‘Team’: An ego-centric approach to cooperative localization. In
The Festival Speech Synthesis System. 2004. http://www.cstr.ed. IEEE International Conference on Robotics and Automation.
ac.uk/projects/festival/. Centre for Speech Technology Research. Taipei, Taiwan, pp. 868–892.
FIPA ACL Message Structure Specification (SC00061G). 2002. Howard, A., Parker, L., and Sukhatme, G. 2004, The SDR experience:
http://www.fipa.org/specs/fipa00061/. Foundation for Intelligent Experiments with a large-scale heterogenous mobile robot team.

Springer
130 Auton Robot (2007) 22:101–132

In 9th International Symposium on Experimental Robotics 2004, International Conference on Robotics and Automation (ICRA),
Singapore. Vol. 4, pp. 3278–3283.
Howard, A. and Roy, N. 2004. Robotics Data Set Repository (RADISH). Logan, B. 1998. Classifying agent systems. In B. Logan and J. Baxter
http://radish.sourceforge.net/index.php. (Eds.), Proceedings of AAAI-98 Conference Workshop on
Jensen, R. and Veloso, M. 1998, Interleaving deliberative and reactive Software Tools for Developing Agents. Menlo Park, California,
planning in dynamic multi-agent domains. In Proceedings of the American Association for Artificial Intelligence.
AAAI Fall Symposium on Integrated Planning for Autonomous Long, M., Murphy, R., and Parker, L. 2003. Distributed multi-agent di-
Agent Architectures, AAAI Press. agnosis and recovery from sensor failures. IEEE/RSJ International
JESS - the expert system shell for the java platform. 2003. Conference on Intelligent Robots and Systems, 3:2506–2513.
http://herzberg.ca.sandia.gov/jess/. Sandia National Laboratories. Lucas, G. 2004. The Rossum Project. http://rossum.sourceforge.net/.
Jia, J., Chen, W., and Xi, Y. 2004. Design and implementation of an MacDonald, B., Yuen, D., Wong, S., Woo, E., Gronlund, R., and Col-
open autonomous mobile robot system. In Proceedings of IEEE lett, T. 2003. Robot programming environments. In ENZCon2003
International Conference on Robotics and Automation (ICRA) 10th Electronics New Zealand Conference. University of Waikato,
2004, Vol. 2, pp. 1726–1731. Hamilton.
Jones, C. and Matarić, M. 2004, Automatic synthesis of MacKenzie, D. and Arkin, R. 1993, Nov. Formal specification for
communication-based coordinated multi-robot systems. In behavior-based mobile robots. Mobile Robots VIII, pp. 94–104.
IEEE/RSJ International Conference on Intelligent Robots and MacKenzie, D. and Arkin, R. 1998. Evaluating the usability of robot
Systems. Sendai, Japan, pp. 381–387. programming toolsets. The International Journal of Robotics
Jung, B. and Sukhatme, G. 2002, Tracking targets using multiple Research, 17(4):381–401.
robots: The effect of environment occlusion. Autonomous Robots, MacKenzie, D., Arkin, R., and Cameron, J. 1997. Multiagent mission
13(3): 191–205. specification and execution. Autonomous Robots, 4(1):29–52.
Kaupp, T. 2005. Orca Robotics. http://orca-robotics.sourceforge.net/. Maes, P. 1990. Situated agents can have goals. In P. Maes (Ed.),
Koker, R., Oz, C., Cakar, T., and Ekiz, H. 2004. A study of neural Designing Autonomous Agents. MIT Press, pp. 49–70.
network based inverse kinematics solution for a three-joint robot. Mallet, A., Fleury, S., and Bruyninckx, H. 2002. A specification
Robotics and Autonomous Systems, 49(3–4):227–234. of generic robotics software components: future evolutions of
Konolige, K. 1997. COLBERT: A language for reactive control in GenoM in the Orocos context. In International Conference on
saphira. In Proceedings of the German Conference on Artificial Intelligent Robotics and Systems, IEEE.
Intelligence. Freiburg, Germany, pp. 31–52. Matarić, M. 2004, Robotics education for all ages. In Proceedings,
Konolige, K. 2000. A gradient method for realtime robot control. AAAI Spring Symposium on Accessible, Hands-on AI and
In Proceedings of the IEEE/RSJ International Conference on Robotics Education.
Intelligent Robotic Systems (IROS). Mayfield, J., Labrou, Y., and Finin, T. 1996. Evaluation of KQML as an
Konolige, K. 2002. Saphira Robot Control Architecture (Tech. Rep.). agent communication language. In M. Wooldridge, J. P. Müller,
Menlo Park, CA, SRI International. and M. Tambe (Eds.), Proceedings on the IJCAI Workshop
Konolige, K., Myers, K., Ruspini, E., and Saffiotti, A. 1997. The Saphira on Intelligent Agents II: Agent Theories, Architectures, and
architecture: A design for autonomy. Journal of Experimental & Languages, Springer-Verlag, Vol. 1037, pp. 347–360.
Theoretical Artificial Intelligence: JETAI, 9(1):215–235. Melchior, N. and Smart, W. 2004. A framework for robust mobile
Kraetzschmar, G., Gassull, G., and Uhl, K. 2004, July. Probabilistic robot systems. In D. W. Gage (Ed.), Proceedings of SPIE: Mobile
quadtrees for variable-resolution mapping of large environments. Robots XVII, Vol. 5609.
In M. I. Ribeiro and J. Santos Victor (Eds.), Proceedings of the 5th Metta, G., Fitzpatrick, P., and Natale, L. 2006. YARP: Yet another
IFAC/EURON Symposium on Intelligent Autonomous Vehicles. robot platform. International Journal on Advanced Robotics
Kraetzschmar, G., Sablatnög, S., Enderle, S., Utz, H., Simon, S., Systems, 3(1):43–48.
and Palm, G. 2000, Integration of multiple representations and Michaud, F. 2005. Engineering Education and the Design of Intelligent
navigation concepts on autonomous mobile robots. In H. Groß, Mobile Robots for Real Use (Submitted to International Journal
K. Debes, and H. Böhme (Eds.), Workshop SOAVE-2000: Selb- of Intelligent Automation and Soft Computing, Special Issue on
storganisation von Adaptivem Verhalten, Vol. 10/643. Ilmenau, Global Look at Robotics Education).
Germany, VDI Verlag. Michaud, F. and Létourneau, D. 2004. Robotflow: Open
Kramer, J. and Scheutz, M. 2003. GLUE—a component connecting Source Robotics Toolkit for Flowdesigner. http://
schema-based reactive to higher-level deliberative layers for robotflow.sourceforge.net/.
autonomous agents. In R. Weber (Ed.), Proceedings of the 16th Michel, O. 2004. Webots: Professional mobile robot simulation.
International FLAIRS Conference, AAAI Press, pp. 22–26. International Journal of Advanced Robotic Systems, 1(1): 39–42.
Labonté, D., Michaud, F., Boissy, P., Corriveau, H., Cloutier, R., and Miro - Middleware for Robots. 2005. http://smart.informatik.uni-
Roux, M. 2005. Evaluation Methodology of User Interfaces for ulm.de/MIRO/index.html. Robotics Group, University of Ulm.
Teleoperated Mobile Robots in Home Environments (Submitted Missionlab v6.0. 2003. http://www.cc.gatech.edu/aimosaic/robot-
to IEEE International Conference on Robotics and Automation). lab/research/MissionLab/. Mobile Robot Laboratory.
LaFary, M. and Newton, C. 2005. Aria html Documentation. Modular Controller Architecture. 2005. http://mca2.sourceforge.net/.
Laukkanen, M. 1999. Evaluation of FIPA-Compliant Agent Plat- Montemerlo, M., Roy, N., and Thrun, S. 2003a. CARMEN, Carnegie
forms. Unpublished master’s thesis, Lappeenranta University of Mellon Robot Navigation Toolkit. http://carmen.sourceforge.net/.
Technology. Montemerlo, M., Roy, N., and Thrun, S. 2003b. Perspectives on
LEGO.com Educational Division—Mindstorms for Schools. 2005. standardization in mobile robot programming: The carnegie
http://www.lego.com/eng/education/mindstorms/default.asp. mellon navigation (CARMEN) toolkit. In IROS 2003. Las Vegas,
LEGO. NV, Vol. 3. pp. 2436–2441.
Lemay, M., Michaud, F., Létourneau, D., and Valin, J. 2004. Au- Moshkina, L. and Arkin, R. 2003. On TAMEing robots. In IEEE
tonomous initialization of robot formations. In IEEE International International Conference on Systems, Man and Cybernetics, Vol.
Conference on Robotics and Automation. 4, pp. 3949–3959.
Lindstrom, M., Orebäck, A., and Christensen, H. 2000, BERRA: Moshkina, L., Endo, Y., and Arkin, R. 2006. Usability evaluation
A research architecture for service robots. In Proceedings of of an automated mission repair mechanism for mobile robot

Springer
Auton Robot (2007) 22:101–132 131

mission specification. In Proceedings of the ACM Conference on Applied Artificial Intelligence, 20(4–5).
Human-robot Interaction (HRI2006), ACM. Scheutz, M. and Andronache, V. 2003. APOC—a framework for
Nesnas, I., Simmons, R., Gaines, D., Kunz, C., Diaz-Calderon, A., and complex agents. In Proceedings of the AAAI Spring Symposium,
Estlin, T. 2006. CLARAty: Challenges and steps toward reusable AAAI Press, pp. 18–25.
robotic software. International Journal on Advanced Robotics Scheutz, M. and Andronache, V. 2004. Architectural mechanisms
Systems, 3(1):23–30. for dynamic changes of behavior selection strategies in
Nesnas, I., Wright, A., Bajracharya, M., Simmons, R., and Estlin, behavior-based systems. IEEE Transactions of System, Man, and
T. 2003. CLARAty and challenges of developing interoperable Cybernetics Part B: Cybernetics, 34(6).
robotic software. In Proceedings of 2003 IEEE/RSJ International Scheutz, M., Andronache, V., Kramer, J., Snowberger, P., and Albert,
Conference on Intelligent Robots and Systems (IROS 2003), Vol. E. 2004. Rudy: A robotic waiter with personality. In Proceedings
3, pp. 2428–2435. of AAAI Robot Workshop, AAAI Press, pp. forthcoming.
Nguyen, G., Dang, T., Hluchy, L., Balogh, Z., Laclavik, M., and Scheutz, M., Schermerhorn, P., Kramer, J., and Middendorff, C. 2006.
Budinska, I. 2002. Agent Platform Evaluation and Comparison The utility of affect expression in natural language interactions in
(Tech. Rep.). Bratislava, Slovakia: Pellucid 5FP IST-2001-34519. joint human-robot tasks. In Proceedings of the ACM conference
Nowostawski, M., Bush, G., Purvis, M., and Cranefield, S. 2000. on human-robot interaction (HRI2006), ACM.
Platforms for agent-oriented software engineering. In J. Dong, Schmidt, D. 1994. The ADAPTIVE communication environment:
J. He, and M. Purvis (Eds.), Proceedings of APSEC 2000. IEEE An object-oriented network programming toolkit for developing
Computer Society Press, pp. 480–488. communication software. In 12th Annual Sun Users Group
Orebäck, A. and Christensen, H. 2003. Evaluation of architectures for Conference. San Francisco, CA, pp. 214–225.
mobile robotics. Autonomous Robots, 14(1):33–49. Silva, A., Romao, A., Deugo, D., and Silva, M. da. 2001. Towards a
Osentoski, S., Manfredi, V., and Mahadevan, S. 2004. Learning hierar- reference model for surveying mobile agent systems. Autonomous
chical models of activity. In IEEE/RSJ International Conference Agents and Multi-Agent Systems, 4:187–231.
on Robots and Systems (IROS 2004). Simmons, R. 1994. Structured control for autonomous robots. IEEE
Pellom, B. and Hacioglu, K. 2003. Recent improvements in the CU Transactions on Robotics and Automation, 10(1):34–43.
SONIC ASR system for noisy speech: The SPINE task. In Simmons, R. 2004. Inter process communication (IPC). http://www-
Proceedings of IEEE International Conference on Acoustics, 2.cs.cmu.edu/afs/cs.cmu.edu/project/TCA/www/ipc/.
Speech, and Signal Processing (ICASSP). Simmons, R., Apfelbaum, D., Fox, D., Goldmann, R., Haigh, K., and
Pfeifer, R. 1988. Artificial intelligence models of emotion. In V. Hamil- Musliner, D. 2000. Coordinated deployment of multiple hetero-
ton, G. H. Bower, and N. H. Frijda (Eds.), Cognitive Perspectives geneous robots. In Proceedings of the IEEE/RSJ International
on Emotion and Motivation, Volume 44 of Series d: Behavioural Conference on Intelligent Robots and Systems (IROS).
and Social Sciences. Kluwer Academic Publishers, Netherlands, Simmons, R., Goldberg, D., Goode, A., Montemerlo, M., Roy, N., and
pp. 287–320. Sellner, B. 2003. GRACE: an autonomous robot for the AAAI
Pineau, J., Montemerlo, M., Pollack, M., Roy, N., and Thrun, S. 2002. robot challenge. AI Mag., 24(2):51–72.
Towards robotic assistants in nursing homes: challenges and Simplified Wrapper and Interface Generator. 2004.
results. In T. Fong and I. Nourbakhsh (Eds.), Workshop Notes http://www.swig.org/.
(WS8: Workshop on Robot as Partner: An Exploration of Social Singh, R. and Sycara, K. 2004. Securing Multi Agent Societies (Tech.
Robots), IEEE International Conference on Robots and Systems. Rep. No. CMU-RI-TR-04-02). Robotics Institute, Carnegie
IEEE, Lausanne, Switzerland. Mellon.
Poggi, A., Rimassa, G., and Turci, P. 2002. What agent middleware can Skubic, M. and and Volz, R. A. 1998. Learning force-based assembly
(and should) do for you. Applied Artificial Intelligence, 16(9–10): skills from human demonstration for execution in unstructured
677–698. environments. In Proceedings of International Conference on
Provost, J., Kuipers, B., and Miikkulainen, R. 2004. Self-organizing Robotics and Automation (ICRA98), pp. 1281–1288.
perceptual and temporal abstraction for robot reinforcement Sloman, A. 1998. What’s an AI toolkit for? In B. Logan and J. Baxter
learning. In AAAI-04 Workshop on Learning and Planning in (eds.), Proceedings of the AAAI-98 Workshop on Software Tools
Markov Processes. for Developing Agents, pp. 1–10.
Pyro, Python Robotics. 2005. http://emergent.brynmawr.edu/pyro/? Sloman, A. 2002. Help Poprulebase.
page = Pyro. Python Robotics. Sloman, A. and Scheutz, M. 2002. A framework for comparing agent
Ricordel, P. and Demazeau, Y. 2000. From analysis to deployment: A architectures. In Proceedings of UK Workshop on Computational
multi-agent platform survey. Engineering Societies in the Agents Intelligence, pp. 169–176.
World. Springer-Verlag, Vol. 1972, pp. 93–105. SOAP version 1.2. 2003. http://www.w3.org/TR/soap12/. W3C XML
Rivard, F. 2005. Localisation relative de robots mobiles opérant en Protocol Working Group.
groupe (Tech. Rep.). Mémoire de maı̂trise, Département de génie Sprouse, J. 2005. Nomadic.sourceforge.net. http://nomadic.
électrique et de génie informatique, Université de Sherbrooke. sourceforge.net/.
Russell, R. 2004. Mobile robot learning by self-observation. Steinfeld, A. 2004. Interface lessons for fully and semi-autonomous
Autonomous Robots, 16(1):81–93. mobile robots. In Proceedings of IEEE International Conference
Russell, S. and Norvig, P. 2002. Artificial Intelligence: A Modern on Robotics and Automation (ICRA) 2004, Vol. 3, pp. 2752–
Approach, 2 ed., Prentice Hall. 2757.
Salter, T., Michaud, F., Dautenhahn, K., Létourneau, D., and Caron, Stentz, A. 2002. CD∗ : A real-time resolution optimal re-planner for
S. 2005. Recognizing interaction from a robot’s perspective. In globally constrained problems. In Proceedings of AAAI 2002, p.
Proceedings IEEE International Workshop on Robot and Human 605.
Interactive Communication, pp. 178–183. Sycara, K., Paolucci, M., Velsen, M.V., and Giampapa, J. 2003.
Scheutz, M. 2004. APOC—An Architecture for the Analysis and The RETSINA MAS infrastructure. Autonomous Agents and
Design of Complex Agents (Ed.) (Forthcoming In Darryl Davis, Multi-Agent Systems, 7(1):29–48.
editor, Visions of Mind). Sycara, K.P. and Zeng, D. 1996. Coordination of multiple intelligent
Scheutz, M. 2006. ADE—steps towards a distributed development software agents. International Journal of Cooperative Information
and runtime environment for complex robotic agent architectures. Systems, 5(2/3):181–212.

Springer
132 Auton Robot (2007) 22:101–132

Tews, A., Matarić, M., and Sukhatme, G. 2003. A scalable approach

to human-robot interaction. In IEEE International Confer-
ence on Robotics and Automation, Taipei, Taiwan, pp. 1665–
1670.
Thrun, S. 2003. Robotic mapping: A survey. In G. Lakemeyer and
B. Nebel (Eds.), Exploring Artificial Intelligence in the New
Millennium. Morgan Kaufmann, San Francisco, CA, USA, pp. 1–
35.
Thrun, S., Fox, D., Burgard, W., and Dellaert, F. 2000. Robust monte
carlo localization for mobile robots. Artificial Intelligence,
128(1–2):99–141. James Kramer is pursuing his Ph.D. in Computer Science and Engi-
Utz, H., Kraetzschmar, G., Mayer, G., and Palm, G. 2005. Hierarchical neering at the University of Notre Dame, South Bend, IN, where he
behavior organization. In Proceedings of IROS 2005. Edmonton, previously earned his M.Sc. in 2005. His primary interests concern the
Canada. role of infrastructure in agent architectures, focussing on the use of re-
Utz, H., Sablatnög, S., Enderle, S., and Kraetzschmar, G. 2002. Miro— flective mechanisms in integrating system and cognitive architectures,
middleware for mobile robot applications. IEEE Transactions particularly concerning their use in failure detection and recovery. These
on Robotics and Automation, Special Issue on Object-Oriented ideas are being implemented in the APOC Development Environment
Distributed Control Architectures, 18(4):493–497. (ADE), of which he is a primary developer.
Utz, H., Stulp, F., and Mühlenfeld, A. 2004. Sharing belief in teams of
heterogeneous robots. In D. Nardi, M. Riedmiller, and C. Sammut
(Eds.), RoboCup-2004: The eighth RoboCup Competitions and
Conferences, Springer Verlag.
Valin, J. and Létourneau, D. 2004. Flowdesigner. http://
flowdesigner.sourceforge.net/.
Varakantham, P., Gangwani, S., and Karlapalem, K. 2002. On handling
component and transaction failures in multi agent systems.
SIGecom Exch, 3(1):32–43.
Vaughan, R., Gerkey, B., and Howard, A. 2003. On device abstractions
for portable, resuable robot code. In Proceedings of IROS 2003,
Las Vegas, Nevada, pp. 2121–2427. Matthias Scheutz received the M.Sc.E. degrees in formal logic and
Vijayakumar, S., D’souza, A., Shibata, T., Conradt, J., and Schaal, computer engineering from the University of Vienna and the Vienna
S. 2002. Statistical learning for humanoid robots. Autonomous University of Technology, respectively, in 1993, and the M.A. and
Robots, 12(1):55–69. Ph.D. of philosophy in philosophy at the University of Vienna, Aus-
Volpe, R., Nesnas, I., Estlin, T., Mutz, D., Petras, R., and Das, H. 2001. tria, in 1989 and 1995 respectively. He also received the joint Ph.D.
The CLARAty architecture for robotic autonomy. In Proceedings in cognitive science and computer science from Indiana University
of the 2001 IEEE Aerospace Conference. Bloomington in 1999. He is an assistant professor in the Department
Walters, D. 2003. Open automation project (OAP). http:// of Computer Science and Engineering at the University of Notre Dame
oap.sourceforge.net/. and director of the Artificial Intelligence and Robotics Laboratory. He
Webots 5. 2005. http://www.cyberbotics.com/. Cyberbotics. has over 80 peer-reviewed publications in artificial intelligence, ar-
White box robotics. 2005. http://whiteboxrobotics.com/. White Box tificial life, agent-based computing, cognitive modeling, foundations
Robotics. of cognitive science, and robotics. His current research interests in-
Wolf, D. and Sukhatme, G. 2005. Mobile robot simultaneous localiza- clude agent-based modeling, complex cognitive and affective robots
tion and mapping in dynamic environments. Autonomous Robots, for human-robot interaction, computational models of human language
19(1):53–65. processing in mono- and bilinguals, distributed agent architectures, and
interactions between affect and cognition.

Springer