Autonomous Robotics:

Action, Perception and

Cognition: Introduction
Gregor Schöner
Institute for Neural Computation
Theory of Cognitive Systems
[email protected]
What comes to your mind when
you hear the word “robot”

Google search “robot” (21 apr 2020)

Nao (robot) - Wikipedia more productive than human workers ... Future Robots and Ensuring Human S… Robots have jumped, raced and rolled a ... Home | Boston Dynamics
en.wikipedia.org information-age.com blogs.3ds.com cnet.com bostondynamics.com

@ght the coronavirus in China ... Social robot - Wikipedia China says AI robots won't lead to ... Could robots be marking your homework ... CES 2020 was full of cute little robots ...
businessinsider.com en.wikipedia.org techinasia.com bbc.com cnet.com

Humanoid robot job apocalypse — or a ... Here are the coolest robots of 2019 s… extend the scope of IoT applications ... The time for putting up with stupid ... Eight cute and kitschy CES 2020 robots ...
pri.org thegadgetYow.com networkworld.com cosmosmagazine.com dezeen.com

Japanese-Israeli venture offers robots ... Robots Might Make Human Workers More ... NAO the humanoid and pro… Will Robots Rob Us From Our Jobs?
[email protected] bloomberg.com softbankrobotics.com industrywired.com

Robots.txt Datei fürs SEO ... Why Ethical Robots Might Not Be Such… Robots could learn to recognise human ... Russia and robots: Steel junk or a ...
neilpatel.com spectrum.ieee.org techxplore.com bbc.com
=> Humanoids (or anthropomorphic) robots
 Those Racist Robots… - Towards Data ... rede@ne personal robots in 2… Biped Robot Timelines – How Long Until ... How Can We Bond With Robots ...
towardsdatascience.com scmp.com emerj.com technologynetworks.com

page 2
Robot companions are … Boston Dynamics, Asimo, Da Vinci, SoFi ... DJI makes push into educational rob… Agility Robotics and Ford team up to ...
vehicle
Biobots: Snakebot, Batbot, and More ...
cnet.com youtube.com asiatimes.com parcelandpostaltechnologyinternational.com youtube.com

legged
robot
The arti@cial skin that allows robots ... What is the future of service robots? Why are we reluctant to trust robots ... Robot at the helm: A space humanoid, an ... two-legged robots and self-driving cars ...
cnn.com eenewseurope.com theguardian.com zdnet.com techcrunch.com

5 Industries Majorly Impacted by ... 4 Robots You Can Use In Real Estate ... Walmart Shows Robots Are As Easy As 123 5 reasons robots aren't going to take ...
analyticsinsight.net corelogic.com.au forbes.com weforum.org

on regular
destroy when they compete with humans ...
marketwatch.com
Toyota Developing Humanoi…
global.toyota
4 Types of Collaborative Robots | RIA ...
robotics.org
A Technology Trend Every Business Must ...
forbes.com
industrial
compliant robot
arms on rst 4
pages
All Robots - ROBOTS: Your Guid… Alphabet X's new Everyday Robo… DENSO Robotics Europe is a market ... Role of Robots in Recruitment ... Robots Film (2005) · Trailer · Kri…
robots.ieee.org theverge.com densorobotics-europe.com careerenlightenment.com kino.de
fi
 in reality, industrial robots are
much more common today than
humanoids or autonomous
vehicles

fundamentally, all factory automatization is a

form of robotics: “programmable”
machines…
 Survey of kinds of robots

other than humanoid or industrial

vironment
simple, single-task
5. Industrial, Personal,
to a remotely
5. Industrial, Personal, and Service Robots

and human
located Service operator.
Robots Robots are able to provide logistics support in office and

ough autonomous vehicles

ustrial environments by transporting materials (packages, medicines, or supplies) or by leading visitors
ocatedhallways.
human operator.
Remotely
transportingExamples
vironments.
Robots
materialsinclude
(packages,
are able
controlled andtomonitored
provide logistics
medicines,
underwater remotely
support
robots are
or supplies)
operated
also in office
able
orvehicles,
by leading
and hazardous or unpleasant
to enter
pipevisitors
cleaning and inspection robots,
dy bomb
controlled and robots.
disposal monitoredSomerobots are also
examples areable to enter
shown in Fig.hazardous
5.5. or unpleasant
clude underwater remotely operated vehicles, pipe cleaning and inspection robots,
Some examples are shown in Fig. 5.5.

Figure 5.5. Examples of service robots.

[photo credits: WTEC
Figurein5.5. Examples of service nal report 2006]
robots.include all the challenges for
e challenges service and personal robotics industrial robotics. Dexterous
fi
some of our own
(older)
autonomous
vehicles
scale agriculture requires machines to cultivate, seed, irrigate, and harvest very large areas of terrain. The

outdoor vehicles
ability to track an autonomous vehicle using global positioning systems (GPS), sensing the soil and plant
conditions in the field, encourages the implementation of robotic vehicles for agricultural or EfieldF
applications. Figure 2.3a shows an example of an agricultural robotic vehicle under development in the
United States. Figure 2.3b shows a large autonomous mining haul truck developed in Australia.

(a) (b)
Figure 2.3. Agricultural robotic vehicle (Int Harv, U.S.) (a). Mining haul truck (ACFR, Australia) (b).

Similar challenges occur in areas of environmental monitoring, where mobile vehicles may move through air,
water, or ground to observe the presence 8of contaminants and track the patterns
2. Robotic Vehiclesand sources of such
pollutants. In large manufacturing facilities, mobility is essential to transport components and subassemblies
during the manufacturing process and a variety of robotic guided vehicles are utilized in these domains.

Figure 2.1. NASA Mars Rover (NASA Jet Propulsion Laboratory (JPL)).

Another example of a hostile and hazardous environment where robotic vehicles are essential tools of work
and exploration is the undersea world. Human divers may dive to a hundred meters or more, but pressure,
cars: autonomous driving
NEWSFOCUS

good,” says Virginia Tech’s Alfred

Wicks. “We’d done our home-
work.” The University of Pennsyl-
vania’s Toyota Prius, Little Ben,
straggles in an hour later. Some-
time past 3:30 p.m., MIT slips in
just before Cornell.
The outcome seems obvious.
Carnegie Mellon spotted Stan-
ford and Virginia Tech a 20-
minute head start and made up
almost all of it. It seems the vic-
tory should be theirs. DARPA
officials will make the final call,
however. And, some participants
grumble, DARPA never fully

Downloaded from www.sciencemag.org on November 24, 2007

explains its judgments.
After you. Stanford’s
Junior and Virginia Make it out to …
Tech’s Odin negotiate But the next morning brings
an intersection.
no surprises. Carnegie Mellon
walks off with the win. Stanford
the University of Central Florida in Orlando. race and the infamous O. J. Simpson low- takes second and $1,000,000, Virginia Tech
They’ve outfitted Knight Rider, a 1996 Sub- speed police chase. takes third and $500,000. “There’s tremen-
aru Outback that belonged to Harper’s wife Each robot has to complete three “mis- dous satisfaction in what the whole field
and has 99,257 miles (159,705 km) on it, with sions” comprising six or seven “submis- accomplished,” Whittaker says. “That was a
just enough gizmos to get around the sions,” such as parking in exactly the right day that stunned the world.” DARPA Direc-
course—they hope. Instead of the spinning space in a lot, traversing an off-road passage, tor Anthony Tether also gushes. “Quite
3D lidar, they use two lidars that see in one or navigating between two places. After each frankly, I watched these things and I forgot
direction and rock them back and forth. “If mission, the robots return to the start area to after a while that there wasn’t anybody in
just one wire falls off, something essential is download the specifications for the next, and there,” he says. “It’s a historic day—’bot on
not going to work,” Harper says. Still, the each machine must travel 60 miles (97 kilo- ’bot for the first time!”
team made the final having invested only meters) in less than 6 hours. Maybe there’s something to the grandiose
$130,000 in the project. At first, the action comes fast and heavy. rhetoric. Now only a Luddite could doubt that
An hour into the race, TerraMax, the hulking soon cars will guide themselves, at least in a
Robots, start your engines! vehicle entered by military contractor pinch to avoid collisions. In fact, the technol-
Race day usually brings the intoxicating Oshkosh Truck Corp. in Wisconsin, turns ogy already seems ripe for low-risk applica-
smell of high-octane fuel and the electri- toward a pillar and gets stuck staring at it. tions, such as automating farm equipment,
fying scream of engines. But not here. At Forty-five minutes later, Central Florida runs and the leading teams are pushing to com-
8:00 a.m., the robots leave the starting straight toward a house. Caroline, the robot mercialize their software. “I think it’s going
area, one by one, like rental cars leaving a from Team CarOLO, the other German squad, to come in bits and pieces,” says Charles
lot. There’s a glitch. Interference from a collides with MIT’s Talos and loses sensors. Reinholtz, leader of the Virginia Tech team
 legged robots
230 C. Site Reports-Europe

Figure C.58. The walking machines built by DillmannCs group.

Up to now 10 six-legged walking machines have been built and distributed: three to museums and seven to
biologically
inspired
robotics

Fig. 3. Salamandra robotica, a salamander robot tha

organization of salamander spinal circuits and the m
is equipped with eight motors for spine undulations
of the walking trot gait of the robot (left) and the sal
of the swimming gait of the robot (left) and the salam

The mechanisms of inter-limb coord

coupling versus mechanical coupling, have
It was shown that stable gaits could be
oscillators,
Fig. 1. Example and with only
of biorobots. (A)indirect coupling
RoboTuna (3
similar to what has been observed in the
Sandfish lizard robot (6). (E) StickyBot (56). (F)
demonstrate that different gaits could be
(I) Cornell robot.
biped When
(74). the Miniature
(J) mass flapping
was placed morewii
neuromuscular model
monkeys, the(20). Permissions:
same gaits emerged as(A) M.
in their
Royal Society; (F, I, J) AAAS; (H) A. Sproewitz.
 snakes, crawlers, climbers
Perhaps the biggest advantage to using the network to connect robots is the ability to connect and harne
physically-removed assets. Mobile robots can react to information sensed by other mobile robots in the ne
room. Industrial robots can adapt their end-effectors to new parts being manufactured up-stream in t
assembly line. Human users can use machines that are remotely located via the network. (See Fig. 7.3.)
obile robot platforms in Dillman>s laboratory. Two SwissLog products are shown on the
extreme right. The ability to network robots also enables fault-tolerance in design. If robots can in fact dynamical
reconfigure themselves using the network, they are more tolerant to robot failures. This is seen in the Intern
where multiple gateways, routers, and computers provide for a fault-tolerant system (although the Internet
bots not robust in other ways). Similarly, robots that can KplugL and KplayL can be swapped in and o
automatically, to provide for a robust operating environment.
ab has developed several articulated, snake-type robots for pipeline inspection. Some are
available and used to inspect water pipes and oil pipelines (including the Alaska pipeline).

Figure C.57. Inspection robot.

oncentrated on enabling the system to work in an unstructured environment. A multi-

Figure 7.2. Robotic modules can be reconfigured to KmorphL into different locomotion systems including
with six links will be used for inspection tasks in sewer pipelines.
wheel-like rolling system (left), a snake-like undulatory locomotion system (right), a four-legged
walking system (bottom).
n
Finally, networked robots have the potential to provide great synergy by bringing together components wi
icated to the development of control systems for four- andcomplementary
six-legged robots,
benefitsas
andwell as the whole greater than the sum of the parts.
making
asis appears to be in the application of artificial muscles (McKibben-type muscles with a
Applications for networked robots abound. The U.S. military routinely deploys unmanned vehicles that a
uction of size and weight, and joint design. Historically, they have fabricated several of the
 ve developed a wide variety of sophisticated technologies for sensing, mapping, and
Figure 2.9 from
showssmall
several of these organisms
vehicles. to major ocean circulation currents.
underwater vehicles, ships
ans at many scales, biological
oth autonomous
b. Coastaland ROV and
Security types, are an increasingly
Environmental important
Monitoring part of this repertoire, and
Systems
n that is unavailable in other ways. Figure 2.2 shows an autonomous underwater vehicle
ER underAUV systems atmay
development be used
Institut as de
Français surveillance
Rechercheand pourobservance of de
l'Exploitation systems
la Merwith both defense and
environmental
rench National Institute implications. Figure 2.10
for Marine Science shows an overview
and Technology. ASTERofwillthe Autonomous
be used for Oceanographic Sensor
p to 3,000Network
meters in depth and
(AOSN) is capable
systems, of carrying
deployed a wide variety
as an experiment of instrumentation
at the for
Monterey Bay Aquarium Research Institute
and biological
(MBARI) sensing and monitoring.
in California, In United
which integrate manyStates research,
different robotic the
and evolution of
sensor resources.
vehicles for deep ocean exploration enabled the discovery of the sunken Titanic and the
at notable shipwreck.

Figure 2.2. IFREMER ASTER autonomous underwater vehicle.

Figure 2.10. Advanced Oceanographic Sensor Network Figure 2.11. HROV (Hybrid ROV)
e and oceans, there are many (MBARI,
applications where human presence is hazardous.
U.S.). projectNuclear
(Johns Hopkins University
amination sites must often be explored and mapped to determine the types and extent
(JHU) of
and Woods Hole
d provide the basis for remediation. Military operations incorporate many different (WHOL), U.S.).
motely operated technologies for air, sea, and ground vehicles. Increasingly, security and
ay usec.networks
Scientific Mission and
of advanced Deepsensors
mobile Ocean Science
that observe and detect potential events that
 with other parts of the military tactical system. The integration of sophisticated computer an

airborne robots
communications architectures is an essential feature of these systems, and the use of algorithms such a
SLAM to interpret complex scenes is an important contribution to these systems. The U.S. is generall
acknowledged as the world leader in military applications of robotic vehicle technologies.

Arthur Sanderson, George Bekey,

Table 2.1
Types of robotic mobility

Figure 2.9. Examples of military and defense robotic vehicles.

Space Robotic Vehicles

The field of space robotics was identified as a topic for separate focus in this study and the major results o
that effort will be presented in Chapter 3. In the context of vehicle technologies, the recent Mars rove
programs have uniquely demonstrated perhaps the most successful deployment of robotics vehicl
technologies to date in any domain of applications. The rovers have landed and explored the surface of Mar
 robotic manipulators, hands

(b) (c)

Robert Ambrose, Yuan Zheng, Brian Wilcox 47

Having hands will be essential in the early advancement of this research, since the learning and association of
knowledge with objects will be done in the robotCs own terms, with the way a tool feels when grasped stored
in sensori-motor space. Key advances in dexterous hands include tactile skins, finger tip load sensing, tendon
drive trains, miniature gearing, embedded avionics, and very recent work in low-pressure fluid power
systems. The fundamental research in biologically inspired actuators will likely transform the nature of this
domain in the next ten to fifteen years.

(e) (f)

ew of the DLR Robots. (a) The DLR-LWR-III equipped with the DLR-Hand-II. (b) The
II that is based on the DLR-LWR-III. (c) The DLR humanoid manipulator Justin. (d)
b, a redesign of the DLR-Hand-II. (e) The DLR-HIT hand, a commercialized version
II. (f) The DLR-Crawler, a walking robot based on the fingers of the DLR-Hand-II.
Figure 4.10. Dexterous arms at DLR, NASA and UMASS.
 some of our own
robotic manipulators
mobile robot
manipulators
198 C. Site Reports-Europe

the rotation, finding the axis of rotation and complying. This concluded a set of six real-time demonstrations,
which is unusual. The fact that the six systems were of such a high caliber should be noted.

Figure C.28. Dexterous arm on mobile base, opening door (left), robot passing through doorway (right).
 our own mobile
robot manipulator

[Arnold: 1998-2000]
 autonomous robotics

auto-nomos: giving laws to oneself

minimally: autonomous robots generate
behavior based on sensory information
obtained from their own on-board sensors
in contrast to industrial robots that are
programmed in a xed and detailed way
fi
 autonomous robotics
but: even an industrial robot uses
autonomous control to reach its
programmed goals…
=> autonomy is expected to go beyond
control, include decisions=qualitative change
of behavior
e.g. avoid obstacle to the left vs. to the right

e.g., reach for one object rather than another

 autonomous robotics

but: we do not expect autonomous robots

to just do whatever “they want”… we
expect to give them “orders”
 autonomous robotics

autonomy as a
“programming interface”:
give instructions to a robot at a
high level, in regular human
language and gesture in a
shared environment…

… and let the autonomous

robot deal with the “details” of
how to achieve goals
why autonomous robots?
 why autonomous robots?

ideas I hear from lay-people

to clean up, to serve drinks..

just generally cool..

robot soldiers..
 toy/entertainment/animation

including therapy (autism)

 assistance robotics

at home, in the work place

collaborate with human users
 autonomous vehicles

…. well, for autonomous transport…

[Amazon robotized
warehouse]
Research on Robotic Vehicles – United States types of sensors. In additio
with other parts of the
In the United States, research on robotic vehicles has emphasized work in the following five areas:
military, re ghting, rescue
Military and Defense Systems
communications architectu
SLAM to interpret compl
U.S. investment in robotic vehicle research has strongly emphasized the developmentacknowledged
of ground, air, as
andthe world
underwater vehicles with military applications. As shown in Figure 2.9, there have been significant
accomplishments in these areas in which large development programs have resulted in capable and reliable
vehicle systems. Many of these systems are deployed in a Lremotely-operatedN mode, that is, a human
controller works interactively to move the vehicle and position based on visual feedback from video or other
types of sensors. In addition, there is a strong emphasis on integration of autonomous probes and observers
the “ideal” application because
with other parts of the military tactical system. The integration of sophisticated computer and
communications architectures is an essential feature of these systems, and the use of algorithms such as
desire to remove human agent
SLAM to interpret complex scenes is an important contribution to these systems. The U.S. is generally

from the scene is consensual …

acknowledged as the world leader in military applications of robotic vehicle technologies.

110 much research B. Site Reports-Asia

The company also introduced three other products: the Basic Robot Education System, the Research and
Education Robot, which included various versions as mentioned earlier (Figure B.10 shows one of them),
and Military Robot. For the latter, Hanool has developed an Anti-Terror Robot called Hanuri-T-ML as shown
in Figure B.11. Like the first three robots, the Research and Education robot and the Anti-Terror robot are all
mobile robots equipped with various sensors for navigation and operation according to the purpose which
they were designed for.

Figure B.10. Research and Education Robot. Figure B.11. Military Robot.
fi
fi
(robot ethics…interesting topic)

may a military robot decide autonomously

to shoot
…. navy ships do that already…

may a autonomous car decide between

avoiding a pedestrian and preventing danger
for car occupants?
fundamental problem: off-loading decisions from user to
designer …
autonomous robotics as a
“playground” of research
autonomous robotics as a
“playground” of research

modern engineering models systems, treating

the remainder stochastically…. autonomous
robotics act in natural environments that are
dif cult to model
fi
autonomous robotics as a
“playground” of research

modern engineering uses modular design

that limits the range over which modules
interact/interfere…autonomous robotics:
requires system integration
autonomous robotics as a
“playground” of research
highly interdisciplinary eld
sensing

perception

mechanics
control

AI/planning

embedded computing
communication / data security

user interfaces
fi
state of the art: current explosion

fast computation makes approach real-time

that used to be not viable
laser range nder… probabilistic approaches
modern software engineering facilitates
programming
… through maturation of technology
fi
4 core problems/challenges

perception
interacting with humans
movement generation
background knowledge
 (1) perception

no autonomy without perception

perception is NOT estimating the stimulus
it is learning about the environment and
extracting meaning=that what enables action
 (1) perception

4 core problems of perception

attention

recognition/classi cation
segmentation

estimation

=> WS lecture course

fi
 (1) perception

much progress in SLAM and variants

exploiting multiple/low level sensors

much progress in computer vision

driven in part by Deep NN
but not as successful in robotic settings: where we have
much experience with few objects rather than little
experience with many objects
 the scene in Fig. 1b, in which no object can be singled out by
(2) interaction with humans
feature reference, this phrase uniquely specifies one of them
A typical relational phrase like the one above consists of a

in part a problem of perception

as well…
perceptually grounding language
intention perception
gesture recognition e.g., “the
(a) (b)red cup to
the left of the green
joint attention cup“
Fig. 1: Visual scenarios affording the use of …spatial language
dialogue management
target (the red cup) and a reference
=> WS (the green cup),
lecture courserelative
emotion recognition
to which a relational term (to the left) is applied. Interpret
ing such a phrase may require that different pairs of objects
(3) back-ground knowledge

implicit knowledge how the world works

how to open a door

that milk is in the fridge

how to grasp a glas vs. a cup vs. a spoon

how to grasp an object to achieve a particular goal

to clear space before moving something to a new

place…
 “background” is a core problem of classical
arti cial intelligence
knowledge bases

reasoning
action planning

architectures
fi
implicit knowledge in behavior based robotics…
the background is in the individual skills and how
they are connected
 (4) movement generation

classical approach
motion planning based on precise world models

using optimal control to address control problems…

but:
high demands on perception and on modeling of plant/
objects

unclear if it works for soft actuation for safe interaction

with humans
need for exible, human like movement and movement
sequences
fl
this is what we’ll cover a lot

exploit analogies with human movement

coordination, movement primitives
exploit analogy with muscle: soft visco-elastic
actuators
 Particular perspective of the
course
We look at autonomous robotics as a
research eld that interacts with the theory
of cognitive systems
1) robots as examples of such systems…
learn about principle problems here
=> integrative framework of dynamical systems

2) robots as tool to test neural models of

cognition and behavior…
=> proof of process account and source of ideas/
discovery of problems
fi
Particular perspective of the
course

dynamical systems
“behavioral dynamics” ...

neural dynamics => WS course on Neural Dynamics

but we will touch on some aspects of neural dynamics in

the “rate code” picture …

(while the WS is focussed mainly on the space code/

population picture)
Particular perspective of the
course

this course is NOT a standard introduction

into autonomous robotics from a technical
point of view
although it provides some elements of that
 Syllabus
dynamical systems tutorial
vehicles: path planning
attractor dynamics approach

other approaches

analogy to navigation in humans and animals

robot arms
kinematics
dynamics

inverse kinematics
 Syllabus

timing
coordination

movement primitives

a neural architecture of movement

motor control
principles of control

human motor control

muscles and re exes
fl