R ES E A RC H

◥ learning to encapsulate models of uncertainty

REVIEW SUMMARY and support further advances in adaptive and
robust control. Learning to manipulate in real-
world settings is costly in terms of both time
ROBOTICS
and hardware. To further elaborate on data-
driven methods but avoid generating examples
Trends and challenges in ON OUR WEBSITE
◥
with real, physical systems,
many researchers use sim-

robot manipulation Read the full article

at http://dx.doi.
ulation environments. Still,
grasping and dexterous
org/10.1126/ manipulation require a
Aude Billard* and Danica Kragic science.aat8414 level of reality that exist-
..................................................
ing simulators are not yet
BACKGROUND: Humans have a fantastic abi- with object occlusion and noisy measurements, able to deliver—for example, in the case of
lity to manipulate objects of various shapes, as well as on adaptive control approaches to modeling contacts for soft and deformable
sizes, and materials and can control the ob- infer an object’s physical properties, so as to objects. Two roads are hence pursued: The
jects’ position in confined spaces with the handle objects whose properties are unknown first draws inspiration from the way humans
advanced dexterity capabilities of our hands. or change as a result of manipulation. acquire interaction skills and prompts robots
Building machines inspired by human hands, to learn skills from observing humans per-
with the functionality to autonomously pick ADVANCES: Roboticists are still working to
forming complex manipulation. This allows

Downloaded from https://www.science.org at Slub Dresden on May 22, 2023

up and manipulate objects, has always been develop robots capable of sorting and pack-
robots to acquire manipulation capabilities in
an essential component of robotics. The first aging objects, chopping vegetables, and folding
only a few trials. However, generalizing the
robot manipulators date back to the 1960s and clothes in unstructured and dynamic environ-
acquired knowledge to apply to actions that
are some of the first robotic devices ever con- ments. Robots used for modern manufactur-
differ from those previously demonstrated re-
structed. In these early days, robotic manipula- ing have accomplished some of these tasks in
mains difficult. The second road constructs
tion consisted of carefully prescribed movement databases of real object manipulation, with
sequences that a robot would execute with no the goal to better inform the simulators and
ability to adapt to a changing environment. As generate examples that are as realistic as pos-
time passed, robots gradually gained the abil- sible. Yet achieving realistic simulation of
ity to automatically generate movement se- friction, material deformation, and other phys-
quences, drawing on artificial intelligence and ical properties may not be possible anytime
automated reasoning. Robots would stack boxes soon, and real experimental evaluation will be
according to size, weight, and so forth, extending unavoidable for learning to manipulate highly
beyond geometric reasoning. This task also re- deformable objects.
quired robots to handle errors and uncertainty
in sensing at run time, given that the slightest OUTLOOK: Despite many years of software
imprecision in the position and orientation of and hardware development, achieving dexter-
stacked boxes might cause the entire tower to ous manipulation capabilities in robots remains
topple. Methods from control theory also be- an open problem—albeit an interesting one,
came instrumental for enabling robots to com- given that it necessitates improved understand-
ply with the environment’s natural uncertainty ing of human grasping and manipulation
by empowering them to adapt exerted forces techniques. We build robots to automate tasks
upon contact. The ability to stably vary forces but also to provide tools for humans to easily
upon contact expanded robots’ manipulation perform repetitive and dangerous tasks while
repertoire to more-complex tasks, such as in- avoiding harm. Achieving robust and flexible
serting pegs in holes or hammering. However, Holding two objects in one hand requires collaboration between humans and robots is
none of these actions truly demonstrated fine dexterity. Whereas a human can grab hence the next major challenge. Fences that
or in-hand manipulation capabilities, and they multiple objects at the same time (top), a currently separate humans from robots will
were commonly performed using simple two- robot (bottom) cannot yet achieve such gradually disappear, and robots will start manip-
ulating objects jointly with humans. To achieve
PHOTOS: LEARNING ALGORITHMS AND SYSTEMS LABORATORY, EPFL

fingered grippers. To enable multipurpose fine dexterity. In this example, a human has placed
manipulation, roboticists focused their efforts the objects in the robot’s hand. this objective, robots must become smooth
on designing humanlike hands capable of using and trustable partners that interpret humans’
tools. Wielding a tool in-hand became a prob- structured settings that still require fences be- intentions and respond accordingly. Further-
lem of its own, and a variety of advanced tween the robots and human operators to ensure more, robots must acquire a better understand-
algorithms were developed to facilitate stable safety. Ideally, robots should be able to work ing of how humans interact and must attain
holding of objects and provide optimality side by side with humans, offering their strength real-time adaptation capabilities. There is also a
guarantees. Because optimality was difficult to carry heavy loads while presenting no danger. need to develop robots that are safe by design,
to achieve in a stochastic environment, from Over the past decade, robots have gained new with an emphasis on soft and lightweight struc-
tures as well as control and planning method-
▪
the 1990s onward researchers aimed to increase levels of dexterity. This enhancement is due to
the robustness of object manipulation at all breakthroughs in mechanics with sensors for ologies based on multisensory feedback.
levels. These efforts initiated the design of perceiving touch along a robot’s body and new
sensors and hardware for improved control mechanics for soft actuation to offer natural The list of author affiliations is available in the full article online.
*Corresponding author. Email: [email protected]
of hand–object contacts. Studies that followed compliance. Most notably, this development Cite this article as A. Billard, D. Kragic, Science 364, eaat8414
were focused on robust perception for coping leverages the immense progress in machine (2019). DOI: 10.1126/science.aat8414

Bilard et al., Science 364, 1149 (2019) 21 June 2019 1 of 1

R ES E A RC H

◥ moving on conveyers can be detected fairly easily

REVIEW by cameras and picked up if fully visible. How-
ever, detection of transparent objects or objects
partially hidden (e.g., when stacked on top of one
ROBOTICS another) remains difficult.
With the need to frequently change the type

Trends and challenges in of goods produced, the robotics industry strives

for multipurpose object grasping and handling
solutions. One step toward this objective is to
robot manipulation provide robots with a choice of grippers varying
in size and strength and to enable robots with
tool-changing mechanisms so that they can select
Aude Billard1* and Danica Kragic2
the correct tool. To determine which tool to use
for a given task, a robot must have knowledge of
Dexterous manipulation is one of the primary goals in robotics. Robots with this capability
an object’s properties, such as shape, weight, ma-
could sort and package objects, chop vegetables, and fold clothes. As robots come to
terial, and so forth. This information is readily
work side by side with humans, they must also become human-aware. Over the past
available in factories where all objects are known.
decade, research has made strides toward these goals. Progress has come from advances
However, this requirement presents a limitation
in visual and haptic perception and in mechanics in the form of soft actuators that offer
for robots in other settings, where the set of objects
a natural compliance. Most notably, immense progress in machine learning has been
to be manipulated may not be known beforehand.
leveraged to encapsulate models of uncertainty and to support improvements in adaptive

Downloaded from https://www.science.org at Slub Dresden on May 22, 2023

and robust control. Open questions remain in terms of how to enable robots to deal What can robots not do today?
with the most unpredictable agent of all, the human.
Although robots are adept at handling rigid

H
objects, they still struggle with flexible materials—
ave you ever found yourself busy forag- onstrate how to do something and what the such as fruits and vegetables or clothing items—
ing in your bag in search of a set of keys? If expected result may be, but we cannot easily that differ in size, weight, and surface proper-
so, you may recall that it took only a few communicate the magnitude of the applied ties. Manipulations that produce a deformation
seconds to find them among the disparate forces and torques or the size of the friction co- (e.g., inserting, cutting, or bending) are particu-
contents of the bag. For certain, you did efficient necessary to satisfy stability conditions. larly difficult, as accurate models of the defor-
not reflect on your abilities and may have carried Still, we find ways of achieving manipulation mations are needed. Industrial grippers often
on this display of unique dexterity through a goals through training and exploration even if use pneumatic vacuum pumps to pick up objects
swift in-hand manipulation, taking out the cor- the end result is not always optimally performed. by sucking. This technique is unbeatable when
rect key and inserting it into the lock even We may also adapt as circumstances dictate (e.g., it comes to grasping an object but is much less
though the corridor lights had gone out. All day tying shoes with an excess of free shoelace or useful for object manipulation (e.g., reorienting
long, our fingers grasp, move, and transform when the ends are quite short), forcing us to the object and placing it in a confined space).
objects and interact with objects in various media deviate from our normal methods. Thus, the One step to address this challenge is to provide
such as air, water, and oil. We do not spend time context in which interactions are performed af- robots with more dexterous hands. Yet creating
thinking about what our hands and fingers are fects various parameters of the execution. hands as dexterous as human hands is difficult,
doing or how the continuous integration of var- Although robotics has made vast progress in owing to a lack of sensors and actuators equiv-
ious sensory modalities—such as vision, touch, mechanical design, perception, and robust con- alent in size, precision, and efficiency to our
proprioception, and hearing—help us outperform trol targeted to grasping and handling objects, skin and muscles.
any other biological system in the breadth of the robotic manipulation is still a poor proxy for hu- Improvements in robots’ dexterity are not lim-
interaction tasks we can execute. Largely over- man dexterity. To date, no robots can easily hand- ited to the engineering of more-capable hands.
looked, and perhaps most fascinating, is the ease wash dishes, button a shirt, or peel a potato. Advanced software programs are required to
with which we perform these interactions, re- analyze in real time the large flux of visual, tac-
sulting in a belief that they are also easy to ac- What can robots do today? tile, and force information and to relate these
complish in artificial systems such as robots. Robots are skilled at picking up and manipu- different senses to recognize objects and model
Manipulating objects is such a ubiquitous ac- lating objects in repetitive and familiar settings their transformations. Additionally, robots need
tivity that we forget how difficult it was to ac- such as industrial assembly setups. In such set- advanced cognitive capabilities to predict where,
quire this competence as a child. Children are tings, the geometry, material properties, and how, and why to manipulate objects. The rest
born with simple grasp reflexes. It takes them weight of the objects are commonly known. of this Review describes why overcoming these
3 years to develop an individuated control of Robots can handle some variation in routine challenges is difficult and where the field of
each finger and another 6 years to display an movements in terms of adapting to small differ- robotics stands today.
adult-equivalent ability for making smooth con- ences in the object properties, but the whole pro-
tact and for planning sequences of manipulation cess is typically optimized to a limited set of Why is designing robotic
skills (1). Even for humans, some dexterous activ- expected variations. In early factory settings, hands difficult?
ities may pose a challenge. For example, tying robot arms followed predetermined trajectories Although research on robot hands has been on-
shoes may be done in various ways, and there and assumed that objects would always appear going for more than five decades (2–4), the most
may be several valid models of how to execute at the same place. Today, robots can adapt their common hand used in many applications to date
such an activity. In addition, we can visually dem- trajectory to retrieve objects at different loca- is still a parallel jaw gripper, usually without any
tions, making it possible for objects to be placed extra sensing. Picking up objects with a gripper
by humans or simply dropped on a conveyer belt devoid of sensing is akin to grasping with the
1
Learning Algorithms and Systems Laboratory, École instead of being deposited at exact positions by tip of your thumb and index finger when both
Polytechnique Fédérale de Lausanne (EPFL), Lausanne, other machines. The classical assembly lines in are numb! This tool may suffice for simple pick-
Switzerland. 2Robotics, Perception and Learning (RPL),
EECS, Royal Institute for Technology (KTH), Stockholm,
which robots were bolted into the floor and placed and-place actions, but not for more-complex
Sweden. one after another, typical for the automobile in- motions such as shuffling keys. Because the
*Corresponding author. Email: [email protected] dustry, can now be made more flexible. Objects human hand performs intricate movements with

Billard and Kragic, Science 364, eaat8414 (2019) 21 June 2019 1 of 8

R ES E A RC H | R E V IE W

Fig. 1. Soft hands. Robotic hands are traditionally made of hard materials with rigid control of fingers. Recent designs aim to mimic the human hand’s
natural compliance by using soft actuators, soft materials, and advanced controllers. (A) Rigid material and actuators and (B) a rigid cover with
partially soft cable-driven actuation: Both hands become soft through software intervention, modulating pressure at the fingertips via tactile feedback.
[Reproduced from (17, 33)] (C) A soft, foldable gripper that can adapt shape and stiffness. [Reproduced from (11)] (D) Soft actuation and material
for a rehabilitation glove that can be worn by a human. [Reproduced from (12)]

ease, it is a natural inspiration for robotics. But

designing robotic hands with sensors and ac-

Downloaded from https://www.science.org at Slub Dresden on May 22, 2023

tuators similar to those of the human hand is
difficult for many reasons.
When constructing anthropomorphic robot
hands, it is challenging to fit all of the necessary
actuators, sensors, and mechanical structure in
the limited available space. Another obstacle is
to keep the total weight of the hand low so that
it satisfies the payload requirements of the
arm to which it is attached. Hence, compared
with human hands, most anthropomorphic
hands and prostheses do not have nearly as
many controllable degrees of freedom (5, 6).
The human hand is soft and flexible, with a
dexterous thumb whose distinctive range of mo- Fig. 2. Sense of touch for robots. (Left) Vision can be used to infer contact forces (red).
tion remains difficult to replicate mechanically [Reproduced from (21)] (Right) Stretchable artificial skin measures contact at knuckles, which may
(7), as the intricate combinations of tendons and be useful for exploring internal parts of objects. [Reproduced from (18)]
muscles differ markedly from traditional serial
robotic joint design (8). Today, robotic hands are
still largely composed of rigid plastic and metal and damping. Human skin is a high-frequency limb segments. Touch detection at the joint
components, with electric motors as actuators. and high-resolution sensor that provides precise (knuckle, elbow, knee) is, however, crucial to de-
This rigidity is partially the cause of the lack of information on normal and tangential forces, tect entrapment. It is also useful to guide explo-
dexterity, as it allows no room for mistakes when information that is critical for grip adjustment. ration inside objects (17). Such contact can be
executing grasps. Rigid fingers closing on an ob- Human skin can also measure stretch and tem- detected only by soft sensors that bend and ex-
ject may easily move rather than grasp an object perature. By contrast, robot hands typically mea- tend along the flexion and extension points of
if its pose is not perfectly estimated, and apply- sure exerted forces through miniature force the limb (Fig. 2, right) (18). Hence, flexible and
ing too much force may crush the object. A grow- sensors placed solely at the fingertips (14). Force stretchable skins are of utmost interest to robot-
ing trend in robotics is the development of soft sensors yield very accurate 3D measurements, icists (19). Prototypes exist in laboratories, and
hands that can conform to an object’s shape, but they cannot easily reveal the exact location we can expect to see their deployment soon,
absorb unexpected forces at contact, and com- of contact. To move objects once held in the hand given the current interest in soft electronics (20).
pensate for load change during manipulation or to hold multiple objects at once (Fig. 1), one As an alternative to using skin, it is possible
(9, 10). needs to measure precise contact points, not just to deduce haptic (contact and force) information
Softness can be achieved through a change in at the fingertips but also along the length and from vision. One can, for instance, infer forces
hardware or software or a combination of both side of the fingers and inside the palm. This can from vision through a dynamic model of contacts
(Fig. 1). Softness from material used to construct be achieved through artificial skins that provide (21) (Fig. 2, left) or use an optical sensor that
hands builds on solutions from 3D manufac- contact measurements all along the limbs. In- renders deformation of an object’s geometry at
turing and materials science. For instance, one terest in artificial skins can be traced back to the a high spatial resolution (22). The necessity of
can manufacture rigid and flexible materials in 1980s (15, 16), but major advances were achieved estimating the exact position of an object, its
a layer-by-layer manner to create foldable fingers in the past decade. local geometry, and other properties such as
that can deploy and retract as needed (11). Cur- At present, we find a variety of affordable weight and weight distribution depends strongly
rently, the low payload and slow speed of these commercial products, several of which can be on the application. It is the interplay among
elastomers restricts manipulation to light objects customized to a robot’s shape. Touch sensors hand design, material, and internal and external
only. As an alternative to generate more power, measure the normal contact force; a few also sensing that offers the appropriate redundancy.
pneumatic or hydraulic actuation may be used provide data on tangential forces, torque, tem- One additional challenge is the need to mea-
(12, 13). perature, vibrations, or surface properties. Never- sure contact at a very high frequency for accu-
Human hands are covered with a multipurpose theless, most touch sensors are rigid, and their rate and timely detection of slip (23). Such high
skin that provides the appropriate level of friction placement is constrained to fingertips and along spatial and temporal resolution, together with

Billard and Kragic, Science 364, eaat8414 (2019) 21 June 2019 2 of 8

R ES E A RC H | R E V IE W

the real-time processing of vision data for ob- thermore, many hands come with no or limited objects, robot vision can help infer geometric
ject tracking, leads to a computational overload means of measuring contact and forces. Thus, and physical properties of known and even
with a massive data stream that must be inter- a change in paradigm is needed to move away unknown objects (26), and this information is
preted in real time. This processing is usually from developing robotic arms devoid of hands important for shaping the aperture of the hand
carried out by a CPU (central processing unit) and hands devoid of arms. We must further and the forces to be applied. Proprioception—
located away from the hand. Alternatively, pro- ensure that hands are developed in a “plug- namely, knowledge of where the robot’s limbs
cessing may be performed on the hand itself and-play” manner and can easily be attached are located—is needed to guide the arm and hand
through the use of a dedicated CPU (24), but and detached through existing tool-switching toward the object, with visual support to contin-
such CPUs have focused solely on processing systems. State-of-the-art force and tactile sen- uously track the object. Touch and force mea-
vision data. Additional studies are needed to sors must become an inherent part of the arm– surements become important once contact has
develop hardware for processing tactile infor- hand system. occurred and the object is held or explored by the
mation in conjunction with vision. Robot dexterity is as much a by-product of hand. The associated control algorithms are used
Dexterous robot hands may hence be realized advances in hardware as it is of advances in soft- to guide the grasp and/or to infer the object’s
by using research in materials science for the ware. It requires suitable algorithms to rapidly physical properties, such as rigidity and mass
design of soft actuation, enabling contact sensing and efficiently process the vast amount of in- distribution, that may have been poorly estimated
along the entire surface of the hand, and by using formation collected through sensors and actua- or unknown previously. Sound has also received
advances in electronics for onboard, real-time tors. At the same time, it needs algorithms to attention recently as a means to infer an invisible
processing of multisensory data. adequately control the movement of a hand in object’s content and to monitor changes in
relation to object, scene, and task properties. We content during manipulation (27).
Design beyond anthropomorphism next review advances in perception, control, and As an example, a robot is tasked with fetching
Although the human hand is fascinating, it does learning for manipulation. a package of milk from a refrigerator. Before the

Downloaded from https://www.science.org at Slub Dresden on May 22, 2023

not have to be the ultimate solution for robotics. robot holds the package in hand, it may not
A human hand design may be desirable for aes- Perception for manipulation know how much milk the package contains nor
thetic reasons; for instance, when designing hand As for humans, robot perception for manipu- the package’s actual weight. Given that the
prostheses or humanoid robots. But this same lation is multimodal (Fig. 4). Vision is instru- package may be made of cardboard, the robot
design may be superfluous for many robots. In- mental for recognizing and localizing objects. needs to know the weight in order to apply a
dustrial hands remain a good solution for spe- When associated with a database of existing suitable grasping force and avoid destroying
cific tasks. Rather than try to replicate the
positioning of human fingers, these hands have
two or three fingers arranged symmetrically
around the palm, a design particularly suited
for industrial screwing.
Robotics keeps oscillating between anthropo-
morphic and traditional industrial designs for
hands. But the gripping systems of simpler ani-
mals may also provide inspiration. For instance,
fish suck in their prey. Adding suction at robots’
fingertips is useful under water, as this technol-
ogy cancels the flow generated by the hand (25).
Why not create hands that both leverage and
go beyond nature? For instance, the human
thumb is amazing, but it creates an asymmetry
that constrains the orientation of the hand for
manipulation. Two thumbs on the same hand,
however, would provide a dexterity beyond
human capability (Fig. 3).

Desiderata for the next generation

of robotic hands Fig. 3. Designing hands beyond human dexterity. Two thumbs would make it possible to execute
The objects around us have been built and screwing and unscrewing motions with one hand rather than two. This capability may be useful
adapted to our hands, which are still rather for robots and humans via prostheses. [Illustrations: Laura Cohen]
small and very robust in comparison with con-
temporary robot hands. Enabling robots to pick
Fig. 4. Manipulation is
up small items such as pens, raisins, screws, and
multimodal. Vision is used
needles is a clear functionality goal. Today, ro-
before contact, whereas
botic arms and hands are commonly developed
haptics and sound are
separately, and integrating them is an engineer-
involved upon contact to
ing job of its own. Industrial arms have sub-
estimate an object’s
stantial payloads but are commonly designed to
physical properties that
be bolted into the floor and are too large to be
cannot be directly observed.
deployed outside industrial settings. The arms
[Photo: Learning Algorithms and
of humanoid robots and robots intended for
Systems Laboratory, EPFL]
fine assembly tasks have low payloads, which
are typically not sufficient to carry a hand and
an object held by the hand. Adding sensing
functionality to arms and hands requires cab-
ling that can quickly become complicated. Fur-

Billard and Kragic, Science 364, eaat8414 (2019) 21 June 2019 3 of 8

R ES E A RC H | R E V IE W

the package. In the case of milk, sound may known objects necessitate additional heuristics fill it with fluid, put it in a dishwasher, or serve
also provide information about the viscosity for the discovery of geometric structures (e.g., it to another person (53) (Fig. 5). Similarly, al-
when the package is shaken, as milk will sound handles, for which a robot would have a can- though a knife, fork, or spoon may be held with
different from another substance, such as yogurt. didate grasp). This challenge is closely related to the same grasp when used to mix soup, this
Over the past few years, major efforts have the classical problems of instance recognition grasp differs from those employed when these
been undertaken to analyze visual information, and categorization in computer vision, but the utensils are used for eating or cutting. To de-
and progress has been considerable. Neverthe- notion that grasping is not an isolated process termine the optimal way to grasp an object,
less, robots still struggle to recognize objects that adds a new dimension. one must understand the purpose of the grasp.
are partially occluded (28), particularly when In addition to being object dependent, grasps Hence, while roboticists aimed to solve the prob-
viewed from a moving camera or when an object are also robot dependent. Moreover, as the num- lem of how to grasp an object, they first had to
moves in a robot’s hands (29). In comparison to ber of degrees of freedom of the hand increases, identify the reason for executing the grasp. To-
developing vision algorithms, much less effort so does the complexity of the control. This is day, researchers consider grasping as part of an
has been devoted to analyzing haptic informa- particularly an issue for anthropomorphic hands. overall plan for object manipulation.
tion, given that solutions for covering entire One avenue of research to simplify the control To determine the correct grasp to use with
hands with haptic sensors are still lacking. To- draws inspiration from biology and promotes the correct tool, one must first have the correct
day, visual and haptic information are still used the use of postural synergies (38). Synergies form tool at their disposal. When in need of a hammer
primarily in a sequential manner [e.g., with vis- a basis of the subspace of effective human move- but no hammer is in reach, a human will instead
ual information provided in the preparation ments in relation to those that are possible by the select the first object sturdy enough to act as a
phase and haptic data provided upon contact kinematics of the body. These have been used as hammer. Future efforts to develop robots that
(30)], and only a few recent works integrate both a tool for robot hand analysis, control, and de- can reason in this manner when the most ap-
modalities for recognition, grasping, in-hand sign choice (39–42). Several studies have also propriate tool is not available will be critical to

Downloaded from https://www.science.org at Slub Dresden on May 22, 2023

adaptation, and shape reconstruction (31–35). demonstrated how underactuated hands can facilitate deployment of robots in natural envi-
In comparison, humans are proficient at alter- be leveraged to grasp and manipulate objects in ronments. Additionally, robots with this capabil-
nating between different senses, from vision to unstructured environments and how this work ity will be able to use tools originally designed
touch and back, and can do so rapidly even if may lead to adaptive hands that are relatively for human dexterity to perform household tasks
these senses change in processing frequency. cheap, lightweight, and easy to control compared without making undesired modifications to our
By contrast, robots still lack the ability to decide with fully actuated hands (43–48). More recent households. How to program such “common
what sensors to use, when to use them, and work has optimized hand design to improve sense” tool use is hence an important avenue
when to switch between sensors. manipulation capabilities (49, 50), providing for research, and some initial work has been
open-source software for such design. Other conducted in this direction (54–57).
Grasping: A stepping stone recent work has suggested that the ability of
Before a robot can manipulate an object in hand, compliant hands to deform in and with the Manipulations that remain difficult
it must be able to grasp its fingers around the environment may reduce the cognitive load of The previous sections detail the many problems
object. If grasping is conceptualized only as get- manipulation (51). Furthermore, this idea can that remain to be solved before robots can per-
ting fingers around an object with no additional be studied systematically using morphological form grasps with a human level of intelligence.
constraints considered, the challenge of grasping computation (52), in which compliant interac- This said, robots are already fairly efficient at
may appear to be solved. However, grasping an tions allow adaptation of behavior to a particular grasping and releasing certain types of objects.
object is a far more daunting problem. For dec- context, without the need for explicit control. They are also capable of performing a variety of
ades, researchers have worked to establish the simple manipulation actions such as throwing
theory of how to form a stable grasp. This be- From grasping to manipulation (58), sliding (59), poking (60), pivoting (61), and
came an intricate mathematical exercise aimed Grasping is not an end on its own; it is also rel- pushing (62). Difficulties arise when these ac-
at determining the minimal number and optimal ated to the task a human or robot is executing. tions must be performed in cluttered environ-
positions of the fingertips on the object’s surface For example, one grasps a cup differently de- ments or require contact-rich interactions (e.g.,
to ensure stability (36). pending on whether the goal is to drink from it, when an object of interest is placed close to or
Although it is valuable, most of this theoretical
work relies on assumptions such as a known 3D
model of the object, a rigid point contact, and no Fig. 5. Grasp functionality. (Top) A
uncertainty in the process. To incorporate uncer- human grasps an item differently
tainty originating from imperfect object models depending on whether the aim is to
and dynamics in the interaction process, we hold it, open the cap, or hand it to
must go beyond modeling a single point contact someone else. (Bottom) Robots can
and pursue substantial advances in the basic also be programmed to hold the
theory. same glass differently depending on
Thus, many of the more recent approaches whether they are tasked with
are data driven (37). To avoid computing an op- handing it to a human or pouring out
timal grasp each time a robot encounters an its contents. [Photos: Learning
object, one can build a database of grasps and Algorithms and Systems
employ methodologies for sampling and rank- Laboratory, EPFL]
ing candidate grasps in real time. This approach
deals with uncertainty in perception and pro-
vides fast and online generation of grasps for
known, familiar, and even unknown objects. Prior
knowledge of object properties determines the
necessary perceptual processing and associated
object representations for generating and rank-
ing grasp candidates. Although this method
works well for known and familiar objects, un-

Billard and Kragic, Science 364, eaat8414 (2019) 21 June 2019 4 of 8

R ES E A RC H | R E V IE W

Fig. 6. Remaining challenges for robot manipulation. Dexterous movement of objects within the hand (left), manipulation of deformable objects (e.g.,
fruits and vegetables) (65) (middle), and manipulation of objects in collaboration with humans (right) present ongoing difficulties. [Photos: Learning
Algorithms and Systems Laboratory, EPFL]

is covered by other objects or is located in a mitives, and planning to model the problem in
confined space such as a shelving unit). It is ne- an efficient manner. This area will gradually
cessary to plan a feasible path and generate a set produce more and more contributions in the
of intermediate actions to ensure no damage to future, given that most of today’s humanoid ro-
the hand or other objects. Today, it is also recog- bots have bimanual capabilities. Furthermore,
nized that perception and control are tightly manipulation does not stop at simply controlling

Downloaded from https://www.science.org at Slub Dresden on May 22, 2023

coupled, and the field of interactive perception the hand; it requires control of the arm, torso,
(63) regards manipulation as a means to perceive and ultimately the entire body (73). The chal-
and perception as a means to achieve better lenges listed above only increase in scale when
manipulation. one wishes to enable a full humanoid robot to
Manipulation actions that generate changes manipulate objects while maintaining its balance
on the object (cutting, crushing) remain partic- (74) (Fig. 7). Finally, control of more-complex
ularly difficult, as they require a model of the manipulation skills that require reasoning, such
deformation and advanced perception to moni- as using an object to retrieve another object, are
tor the alterations (64). To facilitate adaptation still in infancy.
to the changes induced, these actions also neces-
sitate that the forces be applied by the hand (e.g., Learning for manipulation
a reduction of friction when unscrewing a bottle Human dexterity is a skill acquired during child-
cap, an increase in viscosity when digging into a hood and further refined throughout life, in
melon) (Fig. 6) (65). Thus, modeling an object’s activities such as playing a musical instrument
friction and viscosity properties is still an im- or practicing a craft. Similarly, robot dexterity
portant open problem. cannot be achieved in the confines of our labor-
In-hand manipulations in which an object is atories. To be able to manipulate the vast array of
moved while being held are also particularly objects that exist throughout the world, robots
complicated. Examples include twirling a pen must be able to learn continuously, adapt their
across fingers or preparing a key to be inserted perception, and control unfamiliar objects.
into a keyhole. These actions comprise an exten- Learning also addresses some challenges linked
sive combination of (re)grasping movements to the lack of accurate models of objects and
and sliding and rotating maneuvers, as well as contact dynamics and the increasing complexity
interactions between two arms and hands, in some of control for robots with large degrees of free-
cases. When discussing such advanced interac- dom. Hence, many of the present approaches to
tions with objects in robotics, we commonly talk dexterous manipulation rely on learning meth-
about intrinsic and extrinsic dexterity. The former odologies in place of control-theoretic approaches.
denotes the ability of the hand to manipulate For instance, learning can be used to embed rep-
objects using its available degrees of freedom. resentations of stable or suitable grasps (75–78), Fig. 7. Whole-body manipulation. Manipulation
Hands with high intrinsic dexterity often mimic which can then be applied to verify stability and of a heavy object by a humanoid robot requires
the structure of the human hand (66). Alterna- generate regrasping motions at run time or to coordination of arms and body to maintain
tively, the hand can be simpler, and the end- catch a fast-moving object (79). Learning is par- balance. [Reproduced from (74)]
effector is designed specifically for a particular ticularly suitable for embedding the dynamic
task (67, 68). Extrinsic dexterity is the ability to nature of grasping and manipulation, as well training data is to test the algorithms in sim-
compensate for the lack of degrees of freedom by as for modeling manipulation of complex non- ulation first and then refine the learning on a
using external support, such as friction, gravity, rigid objects. Learning has been used to model real platform; e.g., for learning dexterous in-hand
and contact surfaces (69). This functionality also contacts (80) and is also beneficial for reducing manipulation (81–83). Training in simulation de-
enables dexterous manipulation with simple pa- control dimensions by determining latent space, pends on having an accurate simulator of the
rallel grippers. as required in bimanual dynamics (65). tasks. Alternatively, robots may learn from image
One of the largely underdeveloped areas in Nevertheless, solving all problems by solely data and videos available on the internet (84) or
robotics is dual-arm or bimanual (70) manip- relying on learning is not a viable solution and from demonstration by a live expert, usually a
ulation, as well as the use of the second hand has certain limitations. First, learning requires human. Yet it may not always be possible to find
and/or arm to support both intrinsic and ex- data for training, and a common approach is to an expert, especially when the tasks are danger-
trinsic dexterity (71). Some recent work in this generate data from trial-and-error experiments. ous or require extreme precision. Hence, although
area (72) proposes integration of object repre- However, this process is tedious and may dam- learning is important, it cannot be the answer to
sentation, definition of simple movement pri- age the robot. A growing trend in providing every problem in robotics.

Billard and Kragic, Science 364, eaat8414 (2019) 21 June 2019 5 of 8

R ES E A RC H | R E V IE W

Manipulating objects in interaction and more adaptive control modes. In addition, from robots will disappear gradually. Robots
and collaboration with humans: Reality roboticists seek guarantees in terms of machine will thus need to be engaged in collaborative
and challenges performance and the use of common evaluation tasks to jointly manipulate objects with humans
Human–robot collaboration in manufacturing scenarios and benchmarks. while adapting to unexpected human behavior.
setups has been deemed crucial for the industry Equipping robots with advanced physical inter-
(85, 86). Although historically, humans were pro- Outlook action capabilities to achieve safe and smooth
hibited from entering the robot’s environment Since the 1960s, substantial progress has been synchronization of motion between machine and
(ISO 10218; ANSI/RIA R15.06-1999), it is now achieved in several areas of robotic manipu- human is still a major hurdle. This objective will
accepted that robots can work in close proximity lation. We have established the basic theory require advances in detailed tracking of human
to and in collaboration with humans. However, of evaluating stability of a grasp, control algo- fine body movement, as well as a better under-
potentially dangerous scenarios may still occur rithms that can adapt to unpredicted situations, standing of how humans collaborate and achieve
and need to be addressed. Currently, human– and changing dynamics when the appropriate joint goals through planning and direct physical
robot collaboration is allowed through the use sensor feedback is available to perform state es- interaction. Furthermore, there is a demand
of manipulators that remain fairly light and are timation. Lately, the field has also seen advances for robots that are safe by design, putting focus
endowed with internal force sensors for detect- in data-driven approaches in which even dexter- toward soft and lightweight structures as well
ing unexpected contact or collision with humans. ous in-hand manipulation can be accomplished, as control and planning methodologies based
For applications that require maneuvering heavy but only for very specific problems and in highly on multisensory feedback. Human ways of acting
weight, robots capable of managing the weight tailored environments. Achievement of robust, will continue to serve as inspiration for future
may be coupled with an external vision system flexible, and adaptive grasping and manipulation robot systems, and robots will serve as a tool for
for monitoring human presence. Yet challenges of completely unknown objects in media such as better understanding humans.
remain for accurately detecting human pres- water and oil (not solely in air) is expected to

Downloaded from https://www.science.org at Slub Dresden on May 22, 2023

ence. At present, the best solution is to com- result in a major manufacturing revolution, which
bine sensing of proximity and force with external will affect most of the work that relies on fine REFERENCES AND NOTES

vision-based monitoring. Nonetheless, the 100% manipulation and high dexterity. However, sys- 1. J. P. Thibaut, L. Toussaint, Developing motor planning over
ages. J. Exp. Child Psychol. 105, 116–129 (2010).
fence-based safety paradigm is gone, and indus- tematic development is ongoing toward several doi: 10.1016/j.jecp.2009.10.003; pmid: 19919864
trial standards now target risk minimization and technologies vital for meeting and exceeding hu- 2. H. Hanafusa, H. Asada, Stable prehension by a robot hand
mitigation (ISO/TS 15066). man dexterity and fine manipulation capabilities. with elastic fingers. T. Soc. Instr. Control Eng. 13, 370–377
In addition to facing a world in which objects First, there is still a need for basic theoretical (1977). doi: 10.9746/sicetr1965.13.370
3. A. Bicchi, Hands for dexterous manipulation and robust
move and change, robots are now expected to development. We must seek to understand and grasping: A difficult road toward simplicity. IEEE Trans.
manipulate these objects in collaboration with model soft point contact and provide stability Robot. Autom. 16, 652–662 (2000). doi: 10.1109/
humans. Interactive and collaborative manipu- rules for both point contacts and surface con- 70.897777
lation adds a new dimension to robot manipu- tacts. We also need to develop a better method 4. R. A. Grupen, T. C. Henderson, I. D. McCammon, A survey
of general-purpose manipulation. Int. J. Robot. Res. 8, 38–62
lation but presents a wealth of challenges (87). for modeling objects whose states change mark- (1989). doi: 10.1177/027836498900800103
For example, when a robot is tasked with hand- edly after manipulation (e.g., a cucumber after 5. A. M. Okamura, N. Smaby, M. R. Cutkosky, “An overview of
ing an object to a person or carrying a large being sliced, an onion after being chopped). A dexterous manipulation” in Proceedings of the 2000 IEEE
object jointly with someone, the robot must thorough description of manipulation and task International Conference on Robotics and Automation
(IEEE, 2000), pp. 255–262.
grasp and move the object carefully and with goals will be required for planning and gener- 6. T. Feix, J. Romero, C. H. Ek, H. B. Schmiedmayer, D. Kragic,
foresight, so that the robot can infer where the ating appropriate intermediate grasping and A metric for comparing the anthropomorphic motion
human will move and the human does not get manipulation actions. This emphasis on theory capability of artificial hands. IEEE Trans. Robot. 29, 82–93
injured. As simple as it may seem, the act of a and planning is also relevant for data-driven (2013). doi: 10.1109/TRO.2012.2217675
7. M. Chalon, M. Grebenstein, T. Wimböck, G. Hirzinger, “The
robot handing an object to a human entails approaches, as we need better tools for sim- thumb: Guidelines for a robotic design” in Proceedings
several complex questions, which have in turn ulating soft bodies and generating relevant of the 2010 IEEE/RSJ International Conference on Intelligent
inspired studies on how to enable a robot to scenarios and examples that include force and Robots and Systems (IEEE, 2010), pp. 5886–5893.
properly perform this task (88–92). These ques- torque information. 8. L. Y. Chang, Y. Matsuoka, “A kinematic thumb model for
the ACT hand” in Proceedings of the 2006 IEEE International
tions range from how to present an object for In addition to the aforementioned modeling Conference on Robotics and Automation (IEEE, 2006),
optimal human grasping to others related to and software aspects, we also seek to achieve pp. 1000–1005.
social factors, such as the role of gaze, social substantial progress in hardware development 9. Y. Li, I. Kao, “A review of modeling of soft-contact fingers and
cues, and awareness of user state. There is no and design. One area of particular relevance is stiffness control for dextrous manipulation in robotics”
in Proceedings of IEEE International Conference on Robotics
agreement on what factors are most important that of robot sensing. It will be important to and Automation (IEEE, 2001), pp. 3055–3060.
in determining how handovers are carried out develop skinlike sensors that are well integ- 10. B. S. Homberg, R. K. Katzschmann, M. R. Dogar, D. Rus,
between two humans, let alone between a robot rated with hand design but do not require ex- “Haptic identification of objects using a modular soft
and a human. Although most research has foc- cessive cabling or add substantial weight. This robotic gripper” in Proceedings of IEEE/RSJ International
Conference on Intelligent Robots and Systems (IEEE, 2015),
used on a robot handing objects to humans, sensing functionality should facilitate force 1698–1705.
there have also been studies on robots taking and torque measurements, determining shear 11. A. Firouzeh, J. Paik, Grasp mode and compliance control
objects from humans (93–95). Additionally, sev- forces to detect and counteract slippage. To of an underactuated origami gripper using adjustable
eral efforts have been aimed at enabling a robot achieve dexterous in-hand manipulation, we also stiffness joints. IEEE/ASME T. Mech. 22, 2165–2173 (2017).
doi: 10.1109/TMECH.2017.2732827
to manipulate objects jointly with humans, and need actuated hands that can be controlled with 12. P. Polygerinos, Z. Wang, K. C. Galloway, R. J. Wood,
the act of carrying objects jointly with humans high frequencies. Such hands must function in C. J. Walsh, Soft robotic glove for combined assistance and
has been demonstrated using both humanoid ro- different media (air, water, and oil) without at-home rehabilitation. Robot. Auton. Syst. 73, 135–143
bots (96–99) and mobile manipulators (100–103). being damaged or needing to be covered by (2015). doi: 10.1016/j.robot.2014.08.014
13. R. Deimel, O. Brock, A novel type of compliant and
Notable recent efforts have explored human– special gloves. Overall, we need hands that are underactuated robotic hand for dexterous grasping.
robot joint manipulation of deformable mate- light, cheap, robust, and easily integrated with Int. J. Robot. Res. 35, 161–185 (2016). doi: 10.1177/
rials (104), helping humans to dress (105), and any type of robotic arm. 0278364915592961
assistive support (106). Finally, an important industrial challenge will 14. A. Bicchi, J. K. Salisbury, D. L. Brock, Contact sensing
from force measurements. Int. J. Robot. Res. 12, 249–262
Hence, for robots to work seamlessly with be to bring robots in closer proximity to humans (1993). doi: 10.1177/027836499301200304
humans, researchers are striving to equip robots and enable safe physical interaction and collab- 15. L. D. Harmon, Automated tactile sensing. Int. J. Robot. Res. 1,
with the tools for better perception of humans oration. Fences that used to separate humans 3–32 (1982). doi: 10.1177/027836498200100201

Billard and Kragic, Science 364, eaat8414 (2019) 21 June 2019 6 of 8

R ES E A RC H | R E V IE W

16. H. R. Nicholls, M. H. Lee, A survey of robot tactile sensing 36. A. Bicchi, V. Kumar, “Robotic grasping and contact: A review” Conference on Intelligent Robots and Systems (IEEE, 2016),
technology. Int. J. Robot. Res. 8, 3–30 (1989). doi: 10.1177/ in Proceedings of the 2000 IEEE International Conference pp. 2765–2770.
027836498900800301 on Robotics and Automation (IEEE, 2000), pp. 348–353. 58. A. Ghadirzadeh, A. Maki, D. Kragic, M. Björkman, “Deep
17. N. Sommer, A. Billard, Multi-contact haptic exploration and 37. J. Bohg, A. Morales, T. Asfour, D. Kragic, Data-driven predictive policy training using reinforcement learning” in
grasping with tactile sensors. Robot. Auton. Syst. 85, grasp synthesis-A survey. IEEE Trans. Robot. 30, 289–309 Proceedings of the 2017 IEEE/RSJ International Conference
48–61 (2016). doi: 10.1016/j.robot.2016.08.007 (2014). doi: 10.1109/TRO.2013.2289018 on Intelligent Robots and Systems (IEEE, 2017),
18. A. P. Gerratt, N. Sommer, S. Lacour, A. Billard, “Stretchable 38. M. L. Latash, Fundamentals of Motor Control (Academic pp. 2351–2358.
capacitive tactile skin on humanoid robot fingers - First Press, 2012). 59. J. Shi, J. Z. Woodruff, K. M. Lynch, “Dynamic in-hand sliding
experiments and results” in Proceedings of the 2014 IEEE 39. A. Bicchi, M. Gabiccini, M. Santello, Modelling natural and manipulation” in Proceedings of the 2015 IEEE/RSJ
International Conference on Humanoid Robots (IEEE, 2014), artificial hands with synergies. Philos. Trans. R. Soc. B 366, International Conference on Intelligent Robots and Systems
pp. 238–245. 3153–3161 (2011). doi: 10.1098/rstb.2011.0152; (IEEE, 2015), pp. 870–877.
19. M. Amjadi, K. U. Kyung, I. Park, M. Sitti, Stretchable, pmid: 21969697 60. P. Agrawal, A. Nair, P. Abbeel, J. Malik, S. Levine, “Learning
skin‐mountable, and wearable strain sensors and 40. C. Y. Brown, H. H. Asada, “Inter-finger coordination and to poke by poking: Experiential learning of intuitive physics”
their potential applications: A review. Adv. Funct. Mater. postural synergies in robot hands via mechanical implementation in Proceedings of IEEE Conference on Neural Information
26, 1678–1698 (2016). doi: 10.1002/adfm.201504755 of principal components analysis” in Proceedings of the 2007 Processing Systems (IEEE, 2016), pp. 5074–5082.
20. S. Xu et al., Soft microfluidic assemblies of sensors, circuits, IEEE/RSJ International Conference on Intelligent Robots and 61. R. Antonova, S. Cruciani, C. Smith, D. Kragic, Reinforcement
and radios for the skin. Science 344, 70–74 (2014). Systems (IEEE, 2007), pp. 2877–2882. learning for pivoting task. arxiv:1703.00472 [cs.RO]
doi: 10.1126/science.1250169; pmid: 24700852 41. M. Ciocarlie, C. Goldfeder, P. Allen, “Dexterous grasping (1 March 2017).
21. T. H. Pham, N. Kyriazis, A. A. Argyros, A. Kheddar, via eigengrasps: A low-dimensional approach to a high- 62. J. Z. Woodruff, K. M. Lynch, “Planning and control for
Hand-object contact force estimation from markerless visual complexity problem” in Robotics: Science and Systems dynamic, non-prehensile, and hybrid manipulation tasks” in
tracking. IEEE Trans. Pattern Anal. Mach. Intell. 40, (RSS), Manipulation Workshop - Sensing and Adapting to Proceedings of the 2017 IEEE International Conference on
2883–2896 (2018). doi: 10.1109/TPAMI.2017.2759736; the Real World (2007); http://projects.csail.mit.edu/ Robotics and Automation (IEEE, 2017), pp. 4066–4073.
pmid: 29989962 manipulation/rss07/paper__dexterous_grasping_via_ 63. J. Bohg et al., Interactive perception: Leveraging action in
22. W. Yuan, S. Dong, E. H. Adelson, Gelsight: High-resolution eigengrasps_a_low_dimensional_approach_to_a_ perception and perception in action. IEEE Trans. Robot. 33,
robot tactile sensors for estimating geometry and high_complexity_problem__ciocarlie.pdf. 1273–1291 (2017). doi: 10.1109/TRO.2017.2721939
force. Sensors 17, 2762 (2017). doi: 10.3390/s17122762; 42. M. G. Catalano et al., Adaptive synergies for the design and 64. M. Gemici, A. Saxena, “Learning haptic representation for

Downloaded from https://www.science.org at Slub Dresden on May 22, 2023

pmid: 29186053 control of the Pisa/IIT softhand. Int. J. Robot. Res. 33, manipulating Deformable food objects” in Proceedings
23. W. Chen, H. Khamis, I. Birznieks, N. F. Lepora, S. J. Redmond, 768–782 (2014). doi: 10.1177/0278364913518998 of the 2014 IEEE/RSJ International Conference on Intelligent
Tactile sensors for friction estimation and incipient slip 43. L. Birglen, T. Laliberté, C. M. Gosselin, Underactuated Robotic Robots and Systems (IEEE, 2014), pp. 638–645.
detection - towards dexterous robotic manipulation: A review. Hands (Springer, ed. 1, 2007), vol. 40. 65. L. P. Ureche, A. Billard, Constraints extraction from
IEEE Sens. J. 18, 9049–9064 (2018). doi: 10.1109/ 44. M. Ciocarlie et al., The velo gripper: A versatile single- asymmetrical bimanual tasks and their use in coordinated
JSEN.2018.2868340 actuator design for enveloping, parallel and fingertip grasps. behavior. Robot. Auton. Syst. 103, 222–235 (2018).
24. D. Bhowmik, K. Appiah, “Embedded vision systems: A review Int. J. Robot. Res. 33, 753–767 (2014). doi: 10.1177/ doi: 10.1016/j.robot.2017.12.011
of the literature” in Applied Reconfigurable Computing. 0278364913519148 66. R. Ozawa, K. Tahara, Grasp and dexterous manipulation of
Architectures, Tools, and Applications, N. Voros et al., Eds. 45. R. R. Ma, W. G. Bircher, A. M. Dollar, “Toward robust, multifingered robotic hands: A review from a control view
(Lecture Notes in Computer Science Series, Springer, 2018), whole-hand caging manipulation with underactuated hands” point. Adv. Robot. 31, 1030–1050 (2017). doi: 10.1080/
pp. 204–216. in Proceedings of the 2017 IEEE International Conference on 01691864.2017.1365011
25. M. R. Cutkosky, “Reach, grasp, and manipulate” in Living Robotics and Automation (IEEE, 2017), pp. 1336–1342. 67. N. Rahman, L. Carbonari, M. D’Imperio, C. Canali,
Machines: A Handbook of Research in Biomimetics 46. P. Kyberd, A. Clawson, B. Jones, The use of underactuation D. G. Caldwell, F. Cannella, “A dexterous gripper for in-hand
and Biohybrid Systems, T. J. Prescott, N. Lepora, in prosthetic grasping. Mech. Sci. 2, 27–32 (2011). manipulation” in Proceedings of the 2016 IEEE International
P. F. M. J. Verschure, Eds. (Oxford Univ. Press, 2018), doi: 10.5194/ms-2-27-2011 Conference on Advanced Intelligent Mechatronics (IEEE,
chap. 30. 47. C. Meijneke, G. A. Kragten, M. Wisse, Design and 2016), pp. 377–382.
26. N. Vahrenkamp, L. Westkamp, N. Yamanobe, E. E. Aksoy, performance assessment of an underactuated hand for 68. W. G. Bircher, A. M. Dollar, N. Rojas, “A two-fingered robot
T. Asfour, “Part-based grasp planning for familiar objects” in industrial applications. Mech. Sci. 2, 9–15 (2011). gripper with large object reorientation range” in Proceedings
Proceedings of the IEEE/RAS 16th International Conference doi: 10.5194/ms-2-9-2011 of the 2017 IEEE International Conference on Robotics and
on Humanoid Robots (IEEE, 2016), pp. 919–925. 48. A. M. Dollar, R. D. Howe, The highly adaptive SDM hand: Automation (IEEE, 2017), pp. 3453–3460.
27. S. Clarke, T. Rhodes, C. G. Atkeson, O. Kroemer, “Learning Design and performance evaluation. Int. J. Robot. Res. 29, 69. N. C. Dafle, A. Rodriguez, R. Paolini, B. Tang, S. S. Srinivasa,
audio feedback for estimating amount and flow of granular 585–597 (2010). doi: 10.1177/0278364909360852 M. Erdmann, M. T. Mason, I. Lundberg, H. Staab,
material” in Proceedings of the 2nd Conference on Robot 49. J. Borràs, A. M. Dollar, Dimensional synthesis of three- T. Fuhlbrigge, “Extrinsic dexterity: In-hand manipulation with
Learning (Microtome Publishing, 2018), pp 529–550. fingered robot hands for maximal precision manipulation external forces” in Proceedings of the 2014 IEEE International
28. Y. Guo, M. Bennamoun, F. Sohel, M. Lu, J. Wan, 3D object workspace. Int. J. Robot. Res. 34, 1731–1746 (2015). Conference on Robotics and Automation (IEEE, 2014),
recognition in cluttered scenes with local surface features: A doi: 10.1177/0278364915608250 pp. 1578–1585.
survey. IEEE Trans. Pattern Anal. Mach. Intell. 36, 2270–2287 50. R. Ma, A. Dollar, Yale openhand project: Optimizing open- 70. N. Vahrenkamp, M. Przybylski, T. Asfour, R. Dillmann,
(2014). doi: 10.1109/TPAMI.2014.2316828; pmid: 26353066 source hand designs for ease of fabrication and adoption. “Bimanual grasp planning” in Proceedings of the 2011 IEEE/
29. S. Ojha, S. Sakhare, “Image processing techniques for object IEEE Robot. Autom. Mag. 24, 32–40 (2017). doi: 10.1109/ RAS International Conference on Humanoid Robots
tracking in video surveillance-A survey” in 2015 IEEE MRA.2016.2639034 (IEEE, 2011), pp. 493–499.
International Conference on Pervasive Computing (IEEE, 51. K. Ghazi-Zahedi, R. Deimel, G. Montúfar, V. Wall, O. Brock, 71. C. Smith et al., Dual arm manipulation-A survey. Robot.
2015), pp 1–6. “Morphological computation: the good, the bad, and the Auton. Syst. 60, 1340–1353 (2012). doi: 10.1016/
30. R. S. Dahiya, G. Metta, G. Cannata, M. Valle, Special issue on ugly” in Proceedings of the 2017 IEEE/RSJ International j.robot.2012.07.005
robotic sense of touch. IEEE Trans. Robot. 27, 385–388 Conference on Intelligent Robots and Systems (IEEE, 2017), 72. S. Cruciani, C. Smith, D. Kragic, K. Hang, “Dexterous
(2011). doi: 10.1109/TRO.2011.2155830 pp. 464–469. manipulation graphs” in Proceedings of the 2018 IEEE/RSJ
31. J. Ilonen, J. Bohg, V. Kyrki, “Fusing visual and tactile sensing 52. R. Pfeifer, J. C. Bongard, How the Body Shapes the Way We International Conference on Intelligent Robots and Systems
for 3-D object reconstruction while grasping” in Proceedings Think: A New View of Intelligence (MIT Press, 2006). (IEEE, 2018), pp. 2040–2047.
of the 2013 IEEE International Conference on Robotics and 53. R. de Souza, S. El-Khoury, J. Santos-Victor, A. Billard, 73. M. T. Mason, Toward robotic manipulation. Annu. Rev. Control
Automation (IEEE, 2013), pp. 3547–3554. Recognizing the grasp intention from human demonstration. Rob. Auton. Syst 1, 1–28 (2018). doi: 10.1146/annurev-
32. H. Dang, P. K. Allen, “Grasp adjustment on novel objects Robot. Auton. Syst. 74, 108–121 (2015). doi: 10.1016/ control-060117-104848
using tactile experience from similar local geometry” j.robot.2015.07.006 74. K. Bouyarmane, K. Chappellet, J. Vaillant, A. Kheddar,
in Proceedings of the 2013 IEEE/RSJ International Conference 54. M. Madry, D. Song, D. Kragic, “From object categories to Quadratic programming for multirobot and task-space force
on Intelligent Robots and Systems (IEEE, 2013), grasp transfer using probabilistic reasoning” in Proceedings control. IEEE Trans. Robot. 35, 64–77 (2018). doi: 10.1109/
pp. 4007–4012. of the 2012 IEEE International Conference on Robotics and TRO.2018.2876782
33. M. Li, Y. Bekiroglu, D. Kragic, A. Billard, “Learning of grasp Automation (IEEE, 2012), pp. 1716–1723. 75. O. B. Kroemer, R. Detry, J. Piater, J. Peters, Combining active
adaptation through experience and tactile sensing” in 55. R. Detry, J. Papon, L. Matthies, “Task-oriented grasping with learning and reactive control for robot grasping. Robot.
Proceedings of the 2014 IEEE/RSJ International Conference on semantic and geometric scene understanding” in Auton. Syst. 58, 1105–1116 (2010). doi: 10.1016/
Intelligent Robots and Systems (IEEE, 2014), pp. 3339–3346. Proceedings of the 2017 IEEE/RSJ International Conference j.robot.2010.06.001
34. K. Hang et al., Hierarchical fingertip space: A unified on Intelligent Robots and Systems (IEEE, 2017), 76. Y. Bekiroglu, J. Laaksonen, J. A. Jørgensen, V. Kyrki,
framework for grasp planning and in-hand grasp adaptation. pp. 3266–3273. D. Kragic, Assessing grasp stability based on learning and
IEEE Trans. Robot. 32, 960–972 (2016). doi: 10.1109/ 56. M. Kokic, J. A. Stork, J. A. Haustein, D. Kragic, “Affordance haptic data. IEEE Trans. Robot. 27, 616–629 (2011).
TRO.2016.2588879 detection for task-specific grasping using deep learning” in doi: 10.1109/TRO.2011.2132870
35. M. M. Zhang, R. Detry, L. Matthies, K. Daniilidis, “Tactile- Proceedings of the 2017 IEEE International Conference on 77. R. Detry, C. H. Ek, M. Madry, J. Piater, D. Kragic, “Learning a
vision integration for task-compatible fine-part manipulation” Humanoid Robots (IEEE, 2017), pp. 91–98. dictionary of prototypical grasp-predicting parts from
in Robotics: Science and Systems (RSS) Workshop on 57. A. Nguyen, D. Kanoulas, D. G. Caldwell, N. G. Tsagarakis, grasping experience” in Proceedings of the 2013 IEEE
Revisiting Contact - Turning a Problem into a Solution (2017); “Detecting object affordances with convolutional neural International Conference on Robotics and Automation
www.seas.upenn.edu/~zmen/MabelZhang_rss2017ws.pdf. networks” in Proceedings of the 2016 IEEE/RSJ International (IEEE, 2013), pp. 601–608.

Billard and Kragic, Science 364, eaat8414 (2019) 21 June 2019 7 of 8

R ES E A RC H | R E V IE W

78. A. Boularias, O. Kroemer, J. Peters, “Learning robot grasping Proceedings of the 21st IEEE International Symposium on transportation task with a human partner” in Proceedings
from 3-D images with markov random fields” in Proceedings Robot and Human Interactive Communication (IEEE, 2012), of the 21st IEEE International Symposium on Robot
of the 2011 IEEE/RSJ International Conference on Intelligent pp. 764–770. and Human Interactive Communication (IEEE, 2012),
Robots and Systems (IEEE, 2011), pp. 1548–1553. 90. C.-M. Huang, M. Cakmak, B. Mutlu, “Adaptive coordination pp. 962–967.
79. S. Kim, A. Shukla, A. Billard, Catching Objects in Flight. strategies for human-robot handovers” in Proceedings of the 100. K. Kosuge, M. Sato, N. Kazamura, “Mobile robot helper”
IEEE Trans. Robot. 30, 1049–1065 (2014). doi: 10.1109/ 2015 Conference on Robotics: Science and Systems (2015); in Proceedings of the 2000 IEEE International Conference
TRO.2014.2316022 www.roboticsproceedings.org/rss11/p31.html. on Robotics and Automation (IEEE, 2000), vol. 1,
80. D. Nguyen-Tuong, J. Peters, Model learning for robot control: 91. M. Cakmak, S. S. Srinivasa, M. K. Lee, S. Kiesler, J. Forlizzi, pp. 583–588.
A survey. Cogn. Process. 12, 319–340 (2011). doi: 10.1007/ “Using spatial and temporal contrast for fluent robot- 101. J. Stuckler, S. Behnke, “Following human guidance to
s10339-011-0404-1; pmid: 21487784 human handovers” in Proceedings of the International cooperatively carry a large object” in Proceedings of the 2011
81. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, Conference on Human-Robot Interaction (IEEE, 2011), p. 489. IEEE/RAS International Conference on Humanoid Robots
B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, 92. P. Basili, M. Huber, T. Brandt, S. Hirche, S. Glasauer, (IEEE, 2011), pp. 218–223.
A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, “Investigating human-human approach and hand-over” in 102. M. Lawitzky, A. Mortl, S. Hirche, “Load sharing in human-
W. Zaremba, Learning dexterous in-hand manipulation. Human Centered Robot Systems (Cognitive Systems robot cooperative manipulation” in Proceedings of the 19th
arxiv:1808.00177 [cs.LG] (1 August 2018). Monographs, Springer, 2009), vol. 6, pp. 151–160. IEEE International Symposium on Robot and Human
82. H. van Hoof, T. Hermans, G. Neumann, J. Peters, “Learning 93. A. Edsinger, C. C. Kemp, “Human-robot interaction for Interactive Communication (IEEE, 2010), pp. 185–191.
robot in-hand manipulation with tactile features” in cooperative manipulation: Handing objects to one another” in 103. Y. Karayiannidis, C. Smith, D. Kragic, “Mapping human
Proceedings of the 2015 IEEE/RAS International Conference Proceedings of the 16th IEEE International Symposium on intentions to robot motions via physical interaction through a
on Humanoid Robots (IEEE, 2015), pp 121–127. Robot and Human Interactive Communication (IEEE, 2007), jointly-held object” in Proceedings of the 23rd IEEE
83. V. Kumar, E. Todorov, S. Levine, “Optimal control with learned pp. 1167–1172. International Symposium on Robot and Human Interactive
local models: Application to dexterous manipulation” in 94. J. Aleotti, V. Micelli, S. Caselli, “Comfortable robot to human Communication (IEEE, 2014), pp. 391–397.
Proceedings of the 2016 IEEE International Conference on object hand-over” in Proceedings of the 21st IEEE 104. D. Kruse, R. J. Radke, J. T. Wen, “Collaborative human-robot
Robotics and Automation (IEEE, 2016), pp. 378–383. International Symposium on Robot and Human Interactive manipulation of highly deformable materials” in Proceedings
84. Y. Yang, Y. Li, C. Fermüller, Y. Aloimonos, “Robot learning Communication (IEEE, 2012), pp. 771–776. of the 2015 IEEE International Conference on Robotics and
manipulation action plans by “watching” unconstrained 95. M. Pan, E. Croft, G. Niemeyer, “Exploration of geometry and Automation (IEEE, 2015), pp. 3782–3787.
videos from the World Wide Web” in Proceedings of the forces occurring within human-to-robot handovers” in 2018 105. Y. Gao, H. J. Chang, Y. Demiris, “Iterative path optimization

Downloaded from https://www.science.org at Slub Dresden on May 22, 2023

29th Conference on Artificial Intelligence (AAAI Press, 2015), IEEE Haptics Symposium (IEEE, 2018), pp. 327–333. for personalized dressing assistance using vision and force
pp. 3686–3693. 96. K. Yokoyama, H. Handa, T. Isozumi, Y. Fukase, K. Kaneko, information” in Proceedings of the 2016 IEEE/RSJ
85. J. Krüger, T. K. Lien, A. Verl, Cooperation of human and F. Kanehiro, Y. Kawai, F. Tomita, H. Hirukawa, “Cooperative International Conference on Intelligent Robots and Systems
machines in assembly lines. CIRP Ann. Manuf. Technol. 58, works by a human and a humanoid robot” in Proceedings of (IEEE, 2016), pp. 4398–4403.
628–646 (2009). doi: 10.1016/j.cirp.2009.09.009 the 2003 IEEE International Conference on Robotics and 106. L. Lu, J. T. Wen, Human-directed coordinated control of an
86. A. Cherubini, R. Passama, A. Crosnier, A. Lasnier, P. Fraisse, Automation (IEEE, 2003), vol. 3, pp. 2985–2991. assistive mobile manipulator. Int. J. Intell. Robot. Appl. 1,
Collaborative manufacturing with physical human-robot 97. E. Berger, D. Vogt, N. Haji-Ghassemi, B. Jung, 104–120 (2017). doi: 10.1007/s41315-016-0005-3
interaction. Robot. Comput.-Integr. Manuf. 40, 1–13 (2016). H. B. Amor, “Inferring guidance information in cooperative
doi: 10.1016/j.rcim.2015.12.007 human-robot tasks” in Proceedings of the 2013 IEEE/RAS AC KNOWLED GME NTS
87. A. Ajoudani et al., Progress and prospects of the human- International Conference on Humanoid Robots (IEEE, 2013), We thank the reviewers for many helpful comments to improve the
robot collaboration. Auton. Robots 42, 957–975 (2018). pp. 124–129. article, A. Kheddar and J. Paik for providing images of their
doi: 10.1007/s10514-017-9677-2 98. M. Bellaccini, L. Lanari, A. Paolillo, M. Vendittelli, “Manual research, and L. Cohen for hand-drawn illustrations. Funding: We
88. M. Huber, A. Knoll, T. Brandt, S. Glasauer, Handing over guidance of humanoid robots without force sensors: acknowledge funding from the European Research Council, the
a cube: Spatial features of physical joint-action. Preliminary experiments with NAO” in Proceedings of the Knut and Alice Wallenberg Foundation, and the Swedish
Ann. N. Y. Acad. Sci. 1164, 380–382 (2009). doi: 10.1111/ 2014 IEEE International Conference on Robotics and Foundation for Strategic Research. Competing interests: The
j.1749-6632.2008.03743.x; pmid: 19645931 Automation (IEEE, 2014), pp. 1184–1189. authors declare no competing interests.
89. J. Mainprice, M. Gharbi, T. Simeon, R. Alami, “Sharing 99. A. Bussy, P. Gergondet, A. Kheddar, F. Keith, A. Crosnier,
effort in planning human-robot handover tasks” in “Proactive behavior of a humanoid robot in a haptic 10.1126/science.aat8414

Billard and Kragic, Science 364, eaat8414 (2019) 21 June 2019 8 of 8

 Trends and challenges in robot manipulation
Aude Billard and Danica Kragic

Science, 364 (6446), eaat8414.

DOI: 10.1126/science.aat8414

Hand it to you
Our ability to grab, hold, and manipulate objects involves our dexterous hands, our sense of touch, and feedback
from our eyes and muscles that allows us to maintain a controlled grip. Billard and Kragic review the progress made

Downloaded from https://www.science.org at Slub Dresden on May 22, 2023

in robotics to emulate these functions. Systems have developed from simple, pinching grippers operating in a fully
defined environment, to robots that can identify, select, and manipulate objects from a random collection. Further
developments are emerging from advances in computer vision, computer processing capabilities, and tactile materials
that give feedback to the robot.
Science, this issue p. eaat8414

View the article online

https://www.science.org/doi/10.1126/science.aat8414
Permissions
https://www.science.org/help/reprints-and-permissions

Use of this article is subject to the Terms of service

Science (ISSN 1095-9203) is published by the American Association for the Advancement of Science. 1200 New York Avenue NW,
Washington, DC 20005. The title Science is a registered trademark of AAAS.
Copyright © 2019 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim
to original U.S. Government Works