Perception and Perspective in Robotics
Paul Fitzpatrick MIT Computer Science and Artificial Intelligence Laboratory Humanoid Robotics Group Overview
Goal To build robots that can interact with novel objects and participate in novel activities Challenge Machine perception can be robust for a specific domain such as face detection, but unlike human perception it is not currently adaptable in the face of change (new objects, changed circumstances) Approach Integrate conventional machine perception and machine learning with strategies for opportunistic development Active perception (sensorimotor toil) Interpersonal influences (theft) This work is implemented on a humanoid robot (Cog, see right). The robot uses the structure of familiar activities to learn about novel elements within those activities, and tracks known elements to learn about the unfamiliar activities in which they are used. Perspective
familiar activities (tasks, games, )
segmentation
Toil Example Active Segmentation
Theft Example Search Activity
Robot observes a human searching for objects, and learns to make a connection between the named target of the search and the object successfully found. The robot has no predefined vocabulary or object set. Human says Find says Find Toma Toma (shows cube) (sees cube) No No (shows car) (sees car) No No (shows bottle) (sees bottle) Yes! Yes (shows cube) (sees cube) Say Say Cube (shows bottle) (sees bottle) Say Say Toma Robot
1 2 3
Object boundaries are not always easy to detect visually, so robot Cog sweeps its arm through ambiguous areas This can cause object motion, which makes boundaries much easier to find Then robot can learn to recognize and segment object without further contact
This is a good basis for adaptable object perception:
active probing affordance exploitation
(rolling)
use constraint of familiar activity to discover unfamiliar entity used within it
reveal the structure of unfamiliar activities by tracking familiar entities into and through them
edge catalog object detection, recognition
manipulator detection
(robot, human)
Perception
familiar entities (objects, actors, properties, )
This work is funded by DARPA under contract number DABT 63-00-C-10102, and by the Nippon Telegraph and Telephone Corporation under the NTT/MIT collaboration agreement
Goal To learn how human-level perception is possible, by trying to build it Challenge Machine perception can be robust for a specific domain, but is not adaptable like human perception Approach Integrate conventional machine perception and machine learning with strategies for opportunistic development Active perception (sensorimotor toil) Interpersonal influences (theft) Development If a robot is engaged in a known activity there may be sufficient constraint to identify novel elements within that activity. Similarly, if known elements take part in some unfamiliar activity, tracking those can help characterize that activity. Potentially, perceptual development is an open-ended loop of such discoveries.
Learning a sorting activity Human shows robot where a collection of disparate objects should go, based on some common criterion (color). Robot demonstrates understanding through verbal descriptions, nods towards target locations.
Kismet What is done on Kismet
Novel Perspective leads to Novel Perception
Learning a search activity Human shows robot examples of search activity by speaking. Robot demonstrates understanding by linking name and object. Learning through a search activity Blah blah
Cog What is done on Cog
Perception and Perspective in Robotics
Paul Fitzpatrick MIT Computer Science and Artificial Intelligence Laboratory Humanoid Robotics Group
An Example Active Segmentation Overview Goal To learn how human-level perception is possible, by trying to build it Challenge Machine perception can be robust for a specific domain, but is not adaptable like human perception Approach Integrate conventional machine perception and machine learning with strategies for opportunistic development Active perception (sensorimotor toil) Interpersonal influences (theft) Experimental Platform Expressive active vision head Kismet and upper-torso humanoid robot Cog Open-ended Development If the robot is engaged in a known activity there may be sufficient constraint to identify novel elements within that activity. Similarly, if known elements take part in some unfamiliar activity, tracking those can help characterize that activity. Potentially, perceptual development is an open-ended loop of such discoveries.
1 2 3
Object boundaries are not always easy to detect visually, so robot Cog sweeps its arm through ambiguous areas This can cause object motion, which makes boundaries much easier to find Then robot can learn to recognize and segment object without further contact
familiar activities
use constraint of familiar activity to discover unfamiliar entity used within it
reveal the structure of unfamiliar activities by tracking familiar entities into and through them
familiar entities (objects, actors, properties, )
Kismet What is done on Kismet
Gives opportunity for much development
active probing segmentation affordance exploitation
(rolling)
Sorting activity Human shows robot where a collection of disparate objects should go, based on some common criterion (color). Robot demonstrates understanding through verbal descriptions, nods towards target locations. Search activity Human shows robot examples of search activity by speaking. Robot demonstrates understanding through verbal descriptions, nods towards target locations.
Cog What is done on Cog
edge catalog object detection, recognition
manipulator detection
(robot, human)
Perception and Perspective in Robotics
Paul Fitzpatrick MIT Computer Science and Artificial Intelligence Laboratory Humanoid Robotics Group
familiar activities
Active Perception To foo foo foo Active Segmentation Solve classic problem
use constraint of familiar activity to discover unfamiliar entity used within it
reveal the structure of Objectunfamiliar activities by always boundaries are not tracking familiar entities easy to detect visually (e.g. yellow car oninto and through them yellow table)
Solution: robot Cog sweeps through ambiguous area familiar entities (objects, actors, properties, ) helps Resulting object motion segmentation Robot can learn to recognize and segment object without further Opportunities abound and contact cascade EgoMap short term memory Robot can perform find the toma of objects and their locations so out of sight is not out of mind style tasks Observes search activity Then uses structure of search activity to learn new properties (object names) Searching and sorting
Active Perception Point 1, 2, 3 Motivation Training examples are currently a necessary condition for achieving robust machine perception. Acquiring those examples is properly the role of perception itself. But a human is typically needed to collect those examples. Active Perception To foo foo foo Active Segmentation Solve classic problem
Sorting task Human shows robot where a collection of disparate objects should go, based on some common criterion (color). Robot demonstrates understanding through verbal descriptions, nods towards target locations. Search task Human shows robot examples of search activity by speaking Robot demonstrates understanding through verbal descriptions, nods towards target locations.
Perception and Perspective in Robotics
Paul Fitzpatrick MIT Computer Science and Artificial Intelligence Laboratory Humanoid Robotics Group
Goal To understand perception by trying to build it Approach Extend machine opportuistic deve The grist: The mill: Examples
perception
to
include
Active perception Interpersonal influences Opportunistic development
Object boundaries are not always easy to detect visually (e.g. yellow car on yellow table) Solution: robot Cog sweeps through ambiguous area Resulting object motion helps segmentation Robot can learn to recognize and segment object without further contact
Opportunities abound and cascade Robot can perform find the toma style tasks Observes search activity Then uses structure of search activity to learn new properties (object names) Searching and sorting
familiar activities
use constraint of familiar activity to discover unfamiliar entity used within it
reveal the structure of unfamiliar activities by tracking familiar entities into and through them
familiar entities (objects, actors, properties, )
EgoMap short term memory of objects and their locations so out of sight is not out of mind
Opportunism Standard approach to machine perception is to develop algorithms which, when provided with sufficient training data, can learn to perform some classification or regression task. Can move one step back and develop algorithms which, given physical opportunities, acquire the training data. Need to design system behavior side-by-side with the perceptual code. Opportunistic Development Suppose there is a property P which can normally not be perceived. But there exists a situation S where it can be. Then the robot can try to get into situation S, and observe P, and relate it to other perceptual variables that are observable
Perception and Perspective in Robotics
Paul Fitzpatrick MIT Computer Science and Artificial Intelligence Laboratory Humanoid Robotics Group
Head (7 DOFs)
Training Data
Left arm (6 DOFs)
Task Learning Mechanis
Right arm (6 DOFs)
Instructor Task Modeling
Sequencing Mod
1 2 3
Eyes (3 DOFs)
Torso (3 DOFs)
Stand (0 DOFs)
Task Grounding
State Grounding
Facial (15 DOFs)
poking object segmentation
Neck (3 DOFs)
Perceptual System Demonstrated Task
affordance exploitation
(rolling)
Perceptual Netw
edge catalog
object detection
familiar activities
manipulator detection
(robot, human) (recognition, localization, contact-free segmentation)
use constraint of familiar activity to discover unfamiliar entity used within it
reveal the structure of unfamiliar activities by tracking familiar entities into and through them
familiar entities (objects, actors, properties, )
EgoMap short term memory of objects and their locations so out of sight is not out of mind
(Speech)
Training Data
Instructor
Task Learning Mechanism
Sequencing Model Task Modeling
Understanding perception by trying to build it Machine perception is very fallible. Robots (and humans) need not just particular perceptual competences, but the tools to forge those competences out of raw physical experiences. Three important tools for extending a robots perceptual abilities whose importance have been recognized individually are related and brought together. The first is active perception, where the robot employs motor action to reliably perceive properties of the world that it otherwise could not. The second is development, where experience is used to improve perception. The third is interpersonal influences, where the robots percepts are guided by those of an external agent. Examples are given for object segmentation, object recognition, and orientation sensitivity; initial work on action understanding is also described.
Demonstrated Task Task Grounding State Grounding
Perceptual System
Perceptual Network
Object boundaries are not always easy to detect visually Solution: Cog sweeps through ambiguous area Resulting object motion helps segmentation Robot can learn to recognize and segment object without further contact
camera image
implicated edges found and grouped
response for each object
