CSC2626

Imitation Learning for Robotics

Florian Shkurti
Week 1: Behavioral Cloning vs. Imitation
 New robotics faculty in CS

Jessica Burgner-Kahrs Animesh Garg Myself Igor Gilitschenski

Today’s agenda

• Administrivia
• Topics covered by the course
• Behavioral cloning
• Imitation learning
• Quiz about background and interests
• (Time permitting) Query the expert only when policy is uncertain
Administrivia
Administrivia
This is a graduate level course

Course website: http://www.cs.toronto.edu/~florian/courses/csc2626w21

Discussion forum + announcements: https://q.utoronto.ca (Quercus)

Request improvements anonymously: https://www.surveymonkey.com/r/LJJV5LY

Course-related emails should have CSC2626 in the subject

Prerequisites
Mandatory:
• Introductory machine learning (e.g. CSC411/ECE521 or equivalent)
• Basic linear algebra + multivariable calculus
• Intro to probability
• Programming skills in Python or C++ (enough to validate your ideas)

Recommended:
• Experience training neural networks or other function approximators
• Introductory concepts from reinforcement learning or control (e.g. value function/cost-to-go)
Prerequisites
Mandatory:
If you’re missing any of
• Introductory machine learning (e.g. CSC411/ECE521 or equivalent) these this is not the course
• Basic linear algebra + multivariable calculus for you.
• Intro to probability
You’re welcome to audit.
• Programming skills in Python or C++ (enough to validate your ideas)

Recommended: If you’re missing this we can

• Experience training neural networks or other function approximators organize tutorials to help you.
• Introductory concepts from reinforcement learning or control (e.g. value function/cost-to-go)
Grading
Two assignments: 50%

Course project: 50%

• Project proposal: 10%
• Midterm progress report: 5%
• Project presentation: 5%
• Final project report (6-8 pages) + code: 30%

Project guidelines
http://www.cs.toronto.edu/~florian/courses/csc2626w21/CSC2626_Project_Guidelines.pdf
Grading
Two assignments: 50% Individual submissions

Course project: 50%

• Project proposal: 10%
• Midterm progress report: 5%
• Project presentation: 5%
• Final project report (6-8 pages) + code: 30%

Project guidelines
http://www.cs.toronto.edu/~florian/courses/csc2626w21/CSC2626_Project_Guidelines.pdf
Grading
Two assignments: 50% Individual submissions

Course project: 50%

• Project proposal: 10%
• Midterm progress report: 5% Groups of 2-3

• Project presentation: 5%
• Final project report (6-8 pages) + code: 30%

Project guidelines
http://www.cs.toronto.edu/~florian/courses/csc2626w21/CSC2626_Project_Guidelines.pdf
 Guiding principles for this course

Robots do not operate in a vacuum. They do not need to learn everything from scratch.
 Guiding principles for this course

Robots do not operate in a vacuum. They do not need to learn everything from scratch.

Humans need to easily interact with robots and share our expertise with them.
 Guiding principles for this course

Robots do not operate in a vacuum. They do not need to learn everything from scratch.

Humans need to easily interact with robots and share our expertise with them.

Robots need to learn from the behavior and experience of others, not just their own.
 Main questions

How can robots incorporate others’

decisions into their own?

How can robots easily understand our

objectives from demonstrations?

How do we balance autonomous

control and human control in the
same system?
 Main questions
Learning from demonstrations
How can robots incorporate others’ Apprenticeship learning
decisions into their own? Imitation learning

Reward/cost learning
How can robots easily understand our Task specification
objectives from demonstrations? Inverse reinforcement learning
Inverse optimal control
Inverse optimization

How do we balance autonomous

control and human control in the Shared or sliding autonomy
same system?
 Applications

Any control problem where:

- writing down a dense cost function is difficult

- there is a hierarchy of decision-making processes

- our engineered solutions might not cover all cases

- unrestricted exploration during learning is slow or dangerous

https://www.youtube.com/watch?v=M8r0gmQXm1Y
 Applications

Any control problem where:

- writing down a dense cost function is difficult

- there is a hierarchy of interacting decision-making processes

- our engineered solutions might not cover all cases

- unrestricted exploration during learning is slow or dangerous

https://www.youtube.com/watch?v=Q3LXJGha7Ws
 Applications

Any control problem where:

- writing down a dense cost function is difficult

- there is a hierarchy of interacting decision-making processes

- our engineered solutions might not cover all cases

- unrestricted exploration during learning is slow or dangerous

https://www.youtube.com/watch?v=RjGe0GiiFzw
 Applications

Any control problem where:

- writing down a dense cost function is difficult

- there is a hierarchy of interacting decision-making processes

- our engineered solutions might not cover all cases

- unrestricted exploration during learning is slow or dangerous Robot explorer

 Applications

Any control problem where:

- writing down a dense cost function is difficult

- there is a hierarchy of interacting decision-making processes

- our engineered solutions might not cover all cases

- unrestricted exploration during learning is slow or dangerous

https://www.youtube.com/watch?v=0XdC1HUp-rU
Back to the future

https://www.youtube.com/watch?v=2KMAAmkz9go https://www.youtube.com/watch?v=ilP4aPDTBPE

Navlab 1 (1986-1989) Navlab 2 + ALVINN (Dean Pomerleau’s PhD thesis, 1989-1993)

30 x 32 pixels, 3 layer network, outputs steering command
~5 minutes of training per road type
ALVINN: architecture

https://drive.google.com/file/d/0Bz9namoRlUKMa0pJYzRGSFVwbm8/view
Dean Pomerleau’s PhD thesis
ALVINN: training set

Online updates via

backpropagation
 Problems Identified by Pomerlau

Test distribution is different Catastrophic forgetting

from training distribution
(covariate shift)
(Partially) Addressing Covariate Shift
(Partially) Addressing Catastrophic Forgetting

1. Maintains a buffer of old (image, action) pairs

2. Experiments with different techniques to ensure diversity and avoid outliers

Behavioral Cloning = Supervised Learning
25 years later

https://www.youtube.com/watch?v=qhUvQiKec2U
How much has changed?

offline

End to End Learning for Self-Driving Cars, Bojarski et al, 2016

 How much has changed?

“Our collected data is labeled with road type, weather condition, and the driver’s
activity (staying in a lane, switching lanes, turning, and so forth).”

End to End Learning for Self-Driving Cars, Bojarski et al, 2016

How much has changed?
How much has changed?

A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots, Giusti et al., 2016
https://www.youtube.com/watch?v=umRdt3zGgpU
How much has changed?

Not a lot for learning lane following with neural networks.

But, there are a few other beautiful ideas that do not involve end-to-end learning.
 Visual Teach & Repeat
Human Operator or
Planning Algorithm

Visual Path Following on a Manifold in Unstructured Three-Dimensional Terrain, Furgale & Barfoot, 2010
 Visual Teach & Repeat
Key Idea #1: Manifold Map

Build local maps relative to the

path. No global coordinate frame.

Visual Path Following on a Manifold in Unstructured Three-Dimensional Terrain, Furgale & Barfoot, 2010
 Visual Teach & Repeat
Key Idea #1: Manifold Map Key Idea #2: Visual Odometry

Build local maps relative to the Given two consecutive images,

path. No global coordinate frame. how much has the camera
moved? Relative motion.

Visual Path Following on a Manifold in Unstructured Three-Dimensional Terrain, Furgale & Barfoot, 2010
Visual Teach & Repeat

https://www.youtube.com/watch?v=_ZdBfU4xJnQ https://www.youtube.com/watch?v=9dN0wwXDuqo

Centimeter-level precision in tracking the demonstrated path over kilometers-long trails.

Today’s agenda

• Administrivia
• Topics covered by the course
• Behavioral cloning
• Imitation learning
• Quiz about background and interests
• (Time permitting) Query the expert only when policy is uncertain
 Back to Pomerleau

(Ross & Bagnell, 2010): How are we sure these errors are not due to
overfitting or underfitting?

1. Maybe the network was too small (underfitting)

2. Maybe the dataset was too small and the network overfit it

Steering commands
where s are image features

Test distribution is different

from training distribution
(covariate shift)
 Back to Pomerleau

(Ross & Bagnell, 2010): How are we sure these errors are not due to
overfitting or underfitting?

1. Maybe the network was too small (underfitting)

2. Maybe the dataset was too small and the network overfit it

Steering commands
where s are image features

Test distribution is different

It was not 1: they showed that even a linear policy can work well.
from training distribution
It was not 2: their error on held-out data was close to training error.
(covariate shift)
 Imitation learning Supervised learning

(Ross & Bagnell, 2010): IL is a sequential decision-making problem.

• Your actions affect future observations/data.

• This is not the case in supervised learning

Supervised Learning

Assumes train/test data are i.i.d.

If expected training error is

Expected test error after T decisions

Test distribution is different

from training distribution
(covariate shift) Errors are independent
 Imitation learning Supervised learning

(Ross & Bagnell, 2010): IL is a sequential decision-making problem.

• Your actions affect future observations/data.

• This is not the case in supervised learning

Imitation Learning Supervised Learning

Train/test data are not i.i.d. Assumes train/test data are i.i.d.

If expected training error is If expected training error is

Expected test error after T decisions Expected test error after T decisions
is up to
Test distribution is different
from training distribution
(covariate shift) Errors compound Errors are independent
DAgger
(Ross & Gordon & Bagnell, 2011): DAgger, or Dataset Aggregation

• Imitation learning as interactive supervision

• Aggregate training data from expert with test data from execution
DAgger
(Ross & Gordon & Bagnell, 2011): DAgger, or Dataset Aggregation

• Imitation learning as interactive supervision

• Aggregate training data from expert with test data from execution

Imitation Learning via DAgger Supervised Learning

Train/test data are not i.i.d. Assumes train/test data are i.i.d.

If expected training error on aggr. dataset is If expected training error is

Expected test error after T decisions is Expected test error after T decisions

Errors do not compound Errors are independent

DAgger

Initial expert trajectories Supervised learning DAgger

https://www.youtube.com/watch?v=V00npNnWzSU
DAgger
DAgger

Q: Any drawbacks of using it in a robotics setting?

DAgger

https://www.youtube.com/watch?v=hNsP6-K3Hn4

Learning Monocular Reactive UAV Control in Cluttered Natural Environments, Ross et al, 2013
Today’s agenda

• Administrivia
• Topics covered by the course
• Behavioral cloning
• Imitation learning
• Quiz about background and interests
• (Time permitting) Query the expert only when policy is uncertain
DAgger: Assumptions for theoretical guarantees
(Ross & Gordon & Bagnell, 2011): DAgger, or Dataset Aggregation

• Imitation learning as interactive supervision

• Aggregate training data from expert with test data from execution

Imitation Learning via DAgger Supervised Learning

Train/test data are not i.i.d. Assumes train/test data are i.i.d.

If expected training error on aggr. dataset is If expected training error is

Expected test error after T decisions is Expected test error after T decisions

Strongly convex loss

No-regret online learner
Errors do not compound Errors are independent
Appendix: No-Regret Online Learners
Intuition: No matter what the distribution of input data, your online policy/classifier will do
asymptotically as well as the best-in-hindsight policy/classifier.

Policy has access to Policy has access to

data up to round i data up to round N

No-regret:
 Appendix: Types of Uncertainty &
Query-Efficient Imitation

Let’s revisit the two main ideas from query-efficient imitation:

1. DropoutDAgger:
Keep an ensemble of learner policies, and only query the expert when they significantly disagree

2. SHIV, SafeDagger, MMD-IL:

(Roughly) Query expert only if input is too close to the decision boundary of the learner’s policy

Need to review a few concepts about different types of uncertainty.

Biased Coin

observations
Biased Coin

how biased is the coin?

Biased Coin

how biased is the coin?

Induces uncertainty in the model, or epistemic uncertainty,

which asymptotically goes to 0 with infinite observations
 Biased Coin

Q: Even if you eventually discover the true model, can you predict if the next flip will be heads?
 Biased Coin

Q: Even if you eventually discover the true model, can you predict if the next flip will be heads?

A: No, there is irreducible uncertainty / observation noise in the system. This is called aleatoric uncertainty.
Gaussian Process Regression

http://pyro.ai/examples/gp.html
Gaussian Process Regression

http://pyro.ai/examples/gp.html

Zero mean prior over functions

Noisy observations
Gaussian Process Regression

No matter how much data we get, this

observation noise will not go to zero

http://pyro.ai/examples/gp.html

Zero mean prior over functions

Noisy observations
Gaussian Process Regression

If we get data here we can reduce

model / epistemic uncertainty

http://pyro.ai/examples/gp.html

Zero mean prior over functions

Noisy observations
Gaussian Process Classification

Gaussian Processes for Machine Learning, chapter 2

Gaussian Process Classification vs SVM

Gaussian Processes for Machine Learning, chapter 2

GP handles uncertainty in f by averaging

while SVM considers only best f for classification.
Model Uncertainty in Neural Networks
Want

But easier to control network weights

Model Uncertainty in Neural Networks
Want

But easier to control network weights

How do we represent posterior over network weights?

How do we quickly sample from it?
Model Uncertainty in Neural Networks
Want

But easier to control network weights

How do we represent posterior over network weights?

How do we quickly sample from it?
Main ideas:

1. Use an ensemble of networks trained on different

copies of D (bootstrap method)

2. Use an approximate distribution over weights

(Dropout, Bayes by Backprop, …)

3. Use MCMC to sample weights

Model Uncertainty in Neural Networks
Want

But easier to control network weights

How do we represent posterior over network weights?

How do we quickly sample from it?
Main ideas:

1. Use an ensemble of networks trained on different

copies of D (bootstrap method)

2. Use an approximate distribution over weights

(Dropout, Bayes by Backprop, …)

3. Use MCMC to sample weights

Model Uncertainty in Neural Networks
Want

But easier to control network weights

How do we represent posterior over network weights?

How do we quickly sample from it?
Main ideas:

1. Use an ensemble of networks trained on different

copies of D (bootstrap method)
Variational inference
2. Use an approximate distribution over weights
(Dropout, Bayes by Backprop, …)

3. Use MCMC to sample weights

Model Uncertainty in Neural Networks
Want

But easier to control network weights

How do we represent posterior over network weights?

How do we quickly sample from it?
Main ideas:

1. Use an ensemble of networks trained on different

copies of D (bootstrap method)
Variational inference
2. Use an approximate distribution over weights
(Dropout, Bayes by Backprop, …)

3. Use MCMC to sample weights