Deep Reinforcement Learning

CS 294 - 112
Course logistics
Class Information & Resources
Sergey Levine Abhishek Gupta Josh Achiam
Assistant Professor PhD Student PhD Student
UC Berkeley UC Berkeley UC Berkeley

• Course website: rll.berkeley.edu/deeprlcourse/

• Piazza: UC Berkeley, CS294-112
• Subreddit (for non-enrolled students):
www.reddit.com/r/berkeleydeeprlcourse/
• Office hours: after class each day (but not today), sign up in advance
for a 10-minute slot on the course website
Prerequisites & Enrollment
• All enrolled students must have taken CS189, CS289, or CS281A
• Please contact Sergey Levine if you haven’t
• Please enroll for 3 units
• Students on the wait list will be notified as slots open up
• Lectures will be recorded
• Since the class is full, please watch the lectures online if you are not enrolled
What you should know
• Assignments will require training neural networks with standard
automatic differentiation packages (TensorFlow by default)
• Review Section
• Josh Achiam will teach a review section in week 3
• You should be able to at least do the TensorFlow MNIST tutorial (if not, come
to the review section and ask questions!)
What we’ll cover
• Full syllabus on course website
1. From supervised learning to decision making
2. Basic reinforcement learning: Q-learning and policy gradients
3. Advanced model learning and prediction, distillation, reward
learning
4. Advanced deep RL: trust region policy gradients, actor-critic
methods, exploration
5. Open problems, research talks, invited lectures
Assignments
1. Homework 1: Imitation learning (control via supervised learning)
2. Homework 2: Policy gradients (“REINFORCE”)
3. Homework 3: Q learning with convolutional neural networks
4. Homework 4: Model-based reinforcement learning
5. Final project: Research-level project of your choice (form a group of
up to 2-3 students, you’re welcome to start early!)

Grading: 40% homework (10% each), 60% project

Your “Homework” Today
1. Sign up for Piazza (see course website)
2. Start forming your final project groups, unless you want to work
alone, which is fine
3. Fill out the enrolled student survey if you haven’t already!
4. Check out the TensorFlow MNIST tutorial, unless you’re a
TensorFlow pro
What is reinforcement learning, and why
should we care?
What is reinforcement learning?
decisions (actions)

consequences
observations
rewards
Examples

Actions: muscle contractions Actions: motor current or torque Actions: what to purchase
Observations: sight, smell Observations: camera images Observations: inventory levels
Rewards: food Rewards: task success measure Rewards: profit
(e.g., running speed)
What is deep RL, and why should we care?
Deep learning: end-to-end training of
expressive, multi-layer models

Deep models are what allow reinforcement

learning algorithms to solve complex problems
end to end!
What does end-to-end learning mean for
sequential decision making?
 perception

Action
(run away)

action
 sensorimotor loop

Action
(run away)
Example: robotics

robotic state
modeling & low-level
control observations estimation
prediction
planning
control
controls
pipeline (e.g. vision)
Example: playing video games

video extract state

low-level
game game API relevant machine for planner
bot control
controls
AI pipeline features behavior
standard
features mid-level features classifier
computer
(e.g. HOG) (e.g. DPM) (e.g. SVM)
vision
Felzenszwalb ‘08

end-to-end training
deep
learning

robotic state
modeling & low-level
control observations estimation
prediction
planning
control
controls
pipeline (e.g. vision)

end-to-end training
deep state
modeling & low-level
robotic observations estimation
prediction
planning
control
controls
learning (e.g. vision)
tiny, highly specialized tiny, highly specialized
“visual cortex” “motor cortex”

no direct supervision
actions have consequences
The reinforcement learning problem
decisions (actions) Actions: motor current or torque
Observations: camera images
Rewards: task success measure (e.g., running speed)

Deep models are what allow reinforcement

Actions: what to purchase
learning algorithms to solve complex problems
Observations: inventory levels
Rewards: profit
end to end! Actions: words in French
Observations: words in English
Rewards: BLEU score

consequences
observations The reinforcement learning problem is the AI problem!
rewards
When do we not need to worry about
sequential decision making?
When your system is making single isolated decision, e.g. classification, regression
When that decision does not affect future decisions
When should we worry about sequential
decision making?
Limited supervision: you know what you want, but not how to get it
Actions have consequences

Common Applications
autonomous driving business operations

language & dialogue

robotics (structured prediction) finance
Why should we study this now?

1. Advances in deep learning

2. Advances in reinforcement learning
3. Advances in computational capability
Why should we study this now?

Tesauro, 1995

L.-J. Lin, “Reinforcement learning for robots using neural networks.” 1993
Why should we study this now?

Atari games: Real-world robots: Beating Go champions:

Q-learning: Guided policy search: Supervised learning + policy
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. S. Levine*, C. Finn*, T. Darrell, P. Abbeel. “End-to-end gradients + value functions +
Antonoglou, et al. “Playing Atari with Deep training of deep visuomotor policies”. (2015).
Reinforcement Learning”. (2013).
Monte Carlo tree search:
Q-learning: D. Silver, A. Huang, C. J. Maddison, A. Guez,
Policy gradients: S. Gu*, E. Holly*, T. Lillicrap, S. Levine. “Deep L. Sifre, et al. “Mastering the game of Go
J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Reinforcement Learning for Robotic Manipulation with deep neural networks and tree
Abbeel. “Trust Region Policy Optimization”. (2015). with Asynchronous Off-Policy Updates”. (2016). search”. Nature (2016).
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap,
et al. “Asynchronous methods for deep reinforcement
learning”. (2016).
What other problems do we need to solve to
enable real-world sequential decision making?
Beyond learning from reward

• Basic reinforcement learning deals with maximizing rewards

• This is not the only problem that matters for sequential decision
making!
• We will cover more advanced topics
• Learning reward functions from example (inverse reinforcement learning)
• Transferring skills between domains
• Learning to predict and using prediction to act
Where do rewards come from?
Are there other forms of supervision?

• Learning from demonstrations

• Directly copying observed behavior
• Inferring rewards from observed behavior (inverse reinforcement learning)
• Learning from observing the world
• Learning to predict
• Unsupervised learning
• Learning from other tasks
• Transfer learning
• Meta-learning: learning to learn
 Imitation learning

Bojarski et al. 2016

More than imitation: inferring intentions

Warneken & Tomasello

 Inverse RL examples

Finn et al. 2016

Prediction
 What can we do with a perfect model?

Mordatch et al. 2015

 Prediction for real-world control

original
video

predictions
Finn et al. 2017
How do we build intelligent machines?
How do we build intelligent machines?
• Imagine you have to build an intelligent machine, where do you start?
Learning as the basis of intelligence
• Some things we can all do (e.g. walking)
• Some things we can only learn (e.g. driving a car)
• We can learn a huge variety of things, including very difficult things
• Therefore our learning mechanism(s) are likely powerful enough to do
everything we associate with intelligence
• But it may still be very convenient to “hard-code” a few really important bits
 A single algorithm?
• An algorithm for each “module”?
• Or a single flexible algorithm?

Seeing with your tongue

Auditory
Cortex
Human echolocation (sonar)

[BrainPort; Martinez et al; Roe et al.]

adapted from A. Ng
What must that single algorithm do?
• Interpret rich sensory inputs

• Choose complex actions

Why deep reinforcement learning?
• Deep = can process complex sensory input
▪ …and also compute really complex functions
• Reinforcement learning = can choose complex actions
Some evidence in favor of deep learning
Some evidence for reinforcement learning
• Percepts that anticipate reward
become associated with similar
firing patterns as the reward
itself
• Basal ganglia appears to be
related to reward system
• Model-free RL-like adaptation is
often a good fit for experimental
data of animal adaptation
• But not always…
What can deep learning & RL do well now?
• Acquire high degree of proficiency in
domains governed by simple, known
rules
• Learn simple skills with raw sensory
inputs, given enough experience
• Learn from imitating enough human-
provided expert behavior
What has proven challenging so far?
• Humans can learn incredibly quickly
• Deep RL methods are usually slow
• Humans can reuse past knowledge
• Transfer learning in deep RL is an open problem
• Not clear what the reward function should be
• Not clear what the role of prediction should be
Instead of trying to produce a
program to simulate the adult
mind, why not rather try to
produce one which simulates the
child's? If this were then subjected general learning
to an appropriate course of algorithm

observations
education one would obtain the

actions
adult brain.
- Alan Turing
environment