Some slides are from: Katerina Fragkiadaki (CMU), Davild Silver

(DeepMind), Hado van Hasselt (DeepMind)

COMP 4901Z: Reinforcement Learning

2.3 Value Function Approximation

Long Chen (Dept. of CSE)

 Two Types of Importance Sampling

• Ordinary Importance Sampling

∑!∈#(%) %!'(!)() &!

! " =
|((")|
• Weighted Importance Sampling

∑!∈#(%) %!'(!)() &!

! " =
∑!∈#(%) %!'(!)()

• Weighted IS is a biased estimation

• For first-visit method with single return, the expectation is !* " rather than !+ " .
• Ordinary IS is an unbiased estimation
• For first-visit method, its estimator is always !+ "
2
 Two Types of Importance Sampling

• Ordinary Importance Sampling

∑!∈#(%) %!'(!)() &!

! " =
|((")|
• Weighted Importance Sampling

∑!∈#(%) %!'(!)() &!

! " =
∑!∈#(%) %!'(!)()

• The variance of the ordinary IS is in general unbounded, whereas in the weighted

estimator the largest weight on any single return is one.
• Suppose the ratio wen ten, the ordinary importance-sampling estimate would be
ten times the observed return.
3
SARSA Algorithm for On-Policy Control

4
 Q-Learning Algorithm for Off-Policy Control

• Q-Learning: ! "! , $! ← ! "! , $! + ' (!"# + ) max ! "!"#, - − !("! , $! )

$

• SARSA: ! ", $ ← ! ", $ + ' ( + )! " % , $% − ! ", $

5
Double Tabular Q-Learning

6
2.3 Value Function
Approximation
 Function Approximation and Deep RL

• The policy, value function, model, and agent state update are all functions
• We want to learn these from experience
• If there are too many states, we need to approximate
• This is often called deep reinforcement learning
• when using neural network to represent these functions

8
 Large-Scale Reinforcement Learning

• In problems with large number of states, e.g.

• Backgammon: 10&' states
• Go: 10#(' states
• Helicopter: continuous state space
• Robots: real world
• Tabular methods that enumerate every single state do not work
• How can we scale up the model-free methods for prediction and control
from the last two lectures?

9
 Value Function Approximation (VFA)

• So far we have represented value function by a lookup table

• Every state " has an entry ! " or
• Every state-action pair (", ,) has an entry - ", ,
• Problem with large MDPs:
• There are too many states and/or actions to store in memory
• It is too slow to learn the value of each state individually
• Solution for large MDPs
• Estimate value function with function approximation
! "; / ≈ !+ " or - ", ,; / ≈ -+ ", ,
• Generalize from seen states to unseen states
• Update parameters / using MC or TD learning
10
 Agent State Update

• When the environment state is not fully observable ("!)*+ ≠ 5! )

• Use the agent state
"! = 7("!,#, $!,#, , 5! ; 9)
with parameters 9
• Henceforth, "! denotes the agent state
• Think of this as either a vector inside the agent,
or, in the simplest case, just the current observation: "! = 5!

11
 Value Function Approximation (VFA)

• Value function approximation (VFA) replaces the table with a general

parameterized form:

• When we update the parameters 9, the values of many states change

simultaneously!
12
 Policy Approximation

• Policy approximation replaces the table with a general parameterized form

13
 Classes of Functions Approximation

• Tabular: a table with an entry for each MDP state

• Linear function approximation
• Consider fixed agent state update (e.g., "! = 5! )
• Fixed feature map: :: " → (*
• Values are linear function of features = >; 9 = 9/ :(>)
• Differentiable function approximation
• = >; 9 is a differentiable function of 9, could be non-linear
• E.g., a convolutional neural network that takes pixel as input
• Another interpretation: features are not fixed, but learnt

14
 Which Function Approximation?

• There are many function approximators, e.g.

• Linear combinations of features
• Neural networks
• Decision tree
• Nearest neighbour
• Fourier/wavelet bases
•…

15
 Classes of Function Approximation

• In principle, any function approximator can be used, but RL has specific

properties,
• Experience is not i.i.d – successive time steps are correlated
• Agent’s policy affects the data it receives
• Regression targets can be non-stationary
• … because of changing policies (which can change the target and the data!)
• … because of bootstrapping
• … because of non-stationary dynamics (e.g., other learning agents)
• … because the world is large (never quite in the same state)

16
 Classes of Function Approximation

• Which function approximation should you choose?

• This depends on your goals:
• Tabular: good theory but does not scale/generalize
• Linear: reasonably good theory, but requires good features
• Non-linear: less well-understood, but scales well
• Flexible, and less reliant on picking good features first (e.g., by hand)
• (Deep) neural nets often perform quite well, and remain a popular
choice

17
 Function Approximator Examples

• Image representation for classification

18
 Function Approximator Examples

• Pixel space

19
 Function Approximator Examples

• Convolutional neural network (CNN) architectures

20
 Function Approximator Examples

• Recurrent neural network (RNN) architectures

21
 Function Approximator Examples

• Recurrent neural network (RNN) architectures

22
Gradient-based Algorithms
 Gradient Descent

• Let ! " be a differentiable function of parameter vector "

• Define the gradient of ! " to be:

%! "
%&"
∇!! " = ⋮
%! "
%&#
• To find a local minimum of ! " , adjust " in direction of the
negative gradient
1
∆" = − ,∇!! "
2
where , is a step-size parameter

24
 Gradient Descent

• Let ! " be a differentiable function of parameter vector "

• Define the gradient of ! " to be:

%! "
%&"
∇!! " = ⋮
%! "
%&#
• Starting from a guess "$
• We consider the sequence "$ , "" , "% , …
"
• s.t. ∆"&'( = − ,∇!! "&
%

• We then have J "$ ≥ J "" ≥ ! "% ≥ ⋯

25
 Value Function Approx. By Stochastic Gradient Descent

• Goal: find parameter vector " minimizing mean-squared error between the true value
function 0) 1 and its approximation 0(1; ")
! & = 5* 0) 1 − 0(1; ") %
Where 6 is a distribution over states (typically induced by the policy and dynamics)
• Gradient descent finds a local minimum
1
∆" = − ,∇!! " = ,5* 0) 1 − 0 1; " ∇!0 1; "
2
• Stochastic gradient descent (SGD), samples the gradient
∆" = , 7+ − 0 1+ ; " ∇!0 1+ ; "

• Note: Monte Carlo return 7+ is a sample for 0) 1+

• Expected update is equal to full gradient update

• We often write ∇0 1+ as short hand for ∇!0 1+ ; " |!,!!

26
 Feature Vectors

• Represent state by a feature vector

?# "
: " = ⋮
?* "
• :: " → (* is a fixed mapping from state (e.g., observation) to features
• Short hand: : 0 = :("! )
• For example:
• Distance of robot from landmarks
• Trends in the stock market
• Piece and pawn configurations in chess
27
 Linear Value Function Approximation

• Represent value function by a linear combination of features

%

! ", $ = & " ! $ = ' &" " $"

"#$

• Objective function is quadratic in parameters 9

A 9 = B1~3 =4 " − : " / 9 &

• Stochastic gradient descent converges on global optimum

• Update rule is particularly simple
∇& ! "' , $ = & "' = & '
∆$ = * !( "' − ! "' ; $ & '

• Update = step-size × prediction error × feature value

28
 Incremental Prediction Algorithm

• Have assumed true value function !+ " given by supervisor

• But in RL there is no supervisor, only rewards
• In practice, we substitute a target for !+ "
• For MC, the target is the return &!
∆/ = 3 &! − ! 5! ; / ∇, ! 5! ; /
• For TD(0), the target is the TD target
∆/ = 3 8!-) + :! 5!-) ; / − ! 5! ; / ∇, ! 5! ; /
• For TD(;), the target is the ;-return &!.
∆/ = 3 &!. − ! 5! ; / ∇, ! 5! ; /

&!. = 8!-) + : .
1 − ; ! 5!-) + ;&!-)
29
 Monte Carlo with Value Function Approximation

• The return D! is an unbiased, noisy sample of true value =4 "!

• Can therefore apply supervised learning to “training data”:
< "#, D# >, < "&, D& >, … , < "/ , D/ >
• For example, using linear Monte-Carlo policy evaluation
∆9 = ' D! − = "! , ; 9 ∇5= "! ; 9
= ' D! − = "! , ; 9 : !
• Linear Monte-Carlo evaluation converges to a local optimum
• Even when using non-linear value function approximation it converges
(but perhaps to a local optimum)

30
Monte Carlo with Value Function Approximation

31
 TD Learning with Value Function Approximation

• The TD-target (!"# + )= "!"#; 9 is a biased sample of true value =4 ("! )

• Can still apply supervised learning to “training data”
< "#, (# + )=("#; 9) >, < "&, (& + )=("&; 9) >, … , < "/ , (/ + )=("/ ; 9) >
• For example, using linear TD(0)
∆9 = ' (!"# + )= "!"#; 9 − = "! , ; 9 ∇5= "! ; 9
= 'J! : !
where J! = (!"# + )= "!"#; 9 − = "! , ; 9 is “TD error”
This is akin to non-stationary regression problem
• But it’s a bit different: the target depends on our parameters!
We ignore the dependence of the target on 9! We call it semi-gradient method!
32
TD Learning with Value Function Approximation

33
 Control with Value Function Approximation

• Policy evaluation: Approximate policy evaluation, !("! , $! ; 9) ≈ !4

• Policy improvement: L-greedy policy improvement

34
 Action-Value Function Approximation

• Should we use action-in, or action-out?

• Action in: ! >, -; 9 = 9/ ?(>, -)
• Action out: ! >; 9 = M?(>) such that ! >, -; 9 = ! >; 9 [-]
• One reuses the same weights, the other the same features
• Unclear which is better in general
• If we want to use continuous actions, action-in is easier (later lecture)
• For (small) discrete action spaces, action-out is common (e.g., DQN)

35
Convergence and Divergence
 Convergence Questions

• When do incremental prediction algorithms converge?

• When using bootstrapping (i.e., TD)?
• When using (e.g., linear) value function approximation?
• When using off-policy learning?
• Ideally, we would like algorithms that converge in all cases
• Alternatively, we want to understand when algorithms do, or do not,
converge

37
 Example of Divergence

• What if we use TD only on this transition?

38
 Example of Divergence

-')$ = -' + *' / + 0! 1 * − ! 1 ∇! 1

= -' + *' / + 0! 1 * − ! 1 2 1
= -' + *' 0 + 02-' − -'
= -' + *' (20 − 1) -'
#
• Consider P! > 0. If ) > &, then P!"# > P! .

=> lim!→7 P! = ∞
39
 Example of Divergence

• Algorithms that combine

• Bootstrapping
• Off-policy learning, and
• Function approximation
… may diverge
• This is sometimes called the deadly triad.
40
 Deadly Triad

• Consider sampling on-policy, over an episode. Update:

∆P = ' 0 + 2)P − P + '(0 + )0 − 2P)
= ' 2) − 3 P
• This multiplier is negative, for all ) ∈ 0, 1
• => convergence (P goes to zero, which is optimal here)

41
 Deadly Triad

• With tabular feature, this is just regression

• Answer may be sub-optimal, but no divergence occurs
• Specifically, if we only update = > (=left-most state):
• = > = P 0 will converge to )= > %
• = > % = P 1 will stay where it was initialized

42
 Deadly Triad

• What if we use multiple-step returns?

• Still consider only updating the left-most state
∆9 = ; < + >?!" − " #

= % & + ( 1 − * " # # + *(& # + "(# ## ) − " # \ = \ 4 = = > 44 = 0

= % 2( 1 − * − 1 .

/
• The multiplier is negative when 2) 1 − W < 1 => W > 1−
01
2
• E.g. where ) = 0.9, then we need W > ≈ 0.45
3
43
 Convergence of Prediction and Control Algorithms

• Tabular control learning algorithms (e.g., Q-learning) can be extended to FA

(e.g., Deep Q Network — DQN)
• The theory of control with function approximation is not fully developed
• Tracking is often preferred to convergence
(i.e., continually adapting the policy instead of converging to a fixed policy)

44
Deep Q Network (DQN)
 Deep Reinforcement Learning

DL: Deep Learning; RL: Reinforcement Learning

• DL: It requires large amounts of hand-labelled training data.
• RL: It can learn from a scalar reward signal that is frequently sparse, noisy
and delayed.
• DL: It assumes the data samples to be independent.
• RL: It typically encounters sequences of highly correlated states.
• DL: It assumes a fixed underlying distribution.
• RL: The data distribution changes as the algorithm learns new behaviors.

Playing Atari with Deep Reinforcement Learning. In NIPS workshop, 2013.

46
 DQN in Atari

• End-to-end learning of values ! >, - from pixels >

• Input state > is stack of raw pixels from last 4 frames
• Output is ! >, - for 18 joystick/button positions
• Reward is change in score for the step

Network architecture and hyperparameters fixed across all games 47

 DQN

• Approximate the optimal action-value function !5 >, - by ! >, -; 9

48
DQN Results in Atari

49
 Temporal Difference (TD) Learning

• Observe state >6 and perform action -6

• Environment provides new state >67/ and reward \6

• TD target: ]6 = (6 + ) ⋅ max !(>67/, -; 9)

8

• TD error: J6 = !6 − ]6 , where !6 = !(>6 , -6 ; 9)

• Goal: Make !6 close to ]6 , for all _. (Equivalently, make J60 small)

50
 Temporal Difference (TD) Learning

• TD error: J6 = !6 − ]6 , where !6 = !(>6 , -6 ; P)

,
/ ;
• TD learning: Find 9 by minimizing ` 9 = ∑96:/ +
9 0

• Online gradient descent:

• Observe (>6 , -6 , \6 , >67/) and compute J6
<;+, /0 < @(A+ , 8+ ; >)
• Compute gradient b6 = <>
= J6 ⋅ <>
• Gradient descent: 9 ← 9 − ' ⋅ b6
• Discard (>6 , -6 , \6 , >67/) after using it

51
 Shortcoming 1: Waste of Experience

• A transition: (>6 , -6 , \6 , >67/)

• Experience: all the transitions, for _ = 1, 2, …
• Previously, we discard (>6 , -6 , \6 , >67/) after using it
• It is a waste.

52
 Shortcoming 2: Correlated Updates

• Previously, we use (>6 , -6 , \6 , >67/) sequentially, for _ = 1, 2, … , to update 9.

• Consecutive states, >6 and >67/, are strongly correlated (which is bad).
• It violates commonly held assumption for stochastic gradient (similar
issue as continual learning!)

53
 Extra Reading Materials

• Playing Atari with Deep Reinforcement Learning. In NIPS workshop, 2013.

• Human-level Control through Deep Reinforcement Learning. Nature, 2015.

55
Thanks & QA?