Overview

1. Function Approximation
Approximate Dynamic Programming
Does ADP converge?

2. Policy Improvement
Policy Improvement Theorem
Policy Iteration
Value Iteration
Q-Learning
SARSA
Deep Networks

Recap: Dynamic Programming

Remember the Bellman update?

Here, and are tables. How can we represent value functions that have infinitely
many states?

Dynamic Programming with Function Approximation

Two ideas:
values can be represented as parametric functions, i.e., (where is a
parameter vector)
since we have infinite states, we need to operate with samples

Dynamic Programming with Function Approximation

Values can be parametrized, i.e.,

Linear parametrization

Neural networks

1 / 12
Dynamic Programming with Function Approximation
Assume you have a dataset of states , actions , rewards , next states and next
actions where and .

Approximated dynamic programming consists of iterate the following equations:

Approximate Dynamic Programming for Policy

Evaluation
1: Input: , number of episodes , and a parameter vector
2: Collect samples using the policy

3: for do

4: Minimize using e.g.,

gradient descent
5: Minimize

6: end for
7: Return .

Approximate Dynamic Programming for Policy

Evaluation
1: Input: , number of episodes , and a parameter vector
2: Collect samples using the policy

3: for do

4: Minimize using
e.g., gradient descent
5:

2 / 12
6: end for

Gradient Descent Approximate Dynamic Programming

Let us compute the gradient of the optimization problem introduced,

This gradient allows us to write an online version of dynamic programming - thus temporal
difference with function approximation.

Gradient Updates
Let be parametric, i.e., ,

where , and .

Temporal Difference with Function Approximation

Temporal Difference with Function Approximation for

Value Functions
1: Input: , number of episodes , vector parameter
2: for do
3: Sample first state

4: for do
5: Sample action

6: Update learning rate

7: Apply a on the environment and receive reward and next state
8:

9:
10: end for
11: end for

3 / 12
Temporal Difference with Function Approximation

Temporal Difference with Function Approximation for

Action-Value

Functions
1: Input: , number of episodes , vector parameter
2: for do

3: Sample first state

4: Sample action
5: for do

6: Update learning rate

7: Apply a on the environment and receive reward and next state
8: Sample

9:
10:

11: end for

12: end for

Convergence
Approximate dynamic programming and temporal difference with function
approximation converge to a biased solution when the parametrization is linear.
When the function approximation is not linear (e.g., neural networks), ADP and TD with
function approximation are not guaranteed to converge.

In either case, their estimate is biased. Such bias can be mitigated by introducing many
parameters, but many parameters cause high estimation variance, which can be only
compensated by using many samples.
State-of-the-art reinforcement learning tends to use deep neural networks with million (or
even billion) of parameters, and use a vast amount of samples.

4 / 12
Our Objective
Our objective is to find a policy that performs better than any other policy. In
mathematical terms:

We call such policy optimal. Note: there might be several optimal policies.

Policy Improvement Theorem

Policy Improvement Theorem

if , then .
Note: the policy improvement theorem is more general in textbooks, however, this
simplified (and more specific) version, still allow us to derive the following corollaries and
algorithms.

Greedy Policies: A policy is said to be greedy (w.r.t. ) if

Tabular Policy Iteration

Tabular Policy Iteration

1: Create a table that represent deterministic policies, and initialize it randomly
2: for do

3: PolicyEvaluation (e.g, use DP for policy evaluation as introduced the previous

lecture)
4: for do
5:

6: end for
7: end for
Question: does this algorithm converge to the optimal policy? Yes (for tabular and ).

However, it is computational expensive because after each policy improvement we need to

evaluate the policy.

5 / 12
Policy Improvement Theorem
Corollary 1 If exists such that , then
.
Corollary 2 (Optimality Bellman Equation) The optimal -function satisfies the following
optimality Bellman equation:

Can we use the optimality Bellman equation to obtain some guarantees on the
convergence of policy iteration, and to derive a more efficient algorithm?

Optimality Bellman Operator

Consider the optimality Bellman operator

The optimality Bellman operator is contractive, and, thanks to the Banach's theorem, we
can state that

Once we obtain the optimal action-value function , obtaining an optimal policy is trivial:

(Note: such policy is deterministic).

On the Contractivity of the Optimality Bellman Operator

Consider the optimality Bellman operator

6 / 12
Value Iteration

Tabular Value Iteration

1: Create two tables and that represent the -functions, and that represents the
policy
2: for do

3: for do
4:

5: end for
6: for do

7:
8: end for
9: Return

10: end for

Value Iteration
Value iteration is basically dynamic programming with the max operator that selects the
action of the next state.
is model-based (it needs knowledge of the transition model and the reward). Similarly to
what we have seen the previous lecture, we can derive a model-free, online version, called
-learning.

Online Algorithms
Like in the previous lecture, we want to devise an online algorithm, that uses samples to
update the -function and the policy.

Online Algorithm
1: Initialize a -function
2: for Episodes do

3: Sample first state

4: for Single episode do
5: Sample action according to a policy (which policy?)

7 / 12
6: Apply a on the environment and receive reward and next state
7: Use to update the -function

8:
9: end for
10: end for

Q-learning
Like in the previous lecture, we can use the online averaging and bootstrapping, i.e.,

where is the reward and is the next state.

-Greedy Policies
We want to obtain an "online" algorithm: i.e., an algorithm that improves the policy while
interacting with the environment.
A popular strategy is to use -greedy policies: policies that select the greedy action with
probability , and select a sub-optimal action with probability .

Such policies select with high probability (usually epsilon is small) good actions, while
keeping exploring and avoiding local minima. (For the most curious: see the exploration-
exploitation trade-off)

Q-learning

Tabular Q-learning
1: Input: , number of episodes
2: Initialize: a table of state-actions visitation , a table of -values
initialized with zeros
3: for Episodes do

4: Sample first state

5: for Single episode do
# Greedy Policy

8 / 12
With probability select , otherwise select a randomly.
# Learning Rate Update

Apply on the environment and receive reward and next state

10: # Bellman Update
11:

12: end for

13: end for

Simulation Time
Let's simulate -learning. We use the "investor MDP" as an example (next slide). There are
three states : rich, well-off, poor .
There are two actions : no-invest, 1 : invest .

We assume constant learning rate , discount factor , and epsilon-greedy

. Episodes are truncated after 5 steps.
I need 5 student:
One student execute the MDP (simulate.py)
One student execute the epsilon-greedy policy (epsilon_greedy.py)
One student compute
One student compute the entire update

One student keeps track of the q-table on the blackboard.

Tabular SARSA
1: Input: , number of episodes

2: Initialize: a table of state-actions visitation , a table of -values

initialized with zeros.
3: for Episodes do
4: Sample first state .

5: With probability select , otherwise select randomly.

6: for Single episode do
7: # Learning Rate Update

8: Apply on the environment and receive reward and next state # Greedy Policy

9 / 12
9: With probability select , otherwise select a random action
.
10: # Bellman Update

11:
12: end for

Q-learning

Q-learning with Function approximation

1: Input: , number of episodes
2: Initialize: a table of state-actions visitation , a table of -values
initialized with zeros

3: for Episodes do
4: Sample first state
5: for Single episode do

6: with probability select , otherwise select a random action

a.
7: Update the learning rate .

8: Apply a on the environment and receive reward and next state

9:
10: end for

11: end for

SARSA with Function Approximation

1: Input: , number of episodes
2: Initialize: a table of state-actions visitation , a table of -values
initialized with zeros.
3: for Episodes do

4: Sample first state .

5: With probability select , otherwise select randomly.
6: for Single episode do

10 / 12
7: Update the learning rate
8: Apply a on the environment and receive reward and next state

9: With probability select , otherwise select randomly.

10:

11: end for

12: end for

SARSA and Q-Learning

SARSA can be seen an online version of policy iteration (the Bellman update only evaluates
the current policy), and the current policy is greedy w.r.t. the q-function.
Q-learning can be seen as an online version of value iteration (the Bellman update
evaluates the greedy policy).

For this reason, SARSA is an on-policy algorithm (i.e., it evaluates the current policy), while
-learning is off-policy, since it evaluates a greedy policy while using an -greedy policy on
the environment.

Deep Q-Network
Q-learning with function approximation tends to be a bit unstable.
DQN aims to mitigate those instabilities by 1) introducing a target q-function, and 2) by
introducing randomized mini-batch updates (replay buffer).
Target q-functions. To stabilize learning it is useful to avoid bootstrapping. The idea of DQN
is to keep two separate functions, as in DP. This can be done by having two separate sets
of parameters and , and .
The targets parameters are updates once in a while .

Replay buffer. In classic -learning, samples are very correlated, as they are obtained by
running the MDP. Using non i.i.d. samples with function approximation is problematic. The
idea of the replay buffer is to store the last samples, and to sample each time a minibatch
of i.i.d. samples from it.

Deep Q-Network

11 / 12
DQN
1: for Episodes do
2: Sample first state

3: for Single episode do

4: With probability select , otherwise select randomly.
5: Update the learning rate .

6: Apply a on the environment and receive reward and next state

7: Append to .
8: Sample a minimatch

9:

10: end for

11: Every episodes, # Target update

12: end for

12 / 12