Policy Gradient Algorithms

Ashwin Rao

ICME, Stanford University

Ashwin Rao (Stanford) Policy Gradient Algorithms 1 / 33

Overview

1 Motivation and Intuition

2 Definitions and Notation

3 Policy Gradient Theorem and Proof

4 Policy Gradient Algorithms

5 Compatible Function Approximation Theorem and Proof

6 Natural Policy Gradient

Ashwin Rao (Stanford) Policy Gradient Algorithms 2 / 33

Why do we care about Policy Gradient (PG)?

Let us review how we got here

We started with Markov Decision Processes and Bellman Equations
Next we studied several variants of DP and RL algorithms
We noted that the idea of Generalized Policy Iteration (GPI) is key
Policy Improvement step: π(a|s) derived from argmaxa Q(s, a)
How do we do argmax when action space is large or continuous?
Idea: Do Policy Improvement step with a Gradient Ascent instead

Ashwin Rao (Stanford) Policy Gradient Algorithms 3 / 33

“Policy Improvement with a Gradient Ascent??”

We want to find the Policy that fetches the “Best Expected Returns”
Gradient Ascent on “Expected Returns” w.r.t params of Policy func
So we need a func approx for (stochastic) Policy Func: π(s, a; θ)
In addition to the usual func approx for Action Value Func: Q(s, a; w )
π(s, a; θ) func approx called Actor, Q(s, a; w ) func approx called Critic
Critic parameters w are optimized w.r.t Q(s, a; w ) loss function min
Actor parameters θ are optimized w.r.t Expected Returns max
We need to formally define “Expected Returns”
But we already see that this idea is appealing for continuous actions
GPI with Policy Improvement done as Policy Gradient (Ascent)

Ashwin Rao (Stanford) Policy Gradient Algorithms 4 / 33

Value Function-based and Policy-based RL

Value Function-based
Learn Value Function (with a function approximation)
Policy is implicit - readily derived from Value Function (eg: -greedy)
Policy-based
Learn Policy (with a function approximation)
No need to learn a Value Function
Actor-Critic
Learn Policy (Actor)
Learn Value Function (Critic)

Ashwin Rao (Stanford) Policy Gradient Algorithms 5 / 33

Advantages and Disadvantages of Policy Gradient approach

Advantages:
Finds the best Stochastic Policy (Optimal Deterministic Policy,
produced by other RL algorithms, can be unsuitable for POMDPs)
Naturally explores due to Stochastic Policy representation
Effective in high-dimensional or continuous action spaces
Small changes in θ ⇒ small changes in π, and in state distribution
This avoids the convergence issues seen in argmax-based algorithms
Disadvantages:
Typically converge to a local optimum rather than a global optimum
Policy Evaluation is typically inefficient and has high variance
Policy Improvement happens in small steps ⇒ slow convergence

Ashwin Rao (Stanford) Policy Gradient Algorithms 6 / 33

Notation

Discount Factor γ
Assume episodic with 0 ≤ γ ≤ 1 or non-episodic with 0 ≤ γ < 1
States st ∈ S, Actions at ∈ A, Rewards rt ∈ R, ∀t ∈ {0, 1, 2, . . .}
a = Pr (s
State Transition Probabilities Ps,s 0
0 t+1 = s |st = s, at = a)
Expected Rewards Ras = E [rt |st = s, at = a]
Initial State Probability Distribution p0 : S → [0, 1]
Policy Func Approx π(s, a; θ) = Pr (at = a|st = s, θ), θ ∈ Rk
PG coverage will be quite similar for non-discounted non-episodic, by
considering average-reward objective (so we won’t cover it)

Ashwin Rao (Stanford) Policy Gradient Algorithms 7 / 33

“Expected Returns” Objective
Now we formalize the “Expected Returns” Objective J(θ)

X∞
J(θ) = Eπ [ γ t rt ]
t=0

Value Function V π (s) and Action Value function Q π (s, a) defined as:
∞
X
V π (s) = Eπ [ γ k−t rk |st = s], ∀t ∈ {0, 1, 2, . . .}
k=t

X∞
π
Q (s, a) = Eπ [ γ k−t rk |st = s, at = a], ∀t ∈ {0, 1, 2, . . .}
k=t

Advantage Function Aπ (s, a) = Q π (s, a) − V π (s)

Also, p(s → s 0 , t, π) will be a key function for us - it denotes the
probability of going from state s to s 0 in t steps by following policy π
Ashwin Rao (Stanford) Policy Gradient Algorithms 8 / 33
Discounted-Aggregate State-Visitation Measure

∞
X ∞
X
t
J(θ) = Eπ [ γ rt ] = γ t Eπ [rt ]
t=0 t=0
∞
X Z Z Z
t
= γ ( p0 (s0 ) · p(s0 → s, t, π) · ds0 ) π(s, a; θ) · Ras · da · ds
t=0 S S A

∞
Z Z X Z
t
= ( γ · p0 (s0 ) · p(s0 → s, t, π) · ds0 ) π(s, a; θ) · Ras · da · ds
S S t=0 A

Definition
Z Z
π
J(θ) = ρ (s) π(s, a; θ) · Ras · da · ds
S A
R P∞
where ρπ (s) = S t=0 γ t · p0 (s0 ) · p(s0 → s, t, π) · ds0 is the key function
(for PG) we’ll refer to as Discounted-Aggregate State-Visitation Measure.
Ashwin Rao (Stanford) Policy Gradient Algorithms 9 / 33
Policy Gradient Theorem (PGT)

Theorem
Z Z
π
∇θ J(θ) = ρ (s) ∇θ π(s, a; θ) · Q π (s, a) · da · ds
S A

Note: ρπ (s) depends on θ, but there’s no ∇θ ρπ (s) term in ∇θ J(θ)

So we can simply sample simulation paths, and at each time step, we
calculate (∇θ log π(s, a; θ)) · Q π (s, a) (probabilities implicit in paths)
Note: ∇θ log π(s, a; θ) is Score function (Gradient of log-likelihood)
We will estimate Q π (s, a) with a function approximation Q(s, a; w )
We will later show how to avoid the estimate bias of Q(s, a; w )
This numerical estimate of ∇θ J(θ) enables Policy Gradient Ascent
Let us look at the score function of some canonical π(s, a; θ)

Ashwin Rao (Stanford) Policy Gradient Algorithms 10 / 33

Canonical π(s, a; θ) for finite action spaces

For finite action spaces, we often use Softmax Policy

θ is an n-vector (θ1 , . . . , θn )
Features vector φ(s, a) = (φ1 (s, a), . . . , φn (s, a)) for all s ∈ S, a ∈ A
Weight actions using linear combinations of features: θT · φ(s, a)
Action probabilities proportional to exponentiated weights:
T
e θ ·φ(s,a)
π(s, a; θ) = P θT ·φ(s,b) for all s ∈ S, a ∈ A
be

The score function is:

X
∇θ log π(s, a; θ) = φ(s, a)− π(s, b; θ)·φ(s, b) = φ(s, a)−Eπ [φ(s, ·)]
b

Ashwin Rao (Stanford) Policy Gradient Algorithms 11 / 33

Canonical π(s, a; θ) for continuous action spaces

For continuous action spaces, we often use Gaussian Policy

θ is an n-vector (θ1 , . . . , θn )
State features vector φ(s) = (φ1 (s), . . . , φn (s)) for all s ∈ S
Gaussian Mean is a linear combination of state features θT · φ(s)
Variance may be fixed σ 2 , or can also be parameterized
Policy is Gaussian, a ∼ N (θT · φ(s), σ 2 ) for all s ∈ S
The score function is:
(a − θT · φ(s)) · φ(s)
∇θ log π(s, a; θ) =
σ2

Ashwin Rao (Stanford) Policy Gradient Algorithms 12 / 33

Proof of Policy Gradient Theorem

We begin the proof by noting that:

Z Z Z
π
J(θ) = p0 (s0 )·V (s0 )·ds0 = p0 (s0 ) π(s0 , a0 ; θ)·Q π (s0 , a0 )·da0 ·ds0
S S A

Calculate ∇θ J(θ) by parts π(s0 , a0 ; θ) and Q π (s0 , a0 )

Z Z
∇θ J(θ) = p0 (s0 ) ∇θ π(s0 , a0 ; θ) · Q π (s0 , a0 ) · da0 · ds0
ZS ZA
+ p0 (s0 ) π(s0 , a0 ; θ) · ∇θ Q π (s0 , a0 ) · da0 · ds0
S A

Ashwin Rao (Stanford) Policy Gradient Algorithms 13 / 33

Proof of Policy Gradient Theorem
Now expand Q π (s0 , a0 ) as Ras00 + S γ · Psa00,s1 · V π (s1 ) · ds1 (Bellman)
R

Z Z
= p0 (s0 ) ∇θ π(s0 , a0 ; θ) · Q π (s0 , a0 ) · da0 · ds0
ZS ZA Z
+ p0 (s0 ) π(s0 , a0 ; θ) · ∇θ (Ras00 + γ · Psa00,s1 · V π (s1 ) · ds1 ) · da0 · ds0
S A S

Note: ∇θ Ras00 = 0, so remove that term

Z Z
= p0 (s0 ) ∇θ π(s0 , a0 ; θ) · Q π (s0 , a) · da0 · ds0
ZS ZA Z
+ p0 (s0 ) π(s0 , a0 ; θ) · ∇θ ( γ · Psa00,s1 · V π (s1 ) · ds1 ) · da0 · ds0
S A S

Ashwin Rao (Stanford) Policy Gradient Algorithms 14 / 33

Proof of Policy Gradient Theorem
Now bring the ∇θ inside the S to apply only on V π (s1 )
R

Z Z
= p0 (s0 ) ∇θ π(s0 , a0 ; θ) · Q π (s0 , a) · da0 · ds0
ZS ZA Z
+ p0 (s0 ) π(s0 , a0 ; θ) γ · Psa00,s1 · ∇θ V π (s1 ) · ds1 · da0 · ds0
S A S

R R R
Now bring the outside S and A inside the inner S
Z Z
= p0 (s0 ) ∇θ π(s0 , a0 ; θ) · Q π (s0 , a0 ) · da0 · ds0
S
Z Z A Z
+ ( γ · p0 (s0 ) π(s0 , a0 ; θ) · Psa00,s1 · da0 · ds0 ) · ∇θ V π (s1 ) · ds1
S S A

Ashwin Rao (Stanford) Policy Gradient Algorithms 15 / 33

Policy Gradient Theorem
· Psa00,s1 · da0 = p(s0 → s1 , 1, π)
R
Note that A π(s0 , a0 ; θ)
Z Z
= p0 (s0 ) ·∇θ π(s0 , a0 ; θ) · Q π (s0 , a0 ) · da0 · ds0
ZS Z A

+ ( γ · p0 (s0 ) · p(s0 → s1 , 1, π) · ds0 ) · ∇θ V π (s1 ) · ds1

S S

Now expand V π (s1 ) to A π(s1 , a1 ; θ) · Q π (s1 , a1 ) · da1

R

Z Z
= p0 (s0 ) ·∇θ π(s0 , a0 ; θ) · Q π (s0 , a0 ) · da · ds0
ZS Z A Z
+ ( γ · p0 (s0 )p(s0 → s1 , 1, π)ds0 ) · ∇θ ( π(s1 , a1 ; θ)Q π (s1 , a1 )da1 )ds1
S S A

Ashwin Rao (Stanford) Policy Gradient Algorithms 16 / 33

Proof of Policy Gradient Theorem

We are now back to when we started calculating gradient of A π · Q π · da.

R

Follow the same process of splitting π · Q π , then Bellman-expanding Q π

(to calculate its gradient), and iterate.
Z Z
= p0 (s0 ) ·∇θ π(s0 , a0 ; θ) · Q π (s0 , a0 ) · da0 · ds0
ZS Z A Z
+ γp0 (s0 )p(s0 → s1 , 1, π)ds0 ( ∇θ π(s1 , a1 ; θ)Q π (s1 , a1 )da1 + . . .)ds1
S S A

This iterative process leads us to:

∞ Z Z
X Z
= γ t ·p0 (s0 )·p(s0 → st , t, π)·ds0 ∇θ π(st , at ; θ)·Q π (st , at )·dat ·dst
t=0 S S A

Ashwin Rao (Stanford) Policy Gradient Algorithms 17 / 33

Proof of Policy Gradient Theorem

P∞ R
Bring t=0 inside the two S , and note that
π (s , a ) · da is independent of t.
R
A ∇ θ π(s t , a t ; θ) · Q t t t

∞
Z Z X Z
= γ t · p0 (s0 ) · p(s0 → s, t, π) · ds0 ∇θ π(s, a; θ) · Q π (s, a) · da · ds
S S t=0 A

R P∞ t def
Reminder that S · p0 (s0 ) · p(s0 → s, t, π) · ds0 = ρπ (s). So,
t=0 γ
Z Z
π
∇θ J(θ) = ρ (s) ∇θ π(s, a; θ) · Q π (s, a) · da · ds
S A

Q.E.D.

Ashwin Rao (Stanford) Policy Gradient Algorithms 18 / 33

Monte-Carlo Policy Gradient (REINFORCE Algorithm)

Update θ by stochastic gradient ascent using PGT

Using Gt = T k−t · r as an unbiased sample of Q π (s , a )
P
k=t γ k t t

∆θt = α · γ t · ∇θ log π(st , at ; θ) · Gt

Algorithm 4.1: REINFORCE(·)

Initialize θ arbitrarily
 episode {s0 , a0 , r0 , . . . , sT , aT , rT } ∼ π(·, ·; θ)
for each
for t ← 0 to T
G← T
 P k−t · r
do k=t γ k
 do
θ ← θ + α · γ t · ∇θ log π(st , at ; θ) · G
return (θ)

Ashwin Rao (Stanford) Policy Gradient Algorithms 19 / 33

Reducing Variance using a Critic

Monte Carlo Policy Gradient has high variance

We use a Critic Q(s, a; w ) to estimate Q π (s, a)
Actor-Critic algorithms maintain two sets of parameters:
Critic updates parameters w to approximate Q-function for policy π
Critic could use any of the algorithms we learnt earlier:
Monte Carlo policy evaluation
Temporal-Difference Learning
TD(λ) based on Eligibility Traces
Could even use LSTD (if critic function approximation is linear)
Actor updates policy parameters θ in direction suggested by Critic
This is Approximate Policy Gradient due to Bias of Critic
Z Z
∇θ J(θ) ≈ ρπ (s) ∇θ π(s, a; θ) · Q(s, a; w ) · da · ds
S A

Ashwin Rao (Stanford) Policy Gradient Algorithms 20 / 33

So what does the algorithm look like?

Generate a sufficient set of simulation paths s0 , a0 , r0 , s1 , a1 , r1 , . . .

s0 is sampled from the distribution p0 (·)
at is sampled from π(st , ·; θ)
st+1 sampled from transition probs and rt+1 from reward func
At each time step t, update w proportional to gradient of appropriate
(MC or TD-based) loss function of Q(s, a; w )
Sum γ t · ∇θ log π(st , at ; θ) · Q(st , at ; w ) over t and over paths
Update θ using this (biased) estimate of ∇θ J(θ)
Iterate with a new set of simulation paths ...

Ashwin Rao (Stanford) Policy Gradient Algorithms 21 / 33

Reducing Variance with a Baseline

We can reduce variance by subtracting a baseline function B(s) from

Q(s, a; w ) in the Policy Gradient estimate
This means at each time step, we replace
γ t · ∇θ log π(st , at ; θ) · Q(st , at ; w ) with
γ t · ∇θ log π(st , at ; θ) · (Q(st , at ; w ) − B(s))
Note that Baseline function B(s) is only a function of s (and not a)
This ensures that subtracting Baseline B(s) does not add bias
Z Z
π
ρ (s) ∇θ π(s, a; θ) · B(s) · da · ds
ZS A Z
π
= ρ (s) · B(s) · ∇θ ( π(s, a; θ) · da) · ds = 0
S A

Ashwin Rao (Stanford) Policy Gradient Algorithms 22 / 33

Using State Value function as Baseline

A good baseline B(s) is state value function V (s; v )

Rewrite Policy Gradient algorithm using advantage function estimate

A(s, a; w , v ) = Q(s, a; w ) − V (s; v )

Now the estimate of ∇θ J(θ) is given by:

Z Z
ρπ (s) ∇θ π(s, a; θ) · A(s, a; w , v ) · da · ds
S A

At each time step, we update both sets of parameters w and v

Ashwin Rao (Stanford) Policy Gradient Algorithms 23 / 33

TD Error as estimate of Advantage Function

Consider TD error δ π for the true Value Function V π (s)

δ π = r + γV π (s 0 ) − V π (s)

δ π is an unbiased estimate of Advantage function Aπ (s, a)

Eπ [δ π |s, a] = Eπ [r +γV π (s 0 )|s, a]−V π (s) = Q π (s, a)−V π (s) = Aπ (s, a)

So we can write Policy Gradient in terms of Eπ [δ π |s, a]

Z Z
π
∇θ J(θ) = ρ (s) ∇θ π(s, a; θ) · Eπ [δ π |s, a] · da · ds
S A

In practice, we can use func approx for TD error (and sample):

δ(s, r , s 0 ; v ) = r + γV (s 0 ; v ) − V (s; v )

This approach requires only one set of critic parameters v

Ashwin Rao (Stanford) Policy Gradient Algorithms 24 / 33
TD Error can be used by both Actor and Critic

Algorithm 4.2: ACTOR-CRITIC-TD-ERROR(·)

Initialize Policy params θ ∈ Rm and State VF params v ∈ Rn arbitrarily

for each episode

 Initialize s (first state of episode)
P←1








 while s is not terminal



  a ∼ π(s, ·; θ)
Take action a, observe r , s 0

 


do 
δ ← r + γV (s 0 ; v ) − V (s; v )


 

 do v ← v + αv · δ · ∇v V (s; v )



θ ← θ + αθ · P · δ · ∇θ log π(s, a; θ)

 

 

P ← γP

 


 

s ← s0
 

Ashwin Rao (Stanford) Policy Gradient Algorithms 25 / 33

Using Eligibility Traces for both Actor and Critic
Algorithm 4.3: ACTOR-CRITIC-ELIGIBILITY-TRACES(·)

Initialize Policy params θ ∈ Rm and State VF params v ∈ Rn arbitrarily

for each
 episode

 Initialize s (first state of episode)
zθ , zv ← 0 (m and n components eligibility trace vectors)




P←1








 while s is not terminal
a ∼ π(s, ·; θ)
 

 
Take action a, observe r , s 0

 


do 
δ ← r + γV (s 0 ; v ) − V (s; v )


 
 
zv ← γ · λv · zv + ∇v V (s; v )

 
do


θ ← γ · λθ · zθ + P · ∇θ log π(s, a; θ)



 
 z
← v + αv · δ · zv

v
 


 


θ ← θ + αθ · δ · zθ
 


 

P ← γP, s ← s 0

 

Ashwin Rao (Stanford) Policy Gradient Algorithms 26 / 33

Overcoming Bias

We’ve learnt a few ways of how to reduce variance

But we haven’t discussed how to overcome bias
All of the following substitutes for Q π (s, a) in PG have bias:
Q(s, a; w )
A(s, a; w , v )
δ(s, s 0 , r ; v )
Turns out there is indeed a way to overcome bias
It is called the Compatible Function Approximation Theorem

Ashwin Rao (Stanford) Policy Gradient Algorithms 27 / 33

Compatible Function Approximation Theorem

Theorem
If the following two conditions are satisfied:
1 Critic gradient is compatible with the Actor score function

∇w Q(s, a; w ) = ∇θ log π(s, a; θ)

2 Critic parameters w minimize the following mean-squared error:

Z Z
π
= ρ (s) π(s, a; θ)(Q π (s, a) − Q(s, a; w ))2 · da · ds
S A

Then the Policy Gradient using critic Q(s, a; w ) is exact:

Z Z
π
∇θ J(θ) = ρ (s) ∇θ π(s, a; θ) · Q(s, a; w ) · da · ds
S A

Ashwin Rao (Stanford) Policy Gradient Algorithms 28 / 33

Proof of Compatible Function Approximation Theorem
For w that minimizes
Z Z
π
= ρ (s) π(s, a; θ) · (Q π (s, a) − Q(s, a; w ))2 · da · ds,
S A
Z Z
ρπ (s) π(s, a; θ) · (Q π (s, a) − Q(s, a; w )) · ∇w Q(s, a; w ) · da · ds = 0
S A
But since ∇w Q(s, a; w ) = ∇θ log π(s, a; θ), we have:
Z Z
π
ρ (s) π(s, a; θ) · (Q π (s, a) − Q(s, a; w )) · ∇θ log π(s, a; θ) · da · ds = 0
S A

Z Z
π
Therefore, ρ (s) π(s, a; θ) · Q π (s, a) · ∇θ log π(s, a; θ) · da · ds
ZS ZA
= ρπ (s) π(s, a; θ) · Q(s, a; w ) · ∇θ log π(s, a; θ) · da · ds
S A

Ashwin Rao (Stanford) Policy Gradient Algorithms 29 / 33

Proof of Compatible Function Approximation Theorem

Z Z
But ∇θ J(θ) = ρπ (s) π(s, a; θ) · Q π (s, a) · ∇θ log π(s, a; θ) · da · ds
S A

Z Z
π
So, ∇θ J(θ) = ρ (s) π(s, a; θ) · Q(s, a; w ) · ∇θ log π(s, a; θ) · da · ds
ZS ZA
= ρπ (s) ∇θ π(s, a; θ) · Q(s, a; w ) · da · ds
S A

Q.E.D.
This means with conditions (1) and (2) of Compatible Function
Approximation Theorem, we can use the critic func approx
Q(s, a; w ) and still have the exact Policy Gradient.
Ashwin Rao (Stanford) Policy Gradient Algorithms 30 / 33
How to enable Compatible Function Approximation
A simple way to enable Compatible Function Approximation
∂Q(s,a;w )
∂wi = ∂ log ∂θ
π(s,a;θ)
i
, ∀i is to set Q(s, a; w ) to be linear in its features.
n n
X X ∂ log π(s, a; θ)
Q(s, a; w ) = φi (s, a) · wi = · wi
∂θi
i=1 i=1

We note below that a compatible Q(s, a; w ) serves as an approximation of

the advantage function.
Z Z n
X ∂ log π(s, a; θ)
π(s, a; θ) · Q(s, a; w ) · da = π(s, a; θ) · ( · wi ) · da
A A ∂θi
i=1
Z n n Z
X ∂π(s, a; θ) X ∂π(s, a; θ)
= ·( · wi ) · da = ( · da) · wi
A ∂θi A ∂θi
i=1 i=1
n Z n
X ∂ X ∂1
= ( π(s, a; θ) · da) · wi = · wi = 0
∂θi A ∂θi
i=1 i=1
Ashwin Rao (Stanford) Policy Gradient Algorithms 31 / 33
Fisher Information Matrix

Denoting [ ∂ log ∂θ
π(s,a;θ)
i
], i = 1, . . . , n as the score column vector SC (s, a; θ)
and assuming compatible linear-approx critic:
Z Z
π
∇θ J(θ) = ρ (s) π(s, a; θ) · (SC (s, a; θ) · SC (s, a; θ)T · w ) · da · ds
S A
= Es∼ρπ ,a∼π [SC (s, a; θ) · SC (s, a; θ)T ] · w
= FIMρπ ,π (θ) · w

where FIMρπ ,π (θ) is the Fisher Information Matrix w.r.t. s ∼ ρπ , a ∼ π.

Ashwin Rao (Stanford) Policy Gradient Algorithms 32 / 33

Natural Policy Gradient

Recall the idea of Natural Gradient from Numerical Optimization

Natural gradient ∇nat
θ J(θ) is the direction of optimal θ movement
In terms of the KL-divergence metric (versus plain Euclidean norm)
Natural gradient yields better convergence (we won’t cover proof)
Formally defined as: ∇θ J(θ) = FIMρπ ,π (θ) · ∇nat
θ J(θ)

Therefore, ∇nat
θ J(θ) = w

This compact result is great for our algorithm:

Update Critic params w with the critic loss gradient (at step t) as:

γ t · (rt + γ · SC (st+1 , at+1 , θ) · w − SC (st , at , θ) · w ) · SC (st , at , θ)

Update Actor params θ in the direction equal to value of w

Ashwin Rao (Stanford) Policy Gradient Algorithms 33 / 33