Chapter 3.

Dynamic Programming

This chapter introduces basic ideas and methods of dynamic programming. 1 It sets
out the basic elements of a recursive optimization problem, describes the functional
equation (the Bellman equation), presents three methods for solving the Bellman
equation, and gives the Benveniste-Scheinkman formula for the derivative of the op-
timal value function. Let’s dive in.

3.1. Sequential problems

Let β ∈ (0, 1) be a discount factor. We want to choose an infinite sequence of
“controls” {ut }∞
t=0 to maximize

∞


β t r (xt , ut ) , (3.1.1)
t=0

subject to xt+1 = g(xt , ut ), with x0 given. We assume that r(xt , ut ) is a concave

function and that the set {(xt+1 , xt ) : xt+1 ≤ g(xt , ut ), ut ∈ Rk } is convex and
compact. Dynamic programming seeks a time-invariant policy function h mapping
the state xt into the control ut , such that the sequence {us }∞ s=0 generated by
iterating the two functions
ut = h (xt )
(3.1.2)
xt+1 = g (xt , ut ) ,
starting from initial condition x0 at t = 0 solves the original problem. A solution in
the form of equations (3.1.2 ) is said to be recursive. To find the policy function h we
need to know another function V (x) that expresses the optimal value of the original
problem, starting from an arbitrary initial condition x ∈ X . This is called the value

1 This chapter is written in the hope of getting the reader to start using the methods
quickly. We hope to promote demand for further and more rigorous study of the
subject. In particular see Bertsekas (1976), Bertsekas and Shreve (1978), Stokey and
Lucas (with Prescott) (1989), Bellman (1957), and Chow (1981). This chapter covers
much of the same material as Sargent (1987b, chapter 1).

– 78 –
 Sequential problems 79

function. In particular, define

∞


V (x0 ) = max
∞
β t r (xt , ut ) , (3.1.3)
{us }s=0
t=0

where again the maximization is subject to xt+1 = g(xt , ut ), with x0 given. Of

course, we cannot possibly expect to know V (x0 ) until after we have solved the
problem, but let’s proceed on faith. If we knew V (x0 ), then the policy function h
could be computed by solving for each x ∈ X the problem

max{r (x, u) + βV (x̃)}, (3.1.4)

u

where the maximization is subject to x̃ = g(x, u) with x given, and x̃ denotes the
state next period. Thus, we have exchanged the original problem of finding an infinite
sequence of controls that maximizes expression (3.1.1 ) for the problem of finding the
optimal value function V (x) and a function h that solves the continuum of maximum
problems (3.1.4 )—one maximum problem for each value of x. This exchange doesn’t
look like progress, but we shall see that it often is.
Our task has become jointly to solve for V (x), h(x), which are linked by the
Bellman equation
V (x) = max{r (x, u) + βV [g (x, u)]}. (3.1.5)
u

The maximizer of the right side of equation (3.1.5 ) is a policy function h(x) that
satisfies
V (x) = r [x, h (x)] + βV {g [x, h (x)]}. (3.1.6)
Equation (3.1.5 ) or (3.1.6 ) is a functional equation to be solved for the pair of un-
known functions V (x), h(x).
Methods for solving the Bellman equation are based on mathematical structures
that vary in their details depending on the precise nature of the functions r and g . 2
2 There are alternative sets of conditions that make the maximization (3.1.4 ) well
behaved. One set of conditions is as follows: (1) r is concave and bounded, and
(2) the constraint set generated by g is convex and compact, that is, the set of
{(xt+1 , xt ) : xt+1 ≤ g(xt , ut )} for admissible ut is convex and compact. See Stokey,
Lucas, and Prescott (1989), and Bertsekas (1976) for further details of convergence
results. See Benveniste and Scheinkman (1979) and Stokey, Lucas, and Prescott
(1989) for the results on differentiability of the value function. In an appendix on
functional analysis, chapter A, we describe the mathematics for one standard set of
assumptions about (r, g). In chapter 5, we describe it for another set of assumptions
about (r, g).
80 Chapter 3: Dynamic Programming

All of these structures contain versions of the following four findings. Under various
particular assumptions about r and g , it turns out that
1. The functional equation (3.1.5 ) has a unique strictly concave solution.
2. This solution is approached in the limit as j → ∞ by iterations on

Vj+1 (x) = max{r (x, u) + βVj (x̃)}, (3.1.7)

u

subject to x̃ = g(x, u), x given, starting from any bounded and continuous initial
V0 .
3. There is a unique and time invariant optimal policy of the form ut = h(xt ),
where h is chosen to maximize the right side of (3.1.5 ). 3
4. Off corners, the limiting value function V is differentiable with

∂r ∂g
V  (x) = [x, h (x)] + β [x, h (x)] V  {g [x, h (x)]}. (3.1.8)
∂x ∂x

This is a version of a formula of Benveniste and Scheinkman (1979). We often

encounter settings in which the transition law can be formulated so that the state
∂g
x does not appear in it, so that ∂x = 0 , which makes equation (3.1.8 ) become

∂r
V  (x) = [x, h (x)] . (3.1.9)
∂x

At this point, we describe three broad computational strategies that apply in

various contexts.

3 The time invariance of the policy function u = h(x ) is very convenient econo-
t t
metrically, because we can impose a single decision rule for all periods. This lets us
pool data across period to estimate the free parameters of the return and transition
functions that underlie the decision rule.
 Sequential problems 81

3.1.1. Three computational methods

There are three main types of computational methods for solving dynamic programs.
All aim to solve the functional equation (3.1.4 ).

Value function iteration. The first method proceeds by constructing a sequence

of value functions and associated policy functions. The sequence is created by iter-
ating on the following equation, starting from V0 = 0 , and continuing until Vj has
converged: 4
Vj+1 (x) = max{r (x, u) + βVj (x̃)}, (3.1.10)
u
5
subject to x̃ = g(x, u), x given. This method is called value function iteration or
iterating on the Bellman equation.

Guess and verify. A second method involves guessing and verifying a solution
V to equation (3.1.5 ). This method relies on the uniqueness of the solution to the
equation, but because it relies on luck in making a good guess, it is not generally
available.

Howard’s improvement algorithm. A third method, known as policy function

iteration or Howard’s improvement algorithm, consists of the following steps:

1. Pick a feasible policy, u = h0 (x), and compute the value associated with oper-
ating forever with that policy:
∞


Vhj (x) = β t r [xt , hj (xt )] ,
t=0

where xt+1 = g[xt , hj (xt )], with j = 0 .

2. Generate a new policy u = hj+1 (x) that solves the two-period problem

max{r (x, u) + βVhj [g (x, u)]},

u

for each x.
3. Iterate over j to convergence on steps 1 and 2.

4 See the appendix on functional analysis for what it means for a sequence of
functions to converge.
5 A proof of the uniform convergence of iterations on equation (3.1.10 ) is contained
in the appendix on functional analysis, chapter A.
82 Chapter 3: Dynamic Programming

In the appendix on functional analysis, chapter A, we describe some conditions

under which the improvement algorithm converges to the solution of Bellman’s equa-
tion. The method often converges faster than does value function iteration (e.g., see
exercise 2.1 at the end of this chapter). 6 The policy improvement algorithm is also
a building block for the methods for studying government policy to be described in
chapter 22.
Each of these methods has its uses. Each is “easier said than done,” because it is
typically impossible analytically to compute even one iteration on equation (3.1.10 ).
This fact thrusts us into the domain of computational methods for approximating
solutions: pencil and paper are insufficient. The following chapter describes some
computational methods that can be used for problems that cannot be solved by hand.
Here we shall describe the first of two special types of problems for which analytical
solutions can be obtained. It involves Cobb-Douglas constraints and logarithmic
preferences. Later in chapter 5, we shall describe a specification with linear constraints
and quadratic preferences. For that special case, many analytic results are available.
These two classes have been important in economics as sources of examples and as
inspirations for approximations.

3.1.2. Cobb-Douglas transition, logarithmic preferences

Brock and Mirman (1972) used the following optimal growth example. 7 A planner
chooses sequences {ct , kt+1 }∞
t=0 to maximize

∞


β t ln (ct )
t=0

subject to a given value for k0 and a transition law

kt+1 + ct = Aktα , (3.1.11)

where A > 0, α ∈ (0, 1), β ∈ (0, 1).

6 The quickness of the policy improvement algorithm is linked to its being an

implementation of Newton’s method, which converges quadratically while iteration
on the Bellman equation converges at a linear rate. See chapter 4 and the appendix
on functional analysis, chapter A.
7 See also Levhari and Srinivasan (1969).
 Sequential problems 83

This problem can be solved “by hand,” using any of our three methods. We begin
with iteration on the Bellman equation. Start with v0 (k) = 0 , and solve the one-
period problem: choose c to maximize ln(c) subject to c + k̃ = Ak α . The solution
is evidently to set c = Ak α , k̃ = 0 , which produces an optimized value v1 (k) =
1 βα
ln A + α ln k . At the second step, we find c = 1+βα Ak α , k̃ = 1+βα Ak α , v2 (k) =
A αβA
ln 1+αβ + β ln A + αβ ln 1+αβ + α(1 + αβ) ln k . Continuing, and using the algebra of
geometric series, gives the limiting policy functions c = (1 − βα)Ak α , k̃ = βαAk α ,
and the value function v(k) = (1 − β)−1 {ln[A(1 − βα)] + 1−βα
βα
ln(Aβα)} + 1−βαα
ln k .
Here is how the guess-and-verify method applies to this problem. Since we already
know the answer, we’ll guess a function of the correct form, but leave its coefficients
undetermined. 8 Thus, we make the guess

v (k) = E + F ln k, (3.1.12)

where E and F are undetermined constants. The left and right sides of equation
(3.1.12 ) must agree for all values of k . For this guess, the first-order necessary
condition for the maximum problem on the right side of equation (3.1.10 ) implies the
following formula for the optimal policy k̃ = h(k), where k̃ is next period’s value and
k is this period’s value of the capital stock:
βF
k̃ = Ak α . (3.1.13)
1 + βF
Substitute equation (3.1.13 ) into the Bellman equation and equate the result to the
right side of equation (3.1.12 ). Solving the resulting equation for E and F gives
F = α/(1 − αβ) and E = (1 − β)−1 [ln A(1 − αβ) + 1−αβ βα
ln Aβα]. It follows that

k̃ = βαAk α . (3.1.14)

Note that the term F = α/(1 − αβ) can be interpreted as a geometric sum α[1 +
αβ + (αβ)2 + . . .].
Equation (3.1.14 ) shows that the optimal policy is to have capital move according
to the difference equation kt+1 = Aβαktα , or ln kt+1 = ln Aβα + α ln kt . That α
is less than 1 implies that kt converges as t approaches infinity for any positive
α
initial value k0 . The stationary point is given by the solution of k∞ = Aβαk∞ , or
α−1 −1
k∞ = (Aβα) .

8 This is called the method of undetermined coefficients.

84 Chapter 3: Dynamic Programming

3.1.3. Euler equations

In many problems, there is no unique way of defining states and controls, and sev-
eral alternative definitions lead to the same solution of the problem. Sometimes the
states and controls can be defined in such a way that xt does not appear in the
transition equation, so that ∂gt /∂xt ≡ 0 . In this case, the first-order condition
for the problem on the right side of the Bellman equation in conjunction with the
Benveniste-Scheinkman formula implies
∂rt ∂gt ∂rt+1 (xt+1 , ut+1 )
(xt , ut ) + (ut ) · = 0, xt+1 = gt (ut ) .
∂ut ∂ut ∂xt+1
The first equation is called an Euler equation. Under circumstances in which the
second equation can be inverted to yield ut as a function of xt+1 , using the second
equation to eliminate ut from the first equation produces a second-order difference
equation in xt , since eliminating ut+1 brings in xt+2 .

3.1.4. A sample Euler equation

As an example of an Euler equation, consider the Ramsey problem of choosing
∞ t
{ct , kt+1 }∞
t=0 to maximize t=0 β u(ct ) subject to ct + kt+1 = f (kt ), where k0
is given and the one-period utility function satisfies u (c) > 0, u (c) < 0, limct 0
u (ct ) = ∞; and where f  (k) > 0, f  (k) < 0 . Let the state be k and the control
be k  , where k  denotes next period’s value of k . Substitute c = f (k) − k  into the
utility function and express the Bellman equation as
 
v (k) = max{u f (k) − k̃ + βv k̃ }. (3.1.15)
k̃

Application of the Benveniste-Scheinkman formula gives

 
v  (k) = u f (k) − k̃ f  (k) . (3.1.16)
Notice that the first-order condition for the maximum problem on the right side of
equation (3.1.15 ) is −u [f (k) − k̃] + βv  (k̃) = 0 , which, using equation v(3.1.16 ),
gives    
u f (k) − k̃ = βu f k̃ − k̂ f  (k  ) , (3.1.17)

where k̂ denotes the “two-period-ahead” value of k . Equation (3.1.17 ) can be ex-

pressed as
u (ct+1 ) 
1=β  f (kt+1 ) ,
u (ct )
 Stochastic control problems 85

an Euler equation that is exploited extensively in the theories of finance, growth, and
real business cycles.

3.2. Stochastic control problems

We now consider a modification of problem (3.1.1 ) to permit uncertainty. Essentially,
we add some well-placed shocks to the previous non-stochastic problem. So long as the
shocks are either independently and identically distributed or Markov, straightforward
modifications of the method for handling the nonstochastic problem will work.
Thus, we modify the transition equation and consider the problem of maximizing
∞


E0 β t r (xt , ut ) , 0 < β < 1, (3.2.1)
t=0

subject to
xt+1 = g (xt , ut , t+1 ) , (3.2.2)

with x0 known and given at t = 0 , where t is a sequence of independently and iden-

tically distributed random variables with cumulative probability distribution function
prob{t ≤ e} = F (e) for all t; Et (y) denotes the mathematical expectation of a ran-
dom variable y , given information known at t. At time t, xt is assumed to be known,
but xt+j , j ≥ 1 is not known at t. That is, t+1 is realized at (t + 1), after ut has
been chosen at t. In problem (3.2.1 )–(3.2.2 ), uncertainty is injected by assuming
that xt follows a random difference equation.
Problem (3.2.1 )–(3.2.2 ) continues to have a recursive structure, stemming jointly
from the additive separability of the objective function (3.2.1 ) in pairs (xt , ut ) and
from the difference equation characterization of the transition law (3.2.2 ). In partic-
ular, controls dated t affect returns r(xs , us ) for s ≥ t but not earlier. This feature
implies that dynamic programming methods remain appropriate.
The problem is to maximize expression (3.2.1 ) subject to equation (3.2.2 ) by
choice of a “policy” or “contingency plan” ut = h(xt ). The Bellman equation (3.1.5 )
becomes
V (x) = max{r (x, u) + βE [V [g (x, u, )] |x]}, (3.2.3)
u

where E{V [g(x, u, )]|x} = V [g(x, u, )]dF () and where V (x) is the optimal value
of the problem starting from x at t = 0 . The solution V (x) of equation (3.2.3 ) can
86 Chapter 3: Dynamic Programming

be computed by iterating on

Vj+1 (x) = max{r (x, u) + βE [Vj [g (x, u, )] |x]}, (3.2.4)

u

starting from any bounded continuous initial V0 . Under various particular regularity
conditions, there obtain versions of the same four properties listed earlier. 9
The first-order necessary condition for the problem on the right side of equation
(3.2.3 ) is  
∂r (x, u) ∂g 
+ βE (x, u, ) V [g (x, u, )] |x = 0,
∂u ∂u
which we obtained simply by differentiating the right side of equation (3.2.3 ), passing
the differentiation operation under the E (an integration) operator. Off corners, the
value function satisfies

 ∂r ∂g
V (x) = [x, h (x)] + βE [x, h (x) , ] V  (g [x, h (x) , ]) |x .
∂x ∂x

In the special case in which ∂g/∂x ≡ 0 , the formula for V  (x) becomes

∂r
V  (x) = [x, h (x)] .
∂x
Substituting this formula into the first-order necessary condition for the problem gives
the stochastic Euler equation
 
∂r ∂g ∂r
(x, u) + βE (x, u, ) (x̃, ũ) |x = 0,
∂u ∂u ∂x

where tildes over x and u denote next-period values.

9 See Stokey and Lucas (with Prescott) (1989), or the framework presented in the
appendix on functional analysis, chapter A.
 Exercise 87

3.3. Concluding remarks

This chapter has put forward basic tools and findings: the Bellman equation and
several approaches to solving it; the Euler equation; and the Beneveniste-Scheinkman
formula. To appreciate and believe in the power of these tools requires more words
and more practice than we have yet supplied. In the next several chapters, we put
the basic tools to work in different contexts with particular specification of return and
transition equations designed to render the Bellman equation susceptible to further
analysis and computation.

Exercise

Exercise 3.1 Howard’s policy iteration algorithm

Consider the Brock-Mirman problem: to maximize
∞


E0 β t ln ct ,
t=0

subject to ct + kt+1 ≤ Aktα θt ,k0 given, A > 0 , 1 > α > 0 , where {θt } is an i.i.d.
sequence with ln θt distributed according to a normal distribution with mean zero
and variance σ 2 .
Consider the following algorithm. Guess at a policy of the form kt+1 = h0 (Aktα θt )
for any constant h0 ∈ (0, 1). Then form
∞


J0 (k0 , θ0 ) = E0 β t ln (Aktα θt − h0 Aktα θt ) .
t=0

Next choose a new policy h1 by maximizing

ln (Ak α θ − k  ) + βEJ0 (k  , θ ) ,

where k  = h1 Ak α θ . Then form

∞


J1 (k0 , θ0 ) = E0 β t ln (Aktα θt − h1 Aktα θt ) .
t=0

Continue iterating on this scheme until successive hj have converged.

Show that, for the present example, this algorithm converges to the optimal
policy function in one step.