Lecture 5: The Bellman Equation

Florian Scheuer

1 Plan
• Prove properties of the Bellman equation

• (In particular, existence and uniqueness of solution)

• Use this to prove properties of the solution

• Think about numerical approaches

2 Statement of the Problem

V ( x ) = sup F ( x, y) + βV (y)
y

s.t. (1)

y ∈ Γ (x)

• Some terminology:

– The Functional Equation (1) is called a Bellman equation.

– x is called a state variable.
– G ( x ) = {y ∈ Γ ( x ) : V ( x ) = F ( x, y) + βV (y)} is called a policy correspondence.
It spells out all the values of y that attain the maximum in the RHS of (1).
– If G ( x ) is single-valued (i.e. there is a unique optimum), G is called a policy
function.

• Questions:

1. Does (1) have a solution?

2. Is it unique?
3. How do we find it?

1
3 The Bellman Equation as a Fixed-Point Problem
• Define the operator T by

T ( f ) ( x ) = sup F ( x, y) + β f (y)
y

s.t. y ∈ Γ (x)

• V can be defined as a fixed point of T, i.e. a function V such that T (V ) ( x ) = V ( x ) ∀x

• Does T have a fixed point? How do we find it?

Assumption 1. (Assumption 4.3 in SLP) X ⊆ Rn is convex. Γ : X ⇒ X is nonempty, compact-

valued and continuous.

Assumption 2. (Assumption 4.4 in SLP) F : X × X → R is continuous and bounded, i.e. ∃ F̄ such

that F ( x, y) < F̄ for all { x, y} with x ∈ X and y ∈ Γ( x ).

• What space is the operator T defined in?

• Define the metric space (S, ρ) by

S ≡ { f : X → R continuous and bounded} (2)

with the norm

k f k = sup | f ( x )|
x∈X

and thus the distance

ρ ( f , g) = k f − gk = sup | f ( x ) − g ( x )| (3)
x∈X

• Notice that T : S → S, i.e. if f is continuous and bounded, then g is continuous and

bounded.

• How do we know this?

1. T ( f ) ( x ) is continuous
– Recall Theorem of the Maximum:
Proposition 1. (Theorem of the Maximum). X ⊆ Rl and Y ⊆ Rm . f : X × Y → R
is continuous. Γ : X → Y is compact-valued and continuous. Then the function
h : X → R defined by
h( x ) = max f ( x, y)
y∈Γ( x )

2
 is continuous; and the correspondence G : X → Y defined by

G ( x ) = {y ∈ Γ( x ) : f ( x, y) = h( x )}

is non-empty, compact-valued and upper hemi-continuous.

– Applied to this problem:
∗ f ( x, y) becomes F ( x, y) + β f (y)
∗ h ( x ) becomes T ( f ) ( x )
2. This is because F is bounded (Assumption 2) and f is bounded.

• Now we want to show that T has a unique fixed point. Two steps:

1. Show that T is a contraction (Blackwell’s sufficient conditions hold)

2. Appeal to contraction mapping theorem

1. Blackwell’s sufficient conditions:

Proposition 2. (Blackwell’s sufficient conditions) X ⊆ Rl and B( X ) is the space of bounded

functions f : X → R, with the sup norm. T is a contraction with modulus β if:
a. [Monotonicity] f , g ∈ B( X ) and f ( x ) ≤ g( x ) for all x ∈ X ⇒ T ( f )( x ) ≤ T ( g)( x ) for all
x ∈ X;
b. [Discounting] There exists some β ∈ (0, 1) such that T ( f + a)( x ) ≤ T ( f )( x ) + βa for all
f ∈ B( X ), a ≥ 0, x ∈ X.

Proof. [You’ll see this in class]

These conditions hold in our problem because

(a) For any x, y

F ( x, y) + β f (y) ≤ F ( x, y) + βg (y)
sup F ( x, y) + β f (y) ≤ sup F ( x, y) + βg (y)
y∈Γ( x ) y∈Γ( x )

T ( f ) (y) ≤ T ( g) (y)

(b)

T ( f + a) ( x ) = sup F ( x, y) + β [ f (y) + a]
y∈Γ( x )

= T ( f ) ( x ) + βa

3
2. Contraction Mapping Theorem

Proposition 3. (Contraction Mapping Theorem). If (S, ρ) is a complete metric space and

T : S → S is a contraction mapping with modulus β, then:
a. T has exactly one fixed point V in S;
b. For any V0 ∈ S, ρ( T n V0 , V ) ≤ βn ρ(V0 , V ), n = 0, 1, 2, ....

Proof.

(a) Let Vn = T n V0 . Since T is a contraction

ρ (V2 , V1 ) = ρ ( TV1 , TV0 ) ≤ βρ (V1 , V0 )

...
ρ (Vn+1 , Vn ) ≤ βn ρ (V1 , V0 )

Therefore for any m > n

ρ (Vm , Vn ) ≤ ρ (Vm , Vm−1 ) + ρ (Vm−1 , Vm−2 ) + · · · + ρ (Vn+1 , Vn )

h i
≤ βm−1 + βm−2 + . . . βn ρ (V1 , V0 )
βn
≤ ρ (V1 , V0 )
1−β

so {Vn } is a Cauchy sequence. Since S is complete, {Vn } must converge to some V ∈ S.

We must now show that V is indeed a fixed point.

ρ ( TV, V ) ≤ ρ ( TV, T n V0 ) + ρ ( T n V0 , V ) ∀n
 
n −1
≤ βρ V, T V0 + ρ ( T n V0 , V ) ∀n

but both of these terms go to zero as n → ∞ which implies that ρ ( TV, V ) = 0. Finally,
we need to show that there is no other fixed point. Suppose there is another fixed point V 0 .
Then
ρ V, V 0 = ρ TV, TV 0 ≤ βρ V, V 0

(the first step is because they are both fixed points). This implies ρ (V, V 0 ) = 0.
(b)
   
ρ ( T n V0 , V ) = ρ TT n−1 V0 , TV ≤ βρ T n−1 V0 , V

≤ ...
≤ βn ρ(V0 , V )

4
 • The only missing step is to show that (S, ρ) defined by (2) and (3) indeed constitutes a
complete metric space. (SLP Thm 3.1). Notice that if we used
Z
ρ ( f , g) = | f ( x ) − g ( x )| dx

then (S, ρ) would NOT be a complete metric space. (SLP exercise 3.6.a.).

Proposition 4. (SLP 4.6) If Assumptions 1 and 2 hold, then T has a unique fixed point in S, i.e.
there is a unique continuous bounded function that solves (1).

Proof. From Contraction Mapping Theorem, knowing that Blackwell’s sufficient conditions
are met.

Proposition 5. The policy correspondence G ( x ) = {y ∈ Γ ( x ) : V ( x ) = F ( x, y) + βV (y)} is

compact-valued and u.h.c.

Proof. From the Theorem of the Maximum

Proposition 6. If Assumptions 1 and 2 hold, then V is the value function of the sequence problem.

Proof. V solves (1) and, because V is bounded, then limT →∞ β T V ( x T ) = 0 ∀ x̃ ∈ Π ( x0 ) , ∀ x0 ∈

X, so the sufficient conditions for Theorem SLP 4.3 hold.

4 Proving Properties of V
Basic structure of these results: proving that the fixed point belongs to some subset of S0 ⊂ S,
i.e. the set of functions that meet certain criteria.

Proposition 7. If (S, ρ) is a complete metric space and S0 is a closed subset of S, then S0 is a complete
metric space

Proof. SLP Exercise 3.6.b.

Proposition 8. (SLP Corollary 1, page 52). Let (S, ρ) be a complete metric space and T : S → S be
a contraction mapping with fixed point V ∈ S.

1. If S0 is a closed subset of S and T ( f ) ∈ S0 for all f ∈ S0 , then V ∈ S0

2. If in addition S00 ⊆ S0 and T ( f ) ∈ S00 for all f ∈ S0 , then V ∈ S00

5
Proof.

1. Choose V0 ∈ S0 . T n (V ) is a sequence in S0 converging to V. Since S0 is closed, V ∈ S0 .

2. Since V ∈ S0 , then T (V ) ∈ S00 . But T (V ) = V so V ∈ S00

• Example:

– S: all continuous functions f : [ a, b] → R

– S0 : all increasing functions f : [ a, b] → R
– S00 : all strictly increasing functions f : [ a, b] → R

• Note: we require the subset S0 to be closed but not the sub-subset S00

• In our example:

1. If T maps increasing functions into increasing functions, then the fixed point must
be an increasing function
2. If T maps increasing functions into strictly increasing functions, then the fixed
point must be a strictly increasing function

– What the result does not say is that if T maps strictly increasing functions into
strictly increasing functions, then the fixed point must be a strictly increasing
function (because the set of strictly increasing functions is not closed)

4.1 V increasing
Assumption 3. (Assumption 4.5 in SLP) F ( x, y) is strictly increasing in x.
Assumption 4. (Assumption 4.6 in SLP) x ≤ x 0 implies Γ( x ) ⊆ Γ( x 0 )

• Do Assumptions (3) and (4) hold in the Neoclassical model?

Proposition 9. (SLP 4.7). Suppose Assumptions (1)-(4) hold. Then V is strictly increasing.

Proof. Let x 0 > x and f ∈ S.

T ( f ) ( x ) = max F ( x, y) + β f (y)
y∈Γ( x )

≤ max F ( x, y) + β f (y)
y∈Γ( x 0 )

< max F x 0 , y + β f (y)

y∈Γ( x 0 )

= T ( f ) x0

This implies that T maps any continuous bounded function into a strictly increasing func-
tion. Proposition 8 gives the result.

6
4.2 V concave
Assumption 5. (Assumption 4.7 in SLP) F ( x, y) is strictly concave.

Assumption 6. (Assumption 4.8 in SLP) Γ is convex

Proposition 10. (SLP 4.8) Suppose Assumptions (1), (2), (5) and (6) hold. Then V is strictly
concave and G is continuous and single-valued.

Proof. We want to show that T maps concave functions into strictly concave functions. Strict
concavity of V follows by Proposition 8.

• Let x0 6= x1 and xθ = θx0 + (1 − θ ) x1 for θ ∈ (0, 1).

• Let y0 ∈ Γ ( x0 ) be such that T ( f ) ( x0 ) = F ( x0 , y0 ) + β f (y0 ) and similarly y1 ∈ Γ ( x1 )

be such that T ( f ) ( x1 ) = F ( x1 , y1 ) + β f (y1 )

• Then:

T ( f ) ( xθ ) ≥ F ( xθ , yθ ) + β f (yθ )
(Γ convex makes xθ , yθ feasible)
> [θF ( x0 , y0 ) + (1 − θ ) F ( x1 , y1 )] + β [θ f (y0 ) + (1 − θ ) f (y1 )]
( f concave and F strictly concave)
= θ [ F ( x0 , y0 ) + β f (y0 )] + (1 − θ ) [ F ( x1 , y1 ) + β f (y1 )]
(rearranging)
= θT ( f ) ( x0 ) + (1 − θ ) T ( f ) ( x1 )
(by assumption)

• G single-valued follows from strict concavity

• G continuous follows from Theorem of the Maximum

5 Is V differentiable? (Benveniste & Scheinkman, 1979)

• Cannot use same proof technique:

– Space of differentiable functions is not closed

– T does not necessarily map f into a differentiable function

• Instead, rely on the following result

7
Proposition 11. Suppose V : X → R is concave. Let x0 be an interior point and D be a neighbor-
hood around x0 . Suppose exists concave differentiable w : D → R such that:
1. w ( x ) ≤ V ( x )

2. V ( x0 ) = w ( x0 )

3. w is differentiable at x0
Then V is differentiable at x0

Proof. Any subgradient p of V at x0 must satisfy;

p · ( x − x0 ) ≥ V ( x ) − V ( x0 ) ≥ w ( x ) − V ( x0 ) = w ( x ) − w ( x0 )

• The first step is because a subgradient of a concave function is always (by definition)
above the function

• the second step is because w ( x ) ≤ V ( x )

• the last step is because w ( x0 ) = V ( x0 )

Since w is differentiable, then p is unique, which implies V differentiable.
• (Graph)

• This result is useful to establish the following:

Proposition 12. (SLP 4.11) Suppose Assumptions (1), (2), (5) and (6) hold and F is continuosly
differentiable. Then V is differentiable and

Vi ( x0 ) = Fi ( x0 , g ( x0 ))

Proof. Define
w ( x ) = F ( x, g ( x0 )) + βV ( g ( x0 ))
• w is concave, differentiable and satisfies

w ( x ) = F ( x, g ( x0 )) + βV ( g ( x0 ))
⇒ w ( x ) ≤ max F ( x, y) + βV (y)
y∈Γ( x )

= V (x)

and
w ( x0 ) = V ( x0 )

• The result then follows from (11)