0% found this document useful (0 votes)
2 views

3_DynamicProgramming

Uploaded by

Kenji Moreno
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
2 views

3_DynamicProgramming

Uploaded by

Kenji Moreno
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 46

Macroeconomic Analysis I

Dynamic Programming 1

Soojin Kim
Georgia State

1
These lecture slides are based on Dirk Krueger’s manuscript.
Soojin Kim Dynamic Programming Macro I 1 / 37
Functional Equations

Consider the Functional Equation

v(x) = max F (x, y) + βv(y) (1)


y∈Γ(x)

Define an operator T :

(T v)(x) = max F (x, y) + βv(y) (2)


y∈Γ(x)

A solution to the functional equation: v ∗ = T v ∗

Soojin Kim Dynamic Programming Macro I 2 / 37


Goals
1. Solving the Functional Equation

• Existence? Uniqueness? Convergence?

Yes to all, if it is a “contraction.”


Contraction Mapping Theorem [CMT]

• When is it a contraction?

If it satisfies “certain conditions.”


Blackwell’s Theorem

Proof Summary:
The functional equation satisfies “the” conditions.
By the Blackwell’s Theorem, it is a contraction.
By the CMT, the solution exists and is unique.
Soojin Kim Dynamic Programming Macro I 3 / 37
Sequential Problem and Functional Equation

Consider the Sequential Problem (SP )


X
w(x0 ) = sup β t F (xt , xt+1 )
{xt+1 }∞
t=1 t=0

s.t. xt+1 ∈ Γ(xt ), x0 ∈ X given

and the Functional Equation (F E)

v(x) = sup F (x, y) + βv(y)


y∈Γ(x)

Soojin Kim Dynamic Programming Macro I 4 / 37


Goals

2. Principle of Optimality

• When are the functions v and w equivalent?

• When are the optimal solutions to the problems equivalent?

Under the Equivalence between (SP ) and (F E),


we can play with the recursive formulation.

3. Application: Dynamic Programming with Bounded Returns

Soojin Kim Dynamic Programming Macro I 5 / 37


The Roadmap
1 Mathematical Preliminaries

2 Contraction Mapping Theorem & Blackwell’s Theorem

3 Theorem of the Maximum

4 Principle of Optimality

5 Dynamic Programming with Bounded Returns

Soojin Kim Dynamic Programming Macro I 6 / 37


The Roadmap
1 Mathematical Preliminaries

2 Contraction Mapping Theorem & Blackwell’s Theorem

3 Theorem of the Maximum

4 Principle of Optimality

5 Dynamic Programming with Bounded Returns

Soojin Kim Dynamic Programming Macro I 7 / 37


Contraction Mapping Theorem, a Preview

Definition
Let (S, d) be a metric space and T : S → S be a function mapping S into itself. The
function T is a contraction mapping if there exists a number β ∈ (0, 1) satsifying

d(T x, T y) ≤ βd(x, y), ∀x, y ∈ S.

The number β is called the modulus of the contraction mapping.

Theorem (Contraction Mapping Theorem)


Let (S, d) be a complete metric space and suppose that T : S → S is a contraction mapping
with modulus β. Then, (a) the operator T has exactly one fixed point v ∗ ∈ S and (b) for
any v0 ∈ S and any n ∈ N, we have d(T n v0 , v ∗ ) ≤ β n d(v0 , v ∗ ).

Soojin Kim Dynamic Programming Macro I 8 / 37


Metric Space

Definition
A metric space is a set S and a function d : S × S → R such that for all x, y, z ∈ S
1 d(x, y) ≥ 0
2 d(x, y) = 0 if and only if x = y
3 d(x, y) = d(y, x) (symmetry)
4 d(x, z) ≤ d(x, y) + d(y, z) (triangle inequality)

• d: metric used to measure the distance between two elements in S

Example.
• S = R, d(x, y) = |x − y|

Soojin Kim Dynamic Programming Macro I 9 / 37


Metric Space, An (Important) Example

Let X ⊆ Rl and S = C(X) be the set of all continuous and bounded functions f : X → R.
Define the metric d : C(X) × C(X) → R as d(f, g) = supx∈X |f (x) − g(x)|. Then (S, d) is a
metric space.

• The operator of our interest T uses as inputs continuous and bounded functions.

• The space C(X) with norm d will be the domain of our operator T .

Soojin Kim Dynamic Programming Macro I 10 / 37


Convergence of Sequences

Definition
Let (S, d) be a metric space. A sequence {xn }∞
n=0 with xn ∈ S for all n is said to
converge to x ∈ S, if for every ε > 0, there exists a Nε ∈ N such that d(xn , x) < ε for all
n ≥ Nε . In this case, we write limn→∞ xn = x.

Definition
A sequence {xn }∞ n=0 with xn ∈ S for all n is said to be a Cauchy sequence, if for each
ε > 0, there exists a Nε ∈ N such that d(xn , xm ) < ε for all n, m ≥ Nε .

Example.
• Let S = R and d(x, y) = |x − y|. Consider a sequence xn = 1/n. This sequence
converges and is a Cauchy sequence.

Soojin Kim Dynamic Programming Macro I 11 / 37


Complete Metric Space

Theorem
Suppose that (S, d) is a metric space and that the sequence {xn }∞
n=0 converges to x ∈ S.
Then the sequence {xn }∞ n=0 is a Cauchy sequence.

Definition
A metric space (S, d) is complete if every Cauchy sequence {xn }∞
n=0 with xn ∈ S for all n
converges to some x ∈ S.

• Note that the limit x needs to lie in S.


• CMT requires (S, d) to be a complete metric space.

Let X ⊆ RL and C(X) be the set of all bounded continuous functions f : X → R with d
being the sup-norm. Then (C(X), d) is a complete metric space.
Soojin Kim Dynamic Programming Macro I 12 / 37
The Roadmap
1 Mathematical Preliminaries

2 Contraction Mapping Theorem & Blackwell’s Theorem

3 Theorem of the Maximum

4 Principle of Optimality

5 Dynamic Programming with Bounded Returns

Soojin Kim Dynamic Programming Macro I 13 / 37


Contraction Mapping

Definition
Let (S, d) be a metric space and T : S → S be a function mapping S into itself. The
function T is a contraction mapping if there exists a number β ∈ (0, 1) satsifying

d(T x, T y) ≤ βd(x, y), ∀x, y ∈ S.

The number β is called the modulus of the contraction mapping.

• A function that is a contraction mapping is automatically a continuous function.

Soojin Kim Dynamic Programming Macro I 14 / 37


Contraction Mapping Theorem

• Let vn = T n v0 ∈ S be the element in S that is obtained by applying the operator T


n-times to v0 .

Theorem (Contraction Mapping Theorem)


Let (S, d) be a complete metric space and suppose that T : S → S is a contraction mapping
with modulus β. Then,

(a) the operator T has exactly one fixed point v ∗ ∈ S; and

(b) for any v0 ∈ S and any n ∈ N, we have

d(T n v0 , v ∗ ) ≤ β n d(v0 , v ∗ ).

Exercise 3-1/ Prove the Contraction Mapping Theorem.

Soojin Kim Dynamic Programming Macro I 15 / 37


Blackwell’s Theorem

How do we know that a function is a contraction mapping?

Theorem (Blackwell’s Theorem)


Let X ⊆ RL and B(X) be the space of bounded functions f : X → R with the d being the
sup-norm. Let T : B(X) → B(X) be an operator satisfying
1 Monotonicity: If f, g ∈ B(X) are such that f (x) ≤ g(x) for all x ∈ X, then
(T f )(x) ≤ (T g)(x) for all x ∈ X
2 Discounting: Let the function f + a for f ∈ B(X) and a ∈ R+ be defined by
(f + a)(x) = f (x) + a. There exists β ∈ (0, 1) such that for all f ∈ B(X), a ≥ 0 and
for all x ∈ X
[T (f + a)](x) ≤ [T f ](x) + βa.
If these two conditions are satisfied, then the operator T is a contraction with modulus β.

Soojin Kim Dynamic Programming Macro I 16 / 37


Neoclassical Growth Model

T v(k) = max U (f (k) − k 0 ) + βv(k 0 )


0≤k0 ≤f (k)

• Metric space: B([0, ∞), d), the space of bounded functions on [0, ∞) with d, the
sup-norm.

• Blackwell’s conditions

1 Check T : B([0, ∞)) → B([0, ∞))

2 Monotonicity: Suppose v(y) ≤ w(y), ∀y...

3 Discounting: T (v + a)(k) = max U (f (k) − k 0 ) + β(v(k 0 ) + a)


0≤k0 ≤f (k)

Exercise 3-2/ Show that under Assumptions 1 and 2 (from before), there exists a unique
fixed point to the functional equation in the Neoclassical Growth model.
Soojin Kim Dynamic Programming Macro I 17 / 37
The Roadmap
1 Mathematical Preliminaries

2 Contraction Mapping Theorem & Blackwell’s Theorem

3 Theorem of the Maximum

4 Principle of Optimality

5 Dynamic Programming with Bounded Returns

Soojin Kim Dynamic Programming Macro I 18 / 37


Behavior of Policy Functions
Consider
T v(x) = sup F (x, y) + βv(y),
y∈Γ(x)

and define
G(x) = {y ∈ Γ(x) : F (x, y) + βv(y) = T v(x)}.

• G(x): the set of choices y that attain the maximum given x

• G(x) need not be single-valued.

Goals

When do we have: if v ∈ C(X) then (T v)(x) ∈ C(X)?

What are the properties of G(x)?


Soojin Kim Dynamic Programming Macro I 19 / 37
A General Problem

Consider

h(x) = max f (x, y),


y∈Γ(x)

and define

G(x) = {y ∈ Γ(x) : f (x, y) = h(x)}.

Soojin Kim Dynamic Programming Macro I 20 / 37


A Few Definitions on Correspondence

Definition
A correspondence Γ : X → Y is lower hemi-continuous (l.h.c) at x if Γ(x) is nonempty
and if, for every y ∈ Γ(x) and every sequence xn → x, there exists N ≥ 1 and a sequence
{yn }∞
n=N such that yn → y and yn ∈ Γ(xn ), all n ≥ N .

Definition
A compact-valued correspondence Γ : X → Y is upper hemi-continuous (u.h.c) at x if
Γ(x) is nonempty and if, for every sequence xn → x and every sequence {yn } such that
yn ∈ Γ(xn ) for all n, there exists a convergent subsequence of {yn } whose limit point y is
in Γ(x).

Definition
A correspondence Γ : X → Y is continuous if it is both upper hemi-continuous and lower
hemi-continuous.
Soojin Kim Dynamic Programming Macro I 21 / 37
Theorem of the Maximum

Theorem (Theorem of the Maximum)


Let X ⊆ RL and Y ⊆ RM , let f : X × Y → R be a continuous function, and let Γ : X → Y
be a compact-valued and continuous correspondence. Then the function h : X → R is
continuous, and the correspondence G : X → Y is nonempty, compact-valued, and u.h.c.

Note: A single-valued correspondence (i.e., a function) that is upper hemi-continuous is


continuous.

Soojin Kim Dynamic Programming Macro I 22 / 37


The Roadmap
1 Mathematical Preliminaries

2 Contraction Mapping Theorem & Blackwell’s Theorem

3 Theorem of the Maximum

4 Principle of Optimality

5 Dynamic Programming with Bounded Returns

Soojin Kim Dynamic Programming Macro I 23 / 37


Sequential Problem and Functional Equation

Consider the Sequential Problem (SP )


X
w(x0 ) = sup β t F (xt , xt+1 ) (3)
{xt+1 }∞
t=1 t=0

s.t. xt+1 ∈ Γ(xt ), x0 ∈ X given

and the Functional Equation (F E)

v(x) = sup F (x, y) + βv(y) (4)


y∈Γ(x)

Soojin Kim Dynamic Programming Macro I 24 / 37


A Few Building Blocks

• X, set of possible values of the state variable

• Γ : X → X, correspondence describing the feasibility constraints

• A = {(x, y) ∈ X × X : y ∈ Γ(x)}, graph of Γ

• F : A → R, return function

• β ≥ 0, discount factor

• Π(x0 ), set of feasible plans, i.e., Π(x0 ) = {{xt }∞


t=0 : xt+1 ∈ Γ(xt )}

• x̄, a generic element of Π(x0 )

Soojin Kim Dynamic Programming Macro I 25 / 37


Principle of Optimality

Assume:
A0. (a) Γ(x) is nonempty for all x ∈ X
n
X
(b) For all x0 and x̄ ∈ Π(x0 ), lim β t F (xt , xt+1 ) exists
n→∞
t=0
(may be ±∞)

Sufficient condition for A0:

• F is bounded above or below and 0 < β < 1; or

• For each x0 ∈ X and x ∈ Π(x0 ), there exist θ ∈ (0, β −1 ) and 0 < c < ∞ such that
F (xt , xt+1 ) ≤ cθt , ∀t.

Soojin Kim Dynamic Programming Macro I 26 / 37


Principle of Optimality
Define un : Π(x0 ) → R as
n
X
un (x̄) = β t F (xt , xt+1 ).
t=0

Then u(x̄) = lim un (x̄) is well-defined and w(x0 ) = sup u(x̄).


n→∞ x̄∈Π(x0 )

Theorem (Values)
Suppose (X, Γ, F, β) satisfy A0. Then,
1 the function w satisfies the functional equation (F E)
2 if for all x0 ∈ X and all x̄ ∈ Π(x0 ), a solution v to the functional equation (F E)
satisfies
lim β n v(xn ) = 0,
n→∞

then v = w.
Soojin Kim Dynamic Programming Macro I 27 / 37
Principle of Optimality

Theorem (Policies)
Suppose (X, Γ, F, β) satisfy A0.
1 Let x̄ ∈ Π(x0 ) be a feasible plan that attains the supremum in the sequential problem.
Then for all t ≥ 0,
w(x̄t ) = F (x̄t , x̄t+1 ) + βw(x̄t+1 ).
2 Let x̂ ∈ Π(x0 ) be a feasible plan satisfying for all t ≥ 0,

w(x̂t ) = F (x̂t , x̂t+1 ) + βw(x̂t+1 ),

and additionally
lim sup β t w(x̂t ) ≤ 0.
t→∞

Then x̂ attains the supremum in (SP ) for the initial condition x0 .

Soojin Kim Dynamic Programming Macro I 28 / 37


The Roadmap
1 Mathematical Preliminaries

2 Contraction Mapping Theorem & Blackwell’s Theorem

3 Theorem of the Maximum

4 Principle of Optimality

5 Dynamic Programming with Bounded Returns

Soojin Kim Dynamic Programming Macro I 29 / 37


The Problem

v(x) = max F (x, y) + βv(y) (5)


y∈Γ(x)

Assume:
A1. X is a convex subset of Rl and the correspondence Γ : X → X is nonempty,
compact-valued, and continuous.

A2. The function F : A → R is bounded and continuous, and 0 < β < 1.

Note:

• Under A1 and A2, A0 holds.

• Principle of Optimality holds.


Soojin Kim Dynamic Programming Macro I 30 / 37
Plan
Let

G(x) = {y ∈ Γ(x) : v(x) = F (x, y) + βv(y)}; (6)


(T v)(x) = max F (x, y) + βv(y) (7)
y∈Γ(x)

We want to show (under some assumptions)

• T has a unique fixed point.

• Policy correspondence G is nonempty and u.h.c.

• v is strictly increasing.

• v is strictly concave and G is a continuous function.

• v is continuously differentiable.
Soojin Kim Dynamic Programming Macro I 31 / 37
A Useful Corollary to the CMT
Theorem (Contraction Mapping Theorem)
Let (S, d) be a complete metric space and suppose that T : S → S is a contraction mapping
with modulus β. Then,

(a) the operator T has exactly one fixed point v ∗ ∈ S; and

(b) for any v0 ∈ S and any n ∈ N, we have

d(T n v0 , v ∗ ) ≤ β n d(v0 , v ∗ ).

Corollary
Let (S, d) be a complete metric space, and let T : S → S be a contraction mapping with
fixed poitn v ∈ S.
If S 0 is a closed subset of S and T (S 0 ) ⊆ S 0 , then v ∈ S 0 .
If in addition T (S 0 ) ⊆ S 00 ⊆ S 0 , then v ∈ S 00 .
Soojin Kim Dynamic Programming Macro I 32 / 37
Theorem

Theorem (Existence, Uniqueness, Policy Continuity)


Let (X, Γ, F, β) satisfy A1 and A2, and let C(X) be the space of bounded and continuous
functions f : X → R, with the sup norm. Then the operator T maps C(X) into itself,
T : C(X) → C(X); T has a unique fixed point v ∈ C(X); and for all v0 ∈ C(X),

kT n v0 − vk ≤ β n kv0 − vk, n = 0, 1, 2, ...

Moreover, given v, the optimal policy correspondence G : X → X defined by (6) is


compact-valued and u.h.c.

Exercise 3-3/ Prove this theorem.

Soojin Kim Dynamic Programming Macro I 33 / 37


Monotonicity of v

Assume:
A3. For each y, F (·, y) is strictly increasing in each of its first l arguments.

A4. Γ is monotone in the sense that x ≤ x0 implies Γ(x) ⊆ Γ(x0 ).

Theorem (Monotonicity)
Let (X, Γ, F, β) satisfy A1 through A4, and let v be the unique solution to the (5). Then, v
is strictly increasing.

Exercise 3-4/ Prove this theorem.

Soojin Kim Dynamic Programming Macro I 34 / 37


Concavity of v and Policy
Assume:
A5. F is strictly concave; that is ∀(x, y), (x0 , y 0 ) ∈ A and ∀θ ∈ (0, 1),
F (θ(x, y) + (1 − θ)(x0 , y 0 )) ≥ θF (x, y) + (1 − θ)F (x0 , y 0 )
and the inequality is strict if x 6= x0 .

A6. Γ is convex; that is for any 0 ≤ θ ≤ 1 and x, x0 ∈ Γ,


y ∈ Γ(x), y 0 ∈ Γ(x0 ) implies θy + (1 − θ)y 0 ∈ Γ(θx + (1 − θ)x0 ).

Theorem (Concavity and Policy)


Let (X, Γ, F, β) satisfy A1, A2, A5, and A6; let v satisfy (5); and let G satisfy (6). Then
v is strictly concave and G is a continuous, single-valued function.

Exercise 3-5/ Prove this theorem.


Soojin Kim Dynamic Programming Macro I 35 / 37
Benveniste and Scheinkman

Theorem (Benveniste and Scheinkman)


Let X ⊆ Rl be a convex set, let V : X → R be concave, let x0 ∈ int(X), and let D be a
neighborhood of x0 . If there is a concave, differentiable function W : D → R with
W (x0 ) = V (x0 ) and with W (x) ≤ V (x) for all x ∈ D, then V is differentiable at x0 and

Vi (x0 ) = Wi (x0 ), i = 1, 2, ..., l,

where Vi (x0 ) denotes the derivative of V with respect to its i-th component, evaluated at x0

Soojin Kim Dynamic Programming Macro I 36 / 37


Differentiability of v

Assume:
A7. F is continuously differentiable on the interior of A.

Theorem (Differentiability)
Let (X, Γ, F, β) satisfy A1, A2, A5, A6, and A7. If x0 ∈ int(X) and g(x0 ) ∈ int(Γ(x0 )),
then the unique fixed point of T , v is continuously differentiable at x0 with

vi (x0 ) = Fi (x0 , g(x0 )) i = 1, 2, ..., l.

Exercise 3-6/ Prove this theorem.

Soojin Kim Dynamic Programming Macro I 37 / 37

You might also like