Dynamic Programming: Quantitative Macroeconomics (Econ 5725)
Dynamic Programming: Quantitative Macroeconomics (Econ 5725)
Raül Santaeulàlia-Llopis
Spring 2011
3 Roadmap
• Environment is stochastic
Q (z 0 , z) = Pr (zt+1 ≤ z 0 |zt = z)
with z0 given.
T
X
E0 β t u(xt , ct )
t=0
where
xt+1 = f (xt , zt , ct )
with x0 given.
gt : X × Z → C , ∀t
T
X
WT (x0 , z0 , πT ) = E0 β t u(xt , gt (xt , zt ))
t=0
where
An individual maximizes,
max WT (x0 , z0 , πT )
gt (xt ,zt )∈C (xt ,zt )
subject to
xt+1 = f (xt , zt , gt (xt , zt ))
given
x0 , z0 and Q(z 0 , z)
then
• there exists a solution (optimal policy) to the problem above,
πT∗ = {g0∗ , g1∗ , ..., gT∗ }, and
∗ ) is also continuous.
• the value function VT (x0 , z0 ) = WT (x0 , z0 , πT
T
X
VT (x0 , z0 ) = E0 β t u(xt , gt∗ (xt , zt ))
t=0
Corollary: If C (xt , zt ) is convex and u(·) and f (·) are strictly concave in
ct , then gt (xt , zt ) is also continuous.
T
X
VT (x0 , z0 ) = max E0 {u(x0 , c0 ) + β t u(xt , ct )}
πT
t=1
T
X
VT (x0 , z0 ) = max E0 {u(x0 , c0 ) + E1 β t u(xt , ct )}
πT
t=1
T
X
VT (x0 , z0 ) = max E0 {u(x0 , c0 ) + max E1 β t u(xt , ct )}
c0 πT −1
t=1
T
X
VT (x0 , z0 ) = max E0 {u(x0 , c0 ) + β max E1 β t−1 u(xt , ct )}
c0 πT −1
t=1
= max E0 {u(x0 , c0 ) + β max WT −1 (x1 , z1 , πT −1 )}
c0 πT −1
• then, after j periods with j + s = T , πs∗ = {gT∗ −s , gT∗ −s+1 , ..., gT∗ } is
the optimal policy (with elements identical to the original one) that
maximizes Ws (xj , zj , πs ).
• First, start from the last period, with 0 periods to go (s=0). Then
the problem is static and reads:
R
β Z VT (f (xT −1 , zT −1 , cT −1 ), zT )dQ(zT , zT −1 )
T → ∞.
Important consequences:
In the infinite horizon case, we can write the Bellman’s equation as:
Z
V (x, z) = max u(x, c) + β V (f (x, z, c), z 0 )dQ(z 0 , z) (2)
c∈C (x,z) Z
where T : C → C.
1 d(ϕ, φ) = 0 ⇐⇒ ϕ = φ
2 d(ϕ, φ) = d(φ, ϕ)
That is, a contraction contracts two points so that their images, T (ϕ) and
T (φ), are closer together than ϕ and φ
1 (monotonicity) ϕ ≤ φ → T ϕ ≤ T φ, ∀ϕ, φ ∈ M
2 (discounting) T (a + ϕ) ≤ aβ + T ϕ, ∀a > 0, ϕ ∈ M
Two results:
For the deterministic case (see SLP chapter 9 for stochastic environments),
If SLP 4.8 holds (i.e., the decision rule is continuous), we can find the optimal
decision rule function using some continuous method of approximation — this
means we may do not need to solve the decision rule at a lot of points. If SLP 4.8
does not hold, there is not guarantee the decision rule can be approximated with
a continuous function, and we will have to use discretization — takes more time.
It is always nice to have continuity in the optimal decision rule and strict
concavity of the value function because then it is easier to search for optimal
decisions. However, many interesting models do not have such property:
• Models with convex costs of adjustment (e.g. fixed cost of changing the size
of the house).
• Models with non-trivial tax schedule functions.
We can characterize the optimal policy function with the usual methods of
calculus, i.e. by differentiation, like in the static case or in the optimal
control problem in continuous time.
Let V be a concave function defined on the set X , let x0 ∈ X , and let N(x0 ) be a
neighborhood of x0 . If there is a concave differentiable function Ω : N(x0 ) → <
such that Ω(x) ≤ V (x), ∀x ∈ N(x0 ) with the equality holding at x0 , then V is
differentiable at x0 and Vx x0 = Ωx x0 .
Vx 0 (x 0 , z 0 ) = ux 0 (x 0 , g ∗ (x 0 , z 0 ))
which tells us that the derivative of the value function is the partial
derivative of the utility function with respect to the state variable
evaluated at the optimal value for the control.
PT −1
≥ limT →∞ { t=0 β t ux 0 (xt∗ , xt+1
∗
∗
D ) − βux 0 (xt , xt+1 ) (xt+1 − xt+1 )
• First, our object of interest are functions, we deal with functional equations.
We will learn how to approximate functions in a nice way that a computer
can store and handle.
• Third, we will review how to find roots to solve (systems of) non-linear
equations.
Before going into numerical techniques, we will have a quick review at how we
can look at aggregate and survey data in next class.