Optimization in Economic Theory

PD Dr. Johannes Paha

University of Hohenheim

winter term 2021/22

[email protected] Optimization in Economic Theory (winter term 2021/22) 1 / 23

 Chapter 10 – Dynamic Programming

Chapter 10
Dynamic Programming

Jan 12 and 19, 2022

Optimization in Economic Theory

PD Dr. Johannes Paha
winter term 2021/22

[email protected] Optimization in Economic Theory (winter term 2021/22) 2 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Overview

Until now we only considered static problems.

In reality, current choices have repercussions on future outcomes.
We will study a method that can be used to solve dynamic problems.
In general, there are more methods available (see the book by Dixit).
In case of discrete time modeling,
dynamic programming is often the most convenient method.
In case of continuous time modeling, it is possible to differentiate with respect to
time, so that the Maximum Principle is more handy.
Despite the methodological and conceptual differences, the results are remarkably
similar to the discrete time case.
Background reading: Simon and Blume (1994, Ch. 24 and 25)

[email protected] Optimization in Economic Theory (winter term 2021/22) 3 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Learning goals

Students
can explain the difference between continuous and discrete time,
can solve a two-period and a multi-period consumption problem in discrete time,
can define a Bellman equation and solve a dynamic programming problem by
applying a three-step procedure,
can solve problems from optimal control theory.

[email protected] Optimization in Economic Theory (winter term 2021/22) 4 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Preliminaries

We can conceptualize the evolution of time in two ways:

Continuous time

t0 t

Discrete time

t=0 t=1 t=2 t=3 t=4 t=5 t=6

[email protected] Optimization in Economic Theory (winter term 2021/22) 5 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Preliminaries

Consider the plot of a function over continuous and over discrete time (e.g., GDP
or population, etc.).
In continuous time the function is differentiable with respect to time, in discrete
time it is not.

[email protected] Optimization in Economic Theory (winter term 2021/22) 6 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Preliminaries

If a variable X depends on time and time evolves continuously,

we write X (t).
We can differentiate this variable with respect to time and get its evolution (a dot
over a variable denotes the time derivative):
dX (t) X (t + ∆) − X (t)
Ẋ (t) = = lim
dt ∆→0 ∆
In discrete time, the derivative does not exist because ∆ = 1 is the smallest value
that ∆ can attain.
Instead, we have that the change of a variable between time t and t + 1 is given
by the difference of the levels:
∆X = Xt+1 − Xt

[email protected] Optimization in Economic Theory (winter term 2021/22) 7 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Growth Rates

Denote the growth rate of X by γX .

The growth rate in continuous time is given by:
Ẋ (t)
γX =
X (t)
The growth rate in discrete time is given by:
Xt+1 − Xt
γX =
Xt
We have the following approximation:
Ẋ (t) Xt+1 − Xt
≈
X (t) Xt
| {z } | {z }
continuous time discrete time
The approximation gets better the closer to zero the growth rate and the smaller
the discrete time step (e.g., quarterly vs. yearly data).

[email protected] Optimization in Economic Theory (winter term 2021/22) 8 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Growth Rates

If a variable X grows at a constant rate,

it is said to grow exponentially:
Ẋ (t)
= γX , X (0) = X0 ⇒ X (t) = X0 e γX ·t ,
X (t)
Xt+1 − Xt
= γX Xt=0 = X0 ⇒ Xt = X0 (1 + γX )t .
Xt
Note that, due to the chain rule, we can also compute the growth rate of a
variable in continuous time by taking the logarithm and time-differentiating:
d ln[X (t)] Ẋ (t)
= .
dt X (t)

[email protected] Optimization in Economic Theory (winter term 2021/22) 9 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Continuous time – Solving differential equations

Simplest case (homogeneous linear first-order differential equation with constant

coefficient): Ẋ (t) = gX (t)
Divide both sides by X (t) and integrate with respect to t:
Ẋ (t)
= g,
X (t)
Z Z
Ẋ (t)
dt = g dt,
X (t)
ln [X (t)] + c0 = gt + c1 , (1)
where c0 and c1 are constants of integration.

[email protected] Optimization in Economic Theory (winter term 2021/22) 10 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Continuous time – Solving differential equations

Exponentiation yields the general solution

e ln[X (t)]+c0 = e gt+c1
X (t) = ce gt ,
where c = ±e c1 −c0 is the constant of integration.
Assume that X (0) = X0 , then c = X0 because X (0) = ce g·0 and, therefore, the
particular solution of the differential equation pins down to

X (t) = X0 e gt .
Note that g is the growth rate of X , which we denoted by gX .

[email protected] Optimization in Economic Theory (winter term 2021/22) 11 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Discrete time – Solving difference equations

Simplest case (homogeneous linear first-order difference equation with constant

coefficient):
Xt+1 = bXt .
Successive iteration:
X1 = bX0 ,
X2 = bX1 = b 2 X0
...
Xt+1 = b t+1 X0 ,
where the last line is the general solution. In this case the particular solution
follows immediately.
Here b is the growth factor of X .
Note the connection to the growth rate b = 1 + γX .

[email protected] Optimization in Economic Theory (winter term 2021/22) 12 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

We start with the simplest case of a two period consumption problem.

Consider a household that lives for two time periods, t and t + 1.
The utility function is defined to increase with consumption in the two periods

U = u(ct , ct+1 ).

The price of the consumption good is constant and normalized to 1.

The household has wealth at at the beginning of period t and income wt = wt+1 in
both periods.
Savings are denoted by st and earn interest r .

[email protected] Optimization in Economic Theory (winter term 2021/22) 13 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

The choice of the household has to satisfy

ct + st = wt + at in period t,
ct+1 = wt+1 + (1 + r )st in period t + 1.
Substituting st = wt + at − ct in the constraint for t + 1 yields
ct+1 = wt+1 + (1 + r )(wt + at − ct )
ct+1 wt+1
⇔ ct + = at + wt + .
1+r 1+r
This is the intertemporal budget constraint.
LHS: total (discounted) lifetime consumption expenditures.
RHS: total (discounted) lifetime income plus initial assets.

[email protected] Optimization in Economic Theory (winter term 2021/22) 14 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

v
The Lagrangian is
wt+1 ct+1
h i
L = u(ct , ct+1 ) + λ at + wt + − ct − .
1+r 1+r
Ignoring non-negativity constraints for now, the FOCs are
Lct = uct (ct , ct+1 ) − λ = 0,
λ
Lct+1 = uct+1 (ct , ct+1 ) −= 0,
1+r
wt+1 ct+1
Lλ = at + wt + − ct − = 0.
1+r 1+r
Plugging uct (ct , ct+1 ) = λ from the first FOC into the second FOC yields

uct+1 (ct , ct+1 )(1 + r ) = uct (ct , ct+1 ).

[email protected] Optimization in Economic Theory (winter term 2021/22) 15 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

The following reformulation has a nice interpretation

uct (ct , ct+1 )
=1+r
uct+1 (ct , ct+1 )
This is the Euler equation characterizing the change in
the optimal consumption level between t and t + 1.
The LHS is the slope of the “intertemporal” indifference curve
in the (ct+1 , ct )-space.
In other words the LHS is the MRS between consumption today
and consumption tomorrow.
The RHS is the slope of the intertemporal budget constraint.

[email protected] Optimization in Economic Theory (winter term 2021/22) 16 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

\ slope -(1 +r)

0 CJ

[email protected] Optimization in Economic Theory (winter term 2021/22) 17 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

Consider the specific iso-elastic utility function

c 1−θ c 1−θ
u(ct , ct+1 ) = t + β t+1 .
1−θ 1−θ
Households discount the future with the discount factor β = 1/(1 + ρ), with ρ
being the discount rate.
The parameter θ determines the curvature of the utility function and is the
coefficient of relative risk aversion.
It measures by how many percent marginal utility declines if
consumption c increases by 1 percent.
Its inverse 1/θ is the intertemporal elasticity of substitution
that measures how willing individuals are to shift consumption
(e.g., to earn higher interest income).

[email protected] Optimization in Economic Theory (winter term 2021/22) 18 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

The intuitive interpretation of the discount factor is that it measures individual

patience.
The intuitive interpretation of the intertemporal elasticity of substitution is that it
measures how willing individuals are to depart from consumption smoothing.
In terms of our graphical illustration, the discount factor determines the location of
the indifference curve and the intertemporal elasticity of substitution determines its
curvature.

[email protected] Optimization in Economic Theory (winter term 2021/22) 19 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

In case of iso-elastic utility we have

uct (ct , ct+1 ) = ct−θ ,
−θ
uct+1 (ct , ct+1 ) = βct+1 .
The Euler equation is then
ct+1 1
= [(1 + r )β] θ .
ct
The LHS is the growth factor of consumption.
The interpretation is now straightforward. Optimal consumption growth (which is
tantamount to saving) is higher if
the interest rate r is higher,
households are more patient (ρ is lower such that β is higher),
households are more willing to shift consumption over time (θ is lower).

[email protected] Optimization in Economic Theory (winter term 2021/22) 20 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Example 2 – A multi-period consumption problem

Now we extend the problem to T periods.

Note that we explicitly allow for T = ∞.
The utility function is T
X
U0 = β t u(ct ).
t=0
Households choose an optimal consumption plan {ct }Tt=0 = {c0 , c1 , . . . cT } to
maximize U0 subject to the lifetime budget constraint
T T
X ct X wt
= a0 + .
(1 + r )t (1 + r )t
t=0 t=0

[email protected] Optimization in Economic Theory (winter term 2021/22) 21 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Example 2 – A multi-period consumption problem

The Lagrangian is " T #

T T
X t
X ct X wt
L= β u(ct ) − λ t
− a0 − .
(1 + r ) (1 + r )t
t=0 t=0 t=0
The FOCs are
λ
Lct = β t u ′ (ct ) − =0 for t = 0, 1, 2, . . . , T ,
(1 + r )t
T T
X wt X ct
Lλ = a0 + − = 0.
(1 + r )t (1 + r )t
t=0 t=0
Note that there are T + 2 FOCs and T + 2 variables, now.

[email protected] Optimization in Economic Theory (winter term 2021/22) 22 / 23

 Chapter 10 – Dynamic Programming Double click here for audio contents

Example 2 – A multi-period consumption problem

Take the FOCs for ct and ct+1 . We have

λ
β t u ′ (ct ) = ,
(1 + r )t
λ
β t+1 u ′ (ct+1 ) = .
(1 + r )t+1
Dividing these two expressions and rearranging yields
u ′ (ct )
= (1 + r )β.
u ′ (ct+1 )
This is again the Euler equation.
Optimal consumption increases if (1 + r )β > 1.
For iso-elastic utility we have the same expression as before:
ct+1 1
= [(1 + r )β] θ .
ct

[email protected] Optimization in Economic Theory (winter term 2021/22) 23 / 23