0% found this document useful (0 votes)
36 views23 pages

Dynamic Programming 1

The document discusses dynamic programming, which is a method used to solve dynamic optimization problems involving intertemporal choice. It begins with an overview of dynamic programming and its applications in discrete and continuous time modeling. It then covers preliminaries on modeling time as discrete versus continuous. The document provides examples of solving simple differential equations in continuous time and difference equations in discrete time. It concludes with an example two-period consumption problem to illustrate dynamic programming.

Uploaded by

Salim Ghalayini
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
36 views23 pages

Dynamic Programming 1

The document discusses dynamic programming, which is a method used to solve dynamic optimization problems involving intertemporal choice. It begins with an overview of dynamic programming and its applications in discrete and continuous time modeling. It then covers preliminaries on modeling time as discrete versus continuous. The document provides examples of solving simple differential equations in continuous time and difference equations in discrete time. It concludes with an example two-period consumption problem to illustrate dynamic programming.

Uploaded by

Salim Ghalayini
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 23

Optimization in Economic Theory

PD Dr. Johannes Paha

University of Hohenheim

winter term 2021/22

[email protected] Optimization in Economic Theory (winter term 2021/22) 1 / 23


Chapter 10 – Dynamic Programming

Chapter 10
Dynamic Programming

Jan 12 and 19, 2022

Optimization in Economic Theory


PD Dr. Johannes Paha
winter term 2021/22

[email protected] Optimization in Economic Theory (winter term 2021/22) 2 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Overview

Until now we only considered static problems.


In reality, current choices have repercussions on future outcomes.
We will study a method that can be used to solve dynamic problems.
In general, there are more methods available (see the book by Dixit).
In case of discrete time modeling,
dynamic programming is often the most convenient method.
In case of continuous time modeling, it is possible to differentiate with respect to
time, so that the Maximum Principle is more handy.
Despite the methodological and conceptual differences, the results are remarkably
similar to the discrete time case.
Background reading: Simon and Blume (1994, Ch. 24 and 25)

[email protected] Optimization in Economic Theory (winter term 2021/22) 3 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Learning goals

Students
can explain the difference between continuous and discrete time,
can solve a two-period and a multi-period consumption problem in discrete time,
can define a Bellman equation and solve a dynamic programming problem by
applying a three-step procedure,
can solve problems from optimal control theory.

[email protected] Optimization in Economic Theory (winter term 2021/22) 4 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Preliminaries

We can conceptualize the evolution of time in two ways:

Continuous time

t0 t

Discrete time

t=0 t=1 t=2 t=3 t=4 t=5 t=6

[email protected] Optimization in Economic Theory (winter term 2021/22) 5 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Preliminaries

Consider the plot of a function over continuous and over discrete time (e.g., GDP
or population, etc.).
In continuous time the function is differentiable with respect to time, in discrete
time it is not.

[email protected] Optimization in Economic Theory (winter term 2021/22) 6 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Preliminaries

If a variable X depends on time and time evolves continuously,


we write X (t).
We can differentiate this variable with respect to time and get its evolution (a dot
over a variable denotes the time derivative):
dX (t) X (t + ∆) − X (t)
Ẋ (t) = = lim
dt ∆→0 ∆
In discrete time, the derivative does not exist because ∆ = 1 is the smallest value
that ∆ can attain.
Instead, we have that the change of a variable between time t and t + 1 is given
by the difference of the levels:
∆X = Xt+1 − Xt

[email protected] Optimization in Economic Theory (winter term 2021/22) 7 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Growth Rates

Denote the growth rate of X by γX .


The growth rate in continuous time is given by:
Ẋ (t)
γX =
X (t)
The growth rate in discrete time is given by:
Xt+1 − Xt
γX =
Xt
We have the following approximation:
Ẋ (t) Xt+1 − Xt

X (t) Xt
| {z } | {z }
continuous time discrete time
The approximation gets better the closer to zero the growth rate and the smaller
the discrete time step (e.g., quarterly vs. yearly data).

[email protected] Optimization in Economic Theory (winter term 2021/22) 8 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Growth Rates

If a variable X grows at a constant rate,


it is said to grow exponentially:
Ẋ (t)
= γX , X (0) = X0 ⇒ X (t) = X0 e γX ·t ,
X (t)
Xt+1 − Xt
= γX Xt=0 = X0 ⇒ Xt = X0 (1 + γX )t .
Xt
Note that, due to the chain rule, we can also compute the growth rate of a
variable in continuous time by taking the logarithm and time-differentiating:
d ln[X (t)] Ẋ (t)
= .
dt X (t)

[email protected] Optimization in Economic Theory (winter term 2021/22) 9 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Continuous time – Solving differential equations

Simplest case (homogeneous linear first-order differential equation with constant


coefficient): Ẋ (t) = gX (t)
Divide both sides by X (t) and integrate with respect to t:
Ẋ (t)
= g,
X (t)
Z Z
Ẋ (t)
dt = g dt,
X (t)
ln [X (t)] + c0 = gt + c1 , (1)
where c0 and c1 are constants of integration.

[email protected] Optimization in Economic Theory (winter term 2021/22) 10 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Continuous time – Solving differential equations

Exponentiation yields the general solution


e ln[X (t)]+c0 = e gt+c1
X (t) = ce gt ,
where c = ±e c1 −c0 is the constant of integration.
Assume that X (0) = X0 , then c = X0 because X (0) = ce g·0 and, therefore, the
particular solution of the differential equation pins down to

X (t) = X0 e gt .
Note that g is the growth rate of X , which we denoted by gX .

[email protected] Optimization in Economic Theory (winter term 2021/22) 11 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Discrete time – Solving difference equations

Simplest case (homogeneous linear first-order difference equation with constant


coefficient):
Xt+1 = bXt .
Successive iteration:
X1 = bX0 ,
X2 = bX1 = b 2 X0
...
Xt+1 = b t+1 X0 ,
where the last line is the general solution. In this case the particular solution
follows immediately.
Here b is the growth factor of X .
Note the connection to the growth rate b = 1 + γX .

[email protected] Optimization in Economic Theory (winter term 2021/22) 12 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

We start with the simplest case of a two period consumption problem.


Consider a household that lives for two time periods, t and t + 1.
The utility function is defined to increase with consumption in the two periods

U = u(ct , ct+1 ).

The price of the consumption good is constant and normalized to 1.


The household has wealth at at the beginning of period t and income wt = wt+1 in
both periods.
Savings are denoted by st and earn interest r .

[email protected] Optimization in Economic Theory (winter term 2021/22) 13 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

The choice of the household has to satisfy


ct + st = wt + at in period t,
ct+1 = wt+1 + (1 + r )st in period t + 1.
Substituting st = wt + at − ct in the constraint for t + 1 yields
ct+1 = wt+1 + (1 + r )(wt + at − ct )
ct+1 wt+1
⇔ ct + = at + wt + .
1+r 1+r
This is the intertemporal budget constraint.
LHS: total (discounted) lifetime consumption expenditures.
RHS: total (discounted) lifetime income plus initial assets.

[email protected] Optimization in Economic Theory (winter term 2021/22) 14 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

v
The Lagrangian is
wt+1 ct+1
h i
L = u(ct , ct+1 ) + λ at + wt + − ct − .
1+r 1+r
Ignoring non-negativity constraints for now, the FOCs are
Lct = uct (ct , ct+1 ) − λ = 0,
λ
Lct+1 = uct+1 (ct , ct+1 ) −= 0,
1+r
wt+1 ct+1
Lλ = at + wt + − ct − = 0.
1+r 1+r
Plugging uct (ct , ct+1 ) = λ from the first FOC into the second FOC yields

uct+1 (ct , ct+1 )(1 + r ) = uct (ct , ct+1 ).

[email protected] Optimization in Economic Theory (winter term 2021/22) 15 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

The following reformulation has a nice interpretation


uct (ct , ct+1 )
=1+r
uct+1 (ct , ct+1 )
This is the Euler equation characterizing the change in
the optimal consumption level between t and t + 1.
The LHS is the slope of the “intertemporal” indifference curve
in the (ct+1 , ct )-space.
In other words the LHS is the MRS between consumption today
and consumption tomorrow.
The RHS is the slope of the intertemporal budget constraint.

[email protected] Optimization in Economic Theory (winter term 2021/22) 16 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

\ slope -(1 +r)

0 CJ

[email protected] Optimization in Economic Theory (winter term 2021/22) 17 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

Consider the specific iso-elastic utility function


c 1−θ c 1−θ
u(ct , ct+1 ) = t + β t+1 .
1−θ 1−θ
Households discount the future with the discount factor β = 1/(1 + ρ), with ρ
being the discount rate.
The parameter θ determines the curvature of the utility function and is the
coefficient of relative risk aversion.
It measures by how many percent marginal utility declines if
consumption c increases by 1 percent.
Its inverse 1/θ is the intertemporal elasticity of substitution
that measures how willing individuals are to shift consumption
(e.g., to earn higher interest income).

[email protected] Optimization in Economic Theory (winter term 2021/22) 18 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

The intuitive interpretation of the discount factor is that it measures individual


patience.
The intuitive interpretation of the intertemporal elasticity of substitution is that it
measures how willing individuals are to depart from consumption smoothing.
In terms of our graphical illustration, the discount factor determines the location of
the indifference curve and the intertemporal elasticity of substitution determines its
curvature.

[email protected] Optimization in Economic Theory (winter term 2021/22) 19 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Example 1 – A two period consumption problem

In case of iso-elastic utility we have


uct (ct , ct+1 ) = ct−θ ,
−θ
uct+1 (ct , ct+1 ) = βct+1 .
The Euler equation is then
ct+1 1
= [(1 + r )β] θ .
ct
The LHS is the growth factor of consumption.
The interpretation is now straightforward. Optimal consumption growth (which is
tantamount to saving) is higher if
the interest rate r is higher,
households are more patient (ρ is lower such that β is higher),
households are more willing to shift consumption over time (θ is lower).

[email protected] Optimization in Economic Theory (winter term 2021/22) 20 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Example 2 – A multi-period consumption problem

Now we extend the problem to T periods.


Note that we explicitly allow for T = ∞.
The utility function is T
X
U0 = β t u(ct ).
t=0
Households choose an optimal consumption plan {ct }Tt=0 = {c0 , c1 , . . . cT } to
maximize U0 subject to the lifetime budget constraint
T T
X ct X wt
= a0 + .
(1 + r )t (1 + r )t
t=0 t=0

[email protected] Optimization in Economic Theory (winter term 2021/22) 21 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Example 2 – A multi-period consumption problem

The Lagrangian is " T #


T T
X t
X ct X wt
L= β u(ct ) − λ t
− a0 − .
(1 + r ) (1 + r )t
t=0 t=0 t=0
The FOCs are
λ
Lct = β t u ′ (ct ) − =0 for t = 0, 1, 2, . . . , T ,
(1 + r )t
T T
X wt X ct
Lλ = a0 + − = 0.
(1 + r )t (1 + r )t
t=0 t=0
Note that there are T + 2 FOCs and T + 2 variables, now.

[email protected] Optimization in Economic Theory (winter term 2021/22) 22 / 23


Chapter 10 – Dynamic Programming Double click here for audio contents

Example 2 – A multi-period consumption problem

Take the FOCs for ct and ct+1 . We have


λ
β t u ′ (ct ) = ,
(1 + r )t
λ
β t+1 u ′ (ct+1 ) = .
(1 + r )t+1
Dividing these two expressions and rearranging yields
u ′ (ct )
= (1 + r )β.
u ′ (ct+1 )
This is again the Euler equation.
Optimal consumption increases if (1 + r )β > 1.
For iso-elastic utility we have the same expression as before:
ct+1 1
= [(1 + r )β] θ .
ct

[email protected] Optimization in Economic Theory (winter term 2021/22) 23 / 23

You might also like