Recursive Problems and Dynamic Programming

MacroeconomÌa II
MaestrÌa en EconomÌa
UTDT

Francisco J. Ciocchini

[email protected]

2023

1/81
General Problem
I Consider the following dynamic problem with a Önite horizon:

max U (k0 ; k1 ; :::; kT +1 ; c0 ; c1 ; :::; cT )

fct ;kt+1 gT
t=0

s. t. F0 (k0 ; k1 ; :::; kT +1 ; c0 ; c1 ; :::; cT ) = 0

F1 (k0 ; k1 ; :::; kT +1 ; c0 ; c1 ; :::; cT ) = 0

:::

FT (k0 ; k1 ; :::; kT +1 ; c0 ; c1 ; :::; cT ) = 0

k0 given

where t 2 f0; 1; :::; T g denotes the time period, ct 2 R, kt 2 R.

I Remarks: ct and kt could be vectors, but weíll stick to real numbers
for simplicity; we could add non-negativity constraints (e.g., ct $ 0);
more generally, equality constraints could be replaced with inequality
constraints (e.g., F0 (k0 ; k1 ; :::; kT +1 ; c0 ; c1 ; :::; cT ) $ 0).
2/81
General Problem

I The solution to the problem will be of the form:

ener
oh
3
dar
c0 = H0 (k0 ) e

:::
cT = HT (k0 )

k1 = J1 (k0 )
:::
kT +1 = JT +1 (k0 )

3/81
General Problem

I Assume the problem is "nicely behaved" and form the Lagrangian:

L = U (k0 ; k1 ; :::; kT +1 ; c0 ; c1 ; :::; cT )

+00 F0 (k0 ; k1 ; :::; kT +1 ; c0 ; c1 ; :::; cT )

+01 F1 (k0 ; k1 ; :::; kT +1 ; c0 ; c1 ; :::; cT )

:::

+0T FT (k0 ; k1 ; :::; kT +1 ; c0 ; c1 ; :::; cT )

4/81
General Problem
I Notation: k & (k0 ; k1 ; :::; kT +1 ) ; c & (c0 ; c1 ; :::; cT ).
I First Order Conditions (FOC):
@U (k;c) PT
@L
@c0 = 0 ) @c0 + s=0 0s @F@c s (k;c)
0
=0
::: :::
@U (k;c) PT
@L
@cT = 0 ) @cT + s=0 0s @F@c s (k;c)
T
=0

@U (k;c) PT
@L
@k1 =0) @k1 + s=0 0s @F@k
s (k;c)
1
=0
::: :::
@U (k;c) PT
@L
@kT +1 =0) @kT +1 + s=0 0s @Fs (k;c)
@kT +1 = 0

@L
@*0 =0) F0 (k; c) = 0
::: :::
@L
@*T =0) FT (k; c) = 0
I We have a system of 3(T + 1) equations to solve for 3(T + 1)
unknowns, c0 ; :::; cT ; k1 ; :::; kT +1 ; 00 ; :::; 0T , in terms of k0 .
I Notice that, in general, the system has to be solved simultaneously
for the 3(T + 1) unknowns. This is usually hard.
5/81
Special Case

I From now on weíll impose restrictions on the objective function and

the functions that deÖne the constraints.

I Assume:
value
↓Scop
U (k; c) = u0 (k0 ; c0 ) + ::: + uT (kT ; cT ) + S(kT +1 )

Ft (k; c) = Gt (kt ; ct ) ( kt+1 t 2 f0; :::; T g

I As weíll see, with these restrictions the problem becomes recursive.

6/81
Special Case
I Under the previous assumptions, our problem becomes:

max u0 (k0 ; c0 ) + ::: + uT (kT ; cT ) + S(kT +1 )

fct ;kt+1 gT
t=0

s. t. k1 = G0 (k0 ; c0 )

k2 = G1 (k1 ; c1 )

:::

kT +1 = GT (kT ; cT )

k0 given

I These constraints are usually called transition equations: given kt ,

kt+1 = Gt (kt ; ct ) tells you the value of kt+1 for each choice of ct .
I The function ut (kt ; ct ) is usually called the one-period return
function at t.
7/81
Special Case
I The special time-separable structure of this problem implies that, for
given k+ , c+ ináuences the future values fkt+1 gTt=+ , and the current
and future returns fut (kt ; ct )gTt=+ , but it does not ináuence their
$1
past values, fkt+1 g+t=0 $1
and fut (kt ; ct )g+t=0 .

I We have:
k+ +1 = G+ (k+ ; c+ )

k+ +2 = G+ +1 (k+ +1 ; c+ +1 )
= G+ +1 (G+ (k+ ; c+ ); c+ +1 )

k+ +3 = G+ +2 (k+ +2 ; c+ +2 )
= G+ +2 (G+ +1 (G+ (k+ ; c+ ); c+ +1 ); c+ +2 )

::: :::

8/81
Special Case

I Also:
u+ (k+ ; c+ )

u+ +1 (k+ +1 ; c+ +1 ) = u+ +1 (G+ (k+ ; c+ ); c+ +1 )

u+ +2 (k+ +2 ; c+ +2 ) = u+ +2 (G+ +1 (G+ (k+ ; c+ ); c+ +1 ); c+ +2 )

::::

I Notice that k+ constitutes a complete description of the position of

$1
the system at t = 4 : past values fct ; kt g+t=0 add no information
beyond that contained in k+ .This is why we call k+ the state variable
at time 4 ; c+ is called the control variable at time 4 .

I Weíll show below that this key property gives the problem a
recursive structure.

9/81
Special Case
I The Lagrangian for the new problem is

L = u0 (k0 ; c0 ) + u1 (k1 ; c1 ) + ::: + uT (kT ; cT ) + S(kT +1 )

+00 [G0 (k0 ; c0 ) ( k1 ]

+01 [G1 (k1 ; c1 ) ( k2 ]

:::

+0T $1 [GT $1 (kT $1 ; cT $1 ) ( kT ]

+0T [GT (kT ; cT ) ( kT +1 ]

I Or:
T
X T
X
L= ut (kt ; ct ) + S(kT +1 ) + 0t [Gt (kt ; ct ) ( kt+1 ]
t=0 t=0

10/81
Special Case
I FOC:
@u0 (k0 ;c0 )
@L
@c0 =0) @c0 + 00 @G0@c
(k0 ;c0 )
0
=0
::: :::
@uT (kT ;cT )
@L
@cT =0) @cT + 0T @GT@c
(kT ;cT )
T
=0

@u1 (k1 ;c1 )

@L
@k1 =0) @k1 ( 00 + 01 @G1@k
(k1 ;c1 )
1
=0
::: :::
@uT (kT ;cT )
@L
@kT =0) @kT ( 0T $1 + 0T @GT@k
(kT ;cT )
T
=0

@L
@kT +1 =0) S 0 (kT +1 ) ( 0T = 0

@L
@*0 =0) k1 = G0 (k0 ; c0 )
::: :::
@L
@*T =0) kT +1 = GT (kT ; cT )

11/81
Special Case

I Now weíll show that the system of Örst-order conditions can be

solved recursively.

I A recursive solution takes the form:

ct = ht (kt )

kt+1 = gt (kt )

where gt (kt ) & Gt (kt ; ht (kt )).

I The functions ht ()) and gt ()) are usually called policy functions (also
called, optimal feedback rules).

12/81
Special Case
I Forward iterations over the functions above give the solution in
terms of k0 :

c0 = H0 (k0 ) ! h0 (k0 )

k1 = J1 (k0 ) ! G0 (k0 ; H0 (k0 ))

c1 = H1 (k0 ) ! h1 (J1 (k0 ))

k2 = J2 (k0 ) ! G1 (J1 (k0 ); H1 (k0 ))

:::

cT = HT (k0 ) ! hT (JT (k0 ))

kT +1 = JT +1 (k0 ) ! GT (JT (k0 ); HT (k0 ))

13/81
Special Case: Recursive Solution

I Start at t = T .
I The FOC for period T are:
@uT (kT ;cT )
@cT
+ "T @GT@c
(kT ;cT )
T
=0

"T = S 0 (kT +1 )

kT +1 = GT (kT ; cT )
I Notice that we have a system of three equations that can be solved
for the three unknowns, cT , kT +1 , and "T , in terms of kT . We get:

cT = hT (kT )

kT +1 = gT (kT ) ! GT (kT ; hT (kT ))

"T = `T (kT )

14/81
Special Case: Recursive Solution
I Now go one step back to t = T " 1.
I The FOC for period T " 1 are:
@uT !1 (kT !1 ;cT !1 ) @GT !1 (kT !1 ;cT !1 )
@cT !1
+ "T "1 @cT !1
=0

@uT (kT ;cT )

@kT
" "T "1 + "T @GT@k
(kT ;cT )
T
=0

kT = GT "1 (kT "1 ; cT "1 )

I We know from our period-T FOC that cT = hT (kT ) and

"T = `T (kT ). Hence, we can write:
@uT !1 (kT !1 ;cT !1 ) @GT !1 (kT !1 ;cT !1 )
@cT !1
+ "T "1 @cT !1
=0

@uT (kT ;hT (kT ))

@kT
" "T "1 + `T (kT ) @GT (k@k
T ;hT (kT ))
T
=0

kT = GT "1 (kT "1 ; cT "1 )

15/81
Special Case: Recursive Solution

I t = T " 1 (cont.)
I We obtained a system of three equations that can be solved for the
three unknowns, cT "1 , kT , and "T "1 , in terms of kT "1 . We get:

cT "1 = hT "1 (kT "1 )

kT = gT "1 (kT "1 ) ! GT "1 (kT "1 ; hT "1 (kT "1 ))

"T "1 = `T "1 (kT "1 )

16/81
Special Case: Recursive Solution
I Now go one step back to t = T " 2.
I The FOC for period T " 2 are:
@uT !2 (kT !2 ;cT !2 ) @GT !2 (kT !2 ;cT !2 )
@cT !2
+ "T "2 @cT !2
=0

@uT !1 (kT !1 ;cT !1 ) @GT !1 (kT !1 ;cT !1 )

@kT !1
" "T "2 + "T "1 @kT !1
=0

kT "1 = GT "2 (kT "2 ; cT "2 )

I We know from the FOC for period T " 1 that cT "1 = hT "1 (kT "1 )
and "T "1 = `T "1 (kT "1 ). Hence, we can write:
@uT !2 (kT !2 ;cT !2 ) @GT !2 (kT !2 ;cT !2 )
@cT !2
+ !T !2 @cT !2
=0

@uT !1 (kT !1 ;hT !1 (kT !1 )) @GT !1 (kT !1 ;hT !1 (kT !1 ))

@kT !1
! !T !2 + `T !1 (kT !1 ) @kT !1
=0

kT !1 = GT !2 (kT !2 ; cT !2 )

17/81
Special Case: Recursive Solution

I t = T " 2 (cont.)
I We obtained a system of three equations that can be solved for the
three unknowns, cT "2 , kT "1 , and "T "2 , in terms of kT "2 . We get:

cT "2 = hT "2 (kT "2 )

kT "1 = gT "2 (kT "2 ) ! GT "2 (kT "2 ; hT "2 (kT "2 ))

"T "2 = `T "2 (kT "2 )

18/81
Special Case: Recursive Solution
I Keep going back in time until t = 0
I The FOC for period 0 are:
@u0 (k0 ;c0 )
@c0
+ "0 @G0@c
(k0 ;c0 )
0
=0

@u1 (k1 ;c1 )

@k1
" "0 + "1 @G1@k
(k1 ;c1 )
1
=0

k1 = G0 (k0 ; c0 )

I We know from the FOC for period 1 that c1 = h1 (k1 ) and

"1 = `1 (k1 ). Hence, we can write:
@u0 (k0 ;c0 )
@c0
+ "0 @G0@c
(k0 ;c0 )
0
=0

@u1 (k1 ;h1 (k1 ))

@k1
" "0 + `1 (k1 ) @G1 (k@k
1 ;h1 (k1 ))
1
=0

k1 = G0 (k0 ; c0 )

19/81
Special Case: Recursive Solution

I t = 0 (cont.)
I We obtained a system of three equations that can be solved for the
three unknowns, c0 , k1 , and "0 , in terms of k0 . We get:

c0 = h0 (k0 )

k1 = g0 (k0 ) ! G0 (k0 ; h0 (k0 ))

"0 = `0 (k0 )

and we are done.

20/81
Dynamic Programming and the Bellman Equation

I Now weíll show that the previous problem can be solved using an
alternative method.

I This method is known as Dynamic Programming.

I It was developed by Richard Bellman during the 1950s.

I Instead of using a Lagrangian weíll use the so-called

Bellman Equation.

21/81
Dynamic Programming and the Bellman Equation
I DeÖne the following sequence of optimization problems, for
t 2 f0; 1; :::; T g :

Wt (kt ) ! max ut (kt ; ct ) + Wt+1 (kt+1 )

ct ;kt+1

s. t. kt+1 = Gt (kt ; ct ) ↑ecuacion

kt given
4/ cada T

with

WT +1 (kT +1 ) = S(kT +1 )

I Weíll show that the solution to this sequence of "small" problems

coincides with the solution to our original "big" problem.

22/81
Dynamic Programming and the Bellman Equation
I Consider the previous problem for t = 0 :

max u0 (k0 ; c1 ) + W1 (k1 )

c0 ;k1

s. t. k1 = G0 (k0 ; c1 )

k0 given

I Knowledge of the function W1 (&) would allow us to easily solve the

problem by evaluating the trade-o§ between c0 and k1 . The solution
would give us the optimal values of c0 and k1 in terms of k0 .
I But we donít know the function W1 (&)!
I And notice that this would be true for any t < T .
I Fortunately, we do know that WT +1 (kT +1 ) = S(kT +1 ). This will
allow us to solve our sequence of problems recursively, starting at
t = T and working backwards in time.

23/81
Dynamic Programming and the Bellman Equation
I Consider period t = T .
I Then:
WT (kT ) ! max uT (kT ; cT ) + WT +1 (kT +1 )
cT ;kT +1

s. t. kT +1 = GT (kT ; cT )

kT given
I Lagrangian:

L = uT (kT ; cT ) + WT +1 (kT +1 ) + "T [GT (kT ; cT ) " kT +1 ]

I FOC:
@uT (kT ;cT )
@L
@cT
=0) @cT
+ "T @GT@c
(kT ;cT )
T
=0

@L
@kT +1
=0) "T = WT0 +1 (kT +1 )

@L
@)T
=0) kT +1 = GT (kT ; cT )

24/81
Dynamic Programming and the Bellman Equation
I t = T (cont.)
I But WT +1 (kT +1 ) = S(kT +1 ) ) WT0 +1 (kT +1 ) = S 0 (kT +1 ). Then:
@uT (kT ;cT )
@cT
+ "T @GT@c
(kT ;cT )
T
=0

"T = S 0 (kT +1 )

kT +1 = GT (kT ; cT )
I This system coincides with the period-T FOC of our original problem.
I Hence, we get:
cT = hT (kT )

kT +1 = gT (kT ) ! GT (kT ; hT (kT ))

"T = `T (kT ) ! S 0 (GT (kT ; hT (kT )))

I We use the same notation as before on purpose, to make explicit that
the solutions coincide.
I Substituting into the objective function we get:
WT (kT ) = uT (kT ; hT (kT )) + S(gT (kT ))
= uT (kT ; hT (kT )) + S(GT (kT ; hT (kT )))
25/81
Dynamic Programming and the Bellman Equation
I Now consider period t = T " 1.
I Then:
WT "1 (kT "1 ) ! max uT "1 (kT "1 ; cT "1 ) + WT (kT )
cT !1 ;kT

s. t. kT = GT "1 (kT "1 ; cT "1 )

kT "1 given
I Lagrangian:

L = uT "1 (kT "1 ; cT "1 ) + WT (kT ) + "T "1 [GT "1 (kT "1 ; cT "1 ) " kT ]
I FOC
@L @uT !1 (kT !1 ;cT !1 ) @GT !1 (kT !1 ;cT !1 )
@cT !1
=0) @cT !1
+ "T "1 @cT !1
=0

@L
@kT
=0) WT0 (kT ) " "T "1 = 0

@L
@)T !1
=0) kT = GT "1 (kT "1 ; cT "1 )

26/81
Dynamic Programming and the Bellman Equation
I t = T " 1 (cont.)
I We know from our period-T problem that

WT (kT ) = uT (kT ; hT (kT )) + S(GT (kT ; hT (kT )))

I Then:
0 @uT (kT ;hT (kT )) @uT (kT ;hT (kT )) 0
WT (kT ) = @kT
+ hT (kT )
!@cT "
0 @GT (kT ;hT (kT )) @GT (kT ;hT (kT )) 0
+S (GT (kT ; hT (kT ))) $ @kT
+ @cT
hT (kT )

@uT (kT ;hT (kT )) @GT (kT ;hT (kT ))

= @kT
+ S 0 (GT (kT ; hT (kT ))) @kT
! "
@uT (kT ;hT (kT )) @GT (kT ;hT (kT ))
+ @cT
+ S 0 (GT (kT ; hT (kT ))) @cT
h0T (kT )

@uT (kT ;hT (kT )) @GT (kT ;hT (kT ))

= @kT
+ `T (kT ) @kT
! "
@uT (kT ;hT (kT )) @GT (kT ;hT (kT ))
+ @cT
+ `T (kT ) @cT
h0T (kT )

@uT (kT ;hT (kT )) @GT (kT ;hT (kT ))

= @kT
+ `T (kT ) @kT

where we used S 0 (GT (kT ; hT (kT ))) = `T (kT ) and set [%] = 0 in the
penultimate line, by the FOC of period T .
I The previous expression can be found directly by means of the
Envelope Theorem.
27/81
Dynamic Programming and the Bellman Equation
I t = T " 1 (cont.)
I Substituting the previous expression for WT0 (kT ) into the FOC at
T " 1 we get:
@uT !1 (kT !1 ;cT !1 ) @GT !1 (kT !1 ;cT !1 )
@cT !1
+ "T "1 @cT !1
=0

@uT (kT ;hT (kT ))

@kT
+ `T (kT ) @GT (k@k
T ;hT (kT ))
T
" "T "1 = 0

kT = GT "1 (kT "1 ; cT "1 )

I But this system coincides exactly with the FOC at T " 1 of our
original problem. Hence:
cT "1 = hT "1 (kT "1 )

kT = gT "1 (kT "1 ) ! GT "1 (kT "1 ; hT "1 (kT "1 ))

"T "1 = `T "1 (kT "1 )

I Substituting into the objective function we get:
WT !1 (kT !1 ) = uT !1 (kT !1 ; hT !1 (kT !1 )) + WT (gT !1 (kT !1 ))
= uT !1 (kT !1 ; hT !1 (kT !1 )) + WT (GT !1 (kT !1 ; hT !1 (kT !1 )))

28/81
Dynamic Programming and the Bellman Equation
I We can keep going backward in time and show that, in each period,
the solution coincides with the recursive solution to our original
problem.

I Bottom Line: we can solve our original "big" problem by solving the
sequence of "small" problems deÖned by the Bellman Equations.

I See Appendix I for a simple problem solved using both lagrangian

and dynamic programming methods.
I See Appendix II for an example of a problem that is not recursive.

I Recall weíve found WT0 +1 (kT +1 ) = 4T and WT0 (kT ) = 4T "1 . More
generally: Wt0 (kt ) = 4t"1 8t ( 1. Hence, the Lagrange multipliers
give the marginal value of the state variable.
I Remark: in our previous argument we assumed without proof that
the functions ht (&) and Wt (&) are di§erentiable; formal conditions for
this have been established in the literature (e.g., Benveniste &
Scheinkman (1979)).
29/81
Dynamic Programming and the Bellman Equation
I Another perspective on the recursive nature of the problem can be
obtained from the following observations.
I First, notice that:
W0 (k0 ) = max fu0 (k0 ; c0 ) + W1 (k1 )g
c0 ;k1

= max fu0 (k0 ; c0 ) + max fu1 (k1 ; c1 ) + W2 (k2 )gg

c0 ;k1 c1 ;k2

= max fu0 (k0 ; c0 ) + max fu1 (k1 ; c1 )

c0 ;k1 c1 ;k2
+ max fu2 (k2 ; c2 ) + W3 (k3 )ggg
c2 ;k3

= max fu0 (k0 ; c0 ) + max fu1 (k1 ; c1 )

c0 ;k1 c1 ;k2
+ max fu2 (k2 ; c2 ) + max fu3 (k3 ; c3 ) + W4 (k4 )gggg
c2 ;k3 c3 ;k4

= max fu0 (k0 ; c0 ) + max fu1 (k1 ; c1 )

c0 ;k1 c1 ;k2
+ max fu2 (k2 ; c2 ) + max fu3 (k3 ; c3 )
c2 ;k3 c3 ;k4
+ :::: + max fuT (kT ; cT ) + S(kT +1 )g:::gggg
cT ;kT +1

where, in each step, the maximization is understood to be subject to

kt+1 = Gt (kt ; ct ), with kt given.
30/81
Dynamic Programming and the Bellman Equation
I Second, we know the solutions to our two problems coincide. Then:

W0 (k0 ) = max fu0 (k0 ; c0 ) + ::: + uT (kT ; cT ) + S(kT +1 )g

fct ;kt+1 gT
t=0

where the maximization is subject to kt+1 = Gt (kt ; ct ) for all

t 2 f0; 1; :::g, with k0 given.

I Combining the two results we get:

max fu0 (k0 ; c0 ) + ::: + uT (kT ; cT ) + S(kT +1 )g

fct ;kt+1 gTt=0

= maxfu0 (k0 ; c0 ) + maxfu1 (k1 ; c1 ) + maxfu2 (k2 ; c2 )

c0 ;k1 c1 ;k2 c2 ;k3

+ maxfu3 (k3 ; c3 ) + :::: + max fuT (kT ; cT ) + S(kT +1 )g:::gggg

c3 ;k4 cT ;kT +1

31/81
Dynamic Programming and the Bellman Equation

I The expression above shows that, for our problem, it is legitimate to

cascade the maximization operator. That is, the large optimization
problem of the left-hand side can be broken up into T + 1 smaller
problems. First, the problem in the innermost brackets is solved and
the optimal values of cT and kT +1 are obtained for any value of kT :
cT = hT (kT ) and kT +1 = gT (kT ). From these we obtain the
optimized value WT (kT ). Then the problem in the second innermost
brackets is solved, the optimizers being cT "1 = hT "1 (kT "1 ) and
kT = gT "1 (kT "1 ), with optimized value WT "1 (kT "1 ). The process
continues until the problem in the outermost brackets is reached.
This procedure is equivalent to iterating on Bellmanís Equation.

I The preceding argument implies the so-called Bellmanís Principle of

Optimality (see below).

32/81
Dynamic Programming and the Bellman Equation
I Bellmanís Principle of Optimality
I Suppose the policy functions fct = ht (kt )gTt=0 solve

max u0 (k0 ; c0 ) + ::: + uT (kT ; cT ) + S(kT +1 )

fct ;kt+1 gTt=0

s. t. kt+1 = Gt (kt ; ct ) t 2 f0; 1; :::; T g

k0 given

I Then fct = ht (kt )gTt=. solve

max u. (k. ; c. ) + ::: + uT (kT ; cT ) + S(kT +1 )

fct ;kt+1 gTt=0

s. t. kt+1 = Gt (kt ; ct ) t 2 f3 ; 3 + 1; :::; T g

k. given

for any 3 2 f0; 1; :::; T g.

33/81
Dynamic Programming and the Bellman Equation

I Bellmanís Principle of Optimality implies that, as time advances,

there is no incentive to depart from the original plan. Hence,
optimal policies have a self-enforcing character.

I Optimal policies with this self-enforcing character are said to be time

consistent.
I This property is a consequence of the recursive nature of the problem
and will not characterize the solutions to more general problems. See
Appendix II for an example with a time-inconsistent solution.

34/81
Dynamic Programming and the Bellman Equation
I Particular case:
ut (kt ; ct ) = 5 t u(kt ; ct )
where 5 2 (0; 1) is a discount factor.

I Bellman Equation:

Wt (kt ) ! max 5 t u(kt ; ct ) + Wt+1 (kt+1 )

ct ;kt+1

s. t. kt+1 = Gt (kt ; ct )

kt given

I DeÖne the current value function:

Wt (kt )
Vt (kt ) !
5t

Remark: V0 (k0 ) = W0 (k0 ).

35/81
Dynamic Programming and the Bellman Equation
I Then:
Wt (kt )
Vt (kt ) = *t

max * t u(kt ;ct )+Wt+1 (kt+1 )

ct ;kt+1
= *t

* t u(kt ;ct )+Wt+1 (kt+1 )

= max *t
ct ; kt+1

Wt+1 (kt+1 )
= max u(kt ; ct ) + *t
ct ;kt+1

= max u(kt ; ct ) + 5 Wt+1 (kt+1 )

* t+1
ct ;kt+1

I Finally:
Vt (kt ) = max u(kt ; ct ) + 5Vt+1 (kt+1 )
ct ;kt+1

I Notice that 4 t is not included in the Örst term of the right-hand side,
but Vt+1 (kt+1 ) is multiplied by 4 to discount it to period t.
36/81
InÖnite Horizon
I Up to now, weíve been dealing with the following problem:

P
T
max ut (kt ; ct ) + S(kT +1 )
fct ;kt+1 gT
t=0 t=0

s. t. kt+1 = Gt (kt ; ct ) t 2 f0; 1; :::; T g

k0 given

I In general, the solution to this problem gives di§erent policy

functions ht ($) and gt ($) to be used at each date.
I This is a consequence of two features of the problem: (i) the horizon
is Önite (T < 1); (ii) the functions ut ($) and Gt ($) have been
permitted to depend on time in arbitrary ways.
I We would like to discover contexts in which the same policy
functions h($) and g($) are used in every period. These are called
time-invariant or stationary policies.

37/81
InÖnite Horizon

I The previous remarks suggest that we consider an inÖnite-horizon

setup, and that we restrict the time dependence of ut ($) and Gt ($)
appropriately.

I From now on, in addition to an inÖnite horizon, weíll assume:

ut (kt ; ct ) = 1 t u(kt ; ct ) 1 2 (0; 1)

Gt (kt ; ct ) = G(kt ; ct )

for every t 2 f0; 1; :::; T g.

38/81
InÖnite Horizon
I Hence, consider the problem:

P
1
max 1 1 t u(kt ; ct ) = u(k0 ; c0 ) + 1u(k1 ; c1 ) + 1 2 u(k2 ; c2 ):::
fct ;kt+1 gt=0 t=0

s. t. kt+1 = G(kt ; ct ) t 2 f0; 1; :::g

k0 given

P
1 P
T
I Notation: ! t u(kt ; ct ) denotes lim ! t u(kt ; ct ).
t=0 T !1 t=0

I Remark: in general,
P
T P
T
max lim ! t u(kt ; ct ) 6= lim max ! t u(kt ; ct ),
T !1 t=0 T !1 t=0
so we canít just take limits on the Önite-horizon solution.

39/81
InÖnite Horizon

I DeÖne the period-1 continuation problem:

P
1
max 1 1 t$1 u(kt ; ct ) = u(k1 ; c1 ) + 1u(k2 ; c2 ) + 1 2 u(k3 ; c3 ):::
fct ;kt+1 gt=1 t=1

s. t. kt+1 = G(kt ; ct ) t 2 f1; 2; :::g

k1 given

I This problem has exactly the same structure as the original one.
I Both the inÖnite horizon and the time invariance of the functions
u(") and G(") are crucial for this.
I Moreover, this property holds for arbitrary periods, t and t + s.

40/81
InÖnite Horizon
I The property described above suggests that the solution to the
original problem will display time-invariant policy functions

ct = h(kt )
kt+1 = g(kt )

I Consequently, the value function will be time invariant too.

I Hence, for any t 2 f0; 1; :::g, the Bellman Equation will be:

V (kt ) & max u(kt ; ct ) + 1V (kt+1 )

ct ;kt+1

s. t. kt+1 = G(kt ; ct )

kt given

I Remark. To emphasize the irrelevance of the time subscript, some

authors use c for the control variable, k for the current-period state
and k0 for its next-period value.

41/81
InÖnite Horizon
I Substituting the constraint into the objective we get:

V (kt ) = max u(kt ; ct ) + 1V (G(kt ; ct ))

ct

I This is a functional equation to be solved for the function V ($).

I A functional equation is an equation in which the unknown is a
function.

I Notice that in the process of solving for V ($) we will be jointly

solving for the policy functions h($) and g($).

I Formally, one should prove that the solution to the Bellman

Equation will coincide with the solution to the original problem.
Details on the required assumptions can be found in Chapter 4 of
Stokey & Lucas (1989).

42/81
InÖnite Horizon
I Value-function Iteration
I This method proceeds by constructing a sequence of value functions
and associated policy functions. The sequence is created by iterating
on
Vj+1 (k) = max u(k; c) + !Vj (G(k; c))
c

starting from some function V0 (e.g., V0 = 0), and continuing until

Vj has converged (i.e., until Vj+1 = Vj ).

I To show that this method works, one deÖnes an operator T by

(T V )(k) = max u(k; c) + !V (G(k; c)). This operator maps Vj into
c
Vj+1 : Vj+1 = T Vj . Under certain conditions, this operator is a
contraction. Then, the contraction-mapping theorem ensures that T
has a unique Öxed point, V = T V . Moreover, the Öxed point can be
calculated by succesive iterations on Vj+1 = T Vj starting with some
function V0 . Again, the details can be found in Chapter 4 of Stokey
& Lucas (1989).
I This method is particularly useful to approximate the solution
numerically (using a computer).

43/81
InÖnite Horizon

I Guess and Verify

I This method involves guessing a solution V to the Bellman Equation
and verifying whether the guess is correct.

I Usually, the method of undetermined coe¢cients is used to verify the

guess.

I Weíll apply these two methods in future lectures. In particular, weíll

use them to solve versions of the neoclassical growth and RBC
models.

44/81
The Euler Equation
I The solution methods described above require that we Önd the value
function V . We show now that it may be possible to solve the
problem without this knowledge.
I Consider the Bellman Equation for the inÖnite-horizon problem:
V (kt ) = max u(kt ; ct ) + 1V (G(kt ; ct ))
ct

I If V were known (and di§erentiable), we could solve the problem in

the right-hand side using standard calculus methods. The FOC for
an interior solution would be:
uc (kt ; ct ) + 1V 0 (G(kt ; ct ))Gc (kt ; ct ) = 0
I The equation above deÖnes, implicitly, the policy function:
ct = h(kt )
I Substituting into the objective we get:
V (kt ) = u(kt ; h(kt )) + 1V (G(kt ; h(kt )))

45/81
The Euler Equation
I Di§erentiating V (kt ) with respect to kt we get:

V 0 (kt ) = uk (kt ; h(kt )) + uc (kt ; h(kt ))h0 (kt )

+ 1V 0 (G(kt ; h(kt )))[Gk (kt ; h(kt )) + Gc (kt ; h(kt ))h0 (kt )]

= uk (kt ; h(kt )) + 1V 0 (G(kt ; h(kt )))Gk (kt ; h(kt ))

+ [uc (kt ; h(kt )) + 1V 0 (G(kt ; h(kt )))Gc (kt ; h(kt ))]h0 (kt )
| {z }
=0 by FOC

I Then:

V 0 (kt ) = uk (kt ; h(kt )) + 1V 0 (G(kt ; h(kt )))Gk (kt ; h(kt ))

I This is known as the Envelope Condition (since it can be obtained

through a direct application of the Envelope Theorem).
I As noted before, conditions for the di§erentiability of V have been
established in the literature (e.g., Benveniste & Scheinkman (1979)).
46/81
The Euler Equation
I In many problems, there is no unique way of deÖning states and
controls, and several alternative deÖnitions lead to the same solution
of the problem.
I Suppose we manage to choose the state and control variables in
such a way that the transition equation does not depend on kt , so

kt+1 = G(ct )

I See Appendix III for the more general case.

I If follows that Gk = 0. Hence, the envelope condition becomes:

V 0 (kt ) = uk (kt ; h(kt ))

= uk (kt ; ct )
I Shifting the expression above one period ahead we get:

V 0 (kt+1 ) = uk (kt+1 ; h(kt+1 ))

= uk (kt+1 ; ct+1 )

47/81
The Euler Equation
I From the FOC we know that:

uc (kt ; ct ) + 1V 0 (kt+1 )Gc (kt ; ct ) = 0

I Substituting V 0 (kt+1 ) = uk (kt+1 ; ct+1 ) in the expression above we

obtain the Euler Equation:

uc (kt ; ct ) + 1uk (kt+1 ; ct+1 )Gc (kt ; ct ) = 0

I If G has an inverse, we can use kt+1 = G(ct ) to get

ct = m(kt+1 ) & G$1 (kt+1 ). Then, the Euler Equation becomes:

uc (kt ; m(kt+1 )) + 1uk (kt+1 ; m(kt+2 ))Gc (kt ; m(kt+1 )) = 0

I The expression above is a second-order di§erence equation in k.

I To solve this equation weíll need two boundary conditions: the initial
condition k0 , and an appropriate transversality condition.
48/81
The Euler Equation

I Notice that weíve managed to eliminate the value function V from

our problem. The cost of doing this is that now we need to solve a
(typically nonlinear) second-order di§erence equation.

I Moreover, in order to solve this equation we need to derive the

appropriate transversality condition that will serve as our second
boundary condition (more on this next class).

I The Guess-Verify method can sometimes be used to solve the Euler

Equation. Instead of guessing the form of the value function, we
guess the form of the policy function h($), and check that the Euler
Equation is satisÖed.

49/81
Stochastic Dynamic Programming
I Consider the (stationary) inÖnite-horizon stochastic problem:
P
1
max 1 E0 1 t u(kt ; ct )
fct ;kt+1 gt=0 t=0

s. t. kt+1 = G(kt ; ct ; "t ) t 2 f0; 1; :::g

k0 ; "0 given
I Et x denotes the mathematical expectation of the random variable x,
conditional on information available at time t.
I f"t g1
t=0 is a sequence of random variables with conditional
distribution function f ("t+1 j"t ) independent of t. It is assumed that
"t becomes known at t, before ct is chosen.
I Notice that kt+1 belongs to the period-t information set, since the
three arguments in the transition function are known at t.
I In some speciÖcations, the transition equation is of the form
kt+1 = G(kt ; ct ; "t+1 ). In this case, "t+1 and kt+1 are unknown at t.
In other speciÖcations, a stochastic shock is allowed to a§ect the
return function: u(kt ; ct ; "t ). The same tools weíll develop here can
be used to study these cases.
50/81
Stochastic Dynamic Programming
I The stochastic problem described above continues to have a
recursive structure, which follows from the additive separability of
the objective function in pairs (kt ; ct ), and from the particular form
assumed for the transition equation. This implies that dynamic
programming methods remain appropriate.
I The Bellman Equation is:

V (kt ; "t ) = max u(kt ; ct ) + 1Et fV (G(kt ; ct ; "t ); "t+1 )g

ct

where we have used the transition equation to eliminate kt+1 .

I Notice that the state vector is now (kt ; "t ).
I One of the state variables (k) is endogenous, the other one (") is
exogenous.
I The conditional expectation in the right-hand side is
Z
Et fV (G(kt ; ct ; "t ); "t+1 )g = V (G(kt ; ct ; "t ); "t+1 )f ("t+1 j"t )d"t+1

where we have assumed that "t is a continuous random variable.

51/81
Stochastic Dynamic Programming

I By solving Bellmanís functional equation weíll Önd the value

function V , and contingent plans of the form:

ct = h(kt ; "t )
kt+1 = g(kt ; "t ) & G(kt ; h(kt ; "t ); "t )

I The function V that solves the Bellman Equation can be found

through iterations on

Vj+1 (kt ; "t ) = max u(kt ; ct ) + 1Et fVj (G(kt ; ct ; "t ); "t+1 )g
ct

starting from an appropriate function V0 (e.g., V0 = 0).

I In some cases it is also possible to use the Guess-Verify method.
I Weíll see an example next class.

52/81
Stochastic Euler Equation
I Consider the problem

V (kt ; "t ) = max u(kt ; ct ) + 1Et fV (G(kt ; ct ; "t ); "t+1 )g

ct

and assume V is known.

I FOC:

uc (kt ; ct ) + 1Et fVk (G(kt ; ct ; "t ); "t+1 )Gc (kt ; ct ; "t )g = 0

I Then:

uc (kt ; ct ) + 1Et fVk (G(kt ; ct ; "t ); "t+1 )g Gc (kt ; ct ; "t ) = 0

I The equation above implicitly deÖnes the contingent plan

ct = h(kt ; "t )
I Substituting into the objective we get:

V (kt ; "t ) = u(kt ; h(kt ; "t )) + 1Et fV (G(kt ; h(kt ; "t ); "t ); "t+1 )g

53/81
Stochastic Euler Equation
I Di§erentiating V (kt ; "t ) with respect to kt we get:
Vk (kt ; "t ) = uk (kt ; h(kt ; "t )) + uc (kt ; h(kt ; "t ))hk (kt ; "t )
! "
Vk (G(kt ; h(kt ; "t ); "t ); "t+1 )&
+ +Et
[Gk (kt ; h(kt ; "t ); "t ) + Gc (kt ; h(kt ; "t ); "t )hk (kt ; "t )]

= uk (kt ; h(kt ; "t ))

+ +Et fVk (G(kt ; h(kt ; "t ); "t ); "t+1 )g Gk (kt ; h(kt ; "t ); "t )
# $
uc (kt ; h(kt ; "t ))+
+ hk (kt ; "t )
+Et fVk (G(kt ; h(kt ; "t ); "t ); "t+1 )gGc (kt ; h(kt ; "t )

I The last term in the right-hand of the expression above equals zero
because of the FOC. Then:
Vk (kt ; "t ) = uk (kt ; h(kt ; "t ))

+!Et fVk (G(kt ; h(kt ; "t ); "t ); "t+1 )g Gk (kt ; h(kt ; "t ); "t )

I This is the Envelope Condition for the stochastic case.

54/81
Stochastic Euler Equation
I Suppose we manage to choose the state and control variables in
such a way that the transition equation does not depend on kt , so
kt+1 = G(ct ; "t ). Then, Gk = 0 and the Envelope Condition
becomes:
Vk (kt ; "t ) = uk (kt ; h(kt ; "t ))
= uk (kt ; ct )
I Shifting the expression above one period ahead we get:
Vk (kt+1 ; "t+1 ) = uk (kt+1 ; h(kt+1 ; "t+1 ))
= uk (kt+1 ; ct+1 )
I From the FOC we know:
uc (kt ; ct ) + 1Et fVk (kt+1 ; "t+1 )g Gc (kt ; ct ; "t ) = 0
I Combining the previous two expressions we obtain the stochastic
Euler Equation:
uc (kt ; ct ) + 1Et fuk (kt+1 ; ct+1 )g Gc (kt ; ct ; "t ) = 0

55/81
Stochastic Euler Equation

I If we can use kt+1 = G(ct ; "t ) to write ct = m(kt+1 ; "t ), the Euler
Equation becomes:

uc (kt ; m(kt+1 ; "t )) + +Et fuk (kt+1 ; m(kt+2 ; "t+1 ))g Gc (kt ; m(kt+1 ; "t ); "t ) = 0

I The expression above is a second-order stochastic di§erence

equation in k (the endogenous state variable).

56/81
Dynamic Programming and the Lucas Critique

I Weíve seen above that the solution to our inÖnite-horizon problem is

a time-invariant policy function h(kt ; "t ) that satisÖes the following
functional equation:

V (kt ; "t ) = u(kt ; h(kt ; "t )) + 1Et fV (G(kt ; h(kt ; "t ); "t ); "t+1 )g

I Notice that, in general, the optimal policy h($) will depend on the
return function u($), the discount factor 1, the probability
distribution of the shocks f ($), and the transition function G($).
I Even if we keep preferences Öxed (u($) and 1), the optimal decision
rule will depend on the transition function G($).
I The implication is that, in dynamic decision problems, it is in general
impossible to Önd a single decision rule h($) that is invariant with
respect to changes in the law of motion G($).

57/81
Dynamic Programming and the Lucas Critique

I In 1976, Robert Lucas criticized a range of econometric policy

evaluation procedures because they used models that assumed
private agentsí decision rules to be invariant with respect to the laws
of motion that they faced.
I Those models took as structural (that is, as invariant under
interventions) such private agentsí decision rules as consumption
functions, investment schedules, portfolio decision rules, and labor
supply schedules.
I The models were routinely subjected to hypothetical policy
experiments that changed the laws of motion G($) by changing
variables that entered private agentsí constraints, such as tax rates,
wage rates, goods prices, and interest rates.
I Those hypothetical experiments violated the principle that an
optimal decision rule h($) is a function of the law of motion G($).

58/81
Appendix I: A Simple Example
I Consider the following cake-eating problem: choose c0 , c1 , c2 , k1 , k2
and k3 , in order to solve

max ln c0 + ln c1 + ln c2 + ln k3

s. t. k1 = k0 ( c0

k2 = k1 ( c1

k3 = k2 ( c2

k0 > 0 given

I We start by solving the problem using the method of Lagrange.

Then we solve it again using Dynamic Programming, and show that
the solutions coincide.

59/81
Appendix I: A Simple Example
I Lagrange
I The Lagrangian is
L = ln c0 +ln c1 +ln c2 +ln k3 +.1 (k0 'c0 'k1 )+.2 (k1 'c1 'k2 )+.3 (k2 'c2 'k3 )

I FOC
1
c0 : c0 = 91

1
c1 : c1 = 92

1
c2 : c2 = 93

k1 : 91 = 92

k2 : 92 = 93

1
k3 : k3 = 93

+ constraints

60/81
Appendix I: A Simple Example
I From the FOC we get:
91 = 9 2 = 9 3
I Then:
1 1 1 1
= = =
c0 c1 c2 k3
I Then:
c0 = c1 = c2 = k3
I Combining the constraints:
k3 = k2 ( c2 = k1 ( c1 ( c2 = k0 ( c0 ( c1 ( c2 )
c0 + c1 + c2 + k3 = k0
I The two previous expressions imply:
c0 + c0 + c0 + c0 = k0 ) 4c0 = k0 )
1
c'0 = k0
4
I Going back to c0 = c1 = c2 = k3 we get:
1
c'1 = c'2 = k3' = k0
4
61/81
Appendix I: A Simple Example
I Substituting c'0 = 14 k0 into the Örst constraint:
k1' = k0 ( c'0 = k0 ( 14 k0 )

3
k1' = k0
4
I Substituting c'1 = 14 k0 and k1' = 34 k0 into the second constraint:
k2' = k1' ( c'1 = 34 k0 ( 14 k0 = 24 k0 )

1
k2' = k0
2
I Finally, substituting the solution into the objective function we
obtain the maximized value of utility:
U ' = ln c'0 + ln c'1 + ln c'2 + ln k3'
U ' = ln( 14 k0 ) + ln( 14 k0 ) + ln( 14 k0 ) + ln( 14 k0 ) = 4 ln( 14 k0 )
U ' = 4 ln( 14 ) + 4 ln k0 )

U ' = (4 ln 4 + 4 ln k0

62/81
Appendix I: A Simple Example
I Dynamic Programming
I At t = 2 :
W2 (k2 ) & max ln c2 + W3 (k3 )
c2 ;k3

s. t. k3 = k2 ( c2

k2 given
with W3 (k3 ) = ln k3 .
I Then:
W2 (k2 ) & max ln c2 + ln k3
c2 ;k3

s. t. k3 = k2 ( c2

k2 given
I Then:
W2 (k2 ) & max ln c2 + ln(k2 ( c2 )
c2
I FOC:
1 1
=
c2 k2 ( c2 63/81
Appendix I: A Simple Example
I Then: k2 ( c2 = c2 ) k2 = 2c2 )
1
c2 = k2
2
I Substituting into the constraint: k3 = k2 ( c2 ) k3 = k2 ( 12 k2 )
1
k3 = k2
2
I Substituting into the objective: W2 (k2&) =' ln( 12 k2 ) + ln( 12 k2 ) )
W2 (k2 ) = 2 ln( 12 k2 ) ) W2 (k2 ) = 2 ln 21 + 2 ln k2 )
W2 (k2 ) = (2 ln 2 + 2 ln k2
I At t = 1
W1 (k1 ) & max ln c1 + W2 (k2 )
c1 ;k2

s. t. k2 = k1 ( c1

k1 given
where W2 (k2 ) was found in the previous step.
64/81
Appendix I: A Simple Example
I Then:
W1 (k1 ) & max ln c1 ( 2 ln 2 + 2 ln k2
c1 ;k2

s. t. k2 = k1 ( c1

k1 given
I Then:
W1 (k1 ) & max ln c1 ( 2 ln 2 + 2 ln(k1 ( c1 )
c1
I FOC:
1 2
=
c1 k1 ( c1
I Then: k1 ( c1 = 2c1 ) k1 = 3c1 )
1
c1 = k1
3
I Substituting into the constraint: k2 = k1 ( c1 ) k2 = k1 ( 13 k1 )
2
k2 = k1
3

65/81
Appendix I: A Simple Example
I Substituting into the objective:
W1 (k1 ) = ln( 13 k1 ) ( 2 ln 2 + 2 ln( 23 k1 )
W1 (k1 ) = ln( 13 ) + ln k1 ( 2 ln 2 + 2 ln( 23 ) + 2 ln k1
W1 (k1 ) = ( ln 3 ( 2 ln 2 + 2 ln 2 ( 2 ln 3 + 3 ln k1 )
W1 (k1 ) = (3 ln 3 + 3 ln k1
I At t = 0
W0 (k0 ) & max ln c0 + W1 (k1 )
c0 ;k1

s. t. k1 = k0 ( c0

k0 given
where W1 (k1 ) was found in the previous step.
I Then:
W0 (k0 ) & max ln c0 ( 3 ln 3 + 3 ln k1
c0 ;k1

s. t. k1 = k0 ( c0

k0 given
66/81
Appendix I: A Simple Example
I Then:
W0 (k0 ) & max ln c0 ( 3 ln 3 + 3 ln(k0 ( c0 )
c0
I FOC:
1 3
=
c0 k0 ( c0
I Then: k0 ( c0 = 3c0 ) k0 = 4c0 )
1
c0 = k0
4
I Substituting into the constraint: k1 = k0 ( c0 ) k1 = k0 ( 14 k0 )
3
k1 = k0
4
I Substituting into the objective:
W0 (k0 ) = ln( 14 k0 ) ( 3 ln 3 + 3 ln( 34 k0 )
W0 (k0 ) = ln( 14 ) + ln k0 ( 3 ln 3 + 3 ln( 34 ) + 3 ln k0
W0 (k0 ) = ( ln 4 ( 3 ln 3 + 3 ln 3 ( 3 ln 4 + 4 ln k0 )
W0 (k0 ) = (4 ln 4 + 4 ln k0

67/81
Appendix I: A Simple Example

I Hence, the recursive solution is:

c0 = 14 k0 [c0 = h0 (k0 )]

k1 = 34 k0 [k1 = g0 (k0 )]

c1 = 13 k1 [c1 = h1 (k1 )]

k2 = 23 k1 [k2 = g1 (k1 )]

c2 = 12 k2 [c2 = h2 (k2 )]

k3 = 12 k2 [k3 = g2 (k2 )]

68/81
Appendix I: A Simple Example
I From the expressions above we obtain the solution in terms of k0 :

c'0 = 14 k0

k1' = 34 k0

c'1 = 13 k1' = 14 k0

k2' = 23 k1' = 12 k0

c'2 = 12 k2' = 14 k0

k3' = 12 k2' = 14 k0
I Notice that this solution coincides with the one we found with the
method of Lagrange.
I As expected, the values of maximized utility coincide:
U ' = W0 (k0 ) = (4 ln 4 + 4 ln k0

69/81
Appendix II: A Non-Recursive Problem
I Consider a utility-maximization problem of the form:
max u0 (k0 ; c0 ; c1 ) + u1 (k1 ; c1 ) + S(k2 )

s. t. k1 = G0 (k0 ; c0 )

k2 = G1 (k1 ; c1 )

k0 given
I Notice that u0 (") depends not only on k0 and c0 but also on c1 .
I Now consider the following particular version of the problem above:
max ln(c0 ( c1 ) + ln c1 + ln k2

s. t. k1 = k0 ( c0

k2 = k1 ( c1

k0 > 0 given

70/81
Appendix II: A Non-Recursive Problem
I We start by solving the problem with the method of Lagrange.
I Notice that the two constraints can be combined into one:

c0 + c1 + k2 = k0

I Lagrangian:

L = ln(c0 ( c1 ) + ln c1 + ln k2 + 9 (k0 ( c0 ( c1 ( k2 )

I FOC
@L 1
@c0 =0) c0 $c1 =9

@L $1 1
@c1 =0) c0 $c1 + c1 =9

@L 1
@k2 =0) k2 =9

@L
@/ =0) c0 + c1 + k2 = k0

71/81
Appendix II: A Non-Recursive Problem
I Solving the system above we get:
1
c'0 = k0
2
1
c'1 = k0
6
1
k2' = k0
3
I Using the Örst-period constraint, k1 = k0 ( c0 , we get:
1
k1' = k0
2
I Evaluating the objective function we get:

U ' = ( ln 54 + 3 ln k0

72/81
Appendix II: A Non-Recursive Problem
I Now suppose we try to solve the problem by backward induction.
I At t = 1 (i.e., t = T ) we face the following problem:

W1 (k1 ) & max ln c1 + ln k2

c1 ;k2

s. t. k2 = k1 ( c1

k1 given

I Solving the problem above we get:

1
c1 = k1
2
1
k2 = k1
2
W1 (k1 ) = (2 ln 2 + 2 ln k1

73/81
Appendix II: A Non-Recursive Problem

I Suppose that, in the problem above, we take k1 = k1' (the solution

found using the method of Lagrange). Since k1' = 12 k0 , we get:
c1 = 12 k1' = 21 12 k0 )
1
e
c1 = k0
4
I Recall c'1 = 16 k0 . Hence:
c1 6= c'1
e
I We can see that dynamic programming does not give the correct
solution to the original problem.

74/81
Appendix II: A Non-Recursive Problem

I The optimal plan (i.e., the solution to the original problem) is time
inconsistent: if given the chance to re-optimize at t = 1, the
decision maker would choose to depart from the original plan. In
other words, the optimal plan lacks the self-enforcing character of
dynamic programming solutions.

I The time inconsistency arises because, given the chance to

re-optimize at t = 1, the decision maker would like to neglect the
e§ect of c1 on the marginal utility of c0 (an e§ect that was taken
into account in the original plan).

75/81
Appendix II: A Non-Recursive Problem
I Letís continue with our backward iteration.
I At t = 0 we face the following problem:
W0 (k0 ) & max ln(c0 ( c1 ) + W1 (k1 )

s. t. k1 = k0 ( c0

c1 = 12 k1

k0 given
with W1 (k1 ) = (2 ln 2 + 2 ln k1
I Since c1 appears in the return function at t = 0, we need to take into
account that c1 = 12 k1 .
I Alternatively, we could write the problem as follows:
W0 (k0 ) ( max ln(c0 ' c1 ) + ln c1 + ln k2
s. t. k1 = k0 ' c0
c1 = 12 k1
k2 = 12 k1
k0 given

76/81
Appendix II: A Non-Recursive Problem
I Then, we need to solve:
W0 (k0 ) & max ln(c0 ( 12 k1 ) ( 2 ln 2 + 2 ln k1
c0 ;k1

s. t. k1 = k0 ( c0

k0 given
I We get:
5
b
c0 = k0
9
b 4
k1 = k0
9
I Substituting b
k1 = 49 k0 into c1 = 12 k1 and k2 = 12 k1 we get:
2
b
c1 = k0
9
b 2
k2 = k0
9

77/81
Appendix II: A Non-Recursive Problem
I Finally, substituting the solution into the objective function we get:
& '
W0 (k0 ) = ( ln 243
4 + 3 ln k0

I Comparing the solution through backward induction with the

optimal plan we get:

c0 = 59 k0
b > c'0 = 12 k0

b
k1 = 49 k0 < k1' = 12 k0

c1 = 29 k0
b > c'1 = 16 k0

b
k2 = 29 k0 < k2' = 13 k0

I And:
& 243 '
W0 (k0 ) = ( ln 4 + 3 ln k0 < U ' = ( ln 54 + 3 ln k0

I As expected, utility is higher under the optimal plan.

78/81
Appendix III: The Euler Equation
I Recall the FOC gives:

uc (kt ; ct ) + 1V 0 (kt+1 )Gc (kt ; ct ) = 0

I The Envelope Condition is:

V 0 (kt ) = uk (kt ; ct ) + 1V 0 (kt+1 )Gk (kt ; ct )

I From the expression above we get:
V 0 (kt ) ( uk (kt ; ct )
V 0 (kt+1 ) =
1Gk (kt ; ct )
I Substituting into the FOC we get:
V 0 (kt ) ( uk (kt ; ct )
uc (kt ; ct ) + 1 Gc (kt ; ct ) = 0
1Gk (kt ; ct )
I Then:
Gk (kt ; ct )
V 0 (kt ) = uk (kt ; ct ) ( uc (kt ; ct )
Gc (kt ; ct )

79/81
Appendix III: The Euler Equation
I Shifting the expression above one period ahead we get:

Gk (kt+1 ; ct+1 )
V 0 (kt+1 ) = uk (kt+1 ; ct+1 ) ( uc (kt+1 ; ct+1 )
Gc (kt+1 ; ct+1 )
I Substituting the previous expression into the FOC we obtain the
Euler Equation:
# $
Gk (kt+1 ; ct+1 )
uc (kt ; ct )+! uk (kt+1 ; ct+1 ) ' uc (kt+1 ; ct+1 ) Gc (kt ; ct ) = 0
Gc (kt+1 ; ct+1 )

I If we can rewrite the transition equation kt+1 = G(kt ; ct ) as

ct = m(kt ; kt+1 ), the Euler Equation becomes:

uc (kt ; m(kt ; kt+1 ))+

" #
uk (kt+1 ; m(kt+1 ; kt+2 ))'
! G (kt+1 ;m(kt+1 ;kt+2 )) Gc (kt ; m(kt ; kt+1 )) = 0
uc (kt+1 ; m(kt+1 ; kt+2 )) Gkc (kt+1 ;m(kt+1 ;kt+2 ))

which is a second-order di§erence equation in k.

80/81
Bibliography

I Sargent, Thomas. Dynamic Macroeconomic Theory, Harvard

University Press, 1987. Chapter 1.

I Stokey, N. and Lucas, R., with Prescott, E. Recursive Methods in

Economic Dynamics, Harvard University Press, 1989.

81/81