RS – Ch 15 – Dynamic Optimization •5/29/2013

Chapter 15
Dynamic Optimization

1
Pierre de Fermat (1601?-1665) L. Euler Lev Pontryagin (1908-1988)

15.1 Dynamic Optimization

• In this chapter, we will have a dynamic system –i.e., evolving over time
(either discrete or continuous time). Our goal: optimize the system.
• We will have a particular optimization problems with the following
features:
1) Aggregation over time for the objective function
2) Variables linked (constrained) across time.
• Example: Planner’s problem – Maximization of consumption over time
T − rt
Max ∫ e C(t)dt
0
Subject to F = F (K ) and ∂F > 0 and ∂ F < 0
2
∂K ∂K 2
Change in capital stock: K& = F − C −δK 2

•1
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.1 Dynamic Optimization

• Notation:
K(t) = capital (only factor of production)
F(K)
( ) =well-behaved outputp function –i.e.,, F’(K)>0,
( ) , and F”(K)<0,
( ) , for K >0
C(t) = consumption
I(t) = investments = F(K) – C(t)
δ = constant rate of depreciation of capital.

• We are not lookingg for a single

g optimal
p value C* , but for values
C(t) that produce an optimal value for the integral (or aggregate
discounted consumption over time).

15.1 Calculus of variations – Classic Example

• The calculus of variations involves finding an extremum (maximum or
minimum) of a quantity that is expressible as an integral.
• Question: What is the shortest path between two points in a plane?
Y kknow the
You h answer -a straighti h liline- but
b you probably
b bl hhave not
seen a proof of this: the calculus of variations provides such a proof.
• Consider two points in the x-y plane, as shown in the figure.
y
y2 y = y(x)
2
y1 1
ds = dx 2
+ dy 2

x
x1 x2

• An arbitrary path joining the points follows the general curve

y = y(x), and an element of length along the path is
ds = dx2 + dy2 .
4

•2
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.1 Calculus of variations – Classic Example

• We can rewrite this as: ds = 1 + y′( x) 2 dx,

dy
which is valid because dy = dx = y′( x)dx. Thus, the length is
dx
2 x2
L = ∫ ds = ∫ 1 + y′( x)2 dx.
1 x1

• Note that we have converted the problem from an integral along a

path, to an integral over x:
2 x2
L = ∫ ds = ∫ 1+ y′(x)2 dx.
1 x1

• We have thus succeeded in writing the problem down, but we need

some additional mathematical machinery to find the path for which L
is an extremum (a minimum in this case).
5

15.1 Calculus of variations – Classic Example

• In our usual minimizing or maximizing of a function f(x), we

would take the derivative and find its zeroes These points of zero
slope are stationary points –i.e.,
i e the function is stationary at those
points, meaning for values of x near such a point, the value of the
function does not change (due to the zero slope).
• Similarly, we want to be able to find solutions to these integrals
that are stationary for infinitesimal variations in the path. This is
called calculus of variations.
• The methods wee will ill develop
de elop are
re called
lled variational
v ri ti n l methods.
th d
• These principles are common, and of great importance, in many
areas of physics (such as quantum mechanics and general
relativity) and economics.
6

•3
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.1 Euler-Lagrange Equations

• We will try to find an extremum (to be definite, a minimum) for an
as yet unknown curve joining two points x1 and x2, satisfying the
integral
g relation:
x2
S = ∫ f [y(x), y' (x),x]dx.
x1
• The function f is a function of three variables, but because the path
of integration is y = y(x), the integrand can be reduced to a
function of just one variable, x.
• To start, let’s consider two curves joining points
y Y ( x) = y( x)+η ( x) ((wrong)
g)
1 andd 2,
2 the
h “right”
“ i h ” curve y(x), ( ) andd a “wrong”
“ ” y2
2
curve: Y ( x) = y( x) + η ( x); η ( x1 ) = η ( x2 ) = 0. 1
y1 y = y(x) (right)
Y(x) that is a small displacement from the “right” x
x1 x2
curve, as shown in the figure.
• We call the difference between these curves as some function h(x). 7

15.1 Euler-Lagrange Equations

• There are infinitely many functions h(x), that can be “wrong.” We
require that they each be longer than the “right” path. To
quantify how close the “wrong” path can be to the “right” one,
let’s write Y = y + αh, so that
x2
S (α ) = ∫
x1
f [Y , Y ' ( x ), x ]dx
x2
= ∫
x1
f [ y + αη , y '+α η ′, x ]dx .
• Now, we can characterize the shortest path as the one for which
the derivative dS/dα = 0 when α = 0. To differentiate the above
equation with respect to α,α we need the partial derivative ∂S / ∂α
via the chain rule ∂f ( y + αη, y′ + αη′, x) ∂f ∂f
=η +η′ ,
∂α ∂y ∂y′
so dS/dα = 0 gives
dS x2 ∂f x2 ⎛ ∂f ∂f ⎞
=∫ dx = ∫ ⎜⎜η +η′ ⎟⎟dx = 0
dα x1 ∂α x1
⎝ ∂y ∂y′ ⎠ 8

•4
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.1 Euler-Lagrange Equations

• The second term in the equation can be by integrated by parts:

x2
x2 ∂f ⎡ ∂f ⎤ x2 d ⎛ ∂f ⎞
∫
x1
η′
∂y′
dx = ⎢η ( x) ⎥ − ∫ η ( x) ⎜⎜
⎣ ∂y′ ⎦ x1 x1
⎟dx,
dx ⎝ ∂y′ ⎟⎠
but the first term of this relation (the end-point term) is zero
because h(x) is zero at the endpoints.
• Our modified equation is:
dS x2 ⎛ ∂f d ∂f ⎞
dα ∫x1
= η ( x)⎜⎜ − ⎟⎟dx = 0.
⎝ ∂y dx ∂y′ ⎠

• This leads us to the Euler-Lagrange equation

∂f d ∂f
− = 0.
∂y dx ∂y′
• Key: the modified equation has to be zero for any h(x). 9

15.1 Euler-Lagrange Equations

• Let’s go over what we have shown. We can find a minimum (more

generally, a stationary point) for the path S if we can find a function
for the path that satisfies:
∂f d ∂f
− = 0.
∂y dx ∂y′
• The procedure for using this is to set up the problem so that the
quantity whose stationary path you seek is expressed as
x2
S = ∫ f [ y(x),
) y′(x),
) x]dx
d,
x1

where f[y(x),y’(x),x] is the function appropriate to your problem.

• Then, write down the Euler-Lagrange equation, and solve for the
10
function y(x) that defines the required stationary path.

•5
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.1 Euler-Lagrange Equations – Example I

• Find the shortest path between two points:

2 x2
L = ∫ ds = ∫ 1+ y′( x)2 dx.
1 x1

• The integrand contains our function f ( y, y′, x) = 1+ y′(x)2 .

• The two partial derivatives in the Euler-Lagrange equation are:
∂f ∂f y′
= 0 and = .
∂y ∂y′ 1+ y′2
d ∂f d y′
• The Euler-Lagrange
g g equation
q gives
g us = = 0.
ddx ∂y′ dx
d 1+ y′2
y′
• This says that = C, or y′2 = C 2 (1 + y′2 ).
1 + y′ 2

=> y′2 = constant (call it m2), so y(x) = mx + b

11
=> A straight line is the shortest path.

15.1 Euler-Lagrange Equations – Example II

• Intertemporal utility maximization problem:
x2
max ∫ [ B − u(c(t ))]dx s.t. c(t ) = f (k (t )) − k ' (t )
x1

B: bliss level of utility

c(t)=c(k(t),k’(t))

• Substitute constraint into integrand:

f(x(t),x’(t),t)= B-u(c(k(t),k’(t))=V(c(k(t),k’(t)))
d
• Euler-Lagrange equation: Vk = V•
dt k
dc
Vk = −u ' (c ) f ' ( k ); V • = Vc •
= −u ' (c )( −1)
k
dk
d d
Vk = V• ⇒ −u' (c) f ' (k) = u' (c) 12
dt k dt

•6
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.1 Euler-Lagrange Equations – Example II

• Repeating Euler-Lagrange equation:
d d
Vk = V• ⇒ −u' (c) f ' (k ) = u' (c)
dt k dt
• If we are given functional forms:
f(k(t) )=k(t)α
u(c(t)) = ln(c(t))
Then,
d
− u ' (c ) f ' ( k ) = u ' (c )
dt
•
1 d 1 c (t )
− αk (t )α −1 = ( )=−
c (t ) dt c (t ) c (t ) 2
•
α −1 c (t )
⇒ αk (t ) = 13
c (t )

15.1 Dynamic Optimization – Calculus of

Variations
• In the general economic framework, F(.) will be the objective
function, the variable x will be time, and y will be the variable to
choose over time to optimize F(.). Changing notation:
t1
•
maxx
∫ F (t, x(t ), x(t ))dt
t0
s.t. x(t0 ) = x0 , x(t1 ) = x1

• Necessary Conditions: Euler-LaGrange

Euler LaGrange Equation

∂F ∂ ⎛⎜ ∂F ⎞⎟
− = 0.
∂x ∂t ⎜⎝ ∂ x• ⎟⎠
14

•7
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.1 Dynamic Optimization – Limitations

• Method gives extremals, but it doesn’t tell maximum or minimum:
- Distinguishing
g g mathematicallyy between max/min is more
difficult.
- Usually have to use geometry to setup the problem.

• Solution curve x(t) must have continuous second-order derivatives

- Requirement from integration by parts.

• We find stationary states, which vary only in space, not in time

-Very few cases in which systems varying in time can be
solved
- Even problems involving time (e.g., brachistochrones) don’t
change in time
15

15.2 Optimal Control Theory

• Optimal control: Find a control law for a given system such that a
certain optimality criterion is achieved.
• Typical example: Minimization of a cost functional that is a
function of state and control variables.
• More general than Calculus of Variations.
• Handles Inequality restrictions on Instruments and State Variables
• Similar to Static Methods (Lagrange Method and KKT Theory)
• Long history in economics: Shell(1967), Arrow(1968) and
Shell(1969).

16

•8
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory

• The dynamic system is described by a state equation represented by:
•
x (t ) = g (t , x (t ), u (t ))
where x(t) = state variable,
variable u(t) = instrument or control variable
• The control aim is to maximize the objective functional:
t1

∫ f (t , x (t ), u (t )) dt
t0

• Usually the control variable u(t) will be constrained as follows:

u(t) ε Ω(t), t ε[t0,t1]
• Boundary Conditions: t0, t1, x(t0) = x0 fixed. Sometimes, x(t1) = xT.
• Sometimes we will also have additional constraints. For example,
h(x(t),u(t),t) ≥ 0
17
• Functional Form Assumption: f(.), g(.) are continuously differentiable.

15.2 Optimal Control Theory

• We can convert the calculus of variation problem into a control
theory problem.
t1
•
• Calculus of variation problem: max ∫ F (t , x (t ),
) x (t )) dt
t0

Define: x’(t)=u(t), x(t0)=x0, x(t1)=x1

t1

Replace in calculus of variation problem: max ∫ F (t , x (t ), u (t )) dt

t0
Notation: x(t) = xt = xt

18

•9
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory

• Form a Lagrangean-type expression, the Hamiltonian
H(t,x(t),u(t),λ(t)) = f(t,x(t),u(t)) + λ g(t,x(t),u(t))
Necessary Conditions (f.o.c.), called the Maximum Principle):
dH df dg
= −λ = 0
du du du
dH •
− = λ (adjoint equation)
dx
.
dH
d •
= x ( t ) = g ( t , x ( t ), u ( t ))
dλ
Boundary (Transversality) Conditions: x(t0) = x0, λ(t1) = 0
The Hamiltonian multiplier, λ(t), is called the co-state or adjoint variable:
It measures the imputed value of stock (state variable) accumulation. 19

15.2 Optimal Control Theory – Pontryagin’s

maximum principle
• Conditions for necessary conditions to work –i.e.,: u(t) and x(t)
maximize: t 1 •

t0
∫ f (t , x(t ), u (t )) dt s.t. x (t ) = g (t , x (t ), u (t ))

- Control variable must be piecewise continuous (some jumps,

discountinuities OK).
- State variable must be continuous and piecewise differentiable.
- f(.) and g(.) first-order differentiable w.r.t. state variable and t,
but not necessarily w.r.t. control variable.
- initial condition finite for state variable.
- if no finite terminal value for state variable, then λ(t1) = 0. 20

•10
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – Pontryagin’s

maximum principle
• Sufficiency: if f(.) and g(.) are strictly concave, then the necessary
conditions
di i are sufficient,
ffi i meaning
i thath any path h satisfying
i f i these
h
conditions does in fact solve the problem posed.

21

15.2 Optimal Control Theory – Example

● Let’s go back to our first example:
Max ∫ e − rt C dt s.t. K& = Q − C − δK
T
t
0
where Q = Q⎛⎜⎝ K ⎞⎟⎠ and ∂Q > 0 and ∂ Q < 0
2
∂K ∂K 2

Form the Hamiltonian:

H = e −rt Ct + λ ⎡⎢⎣Q − C −δK ⎤⎥⎦ + λ&K (1) (boundaries: K and K .)
0 T
F.o.c.:
F
∂H = e −rt − λ = 0 => e−rt = λ (2)
∂C
∂H = λ ⎛⎜⎜ ∂Q −δ ⎞⎟⎟ + λ& = 0 (3)
∂K ⎜ ∂K
⎝
⎟
⎠
∂H = K& = Q − C − δK (4) 22
∂λ

•11
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – Example

From (3) and (2) ∂Q (K *) = − λ& + δ = r + δ (5)

∂K λ

=> At the optimal (K*), the marginal productivity of capital

should equal the total user cost of capital, which is the sum
of the interest and the depreciation rates.

The capital stock does not change when it reaches K*.

We have the following dynamics:

Ct should be reduced if Kt < K* (dQ/dK too low)
Ct should be increased if Kt > K* (dQ/dK too high)

23

15.2 Optimal Control Theory – Example

• Dynamics of the system
Following dynamics:
Ct should be reduced if Kt < K* ((dQ/dK
Q/ too low))
• Ct should be increased if Kt > K* (dQ/dK too high)

C*

K1 K* K2

24

•12
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – More State and

Control Variables
• We can add to the problem more state and Control variables.
Analogy can be made to multivariate calculus.
• Objective Functional:
t1
∫ t0
f (t, x1 (t), x 2 (t), u 1 (t), u 2 (t))dt

Constraints: x& 1 (t) = g1 (t, x1 (t), x 2 (t), u1 (t),u 2 (t))

x& 2 (t) = g 2 (t
(t, x1 (t),
(t) x 2 (t)
(t),u
u1 (t)
(t),u
u 2 (t))

Boundary Conditions: t0, t1, x1(t0)=x10, x2(t0) = x20; fixed

Free Endpoints for x1 and x2
25

15.2 Optimal Control Theory – More State and

Control Variables
• Form Hamiltonian:
H(t,x1(t),x2(t),u1(t),u2(t),λ1(t), λ2(t))= f(t,x1(t),x2(t),u1(t),u2(t))+λ1g1(t,x1(t),x2(t),u1(t),u2(t))
+λ2g2(t,x1(t),x2(t),u1(t),u2(t))

F.o.c.: ∂f ∂g1 ∂g 2
ui : + λ1 + λ2 =0 i = 1,2
∂u i ∂u i ∂u i
∂H ⎧ ∂f ∂g1 ∂g 2 ⎫
λ& i = − = −⎨ + λ1 + λ2 ⎬
∂x i ⎩ ∂x i ∂x i ∂x i ⎭
∂H
x& i =
∂λ i
Boundary (Transversality) Conditions: x1(t0)=x10, x2(t0)=x20, λ1(t1)=λ2(t1)= 0
Fixed Endpoint Problem: Add the boundary condition: x(t1) = x* 26

•13
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – General Problem

• More General Problem: Let’s add a terminal-value function
and inequality restrictions.
Obj i F
Objective Functional:
i l
t1

max ∫ f (t , x (t ), u (t )) dt + ϕ (t , x (t1 )) s.t.

t0
•
x i (t ) = g i (t , x (t ), u (t )), xi (t 0 ) = xi 0 , i = 1,..., n
xi (t1 ) = xi1 , i = 1,..., q
xi (t1 ) free , i = q + 1,..., r
xi (t1 ) ≥ 0, i = r + 1,..., s
K ( x1 (t ),..., xn (t )) ≥ 0, at t1
1≤ q ≤ r ≤ s ≤ n
27
x = n-dimensional vector, u = m-dimensional vector

15.2 Optimal Control Theory – General Problem

F.o.c.:
∂H
x& i = = g i (t, x,u),i = 1,K,n
∂λ i
⎧ ∂f n ∂g ⎫
λ& i = − ⎨ + ∑ λ i j ⎬ , j = 1,K, n
⎩ ∂x i j=1 ∂x i ⎭
∂f n
∂g
+ ∑ λ k k = 0, j = 1,K,m
∂u j k =1 ∂u j

Note: H(t,x*,u,λ) is maximized by u=u*

28

•14
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – General Problem

Transversality Conditions

(i) xi(t1) free: ∂ϕ

λ i (t1 ) =
∂x i
∂ϕ ⎡ ∂ϕ ⎤
(ii) xi(t1) ≥ 0: λ i (t1 ) ≥ , x i (t1 ) ⎢λ i (t1 ) − =0
∂x i ⎣ ∂x i ⎥⎦

∂ϕ ∂K
(iii) K(xq(t1),…,xn(t1)) ≥ 0: λ i (t1 ) = +p , i = q,K, n
∂x i ∂x i
p ≥ 0, pK = 0
∂ϕ ∂K
(iv) K(xq(t1),…,xn(t1)) = 0: λ i (t1 ) = +p , i = q,K, n
∂x i ∂x i
n
(v) t1 is free: at t: f + ∑ λ i g i + ϕt = 0
i =1 29

15.2 Optimal Control Theory – General Problem

n
(vi) T ≥ t1: f + ∑ λ i g i + ϕi ≥ 0
i =1

at t1, with strict equality if T > t1, if T - t1 >0 is required

∂ϕ ∂K
(vii) K(xq(t1),…,xn(t1),t1) ≥ 0: λ i (t1 ) = +p , i = q,K, n
∂x i ∂x i
n
∂K
f + ∑ λ i g i + ϕt + p =0
i =1 ∂t1
p ≥ 0, K ≥ 0, pK = 0, t = t1

30

•15
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – General Problem

• State Variable Restrictions: k(t,x) ≥ 0
t1

max ∫ f (t , x (t ), u (t )) dt + ϕ ( x (t1 ) s.t.

t0
•
s.t. x (t ) = g (t , x (t ), u (t )), x ( t 0 ) = x 0 , k (t , x ) ≥ 0
• Form Hamiltonian:
H = f(t,x,u) + λg(x,u,t) + ηk(t,x)
Opti lit Conditions:
Optimality C diti

u :H u = f u + λg u = 0
λ& = − H x = −(f x + λg x + ηk x )
λ (t1 ) = ϕx (x(t1 )), η ≥ 0, ηk = 0 31

15.2 OCT – Current Value Hamiltonian

• Often, in economics, we have to optimized over discounted

sums, subject to our usual dynamic constraint:
T
∫ 0
e − rt f (t,
(t x, u)dt
)dt
x& = g(t, x, u), x(0) = x 0
Form Hamiltonian:
H = e-rt f(t,x,u) + λ g(t,x,u)
Optimality
p y Conditions:

u :H u = e − rt f u + λg u
λ& = − H = −e − rt f − λg
x x x

λ (T) = 0
32

•16
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 OCT – Current Value Hamiltonian

• Sometimes, it is convenient to eliminate the discount factor. The
resulting system involves current, rather than, discounted values of
various magnitudes.
Hamiltonian: H = e-rt f(t,x,u) + λ(t) g(t,x,u)
• • •
Define m(t) = ertλ(t) =>m = rertλ + ert λ = rm + ert λ
Current Value Hamiltonian: % = e rt H = f (t, x,u) + mg(t, x, u)
H
%
∂H
Optimality Conditions: = f u + mg u = 0,
0 m& − rm = −f x − mg x
∂u
Note: If f(.) and g(.) are autonomous, this substitution leads to
autonomous transition equations describing the dynamics of the
system.
33

15.2 OCT – Current Value Hamiltonian - Example

• Using the usual notation, we want to maximize utility over time:
t1

max J = ∫ e − ρtU (C (t )) dt s.t.

t0
•
K (t ) = F ( K ) − C − ∂K , K (0) = K 0 , K (T ) = K T , C (t , x ) ≥ 0
• Form Current Value Hamiltonian:
~
H = e ρt H = U (C (t )) + m ( F ( K ) − C − ∂K )
~

• Optimality Conditions: dH dH
= = U ' (C ) − m = 0
dC dC
•
m − ρm = −U ' (C ) − m ( F ' ( K ) − δ )
• Note: The current value Hamiltonian consists of two terms: 1)
utility of current consumption, and 2) net investment evaluated by34
price m, which reflects the marginal utility of consumption.

•17
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 OCT – Current Value Hamiltonian - Example

• The static efficiency condition:
U’(C(t))=m(t),
maximizes the value of the Hamiltonian at each instant of time
myopically, provided m(t) is known.

• The dynamic efficiency condition:

•
m − ρm = −U ' (C ) − m ( F ' ( K ) − δ )
forces the price m of capital to change over time in such a way that
the
h capital
i l stockk always
l yields
i ld a net rate off return, which hi h iis equall to
the social discount rate ρ . That is,
dm = [ −U ' (C ) − m ( F ' ( K ) − δ )]dt + ρmdt
• There is a long-run foresight condition which establishes the
terminal price m(T) of capital in such a way that exactly the terminal
35
capital stock K(T) is obtained at T .

15.2 Optimal Control Theory – Application

• Equilibria in Infinite Horizon Problems

∞ •
max ∫ e − rt f ( x (t ), s t x (t ) = g ( x (t ),
) u (t )) dt s.t. ) u (t )),
)) x (t 0 ) = x 0
t0
~
Hamiltonian: H = f(x,u) + m g(x,u)
Optimality Conditions: H u = 0, m
& − rm = − H x

Transversality Conditions: lim m(t)x(t) = 0

t →∞

lim e − rt m(t)x(t) = 0
t →∞

Problems of this sort, if they are of two dimensions, lead to phase

plane analysis.
36

•18
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – Application

m x& = 0

& =0
m

x
37

15.2 Optimal Control Theory – Sufficiency

• Sufficiency: Arrow and Kurz (1970)
If the maximized Hamiltonian is strictly concave in the state
variables any path satisfying the conditions above will be
variables,
sufficient to solve the problem posed.
~
Hamiltonian: H = f(x,u) + mg(x,u)
Optimality Conditions: H u = 0, m
& − rm = − H x

⇒ u = u(m,
ˆ x)
Maximized Hamiltonian:

H* = f (x, u(m,
ˆ x)) + mg(x, u(m,
ˆ x))

38

•19
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – Applications

• Example I: Nerlove-Arrow Advertising Model

Let G(t) ≥ 0 denote the stock of goodwill at time t .
•
G = u − ∂G , G (0) = G 0
where u = u(t ) ≥0 is the advertising effort at time t measured in dollars
per unit time. Sales S are given by
S = S(p,G,Z),
where p is the price level and Z other exogeneous variables.
Let c(S) be the rate of total production costs
costs, then,
then total revenue net of
production costs is:
R(p,G,Z) = p S(p,G,Z) – c(S)
Revenue net of advertising expenditure is: R(p,G,Z) - u
.
39

15.2 Optimal Control Theory – Applications

• The firm wants to maximize the present value of net revenue
streams discounted at a fixed rate ρ:

•
subject to G = u − ∂ G , G (0) = G 0
• Note that the only place p occurs is in the integrand, which we can
maximize by first maximizing R w.r.t p holding G fixed, and then
maximized the result with respect to u. Thus,

• Implicitly, we get p*(t) = p(G(t),Z(t))

• Define π(G,Z)=R(p*,G,Z). Now, J is a function of G and Z only.
For convenience, assume Z is fixed. 40

•20
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – Application

• Solution by the Maximum Principle

dH
= 0
du

The adjoint variable λ(t) is the shadow price associated with the goodwill
at time t . The Hamiltonian can be interpreted as the dynamic profit rate
which consist of two terms:
(i) the current net profit rate π(G) - u.
(ii) the value of the new goodwill created by
41
advertising at rate u.

15.2 Optimal Control Theory – Applications

The second equation corresponds to the usual equilibrium relation for

investment in capital goods:

It states that the marginal opportunity cost of investment in goodwill,

, should equal the sum of the marginal profit
from increased goodwill and the capital gain, λ(ρ +δ)dt .

42

•21
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – Application

Define as the elasticity of demand with respect to

goodwill and after some algebra, we can derive:

We also can obtain the optimal long-run stationary equilibrium .

Again, after some simple algebra, we obtain: as

43

15.2 Optimal Control Theory – Applications

The property of is that the optimal policy is to go to as fast as
possible.
If , it is optimal
p to jjumpp instantaneouslyy to byy applying
pp y g an
appropriate impulse at t =0 and set for t >0.
If , the optimal control u*(t)=0 until the stock of goodwill
depreciates to the level , at which time the control switches to
and stays at this level to maintain the level of
goodwill. See Figure.

44

•22
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – Applications

45

15.2 Optimal Control Theory – Applications

• Example II: Neoclassical Growth Model (Robert Solow)
∞ C t1−σ
Preferences: ∫ e − ρt dt
0 1−σ

Capital accumulation: K& t = Yt − N t Ct − δK t

Technology: Yt = At K tα N t1−α assume At = 1 N t = 1

max
Form current value Hamiltonian:
Ct1−σ
H (c, K , θ ) =
1−σ
1−
[
+ θ t K tα − Ct − δK t −1 ] (1)
C is control, K is state variable, θ is adjoint variable.
First order conditions: ∂∂CH = 0 => C σ = θ (2) t
−
t
t

θ&t = ρθt − θ t [αK tα −1 − δ ]

∂H t
θ&t = ρθ t −
∂K t
=> (3)
K& t = K tα − Ct − δK t (4) 46

•23
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – Applications

Lim
Transversality condition e − ρtθ t K t = 0 (5)
t →∞

Characterization of the balanced growth path: Capital stock,

consumption and the shadow price of capital remain constant
&
in the balanced growth path CC = g ; KK = g and θθ = gθ . From (3)
& & t
c K
t

θ&t
θt
[
= ρ − αK tα −1 − δ ] => αK tα −1 θ&
= ρ − t +δ
θt
(6)
S ce the
Since t e RHSHS iss constant
co sta t , therefore
t e e o e LHS
HS also
a so should
s ou d be
K&
constant K = 0 . If capital stock is not growing output is not
& &
growing YY = 0 and consumption is not growing CC = 0 .

θ&t C& θ&t

From (2) θt
= −σ t
Ct
=> θt
=0 47

15.2 Optimal Control Theory – Applications

Recall that from (2): C σ = θ => C = θ
σ
t
−
t t t

Then, we have a 2x2 non-linear system of differential equations:

σ
K& t = K tα − θ t − δK t
θ&t
θt
[ ]
= ρ − αK tα −1 − δ = ( ρ + δ ) − αK tα −1

This system can be shown as a phase diagram in the (θ , K ) space, t t

to analyze the transition dynamics of the shadow price θ and t

capital.

From (6), we can calculate the steady state value for Kt:
K * = [( ρ + δ ) / α ]α (α<1)−1

48

•24
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.2 Optimal Control Theory – Applications

In (θ , K ) space the transition dynamics of the shadow
t t

price θ and capital.

t

θ&t = 0

θ&t < 0 θ&t > 0

(θ t )

K > K * > K'.

K& = 0
K*
K& > 0

K& < 0

49
*
K K' K

15.2 Optimal Control Theory – Applications

Putting all these things together the convergence to the
steady state can be summarised in the following diagram.
K& = 0
θ& = 0

θ I II

IV

III

K* K' K
Convergence to the steady state lies in region I and III as
shown by the double arrow red line.
50

•25
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.3 Discrete Time Optimal Control

• We change from continuous measured variables to discrete measured
variables.
• State Variable Dynamics: Δ xt = xt+1 – xt = f(xt, ut, t),
t) x0 given
t =T −1
• Objective Functional: max J = ∑ u(x , t) + S (x
t =0
t T )
• Form a Lagrangean:
T −1 T −1
L = ∑ u ( xt , t ) + S ( xT ) + ∑ λt [ f ( xt , u t , t ) − xt +1 + xt ]
t =0 t =0

• Define
D fi the
h H il i Ht: Ht =H(x
Hamiltonian H( t,ut,λλt+1,t)) = u(x
( t,ut,t)) + λt+1 f(x
f( t,ut,t))
• Now, re-write the Lagrangean:
T −1
L = S ( xT , T ) + ∑ { H t − λ t +1 ( x t +1 − x t )}
t =0
51

15.3 Discrete Time Optimal Control

• F.o.c.: dL dH t
= =0
du t du t
dL dH t
= − ( x t +1 − x t ) = 0
d λ t +1 d λ t +1
dL dH t
= + λ t +1 − λ t = 0
dx t dx t
dL dS
= − λT = 0
dx T dx T

• Optimality Conditions: Hu= 0

Δ xt = xt+1 - xt = dH/dλt+1 = f(xt,ut,t)
Δ λt = -dH/dxt = -du/dxt, - λt+1 df/dxt
Boundary Conditions: x0 given and λT = dS/dxT 52

•26
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.3 Discrete Time Optimal Control - Example

• Consider an production-inventory discrete problem.
Let Ik, Pk and Sk be the inventory, production, and demand at time k ,
respectively. Let I0 be
b the
h initial inventory, let andd bbe the
h goal levels
of inventory and production, and let h and c be inventory and
production cost coefficients.
The problem is:

subject to

Form the Hamiltonian:

53

15.3 Discrete Time Optimal Control - Example

The adjoint variable satisfies:

To maximize the Hamiltonian, let us differentiate w.r.t. to production:

Since production must be nonnegative, we obtain the optimal production

as

These expressions determine a two-point boundary value problem. For a

given set of data, it can be easily solved numerically. If the constraint Pk
≥ 0 is dropped it can be solved analytically.

54

•27
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.3 Discrete Time Optimal Control

• Discrete Calculus of Variations
∞
• We have an intertemporal problem: max ∑ β t g ( xt +1 , xt )
t =0

Trick: Differentiate the objective function with respect to xt at time

t and t-1.
Objective Function at time t: βtg(xt+1,xt)
Optimality Condition at time t: βtgx(xt+1,xt)
Optimality Condition at time t-1: βt-11 gx(xt,xt-1)
Complete Condition: β t g x t (x t +1 , x t ) + β t −1g x t (x t , x t −1 ) = 0
g x t (x t +1 , x t ) + β−1g x t (x t , x t −1 ) = 0
∂g
Transversality Condition: lim βT xT = 0 55
T →∞ ∂x T

15.3 Discrete Time Optimal Control

• Example I: Cash Flow Maximization by a Firm

∞
max ∑ βt [ P[f (L t ) − C(L t +1 − L t )] − WL t ],0 < β < 1
t =0

P: Profit function, Lt: Labor at time t, W: wage rate.

Euler Equation (f.o.c.):

β t [ P[f ′(L t ) + C′(L t +1 − L t )] − W ] − β t −1PC′(L t − L t −1 ) = 0

P[f ′(L t ) + C′(L t +1 − L t )] − W − β−1PC′(L t − L t −1 ) = 0

56

•28
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.3 Dynamic Programing

• Alternative way to solve intertemporal problems

• Equivalent
q in manyy contexts to methods alreadyy seen
• Typical problem:
T
max ∑ β t u (ct ) + V0 ( kT +1 ) s.t.
t =0

c t + k t +1 − (1 − δ ) k t = f ( k t )

g g
• Lagrangean Formulation:
T T
∑ βt u(ct ) + V% 0 (k T+1 ) + ∑ λ% t [f (k t ) + (1 − δ)k t − ct − k t +1 ]
t =0 t =0

57

15.3 Dynamic Programing

T T
∑ βt u(ct ) + V% 0 (k T+1 ) + ∑ λ% t [f (k t ) + (1 − δ)k t − ct − k t +1 ]
t =0 t =0

O i li C
Optimality Conditions:
di i
c t : β t u ′(c t ) − λ% t = 0
k t +1 : λ% t +1 [ f ′(k t +1 ) + (1 − δ) ] − λ% t = 0, 0 < t < T
% ′ (k ) − λ% ≤ 0, ⎡ V % ⎤
⎣ 0′ (k T +1 ) − λ t ⎦ k T +1 = 0
k T +1 : V %
0 T +1 t

c t + k t +1 − ((1 − δ)k
) t = f ((k t )

Eliminate the multiplier to get

β u ′(c t +1 ) [ f ′(k t +1 ) + (1 − δ ) ] = u ′(c t )
% ′ (k ) − λ% ≤ 0, ⎡ V % ⎤
⎣ 0′ (k T +1 ) − λ t ⎦ k T +1 = 0
V %
0 T +1 t
58
c t + k t +1 − (1 − δ )k t = f (k t )

•29
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.3 Dynamic Programing

Problem can be solved recursively:
• First solve the problem at t = T
• Choose cT and kT+1 to maximize: % (k )
βT u(cT ) + V0 T +1

subject to cT + k T +1 − (1 − δ)k T = f (k T ) kT given

⇒ cT = cT (k T ), k T +1 = k T +1 (k T )
Now solve the period T-1 problem
• Choose
Ch cT-1 andd kT to maximize
i i

βT −1u(cT −1 ) + βT u(cT (k T )) + V
% (k (k ))
0 T +1 T

subject to cT −1 + k T − (1 − δ)k T −1 = f (k T −1 ) kT-1 given

59

15.3 Dynamic Programing

Continue solving backwards to time 0.
The same optimality conditions arise from the problem:

V1 (k T ) = m ax u(cT ) + β V0 (k T +1 )
cT ,k T +1
subject to cT + k T +1 − (1 − δ)k T = f (k T )
% (k ) / βT +1
V0 (k T +1 ) = V0 T +1

Optimality Conditions:
u′(cT ) = λ T
β V0′ (k T +1 ) − λ T ≤ 0, [β V0′ (k T +1 ) − λ T ] k T +1 = 0
cT + k T +1 − (1 − δ)k T = f (k T )
These are the same conditions as before if we define λ T = λ% T / βT60

•30
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.3 Dynamic Programing

Envelope Theorem implies V1′(k T ) = λ T [ f ′(k T ) + (1 − δ) ]

Given the constraint and kT, V1((kT) is the maximized value of

u(cT ) + β V0 (k T +1 )
Period T-1 problem is equivalent to maximizing
u(cT −1 ) + β V1 (k T )
with the same constraint at T
T-11 and kT-1
T 1 given:

V2 (k T −1 ) = m ax u(cT −1 ) + β V1 (k T )
cT −1 ,k T

subject to cT −1 + k T − (1 − δ)k T −1 = f (k T −1 )
61

15.3 Dynamic Programing

Optimality Conditions:
u′(cT −1 ) = λ T −1
β V1′(k T ) = λ T −1
cT −1 + k T − (1 − δ)k T −1 = f (k T −1 )
The envelope theorem can be used to eliminate V1′
u′(cT −1 ) = λ T −1
βλ T [ f ′((k T ) + (1
( − δ)] = λ T −1
cT −1 + k T − (1 − δ)k T −1 = f (k T −1 )

The period T-1 envelope condition is

V2′ (k T −1 ) = λ T −1 [ f ′(k T −1 ) + (1 − δ ) ] 62

•31
RS – Ch 15 – Dynamic Optimization •5/29/2013

15.3 Dynamic Programing

This process can be continued giving the following Bellman

Equation:
Vj+1 (k T − j ) = m ax u(cT − j ) + βVj (k T − j+1 )
cT ,k T

subject to cT − j + k T − j+1 − (1 − δ)k T − j = f (k T − j ) kT-j given

• Bellman’s Principle of Optimality

The fact that the original problem can be written in this recursive way
leads us to Bellman’s Principle of Optimality:
An optimal policy has the property that whatever the initial state and
initial decision are, the remaining decisions must constitute an optimal
policy with regard to the state resulting from the first decision.
63

Ken Arrow’s office

64

•32