Calculus of Variations

Summer Term 2014

Lecture 18

11. Juli 2014

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 1 / 25
Purpose of Lesson:

To introduce Pontryagin’s Maximum Principle (PMP)

To discuss several PMP examples

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 2 / 25
Pontryagin’s Maximum Principle

Modern optimal control theory often starts from the PMP. It is a simple,
coincise condition for an optimal control.

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 3 / 25
General control problem
Minimize functional
Zt1
J[x, u] = F0 (t, x, u)dt
t0

subject to constraints ẋ = F(t, x, u), or more fully,

ẋi = Fi (t, x, u)

notice no dependence on ẋ in F0
this differs from many CoV problems

no dependence on ẋ in Fi because we rearrange the equations so

that derivatives are on the LHS.

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 4 / 25
Pontryagin’s Maximum Principle (PMP)

Let u(t) be an admissible control vector that transfers (t0 , x0 ) to a

target (t1 , x(t1 )). Let x(t) be the trajectory corresponding to u(t).
In order that u(t) be optimal, it is necessary that there exists
p(t) = (p1 (t), p2 (t), . . . , pn (t)) and a constant scalar p0 such that
p and x are the solution to the canonical system

∂H ∂H
ẋ = and ṗ = −
∂p ∂x
n
P
where the Hamiltonian is H = pi Fi with p0 = −1
i=0
H(t, x, u, p) > H(t, x, u
b, p) for all alternate controls u
b

all boundary conditions are satisfied

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 5 / 25
PMP proof sketch-1
Consider the general problem: minimize functional
Zt1
J[x, u] = F0 (t, x, u)dt
t0

subject to constraints
ẋi = Fi (t, x, u).
We can incorporate the constraints into the functional using the
Lagrange multipliers λi , e.g.
Zt1
Jb = L(t, x, ẋ, u)dt
t0
Zt1 n
X
= F0 (t, x, u)dt + λi (t) [ẋi − Fi (t, x, u)] dt
t0 i=1

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 6 / 25
PMP proof sketch-2

Given such a function we get (by definition)

∂L
pi = = λi .
∂ ẋi

So we can identify the Lagrange multipliers λi with the generalized

momentum terms pi
1 the pi are known in economics literature as marginal valuation of
xi or the shadow prices
2 shows how much a unit increment in x at time t contributes to the
optimal objective functional Jb
3 the pi are known in control as co-state variables (sometimes
written as zi ).

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 7 / 25
PMP proof sketch-3

By definition (in previous lecture) the Hamiltonian is

n
X
H(t, x, p, u) = pi ẋi − L(t, x, ẋ, p, u)
i=1
Xn n
X
= pi ẋi − F0 (t, x, u) − λi (t) [ẋi − Fi (t, x, u)]
i=1 i=1
n
X
= −F0 (t, x, u) + pi Fi (t, x, u)
i=1

because λi = pi , so the ẋi terms cancel. The final result is just the
Hamiltonian as defined in the PMP.

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 8 / 25
PMP proof sketch-4
From previous slide the Hamiltonian can be written
n
X
H(t, x, p, u) = −F0 (t, x, u) + pi Fi (t, x, u)
i=1

which is the Hamiltonian defined in the PMP. Then the canonical E-L
equations (Hamilton’s equations) are

∂H dxi ∂H dpi
= and =− .
∂pi dt ∂xi dt
∂H dxi
Note that the equations ∂pi = dt just revert to

Fi (t, x, u) = ẋi

which are just the system equations.

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 9 / 25
PMP proof sketch-5

Finally, note that Hamilton’s equations above only relate xi and pi .

What about equations for ui ?
Take the conjugate variable to be zi , and we get (by definition) that

∂L
zi = =0
∂ u̇i

and the second of Hamilton’s equations is therefore

∂H dzi
=− =0
∂ui dt

which suggests a stationary point of H WRT ui .

In fact we look for a maximum (and note this may happen on the
bounds of ui ).

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 10 / 25
PMP Example: plant growth

Example 18.1 (Plant growth-1)

Plant growth problem:

market gardener wants to plants to grow toa fixed height 2 within a

fixed window of time [0, 1]

can supplement natural growth with lights (at night)

growth rate dictates

ẋ = 1 + u
cost of lights
Z1
1 2
J[u] = u dt
2
0

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 11 / 25
PMP Example: plant growth

Example 18.1 (Plant growth-2)

Minimize
Z1
1 2
J[u] = u dt
2
0

subject to x(0) = 0 and x(1) = 2 and

ẋ = F1 (t, x, u) = 1 + u.

Hamiltonian is

H = −F0 (t, x, u) + pF1 (t, x, u)

1
= − u 2 + p (1 + u) .
2

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 12 / 25
PMP Example: plant growth

Example 18.1 (Plant growth-3)

Hamiltonian is
1
H = − u 2 + p (1 + u) .
2
Canonical equations

∂H dx ∂H dp
= and =−
∂p dt ∂x dt
⇓ ⇓
1 + u = ẋ 0 = −ṗ

LHS ⇒ system DE
RHS ⇒ ṗ = 0 means that p = c1 where c1 is a constant.

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 13 / 25
PMP Example: plant growth

Example 18.1 (Plant growth-4)

Maximum principle requires H be a maximum, for which

∂H
= −u + p = 0.
∂u
So u = p, and ẋ = 1 + u so

x = (1 + c1 ) t + c2 .

The solution which satisfies x(0) = 0 and x(1) = 2 is

x = 2t.

So u = c1 = 1, and the optimal cost is 21 .

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 14 / 25
PMP and natural boundary conditions

Typically we fix t0 and x(t0 ), but often the right-hand boundary

condition is not fixed, so we need natural boundary conditions.

Here, they differ from traditional CoV problems in two respects:

The terminal cost φ

The function F0 is not explicitly dependent on ẋ.

The resulting natural boundary conditions are

X  ∂φ     
 ∂φ 
+ pi δxi 
 + − H δt  =0
∂xi t=t1 ∂t t=t1
i

for all allowed δxi and δt.

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 15 / 25
PMP and natural boundary conditions
The resulting natural boundary condition is
X  ∂φ     
 ∂φ 
+ pi δxi 
 + − H δt  = 0.
∂xi t=t1 ∂t t=t1
i

Special cases
when t1 is fixed and x(t1 ) is completely free we get
  
∂φ 
+ pi δxi  = 0, ∀i
∂xi t=t1

when x(t1 ) is fixed, δxi = 0, and we get

  
∂φ 
− H δt  = 0.
∂t t=t1

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 16 / 25
Example: stimulated plant growth
Example 18.2 (Stimulated plant growth-1)
Plant growth problem:

market gardener wants to plants to grow as much as possible

within a fixed window of time [0, 1]
supplement natural growth with lights as before
growth rate dictates ẋ = 1 + u
cost of lights
Z1
1 2
J[u] = u (t)dt
2
0

value of crop is proportional to the height

φ(t1 , x(t1 )) = x(t1 ).

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 17 / 25
Plant growth problem statement

Example 18.2 (Stimulated plant growth-2)

Write as a minimization problem

Z1
1 2
J[x, u] = −x(t1 ) + u dt
2
0

subject to x(0) = 0, and

ẋ = 1 + u.

the terminal cost doesn’t affect the shape of the solution

but we need a natural end-point condition for t1 .

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 18 / 25
Plant growth: natural BC

Example 18.2 (Stimulated plant growth-3)

The problem is solved as before, but we write the natural boundary
condition at x = t1 as
 
∂φ 
+ pi  = 0, ∀i
∂xi t=t1

which reduces to 
−1 + pt=t = 0.
1

Given p is constant, this sets p(t) = 1, and hence the control u = 1 (as
before).

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 19 / 25
Autonomous problems

Autonomous problems have no explicit dependence on t.

time invariance symmetry

hence H is constant along the optimal trajectory

if the end-time is free (and the terminal cost is zero) then the
transversality conditions ensure H = 0 along the optimal trajectory.

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 20 / 25
PMP example: Gout
Example 18.3 (Gout-1)
Optimal treatment of Gout:
disease characterized by excess of uric acid in blood
define level of uric acid to be x(t)
in absence of any control, tends to 1 according to

ẋ = 1 − x

drugs are available to control disease (control u)

ẋ = 1 − x − u

aim to reduce x to zero as quickly as possible

drug is expensive, and unsafe (side effects)

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 21 / 25
PMP example: Gout
Example 18.3 (Gout-2)
Formulation: minimize
Zt1
1 2 
J[u] = k + u 2 dt
2
0

given constant k that measures the relative importance of the drugs

cost vs the terminal time.
End-conditions are x(0) = 1, and we wish x(t1 ) = 0, with t1 free. The
constraint equation is
ẋ = 1 − x − u,
Hamiltonian
1 2 
H=− k + u 2 + p (1 − x − u) .
2

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 22 / 25
PMP example: Gout
Example 18.3 (Gout-3)
Canonical equations

∂H dx ∂H dp
= and =−
∂p dt ∂x dt
⇓ ⇓
1 − x − u = ẋ −p = −ṗ

LHS ⇒ system DE
RHS ⇒ ṗ = p has solution p = c1 et .
Now maximize H WRT the u, i.e., find stationary point

∂H
= −u − p = 0
∂u
So, u = −p = −c1 et .

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 23 / 25
PMP example: Gout
Example 18.3 (Gout-4)
Note
this is an autonomous problem so H = const

this is a free end-time problem, so H = 0.

Substitute values of p and u into H for t = 0 (i.e. p = c1 = −u, and

x(0) = 1), and we get

1 2 2

H=− k + u + p (1 − x − u)
2
k 2 c2
= − − 1 − c12
2 2
=0

and so c1 = ±k .

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 24 / 25
PMP example: Gout
Example 18.3 (Gout-5)
Finally solve ẋ = 1 − x − u where u = −ket to get

k t k −t
x =1− e + e = 1 − k sinh t
2 2
The terminal condition is x(t1 ) = 0, and so

t1 = sinh−1 (1/k )

when k is small the prime consideration is to use a small amount

of the drug, and as k → 0 then t1 → ∞
no optimal for k = 0
when k is large, we want to get to a safe level as fast as possible,
so as k → ∞ we get t1 ∼ 1/k .

c Daria Apushkinskaya 2014 () Calculus of variations lecture 18 11. Juli 2014 25 / 25