Optimization and Control: Examples Sheet 3 This is the eikonal equation of geometric optics; a short-wavelength form of the wave

Continuous-time Models equation. How is the optimal direction at a given point determined from F ?
[Hint: Show that the DP equation is inf u:|u|=1 [1 + v(x)u⊤ ∇F ] = 0. Then use a
1. [lecture 12] Consider the continuous-time system with scalar state variable, plant Cauchy-Schwartz inequality to show that the infimum is achieved by u = −∇F/|∇F |.]
Rh
equation ẋ = u and cost function Q 0 u2 dt + x(h)2 . By writing the DP equation
in infinitesimal form and taking the appropriate limit, show that the value function
satisfies  
∂F 2 ∂F 5. [lecture 13] Consider the optimal control problem:
0= + inf Qu + u , s > 0.
∂t u ∂x Z T
1 2
Show that F and the optimal control with time s to go are minimize 2 u(t) dt subject to ẋ1 = −x1 + x2 , ẋ2 = −2x2 + u ,
0
Qx2 x where u is unrestricted, x1 (0) and x2 (0) are known, T is given and x1 (T ) and x2 (T ) are
F = , u=− .
Q+s Q+s to be made to vanish. Rewrite the problem in terms of new variables, z1 = (x1 + x2 )et
and z2 = x2 e2t and then show that the optimal control takes the form u = λ1 et + λ2 e2t ,
for some constants λ1 and λ2 . Find equations for x1 (0), x2 (0) in terms of λ1 , λ2 , and
2. [lecture 12] Consider the ‘inertial’ system (generalising Example 1) in which the T , which you could in principle solve for λ1 , λ2 in terms of x1 (0), x2 (0) and T .
two components of the state vector obey ẋ1 = x2 and ẋ2 = u and the cost function is Compare a linear feedback controller of the form u(t) = −k1 x1 (t) − k2 x2 (t),
Rh
Q 0 u2 dt + x1 (h)2 . Show that the value function and optimal control are where k1 and k2 are constants. Show that with this controller x1 and x2 cannot be
Q 1 made to vanish in finite time. Discuss the choice of optimal control with a quadratic
Fs = (x1 + sx2 )2 , u=− s(x1 + sx2 ). performance criterion as opposed to linear feedback control, indicating which is likely
Q + s3 /3 Q + s3 /3
to be more appropriate in given circumstances.
Note that this is consistent with the discrete-time assertion of the example on Examples
Sheet 2. Hint: Guess a solution of the form Fs (x) = (x1 + sx2 )2 πs .

6. [lecture 13] A princess is jogging with speed r in the counterclockwise direction
3. [lecture 12] Miss Prout holds the entire remaining stock of Cambridge elderberry around a circular running track of radius r, and so has a position whose horizontal and
wine for the vintage year 1959. If she releases it at rate u (in continuous time) she vertical components at time t are (r cos t, r sin t), t ≥ 0. A monster who is initially
realises a unit price p(u). She holds an amount x at time 0R and wishes to release this located at the centre of the circle can move with velocity u1 in the horizontal direction
∞
in such a way as to maximize her total discounted return, 0 e−αt up(u) dt. Consider and u2 in the vertical direction, where both velocities have a maximum magnitude of
−γ
the particular case p(u) = u , where the constant γ is positive and less than one. 1. The monster wishes to catch the princess in minimal time.
Show that the value function is proportional to a power of x and determine the optimal Analyse the monster’s problem using Pontryagin’s maximum principle. By consid-
release rule in closed-loop form (i.e., as a function of the present stock level.) ering feasible values for the adjoint variables, show that whatever the value of r the √
[Hint: The answers are F (x) = (γ/α)γ x1−γ , u = αx/γ. However, you should try monster should always set at least one of |u1 | or |u2 | equal to 1. Show that if r = π/ 8
to derive these answers from the DP equation; not simply substitute them into the DP then the monster catches the princess in minimal time by adopting the uniquely optimal
equation and check that they work.] policy u1 = 1, u2 = 1. Is the optimal policy always unique?
[Hint: Let x1 and x2 be the differences in the horizontal and vertical directions
between the positions of the monster and princess.]
4. [lecture 12] Let the vector x denote the Cartesian co-ordinates of a particle moving
in Rd . When at position x the particle moves with speed v(x) and in a direction that 7. [lecture 14] In the neoclassical economic growth model, x is the existing capital per
can be chosen. The equation of motion is thus ẋ = v(x)u, where u is a unit vector to worker and u is consumption of capital per worker. The plant equation is
be chosen afresh at each position x. Let F (x) denote the minimal time taken for the ẋ = f (x) − γx − u, (4)
particle to reach a set D from a point x outside it. Show that after minimizing over u
the dynamic programming equation for F implies that |∇F (x)| = v(x)−1 ; i.e., where f (x) is the production per worker, and −γx represents depreciation of capital
and change in the size of the workforce. We wish to choose u to maximize
d  2
X ∂F Z T
= v(x)−2 .
∂x j e−αt g(u) dt,
j=1 t=0

9 10
where g(u) measures utility, is strictly increasing and concave, and T is prescribed. It P
is convenient to take a Hamiltonian

H = e−αt [g(u) + λ(f (x) − γx − u)] , y

c(y)
thereby including a discount factor in the definition of λ and expressing F in terms of
present value. θ
Show that the optimal control satisfies g ′ (u) = λ (assuming the maximum is at a
stationary point) and x
λ̇ = (α + γ − f ′ )λ. (5)
Hence show that the optimal consumption obeys
Figure 1: Zermelo’s problem
1 g ′′ (u)
u̇ = [f ′ (x) − α − γ], where σ(u) = − ′ > 0. (6)
σ(u) g (u) (ii) Suppose the boatman wishes to reach a given point P on the opposite bank in
minimal time. Show that he should follow the heading
(σ is called the ‘elasticity of marginal utility.’)  
Characterise an equilibrium solution, i.e., an x(0) = x̄ such that the optimal −1 λ1 v
θ = cos ,
trajectory is x(t) = x̄, t ≥ 0, and show that this x̄ is independent of g. 1 − λ1 c(y)
where λ1 is a parameter chosen to make his path pass through the target point.
8. [lecture 14] An aircraft flies in straight and level flight at height h, so that lift L
balances weight mg. The mass rate of fuel consumption is proportional to the drag, 10. [lecture 15] Customers arrive at a queue as a Poisson process of rate λ. They are
and may be taken as q = av 2 + b(Lv)−2 , where a and b are constants and v is the speed. served at rate u = u(x), where x denotes the current size of the queue. Suppose that
Thus cost is incurred as rate ax+ bu where a, b > 0. The service rate u is the control variable.
b
ṁ = −q = −av 2 − . The dynamic programming equation in the infinite horizon limit is then
(mgv)2
Find a rule for determining v in terms of m if (i) the distance flown is to be maximized, γ = inf {ax + bu(x) + λ[f (x + 1) − f (x)] + u(x)1x>0 [f (x − 1) − f (x)]}
u
(ii) the time spent flying at height h (until fuel is exhausted) is to be maximized.
where γ denotes the average rate at which cost is incurred under the optimal policy and
"  1/4  1/4 # where f (x) denotes the extra cost associated with starting from state x. (Here 1x>0 = 0
3b b
Hint: Answers are (i) v = , and (ii) v = . if x = 0, and 1x>0 = 1 if x = 1, 2, 3, . . . .) Give a brief justification of this equation.
a(mg)2 a(mg)2 Show that under the constraint that u is a fixed constant, independent of x, and
greater that λ then, for some C, there is a solution of the form
aλ ax(x + 1)
9. [lecture 14] In what is known as Zermelo’s problem, a straight river has current c(y), γ= + bu, f (x) = C + .
u−λ 2(u − λ)
where y is the distance from the bank from which a boat is leaving. The boat then
crosses the river at constant speed v relative to the water, so that its position (x, y) i.e., such that f (x) does not grow exponentially in x (which is needed to ensure that
satisfies ẋ = v cos θ + c(y), ẏ = v sin θ, where θ is the heading angle indicated in the (1/t)Ef (xt ) → 0 as t → ∞ and hence, similarly as in the proof for a discrete time
diagram. model, that γ can be shown to be the time-average cost.) What is the optimal constant
(i) Suppose c(y) > v for all y and the boatman wishes to be carried downstream as service rate?
little as possible in crossing. Show that he should follow the heading Suppose now that we allow u to vary with x, subject to the constraint m ≤ u ≤ M ,
where M > λ. What is the policy which results if we carry out one stage of policy
θ = cos−1 (−v/c(y)). improvement to the optimal constant service policy?

11 12