EE6412: Optimal Control January - May 2023

Homework Set #4
1. Investigate the extremals of the functional
ˆ a
y(x)
J(y) = 0 2
dx, y(0) = 1, y(a) = A,
0 (y (x))

where a > 0 and 0 < A < 1. Show that two extremals go through every pair of points (0, 1)
and (a, A). Which of these two extremals corresponds to a weak minimum?

2. A pendulum consists of a mass m at the end of a massless stick of length l. The other end
of the stick is made to oscillate vertically with a position given by y(t) = Acos(ωt), where
A  l. See Figure 1. It turns out that if ω is large enough, and if the pendulum is initially
nearly upside-down, then surprisingly it will not fall over as time goes by. Instead, it will
(sort of) oscillate back and forth around the vertical position. Find the equation of motion
for the angle of the pendulum (measured relative to its upside-down position). Hint: Use the
Hamilton’s principle of least action.

Figure 1: Inverted Pendulum

3. Consider the following optimal terminal time problem

 ˆ T  
−ρt −ρT
J(T ) = max − ce dt + e x(T ) ,
T 0

where ẋ(t) = b, x(0) = x0 is given and the constants satisfy b > 0, c > 0 and ρ > 0.

(a) Find the optimal terminal time T using calculus of variations approach. Hint: Notice,
there is no control variable here!
(b) Find the optimal terminal time T using Pontryagin’s maximum principle.

1
4. Consider the following optimal control problem
ˆ T
minu 1 dt
0
ẋ1 (t) = x2 (t)
ẋ2 (t) = −x1 (t) + u(t)
x1 (0) = 1, x2 (0) = 1, x1 (T ) = 0, x2 (T ) = 0
u(t) ∈ [−1, 1], T is free

(a) Show that there does not exist singular controls.

(b) Show that the optimal control must switch at least every π time units
(c) Plot the optimal state trajectory in (x1 , x2 )-plane.
(d) Find the optimal terminal time T ∗ .

Hint: The Hamiltonian is linear in control, this implies that the optimal control is Bang-Bang
type.

5. Consider the machine state dynamics

ẋ(t) = −δx(t) + u(t), x(0) = x0 > 0.

where δ > 0 is the rate of deterioration of the machine state and u(t) is the rate of machine
maintenance at time t. Find the optimal maintenance rate:
ˆ T
φx(t) − 21 u2 (t) dt.


max J(u), J(u) := S x(T ) +
u 0

where φ > 0 with φx(t) representing the profit rate when the machine state is x(t), 12 u2 (t)
is the cost of maintaining the machine at rate u(t), T is the time horizon, and S > 0 is the
salvage or terminal value of the machine for each unit of the machine state at time T . Fur-
thermore, show that the optimal maintenance rate decreases, increases, or remains constant
over time depending on whether the difference S − φδ is negative, positive, or zero, respec-
tively.
Hint: Convert the maximization problem as the standard minimization problem and use Pon-
tryagin’s maximum principle

6. Consider the following optimal control problem.

ˆ 1
1 2 2


min J(u), J(u) = 2
x (t) + u (t) dt
u 0
3
subject to ẋ(t) = −x (t) + u(t), x(0) = 5.

(a) Formulate the necessary conditions for optimality using Pontryagin’s maximum prin-
ciple.

2
 (b) Formulate the non-linear two point boundary value problem and solve it using the Mat-
lab routine bvp4c.
(c) Modify the above free end point problem as a fixed end point problem by taking x(1) =
1. Solve the associated two point boundary value problem using the Matlab routine
bvp4c.

7. In textbook [T2], read section 5.2, page 219 on Linear Tracking Problems. Go through
examples 5.2.3 and 5.2.4.
Consider 2 agents (e.g., robots) denoted by {r1 , r2 }. We assume that the agents interact in a
two-dimensional plane. The dynamics of agents is given by
Agent 1 : ẋ1 (t) = u1x (t), ẏ1 (t) = u1y (t)
Agent 2 : ẋ2 (t) = u2x (t), ẏ2 (t) = u2y (t)
where (x1 (t), y1 (t)) ∈ R2 and (x2 (t), y2 (t)) ∈ R2 is the position vectors at time t of the
agents r1 and r2 respectively. (u1x (t), u1y (t)) and (u2x (t), u2y (t)) represent the control inputs
at time t of the agents r1 and r2 respectively. (x10 , y10 ) ∈ R2 and (x20 , y20 ) ∈ R2 represent
the initial position vectors of the agents r1 and r2 respectively.
The objective of agent r1 is to minimize its distance with the agent r2 while simultaneously
minimizing its control effort, that is,

Jr1 (u1x , u1y ) = (x1 (T ) − x2 (T ))2 + (y1 (T ) − y2 (T ))2

ˆ T
(x1 (t) − x2 (t))2 + (y1 (t) − y2 (t))2 + u21x (t) + u21y (t) dt


+
0

The objective of agent r2 is to track a prespecified path given by Y (t) = (sin(αt), cos(αt))
while minimizing its control efforts, that is,

Jr2 (u2x , u2y ) = (x2 (T ) − sin(αT ))2 + (y2 (T ) − cos(αT ))2

ˆ T
(x2 (t) − sin(αt))2 + (y2 (t) − cos(αt))2 + u22x (t) + u22y (t) dt


+
0

The agents can coordinate their actions towards achieving their objectives by jointly solving
the following optimization problem

J(u1x , u1y , u2x , u2y ) = Jr1 (u1x , u1y ) + Jr2 (u2x , u2y ).

(a) Formulate the linear quadratic formulation of the joint optimization problem.
(b) Formulate the Riccati differential equation.
(c) Choose initial locations and parameter values, and provide sample numerical simula-
tions. Provide a detailed illustration of the results and explain your observations. Use
Matlab routine ode45 for solving the Riccati differential equation.

3
 (d) Compute the state, control and co-state trajectories using the solution of the Riccati
differential equations (obtained in step (c)) and plot them.
(e) Take Y (t) = (0, 0),
ˆ T
(x1 (t) − x2 (t))2 + (y1 (t) − y2 (t))2 + u21x (t) + u21y (t) dt


Jr1 (u1x , u1y ) =
0

and
ˆ T
(x2 (t) − 0)2 + (y2 (t) − 0)2 + u22x (t) + u22y (t) dt.


Jr2 (u2x , u2y ) =
0

Repeat steps (a), (b) and (c). Provide your observations for large values of T .

8. (Sufficient conditions for optimality) Consider the optimal control problem

ˆ T
OPT : min J(u), J(u) = K(T, x(T )) + L(t, x(t), u(t))dt
u∈U 0
subject to ẋ(t) = f (t, x(t), u(t)), x(0) = x0 is given, x(T ) is free.

Assume that there exist control u? (t) and the corresponding state trajectory (generated by
u? (t)), given by x? (t) and a multiplier process p(t) satisfy the following conditions
∂H ?
ẋ? (t) = (t, x? (t), p(t)), x? (0) = x0
∂p
∂H ∗ ∂K
ṗ(t) = − (t, x? (t), p(t)), p(T ) = − (T, x? (T )),
∂x ∂x
where H ? (t, x(t), p(t)) = H(t, x(t), u? (t), p(t)) and u? (t) satisfies
u? (t) = arg maxw H(t, x(t), w, p(t)). If H ? (t, x(t), p(t)) is strictly concave in x(t) for each
t and p(t) and K(T, x(T )) is strictly convex in x(T ), then u? (t), t ∈ [0, T ] is optimal for the
problem OPT.
Hint: use envelope theorem, which says that if H ? (t, x(t), p(t)) = H(t, x(t), u? (t), p(t))
?
then ∂H ∂x
(t, x(t), p(t)) = ∂H
∂x
(t, x(t), u? (t), p(t)) = ∂H
∂x
(t, x(t), u(t), p(t)) u(t)=u? (t) .
use the idea that a differentiable convex function g : Rn → R satisfies g(z) − g(z ? ) ≥
dg 0
dz
(z ? ) (z − z ? ) for any z, z ? ∈ Rn .

9. (Discrete-time maximum principle) Consider the following discrete-time controlled system

xt+1 = f (t, xt , ut )

where xt ∈ Rn , ut ∈ Rm for t = 0, 1, 2, · · · , T − 1 and the initial condition x0 = α ∈ Rn

is given and x(T ) is free. The objective is to minimize the following stage additive cost
functional
T −1
X
J(u) = h(T, x(T )) + L(t, xt , ut )
t=0

4
(a) Assume the Hamiltonian function as H(t, xt , ut , pt+1 ) = p0t+1 f (t, xt , ut ) − L(t, xt , ut ).
Derive the necessary conditions for optimality as:
∂H
x∗t+1 = (t, x?t , u?t , pt+1 ). x?0 = α
∂pt+1
∂H ∂h
pt = (t, x?t , u?t , pt+1 ), pT = − (T, x?T )
∂xt ∂xT
∂H
(t, x?t , u?t , pt ) = 0
∂u
Hint: Collecting the decision variables (u0 , u1 , · · · , uT −1 ) as one variable u ∈ RmT ,
and (x0 , x1 , · · · , xT ) as x ∈ Rn(T +1) the discrete-time optimal control problem can be
seen as a equality constrained finite-dimensional optimization on RmT +n(T +1) .
(b) In the necessary conditions what differences did you observe with the continuous-time
case.