Pontryagins Minimum Principle.

Optimal
Control with Constraints on Inputs

1
Numerical determination of optimal
trajectories

1.1

Two point boundary value problem

In previous chapter variational techniques were used to derive necessary conditions for optimal control. The problem was stated as follows:
Consider the problem of minimizing the performance measure
J = h(x(tf ), tf ) +

tf

t0

g(x(t), u(t), t)dt

(1.1)

x(t0 ) = x0

(1.2)

subject to:
x = f(x(t), u(t), t),

The problem is to find an admissible control u (t) that causes the system
(1.2) to follow an admissible trajectory x (t) that minimizes the performance
measure (1.1).
The hamiltonian, has been defined as:
H(x(t), u(t), (t), t) = g(x(t), u(t), t) + T (t)f(x(t), u(t), t)
Assuming that the state and control variables are not constrained, that
the final time tf is fixed and the final state is free, we can summarize the
two-point boundary value problem (TPBVP) that results from the variational
approach by the equations, (Kirk, 2004):
H
= f(x , u , t)

H
=
x

x =

(1.3)

(1.4)

H
u
x (t0 ) = x0
h
(tf ) =
(x (tf ))
x
0 =

(1.5)
(1.6)
(1.7)

From these five sets of conditions it is desired to obtain an explicit relationship for x (t) and u (t), t [t0 , tf ]. The difficulty of solving the differential equations in this case is caused by the combination of split boundary
values and nonlinear differential equations.
Note that the differential equations (1.3) and (1.4) should be solved, in
general, simultaneously because they depend on the same variables (the
states and the adjoint variables). The initial or final conditions, however,
are split as shown in (1.6) and (1.7), as the first ones are the values of the
states at the initial time and the rest are the values of the adjoint variables
at the final time. A classical numerical method for integration cannot be
applied.
The algorithm (Kirk, 2004) is based on the observation that u (t) which
minimizes the hamiltonian will minimize J. Thus, the numerical algorithm
will determine the control u which makes the first derivative of the hamiltonian equal to zero.
Write the hamiltonian H(x(t), u(t), (t)).
H(x(t), u(t), (t), t) = g(x(t), u(t), t) + T f(x(t), u(t), t)

(1.8)

Calculate the first derivative of the hamiltonian with respect to u:

H/u.
Write the differential equations of the adjoint variables:
H
,
i =
xi
Determine the final conditions for (tf )
Start with a random discrete approximation for u(t), t [0, tf ]. For
example divide the interval [t0 , tf ] into N subintervals and consider the
control u constant on each subinterval.
u(t) = u(tk ), k = 1..N

u(t)

uk
initial guess

uk

optimal control u*
1
t0

tf

k
t

Figure 1.1: Discrete control variable

Minimize H(u) using a Matlab function or a numerical algorithm to
determine the minimum (for example gradient methods). The function
to be minimized has to solve at each step:
Using the vector u integrate the state equations (use Runge-Kutta
or Euler methods), with the initial conditions x(0) = x0 to obtain
a state trajectory constant on N subintervals.
Calculate (tf )
Solve the adjoint system of differential equations reversely, from
tf to t0 .
Using u, x and calculate the first derivative of the hamiltonian.
The procedure to solve the optimal control problem using the steepest
descent method can be summarized as follows:
Select a discrete approximation of the control variable u(t), t = [t0 , tf ],
by subdividing the interval [t0 , tf ] into N subintervals and considering
the control variable as being piecewise constant during each of these
subintervals.
Using the N control values, integrate the state equations (1.3) from t0
to tf , with initial conditions (1.6) and obtain a state trajectory as a
piecewise constant vector function.
Calculate (tf ) by substituting x(tf ) into (1.7).
Integrate (1.4) from tf to t0 , with final conditions (1.7) using the
piecewise-constant values of the states calculated before.

 If

H
k
(1.9)
u
where is a small positive constant, terminate the iterative procedure
and output the extremal state and control values.
k

If the stopping criterion (1.9) is not satisfied, generate a new piecewise

constant control, function given by:
uu

H
u

where is the step of the steepest descent method.

(1.10)

2
Pontryagins Minimum Principle

2.1

Constrained optimal control and the minimum principle

We have assumed that the admissible controls and states are not constrained
by any boundaries; however, in realistic systems such constraints commonly
occur. Physically realizable controls generally have magnitude limitations:
the thrust of a rocket engine cannot exceed a certain value; motors, which
provide torque, saturate; attitude control mass expulsion systems are capable of providing a limited torque. State constraints often arise because of
safety or structural restrictions: the current in an electric motor cannot exceed a certain value without damaging the windings; the turning radius of a
maneuvering aircraft cannot be less than a specified minimum value (Kirk,
2004).
The general approach in which we consider the effect of control constraints
and show the necessary conditions leads to the Pontryagins minimum principle.
u(t)

inadmissible
region
u(t)
admissible
region
t0

tf

Figure 2.1: An extremal control that is constrained by a boundary

Figure 2.1 shows an example of bounded controls. All functions bounded

by vertical lines between t0 and tf and placed in the admissible region are
admissible controls.
It can be demonstrated (Kirk, 2004) that a necessary condition for u (t)
to minimize the functional J is:
H(x (t), u (t), (t), t) H(x (t), u(t), (t), t)

(2.1)

for all t [t0 , tf ] and for all admissible controls.

Equation (2.1) which indicates that an optimal control must minimize the
hamiltonian is called Pontryagins minimum principle.
Notice that we have established a necessary condition but not (in general)
sufficient condition for optimality. An optimal control must satisfy Pontryagins minimum principle; however, there may be controls that satisfy the
minimum principle that are not optimal.
Let us now summarize the principal results.
The problem is to find a control u (t) U (where U is the class of
admissible controls), which causes the system

x(t)
= f(x(t), u(t), t)

(2.2)

to follow an admissible trajectory that minimizes the performance measure

J(u) = h(x(tf ), tf ) +

tf

t0

g(x(t), u(t), t)dt

(2.3)

In terms of the hamiltonian:

H(x(t), u(t), (t), t) = g(x(t), u(t), t) + T (t)f(x(t), u(t), t)

(2.4)

the necessary conditions for u (t) to be optimal are:

H
(x (t), u(t) , (t), t) = f(x (t), u(t) , t)

H
(t) =
(x (t), u(t) , t)
x

x (t) =

H(x (t), u (t), (t), t) H(x (t), u(t), (t), t)

(2.5)
(2.6)
(2.7)

for all admissible controls u(t) and for all t [t0 , tf ]

The following boundary conditions hold, when the final state is free and
the final time is fixed:
h
(x (tf ), tf ) = (tf )
(2.8)
x
6

and:

h
(x (tf ), tf ) = 0
x
when the final state is fixed and the final time is free.
It should be emphasized that:
H(x (tf ), u (tf ), (tf ), tf ) +

(2.9)

u (t) is a control that causes H(x (t), u (t), (t), t)to assume its global,
or absolute minimum.
Equations (2.5), (2.6), (2.7), (2.8) consitute a set of necessary conditions for optimality; these conditions are not, in general, sufficient.
In addition, the minimum principle, although derived for controls with
values in a closed and bounded region, can also be applied to problems in
which the admissible controls are not bounded. This can be done by viewing
the unbounded control region as having arbitrarily large bounds, thus ensuring that the optimal control will not be constrained by boundaries. In this
case, for u (t) to minimize the hamiltonian it is necessary (but not sufficient)
that
H
(x (t), u (t), (t), t) = 0
(2.10)
u
If equation (2.10) is satisfied, and the matrix:
2H
(x (t), u (t), (t), t)
u2
is positive definite, this is sufficient to guarantee that u (t) causes H to be a
local minimum; if the hamiltonian can be expressed in the form:
1
H(x(t), u(t), (t), t) = f(x, , t) + c(x, , t)T u(t) + uT R(t)u
2

(2.11)

where c is a m 1 array that does not have any terms containing u(t), then
2
satisfaction of (2.10) and uH2 be positive definite are necessary and sufficient
for H(x (t), u (t), (t), t) to be a global minimum.
For H of the form (2.11)
2H
(x (t), u (t), (t), t) = R(t)
u2
thus, if R(t) is positive definite
u (t) = R1 (t)c(x (t), (t), t)
minimizes the hamiltonian.
7

Example 2.1 (Weber, 2000) Consider the problem of accelerating a skateboard in such a way as to maximize the total distance traveled in a given time
T , minus the effort expended. Denote x1 (t) the distance traveled at time t.
The speed x2 (t) is the first derivative of x1 , and let the acceleration be u(t),
the first derivative of x2 . The dynamical system that describes the problem
is:
x 1 (t) = x2 (t)
x 2 (t) = u(t)

(2.12)

where x1 (0) = 0, x2 (0) = 0. The performance measure to be minimized in

this case is:
1Z T 2
J = x1 (T ) +
u (t)dt
(2.13)
2 1
Note that the first term in J is the negative distance traveled in time T ,
i.e. while this term will be minimized, the distance x1 (T ) is maximized. The
squared acceleration integrated over the entire time interval corresponds to
the effort expended in time T .
The hamiltonian is
1
H(x, u, ) = u2 (t) + 1 (t)x2 (t) + 2 (t)u(t)
2

The control u that minimizes the hamiltonian is computed from

H
= u(t) + 2 (t) = 0
u
or
u (t) = 2 (t)
This control will minimize indeed the hamiltonian since the second derivative
is positive, or
2H
=1>0
u2
The costates are computed from:
H
=0
1 (t) =
x1
H
2 (t) =
= 1 (t)
(2.14)
x2
Since the final states are free and the final time is given (T ), the final
conditions for 1,2 are given by:
h(x(T ))
= 1
x1 (T )
h(x(T ))
=0
2 (T ) =
x2 (T )
1 (T ) =

(2.15)

where
h(x(T )) = x1 (T )
From (2.14) and (2.15) we obtain:
1 (t) = 1
2 (t) = t T

(2.16)

Because u (t) = 2 (t), we obtain the optimal control law as:

u (t) = T t
which decreases linearly with time and reaches zero at the final time.
Example 2.2 Let us now illustrate the effect on the necessary conditions of
constraining the admissible control values. Consider the system having the
state equations:
x 1 (t) = x2 (t)
x 2 (t) = x2 (t) + u(t)

(2.17)

with the initial conditions x(t0 ) = x0 . The performance measure to be minimized is:
Z tf
1 2
J(u) =
[x1 (t) + u2 (t)]dt
(2.18)
t0 2
tf is specified and the final state x(tf ) is free.
a) Find the necessary conditions for an unconstrained control to minimize
J.
1
H(x, u, ) = [x21 (t) + u2 (t)] + 1 x2 (t) + 2 (t)(x2 (t) + u(t)) (2.19)
2
The costate equations are:
H
1 (t) =
= x1 (t)
x1
H
2 (t) =
= 1 (t) + 2 (t)
x2

(2.20)

Since the control values are unconstrained, it is necessary that:

H
= u (t) + 2 (t) = 0 u (t) = 2 (t)
u
9

(2.21)

Notice that the hamiltonian is of the form (2.11) and

2H
=1>0
u2
therefore u (t) does minimize the hamiltonian.
The costate boundary conditions are:
h
(x(tf ), tf ) = 0
x1
h
2 (tf ) =
(x(tf ), tf ) = 0
x2

1 (tf ) =

(2.22)

since h does not appear in J.

b) Find necessary conditions for optimal control if:
1 u(t) 1, f or all t [t0 , tf ]

(2.23)

The state and costate equations and the boundary conditions for (tf )
remain unchanged; however u must be selected to minimize
1
1 2

H(x , u, ) = x2
1 + u + 1 x2 + 2 (x2 + u)
2
2
subject to the constraining relations (2.23). To determine the control
that minimizes H, we first separate all of the terms containing u(t)
1 2
u (t) + 2 (t)u(t)
2

(2.24)

from the hamiltonian. For times when the optimal control is unsaturated, we have:
u (t) = 2 (t)
as in part a); clearly this will occur when |2 (t)| 1. If however, there
are times when |2 (t)| > 1 then, from (2.24) the control that minimizes
H is:
(
1 f or 2 (t) > 1

u (t) =
1 f or 2 (t) < 1
Thus u (t) is the saturated function of 2 (t) pictured in Figure 2.2. For
the constrained control we have:

1 f or 2 (t) > 1
u (t) = 2 (t) f or 1 2 (t) 1

1 f or 2 (t) < 1
10

(2.25)

u(t)
1
s

l2*(t)

-1
-1

Figure 2.2: Saturated control

To determine u (t) explicitely, the state and costate equations must be
solved. Because of the differences in equations (2.21) and (2.25) the
state-costate trajectories in the two cases will be the same only if the
initial state values are such that the bounded control does not saturate.
If this situation occurs, the control constraints do not affect the solution.
Note that the optimal control history for part b) cannot be determined,
in general, by calculating the optimal control history for part a) and allowing
to saturate whenever the stipulated boundaries are violated.

2.2

Additional necessary conditions

Other necessary conditions may be obtained for the special case when the
hamiltonian does not depend explicitly on time, or the optimal control problem is stated as follows:
Determine the optimal control u (t) that causes the system
x(t)

= f(x(t), u(t)), t0 t tf

(2.26)

to follow an optimal trajectory x (t), while minimizing the performance measure:

Z tf
J = h(x(tf )) +
g(x(t), u(t))dt
(2.27)
t0

Additional condition 1: If the final time tf is fixed, the hamiltonian is

constant along the optimal trajectory, or:
H(x (t), u (t), (t)) = C, t0 t tf
11

(2.28)

Additional condition 2: If the final time tf is free, the hamiltonian is zero

along the optimal trajectory, or:
H(x (t), u (t), (t)) = 0, t0 t tf

(2.29)

One approach to prove the relation (2.28) can be to differentiate the

hamiltonian with respect to time:

d
d 
H(x(t), u(t), (t)) =
g(x(t), u(t)) + T (t)f(x(t), u(t)) =
dt
dt
"
#T
"
#T
"
#T
d(t) H
du(t) H
dx(t) H
=
+
+
=
dt
x
dt

dt
u
H
H
T H + u(t)
T
T
= x(t)
+ (t)
x

u
Using the necessary conditions for the optimal trajectory x (t):

(t) =
x

and knowing that the optimal control u (t) satisfies:

x (t) =

H
=0
u

(2.30)

(2.31)
(2.32)

(2.33)

the relation (2.30) becomes:

dH
= x (t)T ( (t)) + (t)T x (t) + u (t)T 0 = 0
(2.34)
dt
which means the hamiltonian is constant when evaluated along the optimal
trajectory.
The relation (2.29) results directly from the boundary condition written
for the case when the final time is free, i.e.:
h
(x (tf ), tf ) = 0
(2.35)
t
When the hamiltonian and the function h do not depend explicitly on time,
the relation (2.35) becomes:
H(x (tf ), u (tf ), (tf ), tf ) +

H(x (tf ), u (tf ), (tf )) = 0

(2.36)

But the hamiltonian is constant along the optimal trajectory, therefore from
(2.36) and (2.28) we obtain:
H(x (t), u (t), (t)) = 0, t0 t tf
12

(2.37)

3
Minimum time problems. Bang-bang control

Example 3.1
Consider the problem of accelerating a skateboard in such way as to bring
the skateboard at rest at given position in minimum time. Denote x1 (t) the
distance traveled at time t, x 1 (t) = x2 (t) the velocity, and x1 (t) = x 2 (t) =
u(t) the acceleration. The model of the dynamical system is:
x 1 (t) = x2 (t); x 2 (t) = u(t)
with the final states specified: x1 (T ) = x2 (T ) = 0. We want to determine
the minimum final time T by minimizing:
J=

1dt

Consider the constraints on control variable 1 u(t) 1.

The hamiltonian is:
H(x, u, ) = 1 + 1 (t)x2 (t) + 2 (t)u(t)
By examining the form of the hamiltonian we can see that the term 2 (t)u(t),
the only one which depends on u(t), will decide the minimum value of H with
respect to u.
The costate 2 (t) is a switching function because the optimal value of
u(t) changes when 2 (t) changes sign. The explanation follows below.
The costate equations are:
(

H
1 (t) = x
=0
1
H
2 (t) = x2 = 1 (t)

13

1 (t) = C1
2 (t) = C1 t + C2

The switching function 2 (t) is a line and therefore it can change sign at
most once. Then:
(
+1, 2 (t) < 0

u (t) =
1, 2 (t) > 0

u*(t)
+1
l2(t)
-1

Figure 3.1: Bang-bang control

For 2 (t) < 0, the term 2 (t)u(t) from the hamiltonian has its minimum
value (negative) when u(t) has its maximum positive value. Then for u (t) =
1 the hamiltonian is minimum for any t [0, T ] (we subtracted the maximum
possible from the term 1 + 1 (t)x2 (t) which is constant with respect to u).
For 2 (t) < 0, the term 2 (t)u(t) will be most negative for u (t) = 1 and
then the hamiltonian reaches the global minimum.
The optimal control is given now in terms of the costate vector (t). We
want the control to be given in terms of the state x(t) so we will have a
closed-loop control system.
Solving the state equations:

x 1 (t) = x2 (t)
x 2 (t) = 1
u (t) = 1

x1 (0) = x10 , x2 (0) = x20

x1 (t) = t2 + x20 t + x10

x2 (t) = t + x20
(3.1)

x220

x22 (t)

1
1
x1 (t) = (x20 t)2 + x10 +
= (x20 t)2 + k1 =
2
2
2
2

x 1 (t) = x2 (t)
u (t) = +1
x 2 (t) = +1

x1 (0) = x10 , x2 (0) = x20

x1 (t) =

x220

(3.2)

x1 (t) = t2 + x20 t + x10

x2 (t) = t + x20

1
1
(t + x20 )2 + x10
= (x20 t)2 + k2 =
2
2
2
14

+ k1

(3.3)
x22 (t)
2

+ k2 (3.4)

The equations (3.2) and (3.4) are a set of parabolas opening about the x1 axis,
as shown in Figure 3.2. Two parabolas pass through each point in x1 x2
x2(t)
k1<0

k1>0

k2<0

k2>0
u*=+1

u*=-1

x1(t)

Figure 3.2: State space trajectories for bang-bang control

space, one for u = 1 and one for u = +1. For a given x(0) = [x10 x20 ]T ,
using only u = 1 or u = +1 may not transfer the system from the initial
state to the origin x(T ) = 0 (the target set).
The parabolas that pass through the origin are the ones for which k1 =
k2 = 0, or, the ones for which the initial conditions satisfy:
x10 =

x220
x2
, for u = 1, or x10 = 20 , for u = +1
2
2

(3.5)

The equation of the curve that transfer the system to the origin are obtained
as follows:

x 1 (t) = x2 (t)
x 2 (t) = 1
u (t) = 1

x1 (T ) = x2 (T ) = 0

x 1 (t) = x2 (t)
x 2 (t) = +1
u (t) = +1

x1 (T ) = x2 (T ) = 0

x1 (t) = (T t)
2
x2 (t) = T t

)
x1 (t) = (tT
2
x2 (t) = t T

x1 (t) =

x1 (t) =

x22 (t)
2

Since t < T , for u = 1 x2 (t) > 0 and for u = +1 x2 (t) < 0. Then
the equation of the switching locus is written as:
1
x1 (t) = x2 (t)|x2 (t)|
2

and is presented in Figure 3.3.

15

x22 (t)
2

(3.6)

x2(t)
u*=-1
x1(t)
u*=+1

Figure 3.3: Switching locus

If, by chance, the initial state lies on the switching locus, then we have
u = 1 accordingly as we have x1 <> 0. If, as in most cases, the initial
state is not on the switching locus, then u = 1 must be chosen to move the
system toward the switching locus. By inspection, it is apparent that, above
the switching locus we have u = 1 and below we have u = +1; a typical
path from an initial state x10 , x20 is shown in the Figure 3.4.
x2(t)
u*=-1

x10,x20
u*=-1
x1(t)

u*=+1
x10,x20

u*=+1

Figure 3.4: Examples of optimal trajectories

The optimal control law is applied according to the following rules:
If x(0) is to the left of the switching curve, apply u = +1 until the
curve is reached, then apply u = 1 until the origin is reached.
If x(0) is to the right of the switching curve, apply u = 1 until the
curve is reached, then apply u = +1 until the origin is reached.
If x(0) is on the switching curve, apply u = sign(x2 (0)) until the
origin is reached.
16

 Apply u = 0 when the origin is reached.

We have obtained the optimal control law at any time t as a function of
the state value. To express the optimal control law in a convenient form, let
us define the switching function, obtained from equation (3.6):
1
s(x(t)) = x1 (t) + x2 (t)|x2 (t)|
2

(3.7)

Notice that:
s(x(t)) > 0 implies x(t) lies on the right of the switching locus
s(x(t)) < 0 implies x(t) lies on the left of the switching locus
s(x(t)) = 0 implies x(t) lies on the switching locus
In terms of this switching function, the optimal control law is:
u = 1 for x(t) such that s(x(t)) > 0
u = +1 for x(t) such that s(x(t)) < 0
u = 1 for x(t) such that s(x(t)) = 0 and x2 (t) > 0
u = +1 for x(t) such that s(x(t)) = 0 and x2 (t) < 0
u = 0 for x(t) = 0
An implementation of this optimal control law is shown in Figure 3.5.
Plant
out

+1

x2

x1

in

-1

0.5x 2(t)|x2(t)| out

-1
+

in

Figure 3.5: Implementation of the time-optimal control law

17

4
Minimum fuel problems

4.1
4.1.1

Minimum fuel problems

The statement of the problem

For a system described by the state equation:

x(t)
= f(x(t), u(t), t), t0 t tf , x(t0 ) = x0

(4.1)

where u(t) is an m 1 control vector, a minimum fuel problem would

require to determine the optimal control vector u (t) that minimizes the
performance measure:
J=

tf

t0

"m
X

i |ui (t)| dt

i=1

(4.2)

while satisfying the constraints:

M ui (t) M, i = 1, 2, ..., m

(4.3)

It is assumed that the fuel consumption is proportional to the magnitude

of the control inputs.

4.1.2

Example of a minimum fuel problem

Example 4.1 The plant described by the differential equation:

x(t)

= x(t) + u(t)

18

(4.4)

is to be transferred from an arbitrary initial state x(0) = x0 to the origin,

x(T ) = 0, by a control that minimizes the performance measure:
J(u) =

|u(t)|dt

(4.5)

The admissible controls are constrained by:

|u(t)| 1

(4.6)

and the final time T is fixed.

A more general case of this problem may be found in (Kirk, 2004). The
solution of the problem is the optimal control signal that minimizes the
hamiltonian, written as:
H(x(t), u(t), (t)) = |u(t)| + (t)(x(t) + u(t))

(4.7)

The terms in the hamiltonian that depend on the control u(t) are given by
(the reduced hamiltonian Hr ):
Hr (u(t), (t)) = |u(t)| + (t)u(t)

(4.8)

Hr = u(t) + (t)u(t), for u(t) 0

Hr = u(t) + (t)u(t), for u(t) 0

(4.9)
(4.10)

or:

If (t) > 1, the minimum value of expression (4.9), which is a positive

quantity, is 0 because u(t) 0; but the minimum of (4.10) is a negative
value and is attained for u (t) = 1.
If (t) = 1, the expression (4.9) has a minimum at 0, with u(t) = 0;
but (4.10) will be zero for any non-positive value of u(t). Thus, any
non-positive value of u(t) will minimize the hamiltonian.
If 0 < (t) < 1, the minimum of (4.9) and (4.10) is zero (they are both
positive) and is attained for u (t) = 0.
If (t) < 1, the minimum value of expression (4.10), which is a positive quantity, is 0 because u(t) 0; but the minimum of (4.9) is a
negative value and is attained for u (t) = +1.

19

>1

1
0.5

2
1

0.5

0
u

0.5

0
1

1.5

0
u

0.5

0
u

0.5

0
u

0.5

1 < < 0

|u|+ u

|u|+ u

0.5

1.5
0<<1

0.5
0
1

0.5

0
u

0.5

1
0.5
0
1

|u|+ u

1
0.5
0
1

0.5

0.5

= 1

1.5
|u|+ u

=1

1.5

|u|+ u

|u|+ u

0
u

0.5

2
0

2
1

< 1

0.5

Figure 4.1: Reduced hamiltonian for various values of

If (t) = 1, the expression (4.10) has a minimum at 0, with u(t) = 0;
but (4.9) will be zero (minimum) for any non-negative value of u(t).
If 1 < (t) < 0, the minimum of (4.9) and (4.10) is zero (they are
both positive) and is attained for u (t) = 0.
The above situations are depicted in Figure 4.1 and the conclusions follow
directly from the plots.
Summarizing, the control signal that minimizes the hamiltonian (4.7) is:

u (t) =

+1, for (t) < 1

1, for (t) > 1
0, for | (t)| < 1
undetermined non-positive value for (t) = +1
undetermined non-negative value for (t) = 1

(4.11)

The costate equation and its solution are:

H

(t)
=
= (t) (t) = Cet
x

20

(4.12)

There are five possible trajectories for (t) that affect the control signal
u (t), depending on the value of the constant of integration C, as shown in
Figure 4.2, (Beale, 2001).

6
C 1
4

0<C<1

*(t) = C et

t1

t2

C=0

1< C < 0

C 1
6
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Figure 4.2: Forms of the costate

Because of the exponential form of the costate, the situation when (t) =
1 cannot occur for a finite time interval. Therefore, the optimal control can
be expressed as:

+1, for (t) < 1

u (t) = 1, for (t) > 1

(4.13)

0, for | (t)| < 1

The form bang-off-bang of the optimal control u (t) as a function of the

costate (t) is shown in Figure 4.3.
u*(t)
1
1

*(t)

-1
-1

Figure 4.3: Bang-off-bang optimal control

Four of the -trajectories are shown in Figure 4.4 and the fifth is (t) =
0 for which u (t) = 0, t [0, T ].
The state trajectories can be found by substituting the admissible values
of the control signal u (t) into the state equation and solving for x(t). This
will determine whether of not the target set can be reached and if it depends
on the initial conditions, (Beale, 2001).
21

u*(t)

*(t)

u*(t) *(t)
*(t)

*(t)
1

t1

-1

-1

u*(t)

(a) C> 1
u*(t)

u*(t)

(b) 0 < C<1

*(t)

u*(t)

*(t)
u*(t)

u*(t)
1

t1

-1

-1

*(t)

*(t)
(c) C< -1

(d) -1 < C<0

Figure 4.4: Forms of the costate and optimal control signals, (?)
u (t) = 0. The solution of the state equation is obtained as follows:
x(t)

= x(t), x(0) = x0 , x(t) = x0 et

(4.14)

The exponential form of the trajectory indicates that the system will
approach the final state zero as time approaches infinity, but it will
never attain the zero value in a finite time. Therefore, when no control
signal is applied, or u (t) = 0, this cannot be the only value used.
u (t) = +1. The solution of the state equation is :
x(t)

= x(t) + 1, x(0) = x0 , x(t) = 1 + (x0 1)et

(4.15)

This state trajectory moves towards 1 as time increases and it will pass
through the value 0 only if x0 < 0.
For a fixed final time T , the state will get to the origin only for a
particular value of the initial condition, determined from:
x(T ) = 0 = 1 + (x0 1)eT x0 = 1 eT
For the rest of the initial conditions switching is needed.

22

(4.16)

u (t) = 1. The solution of the state equation is :

x(t)

= x(t) 1, x(0) = x0 , x(t) = 1 + (x0 + 1)et

(4.17)

This state trajectory moves towards 1 as time increases. It will pass

through the value 0 only if x0 > 0.
The state will attain the origin using only the control value u (t) = 1,
in a fixed time T , only for the initial state obtained as:
x(T ) = 0 = 1 + (x0 + 1)eT x0 = 1 + eT
*

(4.18)

x (t) = 1+(x 1) e
0

x >0
0

1
0
*

x <0

u (t) = +1

1
2
3
0

0.5

1.5
(a)
*

2.5

x (t) = 1+(x +1) e

0

3
2
1

x >0

u (t) = 1

0
1
x <0
2
0

0.5

1.5
(b)

2.5

Figure 4.5: State trajectories for different initial conditions. (a) u (t) = +1
(b) u (t) = 1
Figure 4.5 (a) and (b) show the state trajectories for u (t) = 1 and
u (t) = +1. They indicate that the target set x(T ) = 0 can be reached
in most of the cases (when the initial condition does not have one of the
values given by (4.16) or (4.18)) only after switching the control value from
u (t) = 0 to u (t) = 1 (if the initial state x0 > 0) or to u (t) = +1 (if the
initial state x0 < 0).

4.1.3

Determining the switching time

.
23

Initial condition x0 is positive: x0 > 0. The control u (t) = 0 will not

drive the system to the origin x(T ) = 0 in a finite time. As shown in
Figure 4.4b), either u (t) = 1 or u (t) = [0, 1] must be used.
Let t = t1 be the switching time from u (t) = 0 to u (t) = 1. The
state value at time t1 is obtained from (4.14) for the final time t = t1 .
x(t1 ) = x0 et1

(4.19)

After switching, the state equation (4.17) will be solved for the time
interval [t1 , T ] with the initial condition x(t1 ):
x(t)

= x(t) 1, x(t1 ) = x0 et1

(4.20)

and the solution is:

x(t) = 1 + (x0 + et1 )et

(4.21)

The switching time t1 is obtained by substituting t = T into (4.21) and

letting it equal to zero (the target set), as shown below:
x(T ) = 1 + (x0 + et1 )eT = 0
(x0 + et1 )eT = 1, x0 + et1 = eT , t1 = ln(eT x0 )

(4.22)
(4.23)

Initial condition x0 is negative: x0 < 0. The switching time t1 can be

obtained for negative initial conditions in the same manner.
In this case (see Figure 4.4d)), the optimal control is zero u (t) = 0 for
t [0, t1 ] and u (t) = +1 for t [t1 , T ].
The switching time results from the calculations given below:
x(t)

= x(t) + 1, x(t1 ) = x0 et1

(4.24)

x(t) = 1 + (x0 et1 )et

(4.25)

x(T ) = 1 + (x0 et1 )eT = 0

(4.26)

x0 et1 = eT ,

24

t1 = ln(eT + x0 )

(4.27)

4.1.4

Open-loop control

From all the calculations above, the optimal control is:

u (t) =

0,
1,
0,
+1,

for
for
for
for

x0
x0
x0
x0

>0
>0
<0
<0

and
and
and
and

t < ln(eT x0 )
ln(eT x0 ) < t < T
t < ln(eT + x0 )
ln(eT + x0 ) < t < T

(4.28)

The optimal control (4.28) is an open-loop form since the current value
of the state x(t) is not used to determine the control signal.
Closed-loop control would be preferable to reduce the effects of disturbances.

4.1.5

Closed-loop control

The closed-loop control law will be obtained by solving the state equation for
u (t) = 1 and then for u (t) = +1, with the final condition x(T ) = 0. This
is based on the observation that during the last part of the time interval
[t1 , T ], the optimal control is either +1 or 1 depending on whether the
initial state is negative or positive.
For x(t) > 0:
x(t)

= x(t) 1, x(T ) = 0, x(t) = 1 + eT t , t > t1

(4.29)

For x(t) < 0:

x(t)

= x(t) + 1, x(T ) = 0, x(t) = 1 eT t , t > t1

(4.30)

During the first part of the interval t [0, t1 ], the optimal control is zero
and
x(t) = x0 et , t < t1
(4.31)
The switching of the control from 0 to 1 occurs when the curve (4.31)
intersect curve (4.29). We denote x(t) from (4.29) with:
s(t) = 1 + eT t

(4.32)

and call it the equation of the switching curve. The control will be 0 for all
state values smaller than s(t) and it will switch to 1 when they become
equal.

25

For negative values of the states x(t) < 0 we apply a similar reasoning
and find out that the control is 0 for 0 < x(t) < s(t) and it switches to +1
for x(t) s(t).
The closed-loop form of the optimal control can be written as:

1, for x(t) s(t)

u (t) = 0, for |x(t)| < s(t)

+1, for x(t) s(t)

(4.33)

Figure 4.6 shows the solution for x0 > 0.

u*(t) x(t)
eT-1
s(t) = eT-t - 1
x0

x0 e- t

x*(t)
t1

-1
u*(t)

Figure 4.6: The optimal trajectory and the optimal control

26

5
Exercises

1. (Owens, 1981) Verify that for the scalar plant

x(t)

= x(t) + u(t), x(0) = 1, |u(t)| M t [0, 1]

(5.1)

the admissible controller u (t) that minimizes the performance measure:

Z 1
J = x(1) +
u2 (t)dt
(5.2)
0

takes the form (open-loop):

(a) if 2M 1:

1
u (t) = et1
2

(5.3)

(b) if 2M < 1:
u (t) =

12 et1 , for 0 t 1 + ln 2M
M, for 1 + ln 2M t 1

(5.4)

2. (Kirk, 2004) A differential equation that describes the leaky reservoir

shown in Figure 5.1 is:
x(t)

= 0.1x(t) + u(t)

(5.5)

where x(t) is the height of the water and u(t) is the net inflow rate of
water at time t. Assume 0 u(t) M.
Find the optimal control law if it is desired to minimize:
J=

100

27

x(t)dt

(5.6)

Figure 5.1: Reservoir

3. (Owens, 1981) Consider the plant with the state equations:
x 1 (t) = x2 (t)
x 2 (t) = u(t)

(5.7)
(5.8)

Find the optimal control law that brings the system from the initial
state x(0) = [1, 0]T to the final state x(1) = [0, 0]T , and minimizes
the performance measure:
1
J=
2

u2 (t)dt

(5.9)

The control is constrained by: M u(t) M.

4. Minimize the performance measure:
J=

x(t)dt

(5.10)

subject to the state equation:

x(t)

= u(t), x(0) = 1

(5.11)

and the control constraint: 1 u(t) 1.

5. Minimize the performance measure:
Z

(5.12)

x(t)

= x(t) + u(t), x(0) = 5

(5.13)

J=

u2 (t) + 3u(t) 2x(t) dt

subject to the state equation:

and the control constraint: 0 u(t) 2.

28

6. (Owens, 1981) Verify that the solution of the problem:

x(t)

= x(t) + u(t), x(0) = 0

(5.14)

1
x(t) + u2 (t) dt
(5.15)
2
0
in the presence of the constraint |u(t)| M is the optimal control:
J = 2x(1) +

u (t) =

(t)
if |(t)| 1
Msign((t)) if |(t)| > 1

(5.16)

where (t) = 1 + et1 is the solution of the costate equation

(t)
= (t) 1, (0) = 2

(5.17)

u (t) = M

(5.18)

Hence verify that:

a)
if

M 1 + e1

b)
u (t) = 1 et1

if

M 2

(5.19)

c)
u (t) =

1 et1 if 0 t ln(M 1)
M
if t > 1 + ln(M 1) > 1

if 1 + e1 M 2.

29

(5.20)

Bibliography

Beale, G. (2001). Optimal control. online, George Mason University.

Kirk, D. E. (2004). Optimal Control Theory. An Introduction. Dover Publications, Inc.
Owens, D. (1981). Multivariable and Optimal Systems. Academic Press.
Weber, R. (2000).
Optimization
www.statslab.cam.ac.uk.

30

and

control.

online

at