Calculus of Variations

and Optimal Control

Control Engineering III

 • Function and Functional

• We discuss some fundamental concepts

associated with functionals along side
with those of functions.

Basic • (a) Function: A variable x is a function of

Concepts a variable quantity t, (written as x(t) =
f(t)), if to every value of t over a certain
range of t there corresponds a value x;
i.e., we have a correspondence: to a
number t there corresponds a number x.
• Note that here t need not be always
time but any independent variable.
 • Consider x(t) = 2t2 + 1.
• For t = 1, x = 3, t = 2, x = 9 and so on.
•
Example • Other functions are
• x(t) = 2t
• x(t1, t2) = t12+t22
 • (b) Functional:
• A variable quantity J is a functional
dependent on a function f(x), written as
J = J (f (x) ), if to each function f (x), there
corresponds a value J, i.e., we have a
Functional correspondence: to the function f (x)
there corresponds a number J.

• Functional depends on several

functions.
Example

• Let x(t) = 2t2 + 1. Then

is the area under the curve x(t). If v(t) is the velocity of a vehicle, then

is the path traversed by the vehicle.

• Thus, here x(t) and v(t) are functions of t, and J is a functional of x(t)
or v(t).
Increment

(a) Increment of a Function: In order to consider optimal values of a

function, we need the definition of an increment.

• DEFINITION The increment of the function f, denoted by Df, is defined

as

• It is easy to see from the definition that Df depends on both the

independent variable t and the increment of the independent
variable Dt, and hence strictly speaking, we need to write the
increment of a function as Df(t, Dt).
Increment

• (b) Increment of a Functional: Now we are ready to define the

increment of a functional.

• DEFINITION The increment of the functional J, denoted by DJ, is

defined as

• Here d x(t) is called the variation of the function x(t). Since the
increment of a functional is dependent upon the function x(t) and its
variation d x(t), strictly speaking, we need to write the increment as D
J(x(t), d x(t)).
Example

• If
find the increment of the function f(t) .

• Solution: The increment D f becomes

Example

• Find the increment of the functional

• Solution: The increment of J is given by

Differential and Variation

• (a) Differential of a Function: Let us define at a point t* the

increment of the function f as

• By expanding D f(t* + Dt) in a Taylor series about t*, we get

• Neglecting the higher order terms in D t,

• Here, df is called the differential of J at the point t*.

 Differential and Variation
• (b) Variation of a Functional: Consider the increment of a functional

• Expanding J(x(t) + d x(t)) in a Taylor series, we get

where,

are called the first variation (or simply the variation) and the second
variation of the functional J, respectively.
• The variation dJ of a functional J is the linear (or first order
approximate) part (in d x(t)) of the increment DJ.
 Optimum of a Function and a Functional

• DEFINITION . Optimum of a Function: A function f (t) is said to have a relative

optimum at the point t* if there is a positive parameter e such that for all
points t in a domain D that satisfy |t - t*| < e, the increment of f(t) has the same
sign (positive or negative).

• In other words, if
D f = f(t) - f(t*) >= 0,
then, f(t*) is a relative local minimum.

• On the other hand, if

D f = f(t) - f(t*) <= 0,
then, f(t*) is a relative local maximum.
(a) Minimum and (b) Maximum of a
Function f (t)
Optimum of a Function and a Functional

• It is well known that the necessary condition for optimum of a function is

that the (first) differential vanishes, i.e., df = 0.

• The sufficient conditions are

• 1. for minimum is that the second differential is positive, i.e., d2f > 0, and

• 2. for maximum is that the second differential is negative, i.e., d2f < 0.

• If d2f = 0, it corresponds to a stationary (or inflexion) point.

Optimum of a Function and a Functional

• DEFINITION. Optimum of a Functional:

• A functional J is said to have a relative optimum at x* if there is a positive e such that for
all functions x in a domain n which satisfy |x - x* | < e, the increment of J has the same
sign.

• In other words, if
• D J = J(x) - J(x*) >= 0,
then J(x*) is a relative minimum.

On the other hand, if

• D J = J(x) - J(x*) <= 0,
then, J(x*) is a relative maximum.

• If the above relations are satisfied for arbitrarily large e, then, J(x*) is a global absolute
optimum.
Optimum of a Function and a Functional

• THEOREM

• For x*(t) to be a candidate for an optimum, the (first) variation of J must be

zero on x*(t), i.e.,
d J(x*(t), d x(t)) = 0
for all admissible values of d x(t).

• This is a necessary condition.

• As a sufficient condition for minimum, the second variation d 2J> 0, and for
maximum d 2J < 0.
The Basic Variational Problem
• Fixed-End Time and Fixed-End State System
• We address a fixed-end time and fixed-end state problem, where
both the initial time and state and the final time and state are fixed
or given a priori.
• Let x(t) be a scalar function with continuous first derivatives and the
vector case can be similarly dealt with. The problem is to find the
optimal function x* (t) for which the functional

has a relative optimum. It is assumed that the integrand V has

continuous first and second partial derivatives w.r.t. all its arguments;
to and t f are fixed (or given a priori) and the end points are fixed, i.e.,
Steps involved in finding the optimal solution to
the fixed-end time and fixed-end state system

• • Step 1: Assumption of an Optimum

• • Step 2: Variations and Increment

• • Step 3: First Variation

• • Step 4: Fundamental Theorem

• • Step 5: Fundamental Lemma

• • Step 6: Euler-Lagrange Equation

Step 1: Assumption of an Optimum
• Let us assume that x*(t) is the optimum attained for the
function x(t).

• Take some admissible function xa(t) = x*(t) +dx(t) close to x*(t),

where dx(t) is the variation of x*(t) as shown in the next figure.

• The function xa(t) should also satisfy the boundary conditions

• and hence it is necessary that

Step 2: Variations and Increment
• Let us first define the increment as

which by combining the integrals can be written as

where,
Step 2: Variations and Increment
• Expanding V in the increment in a Taylor series about
the point x*(t) and , the increment DJ becomes (note the cancelation
of 𝑉(𝑥 𝑡 , 𝑥̇ 𝑡 , 𝑡)
 Step 3: First Variation
• Now, we obtain the variation by retaining the terms that are linear in d x(t)
and
as

• To express the relation for the first variation entirely in terms containing d x(t)
(since d x(t) is dependent on d x(t)), we integrate by parts the term involving d
x(t) as (omitting the arguments in V for simplicity)
Step 3: First Variation

• Using the relation

for boundary variations in the last equation, we get
Step 4: Fundamental Theorem

• Fundamental Theorem:
• We now apply the fundamental theorem of the calculus of
variations: the first variation of J must vanish for an
optimum.

• That is, for the optimum x*(t) to exist, dJ(x*(t), d x(t)) = 0.

• Thus the previous relation becomes

Step 5: Fundamental Lemma

• To simplify the condition obtained in the last equation, let us take

advantage of the following lemma called the fundamental lemma of
the calculus of variations
Step 6: Euler-Lagrange Equation

• Applying the previous lemma, a necessary condition for x*(t) to be an

optimal of the functional J is

or in simplified notation omitting the arguments in V,

for all t [to, tf]. This equation is called Euler equation, first published in
1741.
Extrema of Functions with Conditions

• We begin with an example of finding the extrema of a function

under a condition (or constraint).

• We solve this example with two methods, first by direct

method and then by Lagrange multiplier method.

• Let us note that we consider this simple example only to

illustrate some basic concepts associated with conditional
extremization.
Extrema of Functionals with Conditions

• Consider the extremization of the performance index in the form of a

functional
Extrema of Functionals with Conditions

• • Step 1: Lagrangian

• • Step 2: Variations and Increment

• • Step 3: First Variation

• • Step 4: Fundamental Theorem

• • Step 5: Fundamental Lemma

• • Step 6: Euler-Lagrange Equation

Step 1: Lagrangian
Step 2: Variations and Increment
Step 3: First Variation
Step 4: Fundamental Theorem
Step 5: Fundamental Lemma
 Step 6: Euler-Lagrange Equation
• Step 6: Euler-Lagrange Equation: Combining the various relations, the
necessary conditions for extremization of the functional subject to the
condition (according to Euler-Lagrange equation) are
Example
 1 Direct Method:
• 1 Direct Method: Here, we eliminate u(t) between the performance index and
the plant to get the functional as

• Now, we notice that the functional absorbed the condition within itself, and
we need to consider it as a straight forward extremization of a functional as
given earlier. Thus, applying the Euler-Lagrange equation

where

We get
 2 Lagrange Multiplier Method
• 2 Lagrange Multiplier Method: Here, we use the ideas developed in the
previous section on the extremization of functions with conditions.
Consider the optimization of the functional with the boundary
conditions under the condition describing the plant. First we rewrite the
condition as

• Now, we form an augmented functional as

where, l(t) is the Lagrange multiplier, and

is the Lagrangian.
• Now, we apply the Euler-Lagrange equation to the previous Lagrangian
to get
• The solution for the set of differential equations with the boundary
conditions for the previous example using the Symbolic Toolbox of
MATLAB, is shown below.
Variational Approach to Optimal Control Systems

• In this section, we approach the optimal control system by

variational techniques, and in the process introduce the
Hamiltonian function, which was used by Pontryagin and his
associates to develop the famous Minimum Principle.
Terminal Cost Problem
 Terminal Cost Problem
• Step 1: Assumption of Optimal Conditions

• Step 2: Variations of Control and State Vectors

• Step 3: Lagrange Multiplier

• Step 4: Lagrangian

• Step 5: First Variation

• Step 6: Condition for Extrema

• Step 7 : Hamiltonian
Step 1: Assumptions of Optimal Conditions
Step 2: Variations of Controls and States
Step 2: Variations of Controls and States
Step 3: Lagrange Multiplier
• Introducing the Lagrange multiplier vector l(t) (also called costate
vector) and using

• we introduce the augmented performance index at the optimal

condition as

• and at any other (perturbed) condition as

Step 4: Lagrangian

• Let us define the Lagrangian function at optimal condition as

• and at any other condition as

Step 4: Lagrangian
• With these, the augmented performance index at the optimal and
any other condition becomes

• Using mean-value theorem and Taylor series, and retaining the

linear terms only, we have
 Step 5: First Variation
• Defining increment D J, using Taylor series expansion, extracting the first
variation dJ by retaining only the first order terms, we get the first
variation as

• Considering the term d x! (t ) in the first variation and integrating

by parts
Step 5: First Variation

• Also note that since x(to) is specified, dx(to) = 0. Thus, using

the last equation, the first variation dJ becomes

(1)
 Step 6: Condition for Extrema
• Step 6: Condition for Extrema: For extrema of the functional J, the first
variation dJ should vanish according to the fundamental theorem of the
CoV.
• Also, in a typical control system, we note that du(t) is the independent
control variation and dx(t) is the dependent state variation.
• First, we choose l(t) = l *(t) which is at our disposal and hence !* such
that the coefficient of the dependent variation dx(t) in dJ be zero. Then,
we have the Euler-Lagrange equation

where the partials are evaluated at the optimal (*) condition.

Step 6: Condition for Extrema

• Next, since the independent control variation du(t) is arbitrary, the

coefficient of the control variation d u(t) in the first variation (1)
should be set to zero. That is

• Finally, the first variation (1) reduces to

• Let us note that the condition (or plant) equation can be written in
terms of the Lagrangian as
Step 6: Condition for Extrema
• In order to convert the expression containing dx(t) in the first
variation expression into an expression containing d xf (see Figure),
we note that the slope of x*(t) + d x(t) at tf is approximated as

• which is rewritten as
• and retaining only the linear (in d) terms in the relation, we have

• Using this expression in the boundary condition, we have the

general boundary condition in terms of the Lagrangian as
Step 7: Hamiltonian

• Step 7: Hamiltonian: We define the Hamiltonian H* (also called the

Pontryagin H function) at the optimal condition as

• Then the Lagrangian !* in terms of the Hamiltonian H* becomes

Step 7: Hamiltonian

• Thus, for the optimal control u*(t), the relation becomes

• for the optimal state x*(t), the relation becomes

leading to

and for the costate l*( t),

Step 7: Hamiltonian

• Finally, using the relation (1), the boundary condition at the optimal
condition reduces to
Different Types of Systems
Different Types of Systems

• Type (a): Fixed-Final Time and Fixed-Final State System: Here, since
tf and x(tf) are fixed or specified (In the next Figure(a), both dtf and
dxf are zero in the general boundary condition, and there is no extra
boundary condition to be used other than those given in the problem
formulation.

• Type (b): Free-Final Time and Fixed-Final State System: Since tf is

free or not specified in advance, dtf is arbitrary, and since x(tf) is fixed
or specified, dxf is zero as shown in the next Figure(b). Then, the
coefficient of the arbitrary dtf in the general boundary condition is
zero resulting in
Different Types of Systems

• Type (c): Fixed-Final Time and Free-Final State System: Here tf is

specified and x(tf) is free (see Figure (c). Then dtf is zero and dx f is
arbitrary, which in turn means that the coefficient of dxf in the
general boundary condition is zero. That is
 Different Types of Systems
• Type (d): Free-Final Time and Dependent Free-Final State System:
• If tf and x(tf) are related such that x(tf) lies on a moving curve d(t) as
shown in Figure, then

• Using this equation, the boundary condition for the optimal condition
becomes

• Since tf is free, dtf is arbitrary and hence the coefficient of dtf is zero.
That is
Different Types of Systems

• Type (e): Free-Final Time and Independent Free-Final State:

• If tf and x(tf) are not related, then dtf and dxf are unrelated, and the
boundary condition at the optimal condition becomes
Summary of Pontryagin Procedure

• Consider a free-final time and free-final state problem with general

cost function (Bolza problem), where we want to minimize the
performance index

• for the plant described by

• with the boundary conditions as

1
2
3
4
Linear Quadratic Optimal
Control Systems
In this chapter, we present the closed-loop optimal control of linear plants or
systems with quadratic performance index or measure.

This leads to the linear quadratic regulator (LQR) system dealing with state
regulation, output regulation, and tracking.

Broadly speaking, we are interested in the design of optimal linear systems with
quadratic performance indices.
 Problem Formulation
• Consider a linear, time-varying (LTV) system

• with a cost functional (CF) or performance index (PI)

• where, x(t) is nth state vector, y(t) is mth output vector, z(t) is mth reference or desired output
vector (or nth desired state vector, if the state x(t) is available), u(t) is rth control vector, and
e(t) = z(t) - y(t) (or e(t) = z(t) - x(t), if the state x(t) is directly available) is the mth error vector.

• A(t) is n×n state matrix, B(t) is n×r control matrix, and C(t) is m×n output matrix.

• We assume that the control u(t) is unconstrained, 0 < m £ r £ n, and all the states and/or
outputs are completely measurable. The preceding cost functional contains quadratic terms in
error e(t) and control u(t) and hence called the quadratic cost functional.
Problem Formulation

• 1. If our objective is to keep the state x(t) near zero (i.e., z(t) o and C
= I), then we call it state regulator system.

• In other words, the objective is to obtain a control u(t) which takes

the plant from a nonzero state to zero state.

• This situation may arise when a plant is subjected to unwanted

disturbances that perturb the state (for example, sudden load
changes in an electrical voltage regulator system, sudden wind gust in
a radar antenna positional control system).

• 2. If our interest is to keep the output y(t) near zero

(i.e., z(t) = 0), then it is termed the output regulator system.
Problem Formulation

• 3. If we try to keep the output or state near a desired state or output,

then we are dealing with a tracking system.

• We see that in both state and output regulator systems, the desired
or reference state is zero and in tracking system the error is to be
made zero.

• For example, consider again the antenna control system to track an

aircraft.
 Problem Formulation
• 1. The Error Weighted Matrix Q(t): In order to keep the error e(t) small and error
squared non-negative, the integral of the expression (½)e’(t)Q(t)e(t) should be
nonnegative and small.
Thus, the matrix Q(t) must be positive semidefinite. Due to the quadratic nature of the
weightage, we have to pay more attention to large errors than small errors.

• 2. The Control Weighted Matrix R(t): The quadratic nature of the control cost
expression (½) u’(t)R(t)u(t) indicates that one has to pay higher cost for larger control
effort.
Since the cost of the control has to be a positive quantity, the matrix R(t) should be
positive definite.

• 3. The Control Signal u(t): The assumption that there are no constraints on the control
u(t) is very important in obtaining the closed loop optimal configuration.
Combining all the previous assumptions, we would like on one hand, to keep the error
small, but on the other hand, we must not pay higher cost to large controls.

• 4. The Terminal Cost Weighted Matrix F(tf): The main purpose of this term is to
ensure that the error e(t) at the final time tf is as small as possible. To guarantee this,
the corresponding matrix F(tf) should be positive semidefinite.
 Problem Formulation
• Further, without loss of generality, we assume that the weighted matrices Q(t),
R(t), and F(t) are symmetric.

• The quadratic cost functional described previously has some attractive features:

• (a) It provides an elegant procedure for the design of closed-loop optimal

controller.

• (b) It results in the optimal feedback control that is linear in state function.

That is why we often say that the "quadratic performance index fits like a glove“

• 5. Infinite Final Time: When the final time tf is infinity, the terminal cost term
involving F(tf) does not provide any realistic sense since we are always
interested in the solutions over finite time.
• Hence, F(tf) must be zero.
Finite-Time Linear Quadratic Regulator (LQR)

• Now we proceed with the linear quadratic regulator (LQR) system, that
is, to keep the state near zero during the interval of interest.

• Consider a linear, time-varying plant described by

• with a cost functional

Finite-Time Linear Quadratic Regulator (LQR)

• We summarize again various assumptions as follows.

• 1. The control u(t) is unconstrained.

• 2. The initial condition x(t =to)=xo is given. The terminal time tf is specified,
and the final state x(tf) is not specified.

• 3. The terminal cost matrix F(tf) and the error weighted matrix Q(t) are n×n
positive semidefinite matrices, respectively; and the control weighted matrix
R(t) is an r×r positive definite matrix.

• 4. Finally, the fraction ½ in the cost functional is associated mainly to cancel

a 2 that would have otherwise been carried on throughout the result.
Finite-Time Linear Quadratic Regulator (LQR)

• We follow the Pontryagin procedure to obtain an optimal solution

and then propose the closed-loop configuration.

• First, let us list the various steps under which we present the method.

• Step 1: Hamiltonian
• Step 2: Optimal Control
• Step 3: State and Costate System
• Step 4: Closed-Loop Optimal Control
• Step 5: Matrix Differential Riccati Equation
Step 1: Hamiltonian

• Step 1: Hamiltonian: Using the definition of the Hamiltonian

along with the performance index, formulate the Hamiltonian as
Step 2: Optimal Control

• Obtain the optimal control u*(t) using the control relation as

• leading to

• where, we used
Step 3: State and Costate System

• Step 3: State and Costate System: Obtain the state and costate equations
as

• Substitute the control relation in the state equation to obtain the (state
and costate) canonical system (also called Hamiltonian system) of
equations

• where
 Step 3: State and Costate System
• The general boundary condition is reproduced here as

• where, S equals the entire terminal cost term in the cost functional.
• Here, for our present system tf is specified which makes dtf equal to zero,
and x(tf) is not specified which makes dxf arbitrary.
• Hence, the coefficient of dxf in becomes zero, that is,

• This final condition on the costate, l*(t f ) together with the initial condition
on the state xo and the canonical system of equations form a two-point,
boundary value problem (TPBVP).
Step 3: State and Costate System

• The state-space representation of the set of relations for the state

and costate system and the control is shown in next figure:
 Step 4: Closed-Loop Optimal Control
• Step 4: Closed-Loop Optimal Control: The state space representation shown in
the last figure prompts us to think that we can obtain the optimal control u*(t)
as a function (negative feedback) of the optimal state x*(t).

• Now to formulate a closed-loop optimal control, that is, to obtain the optimal
control u*(t) which is a function of the costate , l*(t) , as a function of the state
x*(t), let us examine the final condition on , x*(t) given by

• This in fact relates the costate in terms of the state at the final time tf. Similarly,
we may like to connect the costate with the state for the complete interval of
time [to, t f]. Thus, let us assume a transformation

where, P(t) is yet to be determined.

• Then, we can easily see that the optimal control becomes
Step 4: Closed-Loop Optimal Control

• Differentiating w.r.t. time t, we get

• Using the transformation in the control, state and costate

system of equations, we get

• Now, substituting state and costate relations, we have

Step 5: Matrix Differential Riccati Equation

• Step 5: Matrix Differential Riccati Equation: Now the last relation

should be satisfied for all and for any choice of the initial state
x*(to).
• Also, P(t) is not dependent on the initial state. It follows that this
equation should hold good for any value of x*(t).
• This clearly means that the function P(t) should satisfy the matrix
differential equation

• This is the matrix differential equation of the Riccati type, and often
called the matrix differential Riccati equation (DRE).
• The matrix DRE can also be written in a compact form as
Step 5: Matrix Differential Riccati Equation

• Comparing the boundary condition and the Riccati transformation,

we have the final condition on P(t) as
Finite-Time Linear Quadratic Regulator:Time- Varying Case: Summary
Finite-Time Linear Quadratic Regulator:Time- Varying Case: Summary
Example

• Let us illustrate the previous procedure with a simple second order

example. Given a double integral system

and the performance index (PI)

obtain the feedback control law.

Solution:

• Comparing the present plant and the PI of the problem with the
corresponding general formulations of the plant and the PI,
respectively, let us first identify the various quantities as

• It is easy to check that the system is unstable. Let P(t) be the 2x2
symmetric matrix

• Then, the optimal control is given by

• where, P(t), the 2x2 symmetric, positive definite matrix, is the
solution of the matrix DRE

• satisfying the final condition

• Simplifying the matrix DRE, we get

• Solving the previous set of nonlinear, differential equations backward

in time with the given final conditions, one can obtain the numerical
solutions for the Riccati coefficient matrix P(t).
%% Solution using Control System Toolbox of
%% the MATLAB.
%% The following file example.m requires
%% two other files lqrnss.m and lqrnssf.m
%% which are given in Aula Global
clear all
A= [0. ,1. ; -2. ,1.] ;
B= [0. ; 1.] ;
Q= [2. ,3. ; 3. ,5.] ;
F=[1. ,0.5;0.5,2.];
R= [.25] ;
tspan=[0 5];
x0=[2. ,-3.];
[x,u,K]=lqrnss(A,B,F,Q,R,x0,tspan);