MECE6388 Optimal Control Theory, 2021 Fall

Dr. Zheng Chen

Chapter 2: Optimal Control of Discrete Time System

2.2 Discrete-time Linear Quadratic Regulator

• The state and costate equations:

Let the plant to be controlled be described by

𝑥𝑥𝑘𝑘+1 = 𝐴𝐴𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝐵𝐵𝑘𝑘 𝑢𝑢𝑘𝑘
with 𝑥𝑥𝑘𝑘 ∈ ℝ and 𝑢𝑢𝑘𝑘 ∈ ℝ𝑚𝑚 . The associated performance index is the
𝑛𝑛

quadratic function:
𝑁𝑁−1
1 1
𝐽𝐽𝑖𝑖 = 𝑥𝑥𝑁𝑁𝑇𝑇 𝑆𝑆𝑁𝑁 𝑥𝑥𝑁𝑁 + � (𝑥𝑥𝑘𝑘𝑇𝑇 𝑄𝑄𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝑢𝑢𝑘𝑘𝑇𝑇 𝑅𝑅𝑘𝑘 𝑢𝑢𝑘𝑘 )
2 2
𝑘𝑘=𝑖𝑖
defined over the time interval of interest [𝑖𝑖, 𝑁𝑁]. The initial plant state is
given as 𝑥𝑥𝑖𝑖 . We assume the 𝑄𝑄𝑘𝑘 , 𝑅𝑅𝑘𝑘 and 𝑆𝑆𝑁𝑁 are symmetric positive semi
definite matrices and that |𝑅𝑅𝑘𝑘 | ≠ 0 for all 𝑘𝑘.

The objective is to find the control sequences 𝑢𝑢𝑖𝑖 , 𝑢𝑢𝑖𝑖+1 , … , 𝑢𝑢𝑁𝑁−1 that
minimizes 𝐽𝐽𝑖𝑖 .

The Hamiltonian function:

1
𝐻𝐻𝑘𝑘 = (𝑥𝑥𝑘𝑘𝑇𝑇 𝑄𝑄𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝑢𝑢𝑘𝑘𝑇𝑇 𝑅𝑅𝑘𝑘 𝑢𝑢𝑘𝑘 ) + 𝜆𝜆𝑇𝑇𝑘𝑘+1 (𝐴𝐴𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝐵𝐵𝑘𝑘 𝑢𝑢𝑘𝑘 )
2

The state and costate equations:

𝜕𝜕𝐻𝐻𝑘𝑘
𝑥𝑥𝑘𝑘+1 = = 𝐴𝐴𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝐵𝐵𝑘𝑘 𝑢𝑢𝑘𝑘 , (1)
𝜕𝜕𝜆𝜆𝑘𝑘+1
𝜕𝜕𝐻𝐻𝑘𝑘
𝜆𝜆𝑘𝑘 = = 𝑄𝑄𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝐴𝐴𝑇𝑇𝑘𝑘 𝜆𝜆𝑘𝑘+1 , (2)
𝜕𝜕𝑥𝑥𝑘𝑘
And the stationary condition:
𝜕𝜕𝐻𝐻𝑘𝑘
0= , 𝑘𝑘 = 𝑖𝑖, … , 𝑁𝑁 − 1 , (3)
𝜕𝜕𝑢𝑢𝑘𝑘
MECE6388 Optimal Control Theory, 2021 Fall
Dr. Zheng Chen

Based on the stationary condition and |𝑅𝑅𝑘𝑘 | ≠ 0

𝑢𝑢𝑘𝑘 = −𝑅𝑅𝑘𝑘−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝜆𝜆𝑘𝑘+1 , (4)
If we can find the costate sequence 𝜆𝜆_𝑘𝑘, then we find the optimal control.

Using (4), equation (1) can be written as

𝑥𝑥𝑘𝑘+1 = 𝐴𝐴𝑘𝑘 𝑥𝑥𝑘𝑘 − 𝐵𝐵𝑘𝑘 𝑅𝑅𝑘𝑘−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝜆𝜆𝑘𝑘+1 , (5)

From (2) and (5), we find that the state and costate are coupled. We
can form a single unforced system
𝑥𝑥𝑘𝑘+1 𝐴𝐴 −𝐵𝐵𝑅𝑅 −1 𝐵𝐵𝑇𝑇 𝑥𝑥𝑘𝑘
� 𝜆𝜆 � = � � �𝜆𝜆 �
𝑘𝑘 𝑄𝑄 𝐴𝐴𝑇𝑇 𝑘𝑘+1
Where 𝐴𝐴𝑘𝑘 = 𝐴𝐴, 𝐵𝐵𝑘𝑘 = 𝐵𝐵, 𝑅𝑅𝑘𝑘 = 𝑅𝑅, 𝑄𝑄𝑘𝑘 = 𝑄𝑄 for time invariant system.

The system described by (6) is difficult to solve since part of it develops

forward and part of it develops backwards in time.
If |𝐴𝐴| ≠ 0, we can rewrite (1) by
𝑥𝑥𝑘𝑘 = 𝐴𝐴−1 𝑥𝑥𝑘𝑘+1 + 𝐴𝐴−1 𝐵𝐵𝑅𝑅−1 𝐵𝐵𝑇𝑇 𝜆𝜆𝑘𝑘+1
Then the system of (6) can be written by
𝑥𝑥𝑘𝑘 𝐴𝐴−1 𝐴𝐴−1 𝐵𝐵𝑅𝑅−1 𝐵𝐵𝑇𝑇 𝑥𝑥𝑘𝑘+1
�𝜆𝜆 � = � −1 � � �
𝑘𝑘 𝑄𝑄𝐴𝐴 𝐴𝐴𝑇𝑇 + 𝑄𝑄𝐴𝐴−1 𝐵𝐵𝑅𝑅−1 𝐵𝐵𝑇𝑇 𝜆𝜆𝑘𝑘+1
This equation develops purely backwards in time. If we know 𝑥𝑥𝑁𝑁 and 𝜆𝜆𝑁𝑁 .
Then we can find 𝑥𝑥𝑘𝑘 and 𝜆𝜆𝑘𝑘 and hence the optimal control. However, we
only are given 𝑥𝑥0 not 𝑥𝑥𝑁𝑁 and 𝜆𝜆𝑁𝑁 .

If 𝑢𝑢𝑘𝑘 = 0 and 𝑄𝑄𝑘𝑘 = 0, then

𝑥𝑥𝑘𝑘+1 = 𝐴𝐴𝑥𝑥𝑘𝑘
𝜆𝜆𝑘𝑘 = 𝐴𝐴𝑇𝑇 𝜆𝜆𝑘𝑘+1
The solutions are
𝑥𝑥𝑘𝑘 = 𝐴𝐴𝑘𝑘 𝑥𝑥0
𝜆𝜆𝑘𝑘 = (𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑘𝑘 𝜆𝜆𝑁𝑁
Therefore
𝜆𝜆𝑇𝑇𝑘𝑘 𝑥𝑥𝑘𝑘 = 𝜆𝜆𝑇𝑇0 𝑥𝑥0 = 𝜆𝜆𝑇𝑇𝑁𝑁 𝑥𝑥𝑁𝑁 = 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑓𝑓𝑓𝑓𝑓𝑓 𝑎𝑎𝑎𝑎𝑎𝑎 𝑘𝑘
MECE6388 Optimal Control Theory, 2021 Fall
Dr. Zheng Chen

Three special cases:

Case A: Zero input cost and the Lyapunov Equation
Let the input 𝑢𝑢𝑘𝑘 = 0 , we want to determine the value of 𝐽𝐽𝑖𝑖 in this
uncontrolled situation, we solve 𝐽𝐽𝑖𝑖 backwards to find 𝐽𝐽𝑁𝑁 , 𝐽𝐽𝑁𝑁−1 , 𝐽𝐽𝑁𝑁−2 , …

When 𝑖𝑖 = 𝑁𝑁
1
𝐽𝐽𝑁𝑁 = 𝑥𝑥𝑁𝑁𝑇𝑇 𝑆𝑆𝑁𝑁 𝑥𝑥𝑁𝑁
2
When 𝑖𝑖 = 𝑁𝑁 − 1 and 𝑢𝑢𝑘𝑘 = 0
𝑁𝑁−1
1 1
𝐽𝐽𝑁𝑁−1 = 𝑥𝑥𝑁𝑁𝑇𝑇 𝑆𝑆𝑁𝑁 𝑥𝑥𝑁𝑁 + � 𝑥𝑥𝑘𝑘𝑇𝑇 𝑄𝑄𝑥𝑥𝑘𝑘
2 2
𝑘𝑘=𝑖𝑖
1 𝑇𝑇 1 𝑇𝑇
= 𝑥𝑥𝑁𝑁 𝑆𝑆𝑁𝑁 𝑥𝑥𝑁𝑁 + 𝑥𝑥𝑁𝑁−1 𝑄𝑄𝑥𝑥𝑁𝑁−1
2 2
Using the plant dynamics 𝑥𝑥𝑁𝑁 = 𝐴𝐴𝑥𝑥𝑁𝑁−1
1 𝑇𝑇
𝐽𝐽𝑁𝑁−1 = 𝑥𝑥𝑁𝑁−1 (𝐴𝐴𝑇𝑇 𝑆𝑆𝑁𝑁 𝐴𝐴 + 𝑄𝑄 )𝑥𝑥𝑁𝑁−1
2
We define anew intermediate variable 𝑐𝑐 an 𝑛𝑛 × 𝑛𝑛 matrix.
𝑆𝑆𝑁𝑁−1 = 𝐴𝐴𝑇𝑇 𝑆𝑆𝑁𝑁 𝐴𝐴 + 𝑄𝑄
Then
1 𝑇𝑇
𝐽𝐽𝑁𝑁−1 = 𝑥𝑥𝑁𝑁−1 𝑆𝑆𝑁𝑁−1 𝑥𝑥𝑁𝑁−1
2
If we repeat this procedure, for time-invarying system.
𝑆𝑆𝑘𝑘 = 𝐴𝐴𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝐴𝐴 + 𝑄𝑄, 𝑘𝑘 < 𝑁𝑁, (1)
For time-varying system:
𝑆𝑆𝑘𝑘 = 𝐴𝐴𝑇𝑇𝑘𝑘 𝑆𝑆𝑘𝑘+1 𝐴𝐴𝑘𝑘 + 𝑄𝑄𝑘𝑘 , 𝑘𝑘 < 𝑁𝑁 , (2)
With boundary condition 𝑆𝑆𝑁𝑁 given as the final state weighting matrix.
This is a discrete Lyapunov equation for 𝑆𝑆𝑘𝑘 , also known as the
observability Lyapunov equation. The zero input performance index over
interval [𝑘𝑘, 𝑁𝑁] is
1
𝐽𝐽𝑘𝑘 = 𝑥𝑥𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘 𝑥𝑥𝑘𝑘
2
We call 𝑆𝑆𝑘𝑘 the performance index kernel sequence. 𝑆𝑆𝑘𝑘 can be calculated
offline without knowing the trajectory 𝑥𝑥𝑘𝑘 . If we know the initial
MECE6388 Optimal Control Theory, 2021 Fall
Dr. Zheng Chen

condition 𝑥𝑥𝑘𝑘 , then the performance index with zero-input over the
interval [𝑘𝑘, 𝑁𝑁] can be computed.

The solution of (1) can be written as

𝑁𝑁−1

𝑆𝑆𝑘𝑘 = (𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑘𝑘 𝑆𝑆𝑁𝑁 𝐴𝐴𝑁𝑁−𝑘𝑘 + � (𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑖𝑖−1 𝑄𝑄𝐴𝐴𝑁𝑁−𝑖𝑖−1

𝑖𝑖=𝑘𝑘
By the lyapunov stability theory, as (𝑁𝑁 − 𝑘𝑘) → ∞, this converges to
∞

𝑆𝑆∞ = �(𝐴𝐴𝑇𝑇 )𝑖𝑖 𝑄𝑄𝐴𝐴𝑖𝑖

𝑖𝑖=0
if the plant is asymptotically stable.

If the plant is stable, the cost over the interval [−∞, 𝑁𝑁] or equivalently
[0, 𝑁𝑁] is given by the steady-state cost.
1
𝐽𝐽∞ = 𝑥𝑥0𝑇𝑇 𝑆𝑆∞ 𝑥𝑥0
2
If 𝐴𝐴 is not stable, the zero input steady state cost can be infinite
(depending on 𝑄𝑄) since ||𝑥𝑥𝑘𝑘 || is unbounded.
In the steady state case, 𝑆𝑆 ≜ 𝑆𝑆𝑘𝑘 = 𝑆𝑆𝑘𝑘+1 when 𝑘𝑘 → ∞
becomes
𝑆𝑆 = 𝐴𝐴𝑇𝑇 𝑆𝑆𝑆𝑆 + 𝑄𝑄
which is called algebraic Lyapunov equation.
MECE6388 Optimal Control Theory, 2021 Fall
Dr. Zheng Chen

By the Lyapunov theory, this equation has a positive semidefinite

solution 𝑆𝑆∞ if 𝐴𝐴 is stable.
If �𝑄𝑄 is defined by
𝑇𝑇
𝑄𝑄 = �𝑄𝑄 ⋅ �𝑄𝑄
then 𝑆𝑆∞ is the unique positive definite solution if 𝐴𝐴 is stable and (𝐴𝐴, �𝑄𝑄)
is observable.

Case B: Fixed final state and open loop control

Let the initial time be 𝑖𝑖 = 0. The initial state 𝑥𝑥0 is given. The terminal
objective is to make 𝑥𝑥𝑁𝑁 match exactly the desired final reference
𝑥𝑥𝑁𝑁 = 𝑟𝑟𝑁𝑁
Since 𝑑𝑑𝑥𝑥𝑁𝑁 = 0,
𝑇𝑇
𝜕𝜕𝜕𝜕
� − 𝜆𝜆𝑁𝑁 � 𝑑𝑑𝑥𝑥𝑁𝑁 = 0
𝜕𝜕𝑥𝑥𝑁𝑁
1
Since 𝑥𝑥𝑁𝑁 = 𝑟𝑟𝑁𝑁 , the final state contribution to 𝐽𝐽0 is a fixed value 𝑟𝑟𝑁𝑁𝑇𝑇 𝑆𝑆𝑁𝑁 𝑟𝑟𝑁𝑁 .
2
It is therefore redundant to include a final state weighting term in 𝐽𝐽0 , so
we set
𝑆𝑆𝑁𝑁 = 0
Let the cost function be
𝑁𝑁−1
1
𝐽𝐽0 = � 𝑢𝑢𝑘𝑘𝑡𝑡 𝑅𝑅𝑢𝑢𝑘𝑘
2
𝑘𝑘=0
and we need to find a control that drives 𝑥𝑥0 exactly to 𝑥𝑥𝑁𝑁 = 𝑟𝑟𝑁𝑁 using
minimum control energy.

The state and costate equation are

𝑥𝑥𝑘𝑘+1 = 𝐴𝐴𝑥𝑥𝑘𝑘 − 𝐵𝐵𝑅𝑅−1 𝐵𝐵𝑇𝑇 𝜆𝜆𝑘𝑘+1
𝜆𝜆𝑘𝑘 = 𝐴𝐴𝑇𝑇 𝜆𝜆𝑘𝑘+1
Since 𝑄𝑄 = 0, the costate equation is decoupled from the state equation
𝜆𝜆𝑘𝑘 = (𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑘𝑘 𝜆𝜆𝑁𝑁
MECE6388 Optimal Control Theory, 2021 Fall
Dr. Zheng Chen

Use this to eliminate 𝜆𝜆𝑘𝑘+1 in (1)

𝑥𝑥𝑘𝑘+1 = 𝐴𝐴𝑥𝑥𝑘𝑘 − 𝐵𝐵𝑅𝑅−1 𝐵𝐵𝑇𝑇 (𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑘𝑘−1 𝜆𝜆𝑁𝑁
Considering this as a first order difference equation with the second term
as the input
𝑘𝑘−1

𝑥𝑥𝑘𝑘 = 𝐴𝐴𝑘𝑘 𝑥𝑥0 − � 𝐴𝐴𝑘𝑘−𝑖𝑖−1 𝐵𝐵𝑅𝑅−1 𝐵𝐵𝑇𝑇 (𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑖𝑖−1 𝜆𝜆𝑁𝑁

𝑘𝑘=0
when 𝑘𝑘 = 𝑁𝑁
𝑁𝑁−1

𝑥𝑥𝑁𝑁 = 𝐴𝐴𝑁𝑁 𝑥𝑥0 − � 𝐴𝐴𝑁𝑁−𝑖𝑖−1 𝐵𝐵𝑅𝑅−1 𝐵𝐵𝑇𝑇 (𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑖𝑖−1 𝜆𝜆𝑁𝑁

𝑖𝑖=0
Since 𝑥𝑥𝑁𝑁 = 𝑟𝑟𝑁𝑁 , solve (2) for 𝜆𝜆𝑁𝑁 we get
−1 (
𝜆𝜆𝑁𝑁 = −𝐺𝐺0,𝑁𝑁 𝑟𝑟𝑁𝑁 − 𝐴𝐴𝑁𝑁 𝑥𝑥0 )
where
𝑁𝑁−1

𝐺𝐺𝑜𝑜,𝑁𝑁 = � 𝐴𝐴𝑁𝑁−𝑖𝑖−1 𝐵𝐵𝑅𝑅−1 𝐵𝐵𝑇𝑇 (𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑖𝑖−1

𝑖𝑖=0
Then the costate is
−1 (
𝜆𝜆𝑘𝑘 = (𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑘𝑘 𝜆𝜆𝑁𝑁 = −(𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑘𝑘 𝐺𝐺0,𝑁𝑁 𝑟𝑟𝑁𝑁 − 𝐴𝐴𝑁𝑁 𝑥𝑥0 )
and the optimal control
𝑢𝑢𝑘𝑘∗ = 𝑅𝑅−1 𝐵𝐵𝑇𝑇 (𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑘𝑘−1 𝐺𝐺0,𝑁𝑁
−1 (
𝑟𝑟𝑁𝑁 − 𝐴𝐴𝑁𝑁 𝑥𝑥0 )
This is the minimum control energy solution to the fixed final state LQR
problem. To verify the control drives 𝑥𝑥0 to 𝑥𝑥𝑁𝑁 = 𝑟𝑟𝑁𝑁 , we get
𝑘𝑘−1

𝑥𝑥𝑘𝑘 = 𝐴𝐴𝑘𝑘 𝑥𝑥0 + � 𝐴𝐴𝑘𝑘−𝑖𝑖−1 𝐵𝐵𝑢𝑢𝑖𝑖

𝑖𝑖=0
where 𝑘𝑘 = 𝑁𝑁 and 𝑢𝑢𝑖𝑖 = 𝑢𝑢𝑖𝑖∗
𝑁𝑁−1
−1 (
𝑥𝑥𝑁𝑁 = 𝐴𝐴𝑁𝑁 𝑥𝑥0 + � 𝐴𝐴𝑁𝑁−𝑖𝑖−1 𝐵𝐵𝑅𝑅−1 𝐵𝐵𝑇𝑇 (𝐴𝐴𝑇𝑇 )𝑁𝑁−𝑖𝑖−1 𝐺𝐺0,𝑁𝑁 𝑟𝑟𝑁𝑁 − 𝐴𝐴𝑁𝑁 𝑥𝑥0 )
𝑖𝑖=0
𝑁𝑁 −1 (
= 𝐴𝐴 𝑥𝑥0 + 𝐺𝐺0,𝑁𝑁 ⋅ 𝐺𝐺0,𝑁𝑁 𝑟𝑟𝑁𝑁 − 𝐴𝐴𝑁𝑁 𝑥𝑥0 ) = 𝑟𝑟𝑁𝑁
𝐺𝐺0,𝑁𝑁 is the weighted reachability gramian of the system.
MECE6388 Optimal Control Theory, 2021 Fall
Dr. Zheng Chen

The system reachability matrix is 𝑈𝑈𝑘𝑘 = [𝐵𝐵 𝐴𝐴𝐴𝐴 … 𝐴𝐴𝑘𝑘−1 𝐵𝐵]

Then
𝑅𝑅−1 ⋯ 0
𝑇𝑇
𝐺𝐺0,𝑁𝑁 = 𝑈𝑈𝑁𝑁 � ⋮ ⋱ ⋮ � 𝑈𝑈𝑁𝑁
0 ⋯ 𝑅𝑅−1
If 𝑅𝑅 = 𝐼𝐼, then 𝐺𝐺0,𝑁𝑁 = 𝑈𝑈𝑁𝑁 ⋅ 𝑈𝑈𝑁𝑁𝑇𝑇 . The optimal control exist if and only if
|𝐺𝐺0,𝑁𝑁 ≠ 0|. Since we assumed |𝑅𝑅| ≠ 0, this is equivalent to 𝑈𝑈𝑁𝑁 having
full rank 𝑛𝑛, where 𝑛𝑛 is the state dimension. Therefore, we can drive any
given 𝑥𝑥0 to any desired 𝑥𝑥𝑁𝑁 = 𝑟𝑟𝑁𝑁 for some 𝑁𝑁 steps if and only if the
system is reachable (controllable)

To compute the reachability gramian, we can form the Lyapunov

equation
𝑃𝑃𝑘𝑘+1 = 𝐴𝐴𝑃𝑃𝑘𝑘 𝐴𝐴𝑇𝑇 + 𝐵𝐵𝑅𝑅−1 𝐵𝐵𝑇𝑇 , 𝑘𝑘 > 0
The solution is
𝑘𝑘−1

𝑃𝑃𝑘𝑘 = 𝐴𝐴𝑘𝑘 𝑃𝑃0 (𝐴𝐴𝑘𝑘 )𝑇𝑇 + � 𝐴𝐴𝑘𝑘−𝑖𝑖−1 𝐵𝐵𝑅𝑅−1 𝐵𝐵𝑇𝑇 (𝐴𝐴𝑇𝑇 )𝑘𝑘−𝑖𝑖−1
𝑖𝑖=0
with 𝑃𝑃0 = 0, we get 𝐺𝐺0,𝑘𝑘 = 𝑃𝑃𝑘𝑘 for any 𝑘𝑘 > 0 .
When 𝑘𝑘 = 𝑁𝑁, 𝐺𝐺0,𝑁𝑁 = 𝑃𝑃𝑁𝑁

Case C: Free final state and closed loop control

The state and costate equations:
𝑥𝑥𝑘𝑘+1 = 𝐴𝐴𝑘𝑘 𝑥𝑥𝑘𝑘 − 𝐵𝐵𝑘𝑘 𝑅𝑅𝑘𝑘−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝜆𝜆𝑘𝑘+1
𝜆𝜆𝑘𝑘 = 𝑄𝑄𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝐴𝐴𝑇𝑇𝑘𝑘 𝜆𝜆𝑘𝑘+1
The control is given as
𝑢𝑢𝑘𝑘 = −𝑅𝑅𝑘𝑘−1 𝐵𝐵𝑇𝑇 𝜆𝜆𝑘𝑘+1
The initial state 𝑥𝑥𝑖𝑖 is given and final state 𝑥𝑥𝑁𝑁 is free.

Since 𝑑𝑑𝑥𝑥𝑁𝑁 ≠ 0, then

𝜆𝜆𝑁𝑁 = 𝑆𝑆𝑁𝑁 𝑥𝑥𝑁𝑁
MECE6388 Optimal Control Theory, 2021 Fall
Dr. Zheng Chen

Assume a linear relation

𝜆𝜆𝑘𝑘 = 𝑆𝑆𝑘𝑘 𝑥𝑥𝑘𝑘 , (1)
For some intermediate sequence of 𝑛𝑛 × 𝑛𝑛 matrices 𝑆𝑆𝑘𝑘
𝑥𝑥𝑘𝑘+1 = 𝐴𝐴𝑘𝑘 𝑥𝑥𝑘𝑘 − 𝐵𝐵𝑘𝑘 𝑅𝑅𝑘𝑘−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝑥𝑥𝑘𝑘+1
𝑥𝑥𝑘𝑘+1 = (𝐼𝐼 + 𝐵𝐵𝑘𝑘 𝑅𝑅𝑘𝑘−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 )−1 𝐴𝐴𝑘𝑘 𝑥𝑥𝑘𝑘 , (2)
Now substitute (1) into the costate equation
𝜆𝜆𝑘𝑘 = 𝑆𝑆𝑘𝑘 𝑥𝑥𝑘𝑘 = 𝑄𝑄𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝐴𝐴𝑇𝑇𝑘𝑘 𝑆𝑆𝑘𝑘+1 𝑥𝑥𝑘𝑘+1
𝑆𝑆𝑘𝑘 𝑥𝑥𝑘𝑘 = 𝑄𝑄𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝐴𝐴𝑇𝑇𝑘𝑘 𝑆𝑆𝑘𝑘+1 (𝐼𝐼 + 𝐵𝐵𝑘𝑘 𝑅𝑅−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 )−1 𝐴𝐴𝑘𝑘 𝑥𝑥𝑘𝑘
Since 𝑥𝑥𝑘𝑘 is generally nonzero,
𝑆𝑆𝑘𝑘 = 𝐴𝐴𝑇𝑇𝑘𝑘 𝑆𝑆𝑘𝑘+1 (𝐼𝐼 + 𝐵𝐵𝑘𝑘 𝑅𝑅𝑘𝑘−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 )−1 𝐴𝐴𝑘𝑘 + 𝑄𝑄𝑘𝑘
Using matrix inversion lemma
𝑆𝑆𝑘𝑘 = 𝐴𝐴𝑇𝑇𝑘𝑘 [𝑆𝑆𝑘𝑘+1 − 𝑆𝑆𝑘𝑘+1 𝐵𝐵𝑘𝑘 (𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝐵𝐵𝑘𝑘 + 𝑅𝑅𝑘𝑘 )−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 ]𝐴𝐴𝑘𝑘 + 𝑄𝑄𝑘𝑘 (3)
This is a backward recursion for the postulated𝑆𝑆𝑘𝑘 . The boundary 𝑆𝑆𝑁𝑁 is
known. The matrices 𝑆𝑆𝑘𝑘 can be precomputed offline. Equation (3) is
called a Riccati equation.

If |𝑆𝑆𝑘𝑘 | ≠ 0 for all 𝑘𝑘, the Riccati equation can be rewritten by

𝑆𝑆𝑘𝑘 = 𝐴𝐴𝑇𝑇𝑘𝑘 (𝑆𝑆𝑘𝑘+1
−1
+ 𝐵𝐵𝑘𝑘 𝑅𝑅𝑘𝑘−1 𝐵𝐵𝑘𝑘𝑇𝑇 )−1 𝐴𝐴𝑘𝑘 + 𝑄𝑄𝑘𝑘

The optimal control

𝑢𝑢𝑘𝑘∗ = −𝑅𝑅𝑘𝑘−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝜆𝜆𝑘𝑘+1 = −𝑅𝑅𝑘𝑘−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝑥𝑥𝑘𝑘+1
with 𝑥𝑥𝑘𝑘+1 = 𝐴𝐴𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝐵𝐵𝑘𝑘 𝑢𝑢𝑘𝑘
𝑢𝑢𝑘𝑘∗ = −(𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝐵𝐵𝑘𝑘 + 𝑅𝑅𝑘𝑘 )−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝐴𝐴𝑘𝑘 𝑥𝑥𝑘𝑘
Define the Kalman gain sequence
𝐾𝐾𝑘𝑘 = (𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝐵𝐵𝑘𝑘 + 𝑅𝑅𝑘𝑘 )−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝐴𝐴𝑘𝑘
the control takes the form
𝑢𝑢𝑘𝑘∗ = −𝐾𝐾𝑘𝑘 𝑥𝑥𝑘𝑘
The Kalman gain is given in terms of the Riccati equation solution 𝑆𝑆𝑘𝑘 and
the system and weighting matrices.
MECE6388 Optimal Control Theory, 2021 Fall
Dr. Zheng Chen

In the free final state linear quadratic (LQ) regulator, the optimal control
is given by a close loop control law.

A more efficient way to compute 𝐾𝐾𝑘𝑘 and 𝑆𝑆𝑘𝑘 by using the recursion
𝐾𝐾𝑘𝑘 = (𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝐵𝐵𝑘𝑘 + 𝑅𝑅𝑘𝑘 )−1 𝐵𝐵𝑘𝑘 𝑆𝑆𝑘𝑘+1 𝐴𝐴𝑘𝑘
𝑆𝑆𝑘𝑘 = 𝐴𝐴𝑇𝑇𝑘𝑘 𝑆𝑆𝑘𝑘+1 (𝐴𝐴𝑘𝑘 − 𝐵𝐵𝑘𝑘 𝐾𝐾𝑘𝑘 ) + 𝑄𝑄𝑘𝑘

Table: Discrete Linear Quadratic Regulator (Final state free)

System model:
𝑥𝑥𝑘𝑘+1 = 𝐴𝐴𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝐵𝐵𝑘𝑘 𝑢𝑢𝑘𝑘 , 𝑘𝑘 > 𝑖𝑖
Performance index:
𝑁𝑁−1
1 1
𝐽𝐽𝑖𝑖 = 𝑥𝑥𝑁𝑁𝑇𝑇 𝑆𝑆𝑁𝑁 𝑥𝑥𝑁𝑁 + �(𝑥𝑥𝑘𝑘𝑇𝑇 𝑄𝑄𝑘𝑘 𝑥𝑥𝑘𝑘 + 𝑢𝑢𝑘𝑘𝑇𝑇 𝑅𝑅𝑘𝑘 𝑢𝑢𝑘𝑘 )
2 2
𝑘𝑘=𝑖𝑖
Assumption:
𝑆𝑆𝑛𝑛 ≥ 0, 𝑄𝑄𝑘𝑘 ≥ 0, 𝑅𝑅𝑘𝑘 ≥ 0
and all three are symmetric
Optimal feedback control:
𝑆𝑆𝑘𝑘 = 𝐴𝐴𝑇𝑇𝑘𝑘 [𝑆𝑆𝑘𝑘+1 − 𝑆𝑆𝑘𝑘+1 𝐵𝐵𝑘𝑘 (𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝐵𝐵𝑘𝑘 + 𝑅𝑅𝑘𝑘 )−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 ]𝐴𝐴𝑘𝑘 + 𝑄𝑄𝑘𝑘
Or 𝑆𝑆𝑘𝑘 = 𝐴𝐴𝑇𝑇𝑘𝑘 𝑆𝑆𝑘𝑘+1 (𝐴𝐴𝑘𝑘 − 𝐵𝐵𝑘𝑘 𝐾𝐾𝑘𝑘 ) + 𝑄𝑄𝑘𝑘 , 𝐾𝐾 < 𝑁𝑁, 𝑆𝑆𝑁𝑁 given.
𝐾𝐾𝑘𝑘 = (𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝐵𝐵𝑘𝑘 + 𝑅𝑅𝑘𝑘 )−1 𝐵𝐵𝑘𝑘𝑇𝑇 𝑆𝑆𝑘𝑘+1 𝐴𝐴𝑘𝑘 , 𝑘𝑘 < 𝑁𝑁
MECE6388 Optimal Control Theory, 2021 Fall
Dr. Zheng Chen

𝑢𝑢𝑘𝑘 = −𝐾𝐾𝑘𝑘 𝑥𝑥𝑘𝑘 , 𝑘𝑘 < 𝑁𝑁

1
𝐽𝐽𝑖𝑖∗ = 𝑥𝑥𝑖𝑖𝑇𝑇 𝑆𝑆𝑖𝑖 𝑥𝑥𝑖𝑖
2

Example: Optimal Feedback Control of a scalar system

The plant is described by
𝑥𝑥𝑘𝑘+1 = 𝑎𝑎𝑥𝑥𝑘𝑘 + 𝑏𝑏𝑢𝑢𝑘𝑘
with performance index
𝑁𝑁−1
1 1
𝐽𝐽𝑖𝑖 = 𝑆𝑆𝑁𝑁 𝑥𝑥𝑁𝑁2 + � (𝑞𝑞𝑥𝑥𝑘𝑘2 + 𝑟𝑟𝑢𝑢𝑘𝑘2 )
2 2
𝑘𝑘=𝑖𝑖
The Riccati equation is
2
2
𝑎𝑎2 𝑏𝑏 2 𝑆𝑆𝑘𝑘+1
𝑆𝑆𝑘𝑘 = 𝑎𝑎 𝑆𝑆𝑘𝑘+1 − 2 + 𝑞𝑞
𝑏𝑏 𝑆𝑆𝑘𝑘+1 + 𝑟𝑟
𝑎𝑎2 𝑟𝑟𝑆𝑆𝑘𝑘+1
= 2 + 𝑞𝑞
𝑏𝑏 𝑆𝑆𝑘𝑘+1 + 𝑟𝑟
The Kalman gain is
𝑎𝑎
𝑎𝑎𝑎𝑎𝑆𝑆𝑘𝑘+1 𝑏𝑏
𝐾𝐾𝑘𝑘 = 2 =
𝑏𝑏 𝑆𝑆𝑘𝑘+1 + 𝑟𝑟 1 + 𝑟𝑟 𝑆𝑆𝑘𝑘+1
𝑏𝑏 2
And the optimal control is
𝑢𝑢𝑘𝑘 = −𝐾𝐾𝑘𝑘 𝑥𝑥𝑘𝑘
The optimal value of the performance index is
1
𝐽𝐽𝑘𝑘 = 𝑆𝑆𝑘𝑘 𝑥𝑥𝑘𝑘2
2
𝑆𝑆𝑘𝑘 is hard to find analytically but for a particular value of 𝑎𝑎, 𝑏𝑏, 𝑞𝑞, 𝑟𝑟, 𝑆𝑆𝑁𝑁 , it
is very easy to compute the optimal control sequence 𝑢𝑢𝑘𝑘∗ .

Case A: No Control Weighting

Let 𝑟𝑟 = 0, meaning that we do not care how much control is used. Then
MECE6388 Optimal Control Theory, 2021 Fall
Dr. Zheng Chen

𝑎𝑎2 𝑟𝑟𝑆𝑆𝑘𝑘+1
𝑆𝑆𝑘𝑘 = 2 + 𝑞𝑞 = 𝑞𝑞
𝑏𝑏 𝑆𝑆𝑘𝑘+1 + 𝑟𝑟
And
𝑎𝑎
𝑏𝑏 𝑎𝑎 𝑎𝑎
𝐾𝐾𝑘𝑘 = 𝑟𝑟 = − , 𝑢𝑢𝑘𝑘 = − 𝑥𝑥𝑘𝑘
1 + 2 𝑆𝑆𝑘𝑘+1 𝑏𝑏 𝑏𝑏
𝑏𝑏
Under the influence of this control
1
𝐽𝐽𝑘𝑘 = 𝑞𝑞𝑥𝑥𝑘𝑘2
2
And closed loop system becomes
𝑥𝑥𝑘𝑘+1 = 𝑎𝑎𝑥𝑥𝑘𝑘 + 𝑏𝑏𝑢𝑢𝑘𝑘 = 0
If we have a given value of 𝑥𝑥𝑘𝑘 for the state at time 𝐾𝐾, then to minimize
the magnitude of the state vector, we solve the state equation for 𝑢𝑢𝑘𝑘 so
that 𝑥𝑥𝑘𝑘+1 = 0

Case B: Very large control weighting

When 𝑟𝑟 → ∞, then
𝑎𝑎2 𝑟𝑟𝑆𝑆𝑘𝑘+1 𝑟𝑟→∞
𝑆𝑆𝑘𝑘 = 2 + 𝑞𝑞 �⎯� 𝑎𝑎2 𝑆𝑆𝑘𝑘+1 + 𝑞𝑞
𝑏𝑏 𝑆𝑆𝑘𝑘+1 + 𝑟𝑟
The solution of this equation is
𝑁𝑁−1

𝑆𝑆𝑘𝑘 = 𝑆𝑆𝑁𝑁 𝑎𝑎2(𝑁𝑁−𝑘𝑘) + � 𝑞𝑞𝑎𝑎2(𝑁𝑁−𝑖𝑖−1)

𝑖𝑖=𝑘𝑘

2(𝑁𝑁−𝑘𝑘)
1 − 𝑎𝑎2(𝑁𝑁−𝑘𝑘)
= 𝑆𝑆𝑁𝑁 𝑎𝑎 +( )𝑞𝑞
1 − 𝑎𝑎2
The Kalman gain
𝑎𝑎
𝑏𝑏 𝑟𝑟→∞
𝐾𝐾𝑘𝑘 = 𝑟𝑟 �⎯� 0
1 + 2 𝑆𝑆𝑘𝑘+1
𝑏𝑏
and 𝑢𝑢𝑘𝑘 = 0
MECE6388 Optimal Control Theory, 2021 Fall
Dr. Zheng Chen

The closed loop system becomes

𝑥𝑥𝑘𝑘+1 = 𝑎𝑎𝑥𝑥𝑘𝑘
If we are very concerned about using too much control, the best policy
is to use none at all.

Case C: No intermediate state weighting

Let 𝑞𝑞 = 0, then we are concerned about making 𝑋𝑋𝑁𝑁2 small without using
too much control energy.
𝑎𝑎2 𝑟𝑟𝑆𝑆𝑘𝑘+1
𝑆𝑆𝑘𝑘 = 2 + 𝑞𝑞
𝑏𝑏 𝑆𝑆𝑘𝑘+1 + 𝑟𝑟
−1
𝑎𝑎2 𝑟𝑟𝑆𝑆𝑘𝑘+1 −1
𝑏𝑏 2 + 𝑟𝑟𝑆𝑆𝑘𝑘+1
𝑆𝑆𝑘𝑘 = 2 ⇒ 𝑆𝑆𝑘𝑘 =
𝑏𝑏 𝑆𝑆𝑘𝑘+1 + 𝑟𝑟 𝑎𝑎2 𝑟𝑟
−1
−1
𝑆𝑆𝑘𝑘+1 𝑏𝑏 2
⇒ 𝑆𝑆𝑘𝑘 = 2 + 2
𝑎𝑎 𝑎𝑎 𝑟𝑟
The solution is
𝑁𝑁−1
𝑏𝑏 2 −2(𝑁𝑁−𝑖𝑖−1)
𝑆𝑆𝑘𝑘−1 = 𝑎𝑎−2(𝑁𝑁−𝑘𝑘) 𝑆𝑆𝑁𝑁−1 + � 2 𝑎𝑎
𝑎𝑎 𝑟𝑟
𝑖𝑖=𝑘𝑘
𝑏𝑏 21 − 𝑎𝑎2(𝑁𝑁−𝑘𝑘)
= 𝑆𝑆𝑁𝑁−1 𝑎𝑎−2(𝑁𝑁−𝑘𝑘)
+ 2(𝑁𝑁−𝑘𝑘)
𝑟𝑟𝑎𝑎 1 − 𝑎𝑎2
𝑆𝑆𝑁𝑁 𝑎𝑎2(𝑁𝑁−𝑘𝑘)
⇒ 𝑆𝑆𝑘𝑘 =
𝑏𝑏 2 1 − 𝑎𝑎2(𝑁𝑁−𝑘𝑘)
1 + 𝑆𝑆𝑁𝑁 � � � �
𝑟𝑟 1 − 𝑎𝑎2