Linear-Quadratic-Gaussian Immune Response Example

Controllers ! Optimal open-loop drug therapy (control)

Robert Stengel ! Optimal closed-loop therapy
Optimal Control and Estimation MAE 546 ! Stochastic optimal closed-loop therapy
Princeton University, 2010
! Importance of estimation as well as control
! Certainty equivalence and separation
! Asymptotic stability of the constant-gain LQG
regulator
! Coupling due to parameter uncertainty
! Robustness (loop transfer) recovery
! Stochastic robustness analysis and design

Copyright 2010 by Robert Stengel. All rights reserved. For educational use only.
http://www.princeton.edu/~stengel/MAE546.html
http://www.princeton.edu/~stengel/OptConEst.html

Immune Response Immune Response with Stochastic

Example with Optimal Feedback Control
(Random Disturbance and Measurement Error Not
Optimal Feedback Simulated)
Control
Low-Bandwidth Estimator High-Bandwidth Estimator
(|W| < |N|) (|W| > |N|)
Open- and Closed-Loop
Open-Loop Optimal Control Optimal Control for 150%
for Lethal Initial Condition Lethal Initial Condition
 The Problem: Choose Control to Minimize Cost,
Immune Response to Random Subject to Dynamic Constraint, Uncertain
Disturbance with Stochastic Disturbances, and Measurement Error
Optimal Control x! ( t ) = Fx(t) + Gu(t) + Lw(t), x ( 0 ) = x o
z ( t ) = Hx ( t ) + n ( t ) Dynamic System
• Disturbance due to
– Re-infection min V ( t o ) = min J t f
u u
( )
– Sequestered “pockets”
( (*t f ! Q 0 # ! x(t) # ,* ,*
of pathogen 1 *
= min E ) E !" xT (t f )S(t f )x(t f ) | I D #$ + E ) ' ! xT (t) uT (t) # % % & dt I D - -
• Noisy measurements 2 u " $ % 0 R && % u(t) &
*+ *+ 0 " $" $ *. *.
• Closed-loop therapy is where
robust Cost Function
I D ( t ) = {x̂ ( t ) , P ( t ) , u ( t )}
• ... but not robust enough:

{ } = P(0)
– Organ death occurs in
E !" x ( 0 ) #$ = x̂ o ; E !" x ( 0 ) % x̂ ( 0 ) #$ !" x ( 0 ) % x̂ ( 0 ) #$
T
one case
• Probability of satisfactory E !" w ( t ) #$ = 0; E !" w ( t ) w T (& ) #$ = W' ( t % & )
therapy can be dim !" x ( t ) #$ = n % 1
E !" n ( t ) #$ = 0; E !" n ( t ) nT (& ) #$ = N' ( t % & )
maximized by stochastic dim !" u ( t ) #$ = m % 1 Dimensions

redesign of controller E !" w ( t ) nT (& ) #$ = 0 Statistics dim !" w ( t ) #$ = s % 1

dim !" z ( t ) #$ = dim !" n ( t ) #$ = r % 1

Linear-Quadratic- Estimator in the

Gaussian Control Feedback Loop
x! ( t ) = Fx(t) + Gu(t) + Lw(t), x ( 0 ) = x o
z ( t ) = Hx ( t ) + n ( t )
! Linear-Gaussian (LG) state estimator adds dynamics to
the feedback signal
Control Law u ( t ) = !C(t)x̂(t)
x!ˆ ( t ) = Fx̂(t) + Gu(t) + K ( t ) "# z ( t ) ! Hx̂ ( t ) $% u ( t ) = !C(t)x̂(t)
Estimator xˆ! ( t ) = Fx̂(t) + Gu(t) + K ( t ) "# z ( t ) ! Hx̂ ( t ) $%
! Thus, it can be viewed as a “compensator”
Control Gain
C ( t ) = R !1GT S ( t ) ! Bandwidth of the compensation is dependent on the
S! ( t ) = !Q ! FT S ( t ) ! S ( t ) F + S ( t ) GR !1GT S ( t ) multivariable signal/noise ratio, PHTN–1

K ( t ) = P(t)HT N !1
Estimator Gain K ( t ) = P(t)H N T !1
! = FP(t) + P(t)FT + LWLT ! P(t)HT N !1HP(t)
P(t)
! = FP(t) + P(t)FT + LWLT ! P(t)HT N !1HP(t)
P(t)
 Scalar LTI Example of Steady-State Signal "Power" = State Estimate Variance = P
Noise "Power" = Measurement Error Variance = N
Estimator Compensation Filter Gain H = Projection from Noise Space to Signal Space

! Dynamic system and x! = x + w; z = Hx + n

measurement ! Constant, scalar PH
K=
filter gain N
! Estimator differential
x!ˆ = x̂ + K ( z ! Hx̂ ) = (1 ! KH ) x̂ + Kz
equation
P2 H 2 2N WN
0 = 2P + W ! ; P2 ! 2 P ! 2 = 0
! Laplace transform sx̂ ( s ) = (1 ! KH ) x̂ ( s ) + Kz ( s ) ! Algebraic Riccati N H H
equation N ( WH 2 +
2
of estimator N " N % WN
sx̂ ( s ) ! (1 ! KH ) x̂ ( s ) = Kz ( s ) P = 2 ± $ 2 ' + 2 = 2 *1 ± 1 +
H #H & H H *) N -,
-

K ') N ! $ +)
! Estimator transfer x̂ ( s ) = z (s) ( 2 #1 + 1 +
WH 2
! Constant, scalar &, H
function "# s ! (1 ! KH ) $% ) H #" N &% )- )' 1 ! WH 2 $ )+
filter gain K=* = ( #1 + 1 + &,
! Low-pass filter
x̂ ( s ) K
N
*) #"
H N &% )
-
=
z ( s ) "# s ! (1 ! KH ) $% .W..
>> N
/
W
N

Expected Values of Dependence of the Stochastic Cost

State and Control Function on the Information Set

! Expected values of the state and control 1 &( (*

tf tf

J=
2 )(
{ } { }
E ' E !" Tr !"S(t f )x(t f )xT (t f ) #$ | I D #$ + % E Tr !"Qx ( t ) xT ( t ) #$ dt + % E Tr !" Ru ( t ) uT ( t ) #$ dt +
are conditioned on the information set 0 0 ,(

! Expand the state covariance

E !" x ( t ) | I D #$ = x̂ ( t )
{
P ( t ) = E "# x ( t ) ! x̂ ( t ) $% "# x ( t ) ! x̂ ( t ) $% | I D
T
}
{ T
}
E !" x ( t ) % x̂ ( t ) #$ !" x ( t ) % x̂ ( t ) #$ | I D = P ( t ) {
= E "# x ( t ) x ( t ) ! x̂ ( t ) x ( t ) ! x ( t ) x̂ ( t ) + x̂ ( t ) x̂T ( t ) $% | I D
T T T
}
{ } { }
E !" x ( t ) x̂T ( t ) #$ | I D = E !" x̂ ( t ) xT ( t ) #$ | I D = x̂ ( t ) x̂T ( t )
... where the conditional expected values
are obtained from a Kalman-Bucy filter { }
P ( t ) = E !" x ( t ) xT ( t ) #$ | I D % x̂ ( t ) x̂T ( t )
or
{ }
E !" x ( t ) xT ( t ) #$ | I D = P ( t ) + x̂ ( t ) x̂T ( t )
 Certainty-Equivalent and Dynamic Constraint on the
Stochastic Incremental Costs Certainty-Equivalent Cost
1 &( *( ! P(t) is independent of u(t); therefore
{ }
tf tf

J= ( ) { }
E ' Tr S(t f ) !" P t f + x̂(t f )x̂T (t f ) #$ + % Tr Q !" P ( t ) + x̂ ( t ) x̂T ( t ) #$ dt + % Tr !" Ru ( t ) uT ( t ) #$ dt +
2 )( ,(
! J CE + J S
0 0
min J = min J CE + J S
u u

! Cost function has two parts ! JCE is

! Certainty-equivalent cost ! Identical in form to the deterministic cost function
! Stochastic increment cost ! Minimized subject to dynamic constraint based on
the state estimate
1 &( *(
tf tf

J CE = { }
E ' Tr !"S(t f )x̂(t f )x̂T (t f ) #$ + % Tr Qx̂ ( t ) x̂T ( t ) dt + % Tr !" Ru ( t ) uT ( t ) #$ dt +
xˆ! ( t ) = Fx̂(t) + Gu(t) + K ( t ) "# z ( t ) ! Hx̂ ( t ) $%
2 () 0 0 (,
1 &( (*
tf

( )
J S = E ' Tr !"S(t f )P t f #$ + % Tr !"QP ( t ) #$ dt +
2 )( 0 ,(

Kalman Filter Provides Estimate Control That Minimizes the

of the State Mean Value Certainty-Equivalent Cost
! Filter residual is a Gaussian white-noise ! Optimizing control history is generated
process by a time-varying feedback control law
x!ˆ ( t ) = Fx̂(t) + Gu(t) + K ( t ) "# z ( t ) ! Hx̂ ( t ) $% u ( t ) = !C(t)x̂(t)
" Fx̂(t) + Gu(t) + K ( t ) & ( t )
! The control gain is the same as the
... which is analogous to the deterministic dynamic deterministic gain
constraint on a deterministic cost function
C ( t ) = R !1GT S ( t )
x! ( t ) = Fx(t) + Gu(t) + Lw(t) ( )
S! ( t ) = !Q ! FT S ( t ) ! S ( t ) F + S ( t ) GR !1GT S ( t ) , S t f given
 Minimum Cost for the
LQG Controller
! By comparison to the previous result

J CE
1 ! T
tf
#
= Tr &S(0)E !" x̂ ( 0 ) x̂ ( 0 ) #$ + % S ( t ) K ( t ) NKT ( t ) dt '
Asymptotic Stability of
'$
the LQG Regulator
2 &" 0

... and
J = J CE + J S
1 &( (*
tf

= Tr 'S(0)E !" x̂ ( 0 ) x̂T ( 0 ) #$ + % S ( t ) K ( t ) NKT ( t ) dt +

2 () 0 (,
1 ! #
tf

+ Tr -S(t f )P(t f ) + % QP ( t ) dt .
2 -" 0 .$

System Equations with System Equations with

LQG Control LQG Control
x! ( t ) = Fx(t) + Gu(t) + Lw(t) " x! ( t ) % " ( F ( GC ) GC % " x (t ) % " % " w (t ) %
$ '=$ '$ ' + $ L 0 '$ '
$ ! (t ) ( F ( KH ) '& $# ! (t ) ' # L (K & $ n ( t )
x!ˆ ( t ) = Fx̂(t) + Gu(t) + K ( t ) "# z ( t ) ! Hx̂ ( t ) $% #
' $
& #
0
& #
'
&

! Upper-block-triangular stability matrix

! ( t ) = x ( t ) " x̂ ( t ) ! LQG system is stable because
! (F – GC) is stable
! ( t ) = ( F " KH ) ! ( t ) + Lw ( t ) " Kn ( t ) ! (F – KH) is stable
! Estimate error affects state response
! Actual state does not affect error response
! Disturbance affects both equally
 Coupling Due To
Parameter Uncertainty
Actual System: {FA ,G A , H A }
Coupling Due to Assumed System: {F,G, H}

x! ( t ) = FA x(t) + G A u(t) + Lw(t); z ( t ) = H A x(t) + n ( t )

Parameter Uncertainty xˆ! ( t ) = Fx̂(t) + Gu(t) + K ( t ) "# z ( t ) ! Hx̂ ( t ) $%

" x! ( t ) % " ( FA ( G AC ) G AC % " x (t ) %

$ '=$ '$ ' +"
& # #( A
$ ! (t ) ' $ " F ( F ) ( ( G A ( G ) C ( K ( H A ( H ) %& "# F + ( G A ( G ) C ( KH %& ' $ ! (t ) '
# &# &

! Closed-loop control and estimator responses

are coupled

Effects of Parameter Doyle!s Counter-Example

Uncertainty on Closed-Loop of LQG Robustness (1978)
Stability
! x!1 $ ! 1 1 $ ! x1 $ ! 0 $ ! 1 $
"# sI n ! ( FA ! G AC ) $% !G AC # &=# &# &+# &u + # &w
sI 2n ! FCL = #" x!2 &% " 0 1 % #" x2 &% " 1 % " 1 %
! "#( FA ! F ) ! ( G A ! G ) C ! K ( H A ! H ) $% {sI n ! "# F + ( G A ! G ) C ! KH $% } Unstable
Plant ! x1 $
= & CL ( s ) = 0 z = !" 1 0 $% # &+n
#" x2 &%
! Uncertain parameters affect closed-loop
eigenvalues Design ! 1 1 $ ! 1 1 $
Matrices
Q = Q# & ; R = 1; W = W # &; N = 1
! Coupling can lead to instability for " 1 1 % " 1 1 %

numerous reasons
( )
C = 2 + 4 + Q !" 1 1 #$ = !" c c #$
! Improper control gain Control and
! 1 # ! k #
Estimator Gains K = (2 + 4 + W )% &=%
! Control effect on estimator block " 1 $ "
&
k $

! Redistribution of damping
Uncertainty in the Control Effect Doyle!s Counter-Example
! 0 $
FA = F; G A = # &; HA = H ! Solution
#" µ &% ! Choose Q and W to be small, increasing allowable
range of µ
( s ! 1) !1 0 0
0 ( s ! 1) µc µc
= s 4 + a3 s 3 + a2 s 2 + a1s + a0 = " CL ( s ) = 0
!k 0 (s ! 1 + k) !1 ! However, .... The counter-example is irrelevant
!k 0 (c + k ) ( s ! 1 + c) because it does not satisfy the requirements for LQ
and LG stability
! Routh!s stability criterion
! The open-loop system is unstable, so it requires
! All coefficients of !(s) must be positive for stability feedback control to restore stability
! µ is nominally equal to 1
! To guarantee stability, Q and W must be positive
! µ can force a1 and a0 to change sign definite, but
! Result is dependent on magnitude of ck
a1 = k + c ! 4 + 2 ( µ ! 1) ck ! 1 1 $ ! 1 1 $
Q = Q# & ; hence, Q = 0; W = W # & ; hence, W = 0
a0 = 1 + (1 ! µ ) ck " 1 1 % " 1 1 %
! Arbitrarily small uncertainty could cause instability
! This should not be surprising, as uncertainty is in the control effect

Loop-Transfer Recovery
(Doyle and Stein, 1979)
! Proposition: LQG robustness would be the same as

Robustness
LQ robustness if the control vector had the same
effect on the state and its estimate
Then

(Loop Transfer) Cx ( t ) and Cx̂ ( t ) produce same expected value of control, E !" u ( t ) #$
but not the same

{
E !" u ( t ) % u ( t ) #$ !" u ( t ) % u ( t ) #$
T
}
Recovery as x̂ ( t ) contains measurement errors but x ( t ) does not

! Therefore, restoring the correct mean value from z(t)

restores closed-loop robustness
! Solution: Increase the assumed “process noise” for
estimator design as follows (see text for details)

W = Wo + ! 2GGT
 Expression of Uncertainty
Uniform Distribution
• Uncertainty may be expressed as variation in of Eigenvalues

– Elements of F

Stochastic – Coefficients of D(s)

– Eigenvalues

Robustness Analysis
– Frequency response/singular values/time
response
• Variation may be

and Design – Deterministic, e.g.,

• Upper/lower bounds
Gaussian Distribution
of Eigenvalues

– Probabilistic, e.g.,
• Gaussian distribution
• Bounded variation is equivalent to
probabilistic variation with uniform
distribution

Probability of Instability Probabilistic Control Design

• Design constant-parameter controller (CPC) for
satisfactory stability and performance in an
uncertain environment
• Monte Carlo Evaluation of simulated system
• Nonlinear mapping from
probability density response with
functions (pdf) of – competing CPC designs [Design parameters = d]
uncertain parameters
to pdf of roots – given statistical model of uncertainty in the plant
• Finite probability of [Uncertain plant parameters = v]
instability with
Gaussian (unbounded)
• Search for best CPC
distribution – Exhaustive search
• Zero probability of – Random search
instability for some
uniform distributions – Multivariate line search
– Genetic algorithm
– Simulated annealing
 Design Outcome Follows Example of an Unstable Plant
Binomial Distribution
• Longitudinal dynamics for a
X-29 Aircraft Forward-Swept-Wing
• Binomial distribution: Satisfactory/Unsatisfactory Demonstrator
• Confidence intervals of probability estimate are functions of #!2gf11 /V "V 2 f12 /2 "Vf13 !g& # g11 g12 &
% ( % (
– Actual probability !45 /V 2 "Vf 22 /2 1 0( 0 0(
F =% ; G =%
% 0 "V 2 f 32 /2 "Vf 33 0 ( %g31 g32 (
– Number of trials % ( % (
$ 0 0 1 0' $0 0'

• Nominal eigenvalues (one unstable)

!1"4 = "0.1± 0.057 j, " 5.15, 3.35

• ! and V have uniform distributions (±30%)

• 10 Aerodynamic coefficients have Gaussian distributions
(" = 30%)
p = [! V f11 f12 f13 f 22 f 32 f 33 g11 g12 g31 g32 ]
Environment Uncontrolled Dynamics Control Effect
Maximum Entropy when Pr = 0.5

LQ Regulators for Probabilities of Instability

the Example for the Three Cases

Q = diag(1,1,1,0); R = (1,1); !1"4 nominal = "35,"5.1,"3.3,".02

• Case a) LQR with low
control weighting #0.17 130 33 0.36&
C=% (
$0.98 "11 "3 "1.1'

• Case b) LQR with high Q = diag(1,1,1,0); R = (1000,1000); !1"4 nominal = "5.2,"3.4,"1.1,".02

control weighting #0.03 83 21 "0.06&
C=% (
$0.01 "63 "16 "1.9 '

• Case c) Case b with gains !1"4 nominal = "32,"5.2,"3.4,"0.01

multiplied by 5 for bandwidth #0.13 413 105 "0.32 &
(loop-transfer) recovery C=% (
$0.05 "313 "81 "1.1" 9.5'
 Stochastic Root Loci for Benchmark Control Problem
the Three Cases (ACC, 1991)
Case a: Low Control Weights

with Gaussian Aerodynamic Uncertainty

• Parameters of 4th-order mass-spring system
– Uniform probability density functions for
• 0.5 < m1, m2 < 1.5
Case b: High Control Weights
• 0.5 < k < 2
• Probability of Instability, Pi
– mi = 1 (unstable) or 0 (stable)
• Probability of Settling Time Exceedance, Pts
– mts = 1 (exceeded) or 0 (not exceeded)
Case c: Bandwidth Recovery

• Probability of Control Limit Exceedance, Pu

• Probabilities of instability with – mu = 1 (exceeded) or 0 (not exceeded)
30% uniform aerodynamic
uncertainty • Design Cost Function
– Case a: 3.4 x 10-4
– Case b: 0 – J = a Pi2 + b Pts2 + c Pu2 a = 1, b = c = 0.01
– Case c: 0

Stochastic LQG Design for Comparison of Design Costs for

Benchmark Control Problem
Benchmark Control Problem
Cost Emphasizes
• SISO Linear-Quadratic-Gaussian Regulators Instability
Cost Emphasizes
Excess Control
Cost Emphasizes Settling-
Time Exceedance
(Marrison)
– Implicit model following with control-rate weighting and
scalar output (5th order) Competition
Designs
LQG
Designs

– Kalman filter with single measurement (4th order)

– Design parameters
• Control cost function weights
• Springs and masses in ideal model
• Estimator weights
– Search
• Multivariate line search
• Genetic algorithm
 Estimation of Minimum Design Cost
Using Jackknife/Bootstrapping
Evaluation
Syllabus - 1
Week Tuesday Thursday
==== ======= ========

1 Overview and Preliminaries Minimization of Static Cost Functions

2 Principles for Optimal Control Principles for Optimal Control

of Dynamic Systems Part 2
[Assignment #1 due]

3 Numerical Optimization of Neighboring-Optimal Control

Dynamic Systems via Linear-Quadratic (LQ) Feedback

4 Dynamic System Stability Linear-Quadratic Regulators

5 Cost Functions and Linear-Quadratic Control System

Controller Structures Design

6 Modal Properties of LQ Systems Spectral Properties of LQ Systems

[Assignment #2 due]

MID-TERM BREAK

Syllabus - 2
Week Tuesday Thursday
==== ======= ========

7 Minimum-Time and Minimum-Fuel Singular-Value Analysis of LQ

Trajectory Optimization Systems
[Assignment #3 due]

8 Probability and Statistics Least-Squares Estimation for

Static Systems

9 Propagation of Uncertainty State Estimation for Discrete-Time

in Dynamic Systems Systems (Kalman filter)
[Assignment #4 due]

10 State Estimation for Continuous-Time Nonlinear State Estimation

Systems (Kalman-Bucy filter) (Extended Kalman filter)

11 Nonlinear State Estimation Adaptive State Estimation

(Quasi-Linear and Sigma-Point Filters) [Assignment #5 due]

12 Stochastic Optimal Control Linear-Quadratic-Gaussian Control

READING PERIOD [Final Paper due on "Dean's Date"]