Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Foundations of Cyber Physical Systems

Soumyajit Dey, Associate Professor,

CSE, IIT Kharagpur

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

CPS Features and Concept Map

▶ Continuous
Dynamics and
Feedback
▶ Compute/Sense/
Communicate in Real
time, calibrate its
effect on Control
▶ Combine AI and
Optimization based
classical techniques
for control
TECHNO
OF LO
TE

▶ Assure safety through

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

Formal Methods yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

CPS Examples

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Course Organization
Topic Week Hours
CPS : Motivational examples and compute platforms 1 2.5
Real time sensing and communication for CPS 2 2.5
Real time task scheduling for CPS 3 2.5
Dynamical system modeling, stability, controller design 4 2.5
Delay-aware Design; Platform effect on Stability/Performance 5 2.5
Hybrid Automata based modeling of CPS 6 2.5
Reachability analysis 7 2.5
Lyapunov Stability, Barrier Functions 8 2.5
Quadratic Program based safe Controller Design 9 2.5
Neural Network (NN) Based controllers in CPS 10 2.5
State Estimation using Kalman Filters (KF) 11 2.5 TE
OF
TECHNO
LO

GY
ITU
IAN INST

KH
ARAGPUR
Attack Detection and Mitigation in CPS 12 2.5

IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Section 1

Introduction to Optimal Controller Design

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Optimal Control
Suppose a dynamical system evolve following
ẋ = f (x, u, t)where x ∈ Rn , u ∈ Rm are states and control inputs for the system

Optimal Control Problem1

Considering a control performance index
RT
J(x(t0 ), u, t0 ) = [L(x(s), u(s), s)ds + V (x(T ), T )], the optimal control problem is to
t0
find the optimal control input u ∗ (t)∀t ∈ [t0 , T ] such that J(x(t0 ), u, t0 ) is minimized,
i.e.
u ∗ (t) = arg min J = arg V (x(t), t)
u(t),t∈[t0 ,T ] u(t),t∈[t0 ,T ] TE
OF
TECHNO
LO

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

1
Ref: Optimal Control, B.D.O. Anderson, J.B. Moore, Prentice Hall
yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Optimal Control : Points to Note

▶ Notice that optimal cost V is independent of the initial control input since the
knowledge of the initial state and time horizon determines the u ∗ (t).
▶ Whatever the initial state and the initial control decision are, the remaining
decisions must be optimal w.r.t. the states resulting from the initial state control
decision to achieve this optimal control policy.
▶ We need to study the evaluation of V for all initial states in the system domain in
order to find such a policy.
▶ A positive loss function and a dynamic programming approach helps in finding this
optimal input from the input space in order to minimize the cost.
▶ LQR is such an optimal control problem with a quadratic loss function for the
linear dynamical systems.
TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Section 2

Linear Quadratic Regulator

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Linear Quadratic Regulator (LQR) : The System

Consider a system:
dx = Axdt + Budt + dvc
Here A, B may be time-varying matrices and vc has mean value of zero. The above
model can be sampled. The input u(t) is constant over the sampling period.
x(t) = Φ(t.kh) + Γ u(kh) + v (kh) y (kh) = Cx(kh) + e(kh)
Here v is process noise and e measurement noise. Both v and e are discrete-time
Gaussian white noise processes with zero mean value and
Zt
d
Φ(t, kh) = A(t)Φ(t, kh), Γ (t, kh) = Φ(t, s)B(s)ds
dt
kh
Zh TE
OF
TECHNO
LO

GY
ITU
IAN INST

KH
ARAGPUR
T AΓ T T
E [v (kh)v (kh)] = R1 = e R1C , E [v (kh)e (kh)] = R12 , E [e(kh)e (kh)] = R2

IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems 0 Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

LQR : The Criteria2

▶ The state x(t) needs to be kept it stable, i.e. keep it around its equilibrium point
of current operating region
▶ For this, control action u(t) is required and it should not be drastically changing
and bounded.
▶ Also the state should not be changing abnormally.
A way to measure the magnitudes of state is to measure its energy

ZNh ZNh
2
J = |x(t)| dt = = x T (t)x(t)dt
0 0
TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
2
Ref: Computer-Controlled Systems:Theory and Design, Karl J. Astrom, Bjorn Wittenmark,

IND
 

19 5 1

Tshinghua University Press, Prentice Hall

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

LQR : The Loss Function

▶ If state vector consists of multiple states, then their contribution to the power
would be different. Hence we need a generic weighting matrix for x, i.e. Q1C
(symmetric, positive definite)
h ZNh i
J=E x T (t)Q1C x(t)dt
0
▶ We also consider the control input in our loss function, i.e.
h ZNh i
J =E (x T (t)Q1C x(t) + 2x T (t)Q12C u(t) + u T (t)Q2C u(t))dt + x T (Nh)Q0C x(Nh)
0

h ZNh  
x(t) i

Q1C Q12C
 TE
OF
TECHNO
LO

GY
ITU
IAN INST

KH
T T

ARAGPUR
=E [x (t) u (t)]Qc , Qc = (all are symmetric)

IND
T
 

u(t) Q12C Q2C 19 5 1

yog, kms kOflm^

0
Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

LQR : The Objective

▶ The objective is to design proper optimal control signals that minimize this loss
function .
h ZNh  
x(t) i
T T
J=E [x (t) u (t)]Qc
u(t)
0

x(t)
▶ Notice that, J is a parabolic function w.r.t .
u(t)
▶ Hence there will be a global minima which we intend to find as part of this optimal
control problem.
TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Sampling the Loss Function

Assuming periodic sampling with h sampling period, following is the discrete version of
the continuous-time loss function
h N−1
X  
x(Nh) i
T T
J =E J(k) + [x (Nh) u (Nh)]Q0C
u(Nh)
k=0
(k+1)h
Z
where, J(k) = (x T (t)Q1C x(t) + 2x T (t)Q12C u(t) + u T (t)Q2C u(t))dt
kh
= (x T (kh)Q1 x(kh) + 2x T (kh)Q12 u(kh) + u T (kh)Q2 u(kh))
(k+1)h
Z (k+1)h
Z
T
where Q1 = Φ (s, kh)Q1C Φ(s, kh)ds, Q12 = ΦT (s, kh)(Q1C Γ (s, kh) + Q12C )ds TE
OF
TECHNO
LO

GY
ITU
IAN INST

KH
ARAGPUR
IND
kh kh 

19 5 1


yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Sampling the Loss Function

(k+1)h
Z
Q2 = (Γ T (s, kh)Q1C Γ (s, kh) + 2Γ T (s, kh)Q12C + Q2C )ds
kh
Since u(t) is constant over t ∈ [kh, (k + 1)h)
h N−1
X 
x(kh)
  
x(Nh) i
T T T T
J=E [x (kh) u (kh)]Q + [x (Nh) u (Nh)]Q0
u(kh) u(Nh)
k=0
 
Q1 Q12
where, Q = T
Q12 Q2
TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

LQR for Deterministic System

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

LQR for Deterministic System

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

LQR for Deterministic System

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

LQR for Deterministic System

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Section 3

Optimal Safe Controller Design

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Safety as Invariance
Consider a harmonic oscillator with
continuous-time
  dynamical system equation
0 −1
ẋ = x
1 0
• The domain: D = (x1 ≥ 0) ∧ (x2 ≥ 0) ∈ R2 ,
• The initial state-space:
X0 = (1 ≤ x1 ≤ 2) ∧ (x2 = 0),
• The unsafe region:
U = (x1 ≤ 2) ∧ (x2 ≤ 2) ∧ (x1 + x2 ≥ 3).
• Notice that to guarantee the system safety
the system design must have the safe set
S = D \ U as invariant TE
OF
TECHNO
LO

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Safety as Invariance
Consider the following Barrier certificate
candidate for this continuous-time
dynamical system
• The barrier function :
B(x1 , x2 ) = x12 + x22 − 4.3. Notice that,
▶ The zero level set of B separates safe
and unsafe operating regions of this
system.
▶ Also, the function is decreasing inside
the domain (and in initial region)
▶ Ḃ ≤ 0 ⇒ S is invariant i.e. the system TECHNO
OF LO
TE

is safe

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Safe Control Design

For a generalized controlled version of such a dynamical system

ẋ = f (x) + g (x)u, here, f , g are globally Lipschitz, and u ∈ Cu ⊂ Rm

For this system, an existance of barrier certificate B(x, u) would mean,

∃u ∈ Cu s.t. B(x, u) < 0 and ∀Ḃ(x, u) ≤ 0 ⇐⇒ S is invariant

Hence, a proper choice of B can make this control strategy

▶ a necessary and sufficient condition for safety invariance (i.e. the sublevel set of
B = 0 is safe and invariant),
▶ and smoothly deters the system trajectory to become unsafe i.e. (discourage
B(x, u) ∀x ∈ D from becoming positive) 3 TE
OF
TECHNO
LO

GY
ITU
IAN INST

KH
ARAGPUR
3
Ref: Control barrier functions: Theory and applications, Ames, Aaron D. Ames, Samuel Coogan,

IND
 

19 5 1

Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, Paulo Tabuada, ECC 2019
yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Performance-aware Control Design

The above strategy ensures the safety invariance of the system but not its performance.
Control Lyapunov Function (CLF)
Let, V (x) : Rn → R be a continuously differentiable function with bounded sublevel
sets4 for any c > 0 such that V (xe ) = 0, v (x) > 0 ∀x ∈ D \ xe and for a γ > 0
∂V (x,u)
∂x f (x) + ∂V∂x
(x,u)
g (x)u ≤ −γV (x)∀x ∈ D, u ∈ Cu .
Then V is a control lyapunov function (CLF) around the equilibrium point xe for the
given dynamical system .

• A control strategy based on CLF that ensures exponential stability with γ decay rate is

∂V (x, u) ∂V (x, u)
Kclf (x) = {u ∈ Cu : f (x) + g (x)u ≤ −γV (x, u)}
∂x ∂x TE
OF
TECHNO
LO

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

4
set of x s.t. V (x) ≤ c
yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Understanding CLF

Making this decay stricter by bounding the

decay of CLF with −γV i.e. V̇ ≤ −γV to
imply exponential stability in place of
asymptotic stability

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Modifying The Safe Control Strategy

Choosing a B s.t. Ḃ ≤ −λB, λ > 0

(i) allows B to increase as long as B < 0
(ii) no further B ↑ as B → 0 as then Ḃ → 0
(iii) if B > 0, Ḃ < 0 ⇒ B ↓ a

a
Note that −λB bounds the positive slope of B in the sublevel
set of B(0) and it gradually reduces as B approaches 0 TE
OF
TECHNO
LO

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Modifying The Safe Control Strategy

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Control Barrier Function (CBF)

CBF
Let S ⊆ D ⊆ Rn be the sublevel set5 of a continuously differentiable function
B : Rn → R for x = 0 then B is a control barrier function (CBF) if there exists a λ > 0
s.t. for the controlled dynamical system
∃u ∈ Cu s.t. ∂B ∂B
∂x f (x) + ∂x g (x)u ≤ −λB(x) ∀x ∈ D

• The safe control strategy based on CBF is

∂B ∂B
Kcbf (x) = {u ∈ Cu : f (x) + g (x)u ≤ −λB(x)}
∂x ∂x
• It ensures B(x, u) ≤ 0 inside S, lim B(x, u) = 0 (∂S denotes boundary of S) with a
x→∂S TE
OF
TECHNO
LO

λ decay rate, and B > 0 outside S

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

5
{x s.t. B(x) ≤ 0}
yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Combining CLF and CBF for Optimal Control

▶ We have already seen formulation of quadratic cost function in LQR Similar way
we formulate a cost function uHu T
▶ The safe and performance-aware control input

u ∗ = arg min (uHu T + pδ 2 )

u∈Cu ,δ>0

subject to the constraints:

∂B ∂B
f (x) + g (x)u ≤ −λB(x)..enforcing safety with CBF
∂x ∂x
∂V ∂V
f (x) + g (x)u ≤ −γV (x) + δ ..enforcing performance with CLF
∂x ∂x
TECHNO

Note: Here δ is a slack variable introduced for relaxing the constraints. No such
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

relaxation for safety constraints corresponding to CBF. 19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Combining CLF and CBF for Optimal Control

▶ Notice that, the objective is non-linear, more specifically quadratic

▶ Like the system dynamics, the constraints are also control-affine, i.e. linear w.r.t.
the control input
▶ So the problem can be solved as quadratic programming (QP)-type non-linear
optimization problem
▶ The derived optimal control input is (i) safe as enforced by the safety invariance of
CBF and (ii) performance-aware as ensured by the CLF.

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Example : Harmonic Oscillator

Consider a harmonic oscillator with
continuous-time
  dynamical
  system equation
0 −1 0
ẋ = x+ u
1 0 1
• The domain:
D = (1 < x1 < 3) ∧ (−1 < x2 < 1) ∈ R2 ,
• The unsafe region:
U = {x ∈ D : (1−(x1 −2)2 )−1 +(1−x22 )−1 < 4}
a
▶ Consider CLF V = x12 + x1 x2 + x22 and a
Ref: "Uniting Control Lyapunov and Control Barrier
2 )−1 −(1−x 2 )−1
▶ CBF B = e −(1−(x1 −2) 2 − e −4 Functions" by Muhammad Zakiyullah Romdlony, Bayu
Jayawardhana TE
OF
TECHNO
LO

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Example : ACC
Consider an an adaptive cruise control system with
continuous-time
    dynamical
  system equation
Ḋ v 0
= + u, m = 1650kg , g = 9.81m/s 2
v̇ − Fmr 1/m
| {z } | {z }
f (x) g (x)
u = wheel force Fw ∈ [−0.3mg , 0.3mg ], and rolling
resistence Fr = 0.1 + 5v + 0.25v 2 .
D2
▶ The unsafe region: U = {x ∈ D : D − 0.6g < 1.8v }
a
▶ Consider CLF V = (v − vd )2 desired speed
vd = 24m/s and a
Ref: "Control Barrier Function based
Quadratic Programs with Application
(14−v )2
▶ CBF B = + 1.8v − D toAdaptive Cruise Control" by Aaron D. Ames, TE
OF
TECHNO
LO

GY
ITU
6.54 Jessy W. Grizzle and Paulo Tabuada

IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur
Introduction to Optimal Controller Design Linear Quadratic Regulator Optimal Safe Controller Design

Example : ACC

TECHNO
OF LO
TE

GY
ITU
IAN INST

KH
ARAGPUR
IND
 

19 5 1

yog, kms kOflm^

Foundations of Cyber Physical Systems Soumyajit Dey, Associate Professor, CSE, IIT Kharagpur