Deterministic Epidemic Models and Optimal

Control

Lawi George

Department of Mathematics
Masinde Muliro University of Science and Technology

March 9, 2023

Lawi G. Basics of Optimal Control

Contents

1 Introduction

2 Linear Continuous models

3 Feedback Particle filter

4 Application

5 Conclusion

Lawi G. Basics of Optimal Control

History
Optimal Control Theory is a recent branch of Mathematics
that can also be seen as an extension of the Calculus of
Variations.
Its origins is associated with the publication of the solution of
the brachistochrone problem by Johann Bernoulli
The word brachistochrone comes from the Ancient Greek
word which means the “shortest time”. The main goal of such
problem is to find the curve between two points, on a vertical
plane, that a sphere without friction covers in the shortest
time
In 1696, Johann Bernoulli challenged the best mathematicians
of his time to solve this problem. Consequently, Johann
Bernoulli himself discovered the solution, as well as his brother
Jacob Bernoulli, Newton, Leibniz and Guillaume François
Antoine - Marquis de l’Hôpital (1661-1704). The solution is a
cycloid arc starting with a vertical tangent.
Lawi G. Basics of Optimal Control
History

Lawi G. Basics of Optimal Control

Basics

Dynamics:
We open our discussion by considering an ordinary differential
equation (ODE) having the form

dx
= f(x(t)), (t > 0)
dt
x(o) = x o

We are here given the initial point x o ∈ Rn and the function

f : Rn → Rn . The unknown is the curve x : [0, ∞) → Rn , which we
interpret as the dynamical evolution of the state of some “system”.

Lawi G. Basics of Optimal Control

Basics

Controlled Dynamics:
We generalize a bit and suppose now that f depends also upon
some “control” parameters belonging to a set A ⊂ Rm ; so that
f : Rn xA → Rn . Then if we select some value a ∈ A and consider
the corresponding dynamics:

dx
= f(x(t), a), (t > 0)
dt
x(o) = x o

we obtain the evolution of our system when the parameter is

constantly set to the value a.
The next possibility is that we change the value of the parameter
as the system evolves. For instance, suppose we define the
function α(t) : [0, ∞) → Rn

Lawi G. Basics of Optimal Control

Basics


a1 , 0 ≤ t ≤ t1

u(t) = a2 t1 ≤ t ≤ t2 (3)

a3 t2 ≤ t ≤ t3 etc


for times 0 < t1 < t2 < t3 ... and parameter values a1 , a2 , a3 , ∈ A

and we then solve the dynamical equation
dx
= f(x(t), u(t)), (t > 0)
dt
x(o) = x o

More generally, we call a function u : [0, ∞) → Rn a control and

the trajectory x the response of the system.

Lawi G. Basics of Optimal Control

Basics

The system may behave differently as the control is varied. We

introduce A = {u : [0, ∞) → A u(.) measurable} to denote the
class of admissible controls such that
u(t) = (u 1 (t), u 2 (t), u 3 (t), ........u m (t))T
Payoffs:
Our overall task will be to determine what is the “best” control for
our system. For this we need to specify a specific payoff (or
reward) criterion. Let us define the payoff functional
Z T
P[u(.)] = r (x(t), u(t))dt + g (x(T ))
o

where x(.) solves the ODE for the control u(.). Here
r : Rn xA → R and g : Rn → R are given and r is called the
running payoff while g is called the terminal payoff. The terminal
time T > 0 is given as well.

Lawi G. Basics of Optimal Control

The Basic Problem
The aim is to determine a control u ∗ (.) which maximizes the
payoff i.e P[u ∗ (.)] ≥ P[u(.)] for all controls u(.) ∈ A. Such a
control u ∗ (.) is called optimal.
Motivational Example: Suppose we own, say, a factory whose
output we can control. Let us begin to construct a mathematical
model by setting x(t)= amount of output produced at time t ≥ 0
We suppose that we consume some fraction of our output at each
time, and likewise can reinvest the remaining fraction. Let us
denote u(t) = fraction of output reinvested at time t ≥ 0.

Lawi G. Basics of Optimal Control

Basics
This will be our control, and is subject to the obvious constraint
that 0 ≤ u(t) ≤ 1 for each time t ≥ 0
Given such a control, the Optimal control problem is to solve
Z T
P[u(.)] = (1 − u(t))x(t)dt
o
.
subject to x = κu(t)x(t),
x(0) = x o
The meaning here is that we wish to maximize our total
consumption of the output, the consumption at any time t being
(1 − u(t))x(t)

dx
= κu(t)x(t)
dt
x(o) = x o

Lawi G. Basics of Optimal Control

The Necessary Conditions
In our basic optimal control problem for ordinary differential
equations, we use u(t) for the control and x(t) for the state. The
state variable satisfies the differential equation which depends on
the control variable:
dx
= g (t, x(t), u(t))
dt
Our basic optimal control problem consists of finding a piecewise
continuous control u(t) and the associated state variable x(t) to
maximize the given objective functional, i.e.,
Z t1
max f (t, x(t), u(t))dt
u(.) to
dx
subject to = g (t, x(t), u(t)),
dt
x(to ) = xo , and x(t1 )free

Lawi G. Basics of Optimal Control

The principle technique for such an optimal control problem is to
solve a set of “necessary conditions” that an optimal control and
corresponding state must satisfy. To derive the necessary
conditions we express our objective functional in terms of the
control: Z t1
J(u) = f (t, x(t), u(t))dt
to

where x = x(u) is the corresponding state(emphasis on the

dependence of x on the control u). The necessary conditions that
we derive were developed by Pontryagin and his co-workers in
Moscow in the 1950’s .
Pontryagin introduced the idea of “adjoint” functions to append
the differential equation to the objective functional.
Adjoint functions have a similar purpose as Lagrange multipliers in
multivariate calculus, which append constraints to the function of
several variables to be maximized or minimized.

Lawi G. Basics of Optimal Control

Thus, we begin by finding appropriate conditions that the adjoint
function should satisfy.
Then, by differentiating the map from the control to the objective
functional, we will derive a characterization of the optimal control
in terms of the optimal state and corresponding adjoint

Lawi G. Basics of Optimal Control

Probabilistic/Bayesian framework

In this framework, the probability law for each problem is

established by means of conditional probability density functions.
Of interest, in this case, are the following:
The evolution of conditional probability density, dp(xt | Yt ).
The evolution of the moments—from which we can obtain the
mean and the covariances.
These have already been obtained for the system (1).

Lawi G. Basics of Optimal Control

Probabilistic/Bayesian framework

The system (1) can be rewritten as follows.

Signal: dxt /dt = f (xt , θ) + G (xt )vt ; t0 ≤ t, (7a)

Measurement: dyt /dt = h(xt ) + wt , t0 ≤ t, (7b)

where
{vt , t > t0 } white Gaussian noise process m×1
yt output vector r ×1
{wt , t > t0 } white Gaussian noise process r ×1

vt ∼ N (0, Q(t)) and wt ∼ N (0, R(t))

Lawi G. Basics of Optimal Control

Conditional density

Theorem 1.1: Evolution of conditional density

For the system (2), the conditional density evolves as follows.

dp = L(p)dt + (h − ĥ)T R −1 (t)(dyt − ĥdt)p, (8)

where
n n
X ∂(pfj ) 1 X ∂ 2 [p(GQG T )ji ]
L(p) = − + .
∂xi 2 ∂xj ∂xi
j=1 j,i=1

Equation (3) is called Kushner’s equation. In the absence of

observations, that is, when R = 0, we obtain the so-called
Kolmogorov’s forward equation—which is also known as the
Fokker-Plank equation

Lawi G. Basics of Optimal Control

Moments

Lemma 1.1: Evolution of moments

For the system (2), the moments, ψ(xt | Yt ), under the con-
ditional density, p, evolve as follows.

T
1 T
d ψb = ψd
x f dt + tr(GQG ψxx )dt+
2 (9)
c − ψ̂ ĥ)T R −1 (t)(dyt − ĥdt),
(ψh

From (4), we can obtain the mean, µt , and the covariance,

Pt , which correspond to the first and second moments,
respectively.
The mean, µt , and the covariance, Pt , provide sufficient
statistics for a Gaussian process.

Lawi G. Basics of Optimal Control

Mean and Covariance

Theorem 1.2: Evolution of the mean and covariance

For the system (2), the mean mean, x̂t , and the covariance,
Pt , satisfy the following equations.

−1
d x̂t = fˆdt + (xd T T
t h − x̂t ĥ )R (t)(dyt − ĥdt), (10)
ˆ ˆ
i fj − x̂i fj )dt + (fi xj − fi x̂j )dt+
(dPt )ij = (xc c
T −1 c
(GQG T )ij dt − (xc i h − x̂i ĥ) R (hxj − ĥx̂j )dt+
(11)
i xj h − x
(x[ i xj ĥ − x̂i xj h − x̂j xi h+
d c c
2x̂i x̂j ĥ)T R −1 (t)(dyt − ĥdt),

Lawi G. Basics of Optimal Control

Mean and Covariance: Continuous-discrete case

Theorem 1.3: Evolution of the mean and covariance

For the system (2) with discrete observations, the mean mean,
x̂t , and the covariance, Pt , satisfy the following equations.
Prediction step

d x̂t = fˆ(xt )dt, tn ≤ t ≤ tn+1 , (12a)

T ˆT T ˆ T
dPt = (E {xt f } − x̂t f )dt + (E {fxt } − f x̂t )dt+
(12b)
E {GQG T }dt,

Update step

E {xtn p(ytn | xtn )}

x̂tn = (13a)
E {p(ytn | xtn )}
E {xtn xtTn p(ytn | xtn )}
Ptn = − x̂tn x̂tTn . (13b)
E {p(ytn | xtn )}

Lawi G. Basics of Optimal Control

Example: A nonlinear scalar SDE 1/2
Example 1: Nonlinear scalar SDE

Consider the following linear Gaussian Itô state space model.

dxt = axt dt + bxt2 dt; t0 ≤ t, (14a)

yt = xt + xt3 +R 1/2
wt ; t0 ≤ t. (14b)

where {wt } is a Gaussian white noise process with E{wt wtT } =

1. Let the xt at time, t0 be xt0 ∼ N (0.4, 0.001). Let, moreover,
xt0 and {wt , t ≥ t0 } be uncorrelated. The analytic solution for
(7a) is:
axt0 e a(t−t0 )
xt = (15)
a + bxt0 (1 − e a(t−t0 ) )

Take a = −0.2, b = 0.2 and R = 0.0001. Estimate xt .

Lawi G. Basics of Optimal Control

Example: A nonlinear scalar SDE 2/2

Figure: A plot of the analytic solution, numerical approximation, filter

estimate and the measurements of the model in Example (1)

Lawi G. Basics of Optimal Control

Linear Continuous Models
The linear version of the system (1) is obtained by replacing
f (xt , θ) with F (t, θ)xt , G (xt ) with G (t) and h(xt ) with
H(t)xt —which leads to:

Signal: dxt = F (t, θ)xt dt + G (t)dβt ; t0 ≤ t, (16a)

Measurement: dyt = H(t)xt dt + dηt ; t0 ≤ t, (16b)

in which
xt state vector n×1
F (t, θ) continuous time-function matrix n×n
θ vector of parameters d ×1
G (t) continuous time-function matrix n×m
{βt , t > t0 } Brownian motion process m×1
yt output vector r ×1
H(t) continuous time-function matrix r ×n
{ηt , t > t0 } Brownian motion process r ×1

Lawi G. Basics of Optimal Control

Mean and Covariance: Linear Models

Theorem 2.1: Evolution of the mean and covariance

For the system (7), the mean, x̂t , and the covariance, Pt ,
satisfy the following equations.

d x̂t = F x̂t dt + Pt H T R −1 (t)(dyt − H x̂t dt), (17a)

dPt = FPt dt + Pt F T dt + GQG T dt − Pt H T R −1 HPt dt.
(17b)

The system (8) is the minimum variance (optimal) filter for

the time-continuous system (7). It is known as Kalman-Bucy
filter.

Lawi G. Basics of Optimal Control

Continuous Smoother: Linear Models

Theorem 2.2: Continuous Linear Smoother

For the system (7), the smoothed estimate, x̂ts , and the
smoothed covariance, Pts , satisfy the following equations.

d x̂ts = F x̂ts dt + GQG T Pt−1 (x̂ts dt − x̂t dt), (18a)

dPts /dt = (F + GQG T Pt−1 )Pts + Pts (F T + Pt−1 GQG T ) − GQG T
(18b)

The system (9) is the Rauch-Tung-Striebel Smoother.

Beginning with the filter estimates of the mean, x̂t , and the
covariance, Pt , at time t the equations (9a-9b) are
integrated backwards in time to obtain smoothed estimates of
mean, x̂ts , and covariance, Pts .

Lawi G. Basics of Optimal Control

Ensemble Kalman-Bucy Filter
Suppose, instead of having a single hypothesis of the state, xt , we
formulate say N hypotheses, Xt := {xti }N i=1 . The EnKB filter for
the linear system (7) is of the following form.
Deterministic: (Bergemann and Reich, 2012; de Wiljes, Reich
and Stannart 2016)

dxti = Fxti dt + G (t)dβti + Pt H T R −1 (t)(dyt − 0.5H(xti + X̂t )dt).

(19)
Stochastic: (Law, Stuart and Zygalakis, 2015; Reich, 2011)

dxti = Fxti dt + G (t)dβti + Pt H T R −1 (t)(dyt − Hxti dt + dwt ).

(20)
The mean, x̂t , and the covariance, Pt , at time t are obtained
empirically as shown.
N N
1 X i 1 X i
x̂t = xt , Pt = (xt − X̂t )(xti − X̂t )T . (21)
N N −1
i=1 i=1

Lawi G. Basics of Optimal Control

Feedback Particle filter

Much like in the deterministic variant of the EnKB filter, the

FPF—for the nonlinear system (1)— has a batch of hypotheses of
the state, Xt := {xti }N
i=1 , which are propagated using the
Statonovich SDE below.

dxti = f (xti , θ)dt + G (xti )dβti + K (xti )(dyt − 0.5H(xti + X̂t )dt). (22)

where K , for the linear SDE, is the Kalman gain.

Lawi G. Basics of Optimal Control

Application: A scalar SDE
Example 1: Scalar SDE

Consider the following linear Gaussian Itô state space model.

dxt = (axt + b)dt + Q 1/2 dvt ; t0 ≤ t, (23a)

1/2
dyt = cxt dt + R dwt ; t0 ≤ t. (23b)

where {vt } and {wt } are Brownian motion processes with,

respectively, E{dvt dvtT } = dt and E{dwt dwtT } = dt. Let
the xt at time, t0 be xt0 ∼ N (0, 0.001). Let, moreover, xt0 ,
{vt , t ≥ t0 } and {vt , t ≥ t0 } be uncorrelated.
Take a = −0.2, b = 0.2, c = 1.01, Q = 0.001,
R = 0.0001. Estimate xt .
Pretend that you do not know a and b. Estimate a and
b.

Lawi G. Basics of Optimal Control

Conclusion

The (linear and non-linear) filtering has been introduced.

An overview of the Feedback Particle filter has been presented.
An application to a scalar SDE has been made both for state
and parameter estimation.
The FPF gives—in the scalar SDE considered—a converging
estimate for the parameters.

Lawi G. Basics of Optimal Control

References
1 Amirhossein T., de Wiljes J., Prashant G., Reich S., (2016)
Kalman Filter and its Modern Extensions for the
Continuous-time Nonlinear Filtering Problem,
arXiv:1702.07241v2[math.OC].
2 Bergemann., Reich S., (2012), An Ensemble Kalman-Bucy
Filter for Continuous Data Assimilation, Meteorolog.
Zeitschrift, vol. 21, pp. 213-219.
3 Jazwinski A., (2007), Stochastic Processes and Filtering
Theory, Academic Press, Inc., New York, Dover edition.
4 Kalman R., Bucy R., (1961), New Result in Linear Filtering
and Prediction Theory, Journal of Basic Engineering.
5 Reich S. and Cotter C., (2015), Probabilistic Forecasting and
Bayesian Data Assimilation, Cambridge University Press.
6 Wiljes J., Reich S., Stannat W., (2016), Long-time Stability
and Accuracy of the Ensemble Kalman-Bucy Filter for Fully
Observed Processes and Small Measurement Noise,
arXiv:1612.06065v1.
Lawi G. Basics of Optimal Control