2011 50th IEEE Conference on Decision and Control and

European Control Conference (CDC-ECC)

Orlando, FL, USA, December 12-15, 2011

On infinite-horizon probabilistic properties

and stochastic bisimulation functions
Ilya Tkachev and Alessandro Abate

Abstract— This work investigates infinite-horizon properties of finding a stochastic bisimulation function (SBF) of two
over discrete-time stochastic models with continuous state processes [9].
spaces. The focus is on understanding how the structural
features of a model (e.g., the presence of absorbing sets) affect
The first problem (Section III) is elaborated by looking
the values of these properties and relate to their uniqueness. at the presence of absorbing sets [11] over regions of
Furthermore, we argue that the investigation of these features the continuous state space. These sets turn out to play
can lead to approximation bounds for the value of such a critical role, both formally and computationally. As for
properties, as well as to improvements on their computation. the second problem, in Section IV we propose a lower
The article employs the presented results to find a stochastic
bisimulation function of two processes.
bound for infinite-time probabilistic invariance by expressing
this property via a finite-time reach-avoid plus an infinite-
I. I NTRODUCTION time invariance around absorbing sets on the state space.
An example is proposed in Section IV-D to elucidate the
The analysis of general state-space stochastic models has
results. With regards to the third problem (seeking a SBF
recently witnessed substantial efforts towards the use of
of two given processes), we put forward in Section V an
formal verification techniques [2], [7]. Recent work has
approach that is distinguished from [9] in that we do not
drawn formal connections between the solution of dynamic
raise assumptions (e.g. linearity, stability) on the dynamics
programming problems, which constitutes the underlying
of the two processes under study – a computational study is
computation engine for a number of studies (such as (prob-
developed in Section V-A to help explain the results. Proofs
abilistic) reachability, invariance, and reach-avoid), and the
are omitted from this manuscript due to space constraints.
verification of related specifications in PCTL logic [3], [13]
(until properties). Along similar lines, control synthesis prob-
lems have been tackled with approaches that can be related to II. P RELIMINARIES
probabilistic model checking [4]. These research efforts have
focused on models as general as stochastic hybrid systems, Consider a discrete-time homogeneous Markov process
namely hybrid models with stochastic continuous dynamics X on the Borel space X together with a Borel σ -
as well as probabilistic jumps and resets. algebra B(X ) (see e.g. [8]). We define the stochastic basis
There are two main approaches to study classes of infinite- (Ω, F , F, P), where F = {Fn }n≥0 and Fn = {Xk , 0 ≤ k ≤ n}
time properties for general state-space processes: 1) as the [6]. For each distribution of the initial state denoted as π,
limit of the iteration of certain operators, or 2) as the we define the probability measure Pπ such that
solution of integral (Bellman-like) equations. Understanding
the properties and the computations related to the solution of Pπ {X0 ∈ A} = π(A),
infinite-horizon properties [1] has lead to a few early results
[13], such as lack of uniqueness or triviality of solutions. for all A ∈ B(X ). An expectation with respect to this
This work looks at discrete-time, general state-space mod- measure is denoted as Eπ . Similarly, we denote by Px (·) =
els (its results can be for instance applied to stochastic P(·|X0 = x) and the corresponding expectation by Ex . The
hybrid systems – dtSHS – as in [2], [4]) and has three main transition kernel associated to the process X is a measure on
objectives. First, it seeks to shed new light over the existence (X , B(X )) for each x ∈ X given by T (A|x) = Px {X1 ∈ A}.
of unique or trivial solutions of infinite-time properties such For a measurable function f : X → R let us put f − (x) =
as probabilistic invariance and reachability. Second, it puts max{0, − f (x)}. We say that f ∈ L if Ex f − (X1 ) < ∞ for any
forward a technique to ease the computation of infinite-time x ∈ X . For f ∈ L a transition operator P and an excessive
invariance, by hierarchically breaking down this property into operator Q are defined as follows:
two subproblems and by approximating their solution. Third,
it exhibits the application of the above results to the problem P f (x) = Ex [ f (X1 )],
(1)
Q f (x) = max{ f (x), P f (x)}.
This work is supported by the European Commission under the
MoVeS project FP7-ICT-2009-257005, by the European Commission under
Marie Curie grant MANTRAS 249295, and by NWO under VENI grant With C (B) we define the class of real-valued functions
016.103.020.
The authors are with the Delft Center for Systems & Control, Technische that are continuous and bounded on B ∈ B(X ) and for a
Universiteit Delft - Delft University of Technology, The Netherlands. measurable function g : X → R and δ ∈ R we define the
Emails: {i.tkachev,a.abate}@tudelft.nl superlevel set Xg (δ ) = {x ∈ X : g(x) > δ }.

978-1-61284-799-3/11/$26.00 ©2011 IEEE 526

 III. P ROBABILISTIC REACHABILITY AND INVARIANCE On the other hand we know that (see e.g. [14, Th.1, p.28])
A. Reachability as an optimal stopping problem
vn (x; Ac ) = max{IAc (x), Pvn−1 (x; Ac )},
On the set Ω we define the following events for any N ≥ 0
and A ∈ B(X ): where v0 (x; Ac ) = IAc (x) and hence
rN (A) = {∃n ∈ [0, N] : Xn (ω) ∈ A}, un (x; A) = min{IA (x), Pun−1 (x; A)} (4)
rA = {∃n ≥ 0 : Xn (ω) ∈ A}.
(compare e.g. [2, Prop.1, p.4]). We can also provide another
We formally show the connection between calculating the way to calculate the functions un : note that, from (4),
probability of these events and solving an optimal stopping (
problem. Let us first define a stopping time [6]. IA (x), if x ∈ Ac ,
un (x; A) =
Definition 1. A random variable τ is called a stopping time Pun−1 (x; A), if x ∈ A,
for the process X if for all n ≥ 0 the event {τ ≤ n} ∈ Fn and hence for all n ≥ 1 we have
P{τ < ∞} = 1. For any subset A ∈ B(X ) we also denote Z
τA = inf{n ≥ 0 : Xn ∈ A} as a first entry time for the set A. un (x; A) = IA (x)Pun−1 (x; A) = IA (x) un−1 (y; A)T (dy|x),
We introduce the following proposition, also discussed in X
[5, Prop.6, p. 6176]. where T is the transition kernel associated to the discrete
Proposition 1. For any N ≥ 0 and A ∈ B(X ) we have time homogenous Markov process X and u0 (x; A) = IA (x).
Let us introduce the operator IA which is defined for
P[rN (A)] = sup E[IA (Xτ )], (2) functions f ∈ L :
τ≤N
Z
where IA (·) : X → {0, 1} is an indicator function of the set IA f (x) = IA (x) f (y)T (dy|x). (5)
A and τ is a stopping time. Moreover, the stopping time X
τ ∗ = τA ∧ N = min{τA , N} is an optimal stopping time in the
optimal stopping problem (2). With regards to the infinite horizon invariance, let us
introduce the event {Xn ∈ A for all n ≥ 0}. It can be easily
Corollary 1. For any A ∈ B(X ), seen that
P[rA ] = lim P[rN (A)]. Px {Xn ∈ A for all n ≥ 0} = 1 − Px [r(Ac )], thus
N→∞

Note that these facts hold over arbitrary probability mea- Px {Xn ∈ A for all n ≥ 0} = u(x; A) := lim un (x; A),
n→∞
sures, in particular over Px , for any x ∈ X .
The characterization of probabilistic reachability as an where the limit exists pointwise for all x ∈ X (cf. [1, Th.3,
optimal stopping problem leads to the following result, which p.263]). We denote v(x; Ac ) = Px [r(Ac )] = lim vn (x; Ac ).
n→∞
indicates a method to compute this quantity and suggests a
different perspective on [4]. C. Trivial solutions of probabilistic invariance

Theorem 1. Let A ∈ B(X ) and n ∈ N. Then Theorem 2. Suppose that for some N we have

Px [rn (A)] = Q n IA (x), uN (x; A) ≤ α < 1

(3)
Px [r(A)] = lim Q IA (x),
n
n→∞ for all x ∈ A, then u(x; A) ≡ 0.
for all x ∈ X , where as in (1), Q f (x) = max{ f (x), P f (x)}. This fact can be exploited in the computation of u(x; A),
and leads us to the question of whether there will be an N
B. Probabilistic invariance – definition
such that supx∈A uN (x; A) < 1. To answer this question we
Probabilistic invariance is dual to the problem of stochastic introduce the notion of an absorbing set [11, Def.IV, p.91].
reachability. It was studied in [2], assuming the process X is
a realization of a dtSHS. For a given set A ∈ B(X ) and a Definition 2. A non-empty set B ∈ B(X ) is called absorb-
time horizon n ∈ N we define corresponding value functions ing if for all x ∈ B the following identity holds

un (x; A) = Px {Xi ∈ A for all 0 ≤ i ≤ n} T (B|x) = 1.

and The absorbing set B ⊆ A is called an absorbing subset of
vn (x; A) = Px [rn (A)]. A. For the set A ∈ B(X ) the set B is the largest absorbing
subset of A if for any absorbing set B0 ⊆ A holds B0 ⊆ B.
Clearly we have the following identity
Definition 3. The transition kernel T is called weakly
un (x; A) = 1 − vn (x; Ac ), continuous [8] if P f ∈ C (X ) whenever f ∈ C (X ). We
where Ac := X \ A. write then T ∈ Cw (X ).

527
D. Invariance and absorbing sets The following theorem provides strict bounds for the
Consider the following family of sets: convergence in (8).

An = {x ∈ A : un (x; A) = 1}. Theorem 4. Suppose that for some N we have

Here we note that A = A0 and A1 ⊆ A0 . Define the following uN (x; A) ≤ α < 1

limit set ∞ for all x ∈ A. Then, for any n,
\
A∞ = An . n
n=0 Nα b N c
w(x; A, B) − wn (x; A, B) ≤ . (9)
Lemma 1. For all n ≥ 0 we have 1−α
Remark 2. To calculate the finite-horizon quantity
An+1 (x) = {x : T (An |x) = 1}. (6)
wn (x; A, B) one can use the discretization technique and
and An+1 ⊆ An . formal bounds provided in [2].
Theorem 3. Let A ∈ B(X ) be a compact set and T ∈ B. Decomposition of the infinite-horizon invariance proba-
Cw (X ). Then bility
u(x; A) = 0
For the reach-avoid problem we derived computational
for all x ∈ X if and only if there are no absorbing subsets bounds for the case when the solution in (8) may be non-
of A. Moreover, if there is an absorbing subset of A then A∞ , trivial. The following theorem shows how to bound invari-
as in Lemma 1, is the largest absorbing subset of A. ance on the infinite horizon by two different problems: using
a reach-avoid value function, and an invariance problem.
Remark 1 (On the computation of (6)). In some verification
problems it is relevant to calculate an absorbing subset Theorem 5. Let T ∈ Cw (X ) and sets A, B ∈ B(X ) be such
of a given set A, or to overapproximate it [3]. Using the that B ⊆ A and A0 \ B is a compact set without absorbing
monotonicity of the procedure in (6) we conclude that if subsets. Then for all x ∈ A0 we have
A0n ⊇ An is an overapproximation of the set An and
u(x; A) ≥ w(x; A, B)ψ(A, B), (10)
A0n+1 ⊇ {x : T (A0n |x) = 1} (7)
where
then A0n+1 ⊇ An+1 . The right-hand side in (7) can be precisely ψ(A, B) = inf u(x; A).
x∈B
overapproximated by means of a state space discretization
algorithm [2] (the requirement on Lipshitz continuity in [2] Note that due to the monotonicity of the sequence
also verifies the continuity assumption in Definition 3). wn (x; A0 , B) we have
IV. R EACH - AVOID PROBLEM u(x; A) ≥ wn (x; A, B)ψ(A, B), (11)
A. Bounds for infinite horizon reach-avoid problem for all n ≥ 0. Thus, the main question is how to find ψ(A, B)
For the sets A, B ∈ B(X ) such that A ∩ B = 0/ we define or how to find bounds for this quantity, which we answer in
the reach-avoid probability as follows the next section. Notice that the definition of ψ(A, B) focuses
w(x; A, B) = Px {∃n > 0 : Xk ∈ A∀k < n, Xn ∈ B or X0 ∈ B}. on points in set B ⊆ A: this directly leads to computational
advantages when calculating this quantity over u(x; A).
The set B is a target (reach) set, the set A is a set of legal
states and the set X \ (A ∪ B) is an illegal (avoid) set. C. Bounds via local Doob’s inequality
To calculate the reach-avoid probability value in w(x; A, B) We leverage the following Doob’s inequality [6], [12]: if
we set up a procedure, as described in [13] for the for some fixed x ∈ X the process g(Xn ) is a non-negative
unbounded-until specification, which is a specification in Px -supermartingale for all n ≤ N, then for all δ > 0 it holds
modal logic that corresponds to the reach-avoid problem. that
Let us define the operator  
1 g(x)
Z Px sup g(Xn ) ≥ δ ≤ sup Ex [g(Xn )] = . (12)
RA,B f (x) = IB (x) + IA (x) f (y)T (dy|x). 0≤n≤N δ 0≤n≤N δ
X Definition 4. A measurable function g ∈ L is said to be
If we denote w0 (x; A, B) = IB (x), then for n ≥ 0 define an excessive function (for a process X or with respect to an
operator P), if
wn+1 (x; A, B) = RA,B wn (x; A, B). Pg(x) ≤ g(x).
This sequence in non-decreasing and bounded from above for all x ∈ X . Let us call the function g excessive in
by 1. Moreover, B ∈ B(X ) if for all x ∈ B it holds that Pg(x) ≤ g(x). The
w(x; A, B) = lim wn (x; A, B), (8) maximal excessive set for the function g is the set
n→∞
where the limit is taken pointwise (see [13, (3.7)]). Eg = {x ∈ X : Pg(x) ≤ g(x)}.

528
Lemma 2. [12, (1.2.40), p. 18] The function g is excessive present next an explicit solution for the infinite horizon
if and only if (g(Xn ))n≥0 is a Px -supermartingale for all invariance problem for a finite space Markov chain.
x∈X. Consider a Markov chain Y with a state space Y =
{yi }Ni=1 . Without loss of generality we put yi = i for 1 ≤ i ≤ N.
Often inequality (12) requires conservative assumptions on
Now, pi j = P(Y1 = j|Y0 = i) form a stochastic matrix P.
function g: indeed, the previous lemma shows that inequality
For any given set A ⊆ Y we consider a vector u, which
(12) is only satisfied for excessive functions g. [10] presents
components
a relaxation of Doob’s inequality for functions that are
excessive only locally (see bottom of Def. 4). ui = P{Yn ∈ A for all n ≥ 0|Y0 = i} (13)
Theorem 6. [10, Theorem 12, p. 71] If g is a measurable denote infinite-horizon invariance probabilities. Vector u has
and non-negative function such that Xgc (δ ) ⊆ Eg for some to be a solution of the following equation
δ > 0 then
u = D(A) · Pu, (14)
 
g(x)
Px sup g(Xn ) ≥ δ ≤ .
n≥0 δ where D(A) is the diagonal matrix such that Dii (A) = 1 if
for all x ∈ Xgc (δ ). and only if i ∈ A. The following holds:

D. Decomposition of probabilistic invariance: an example Proposition 2. Equation (14) has a unique solution u = 0 if
and only if there is no an absorbing subset of A.
This example displays the use of the bounds in Sections
IV-B and IV-C over a linear Gaussian discrete-time system. Let us now look to the instance where the chain admits
Let X = Rm and M is a symmetric positive semidefinite an absorbing subset A0 ⊆ A, and let us also suppose that
matrix. Let us consider the following Markov process X: A0 is the largest absorbing subset of A, so A \ A0 has no
absorbing subsets. Without loss of generality suppose that
Xn+1 − Xn = µ(Xn ) + σ (Xn )ξn , A0 = [1, m] = 1, 2, ..., m and A = [1, n] where m ≤ n ≤ N since
where µ : Rm → Rm , σ : Rm → Rm×k and ξn is a standard k- it is always possible to sort the elements of set Y in order
dimensional Gaussian random vector, ξn0 ⊥ ξn00 for all n0 6= to obtain this form. Clearly
n00 . We select a function g with a quadratic form g(x) =
(
1, for 1 ≤ i ≤ m,
xT Mx. Suppose that for some δ 0 > 0 the following inequality ui = (15)
0, for n + 1 ≤ i ≤ N.
holds, for all x ∈ Xgc (δ 0 ):
We rewrite the equation (14) in the following form
µ T (x)Mµ(x) + 2µ T (x)Mx + Tr(σ T (x)Mσ (x)) ≤ 0.
(IN − D(A) · P)u = Qu = 0,
Then, by application of the inequality in (12), for all δ ≤ δ 0
we have where we denote an identity matrix of dimension N with IN .
Obviously det(Q) = 0, so 1 is an eigenvalue of the matrix
xT Mx xT Mx
v(x; Xg (δ )) ≤ i.e. u(x; Xgc (δ )) ≥ 1 − . D(A) · p. We put (qi j )Ni,j=1 to be elements of the matrix Q.
δ δ As we discussed, only the elements ui for m + 1 ≤ i ≤ n are
We show how this bounds can be improved in the case unknown. We can compute them from the equation Qu = 0:
when M is positive definite, by leveraging the decomposition m n
technique in Section IV-B. We select sets A = Xgc (δ ) and ∑ qi j + ∑ qi j u j = 0 (16)
B = {x ∈ X |g(x) < ε} for ε < δ . By Theorem 6, we obtain j=1 j=m+1

ε for m + 1 ≤ i ≤ n. This equation has a unique solution if the

ψ(A, B) ≥ 1 − .
δ coefficients qi j for m + 1 ≤ i, j ≤ n form a matrix with non-
Furthermore, we can use the inequality in (10), obtaining zero determinant. Indeed, they form the [m + 1, n] × [m + 1, n]
block of the matrix Q, which we denote as Q0 . Let us show
 ε
u(x; A0 ) ≥ w(x; A, B) 1 − . that the determinant is non-zero by contradiction. Assume
δ
the contrary, det(In−m − Q0 ) = 0 which implies that det(IN −
Finally, the term w(x; A, B) can be calculated with any D([m + 1, n]) · P) = 0.
accuracy since A = A0 \ B does not contain the origin, which But this is equivalent to the statement that the equation
is the only absorbing subset of X , thus the conditions of u0 = D([m + 1, n]) · Pu0 has non-unique solution, which is not
Theorem 4 are satisfied. true since A0 is a largest absorbing subset of A and hence,
E. Probabilistic invariance over finite state spaces the interval [m + 1, n] of elements of Y does not contain
absorbing subsets.
In the previous section we have described how to find In conclusion, we can summarize the discussion above
bounds for ψ(A, B) in the case when A and B are sublevel with the following proposition.
sets of some locally excessive function g. In general one
can apply discretization techniques [2] to estimate ψ(A, B) Proposition 3. In the presence of an absorbing set A0 the
by working with a finite space Markov chain approximation. solution the invariance problem (13) over A, A0 ⊆ A, is a non-
With focus on this latter class of probabilistic models, we trivial solution of (14) and is uniquely defined by (15)-(16).

529
 V. S TOCHASTIC B ISIMULATION F UNCTIONS is the smallest SBF of the function g, i.e. ϕ is an SBF of g and
Consider a stochastic model S1with associated discrete- for all ϕ ∗ which is an SBF of g it holds that ϕ(x) ≤ ϕ ∗ (x)
time, (time-)homogeneous strong Markov process X 1 , and a for all x ∈ X .
second model denoted by S 2 and X 2 . (S 1 may be thought Remark 3. Note that the limit in (19) exists pointwise for all
of as a concrete model, whereas S 2 as its abstraction.) We x ∈ X . Indeed, for any fixed x, we have that ϕn+1 (x) ≥ ϕn (x)
are interested in quantifying “how close” S 1 and S 2 are, and due to the monotonicity of the sequence (ϕn (x))n≥0 the
which can be done by selecting a function g inducing a limit lim ϕn (x) always exists, being either finite or infinite.
metric over the distance between their trajectories [9]. The n→∞
Note also that ϕ(x) defined in (19) is unique.
quality of the abstraction is characterized, over a finite and
Alternatively, for a function g ∈ L , its smallest excessive
an infinite time horizon, by the following quantities:
  majorant ϕ(x) satisfies the following equation
VδN (x) = Px sup g(Xn ) ≥ δ (17) ϕ(x) = max{g(x), Pϕ(x)}, (20)
0≤n≤N

and   which solution, however, is not unique (see [14, Rem.1,

Vδ (x) = Px sup g(Xn ) ≥ δ , (18) p.43]): if g(x) ≤ C < ∞ for all x ∈ X , then for any constant
n≥0 C0 ≥ C the function ϕ(x) ≡ C0 admits (20).
where g is a measurable non-negative function for X =
It is in general difficult to characterize analytically the
(X 1 , X 2 ), a Markov process for the joint system S =
smallest SBF, as well as to provide strict bounds for the
(S 1 , S 2 ). For a finite horizon N, it can be noticed that
numerical calculations in (19) or (20). On the other hand one
VδN (x) = Px [rN (Xg (δ ))], can find Vδ (x) using the fact that Vδ (x) = v(x; Xg (δ )) = 1 −
u(x; Xgc (δ )). Notice the similarity of the recurstions in (3)
thus VδN (x) can be calculated using (3).
and (19), tailored to calculate respectively v(x; Xg (δ )) and
Next, we focus on the infinite horizon case, where
ϕ(x). Since the SBF gives only bounds on the reachability
Vδ (x) = Px [r(Xg (δ ))] = v(x; Xg (δ )). probability, the only advantage of characterizing an SBF in
the discrete-time case is that, once obtained, it can be applied
However, as discussed, this latter quantity is in general
to any set of the form Xg (δ ), whereas the reachability
difficult to precisely quantify. Alternatively, one can exploit
probability would require being calculated for each of these
the use of a stochastic bisimulation function (SBF), as
sets separately,
developed for continuous-time models in [9].
Let us assume that there exists min g(x) = δ∗ and more-
x∈X
Definition 5. Let the measurable function ϕ : X → R+ over that for all δ > δ∗ , we have that Xgc (δ ) is a compact
satisfy the following conditions set and T ∈ Cw (X ). By Theorem 3 we know that if there
1) (ϕ(Xn ))n≥0 is a Px -supermartingale for any fixed x ∈ are no absorbing subsets of Xgc (δ ) then u(x; Xgc (δ )) ≡ 0
X, and the bisimulation is not valid over the infinite horizon.
2) ϕ(x) ≥ g(x) for all x ∈ X . Thus we conclude that to find a finite SBF over an infinite
Then ϕ is an SBF for the function g (with respect to the horizon for the processes X 1 and X 2 we need for Xgc (δ ) to
process X). contain an absorbing subset.
If ϕ is an SBF of g then It is interesting to note that also [9] considers cases with
    the origin as an absorbing set. Due to the monotonicity and
Px sup g(Xn ) ≥ δ ≤ Px sup ϕ(Xn ) ≥ δ , compactness of the family Xgc (δ ), that it is equivalent to
n≥0 n≥0 the following statement: Xgc (0) = {x ∈ X : g(x) = 0} has
to contain an absorbing subset. If the function g is locally
thus, using Doob’s inequality, we obtain Vδ (x) ≤ ϕ(x)
δ . excessive (i.e. it is excessive on Xgc (δ ) for some δ > δ∗ )
Let us again employ excessive functions as in the follow-
then for the overapproximation of Vδ (x) one can use the
ing definition.
decomposition technique provided in Section IV-B.
Definition 6. [14, Def.2, p.39] An excessive function f is
said to be an excessive majorant of the measurable function A. Stochastic Bisimulation Functions: a case study
g if f (x) ≥ g(x) on X . An excessive majorant f of g is said We consider a discrete-time approximation of the follow-
to be the smallest excessive majorant of g if f (x) is less than ing stochastic differential equation
or equal to any excessive majorant of g.
dxt = αxt dt + σ xt dwt ,
Clearly, ϕ is an SBF of g if and only if ϕ is an excessive
majorant of g. The following theorem holds over the smallest where we select α = −1 and σ = 0.3. The time discretization
excessive majorant of a function. is obtained by first-order Euler-Maryuama scheme. For two
Theorem 7. Let a function g ∈ L then the function discretization steps ∆t and ∆t2 we obtain respectively two
discrete time processes, denoted as Xn and Xn0 . In order to
ϕ(x) = lim Q n g(x) (19) set up a relation over the distance between the two processes,
n→∞

530
 0 and select the distance function
we consider Xn and Yn = X2n Leveraging the decomposition approach, from (11) we obtain
as follows: the bound V0.1 (0.15, 0.15) ≤ 0.43.
The direct calculations of reachability iterations give
g(x, y) = (x2 + 1)((x − y)2 + 1) − 1. V0.1 (0.15, 0.15) ≈ 0.4. Notice that the straightforward com-
putation does not allow defining a bound.
In the above expression we employ a distance term between x
Finally, the numerical calculation of the smallest SBF φ by
and y, additionally the term (x2 + 1) ensures the compactness
(19) give a very conservative bound: V0.1 (0.15, 0.15) < 1.04.
of sublevel sets, and −1 is used to rescale the minimum of
g to be equal to 0 (see discussion above). VI. F UTURE WORK
The dynamics of the two processes are described by Current work is focused on the characterization of absorb-
√ ing sets and on the properties they entail. This is approached
Xn+1 = Xn (1 + αdt + σ ξn dt),
r ! from three main directions.
(αdt)2 dt First, the combination of Theorem 3 and Proposition 2 for the
Yn+1 = Yn 1 + αdt + +σ (ηn + ζn )
4 2 Borel state space gives a useful criterion relating absorbance
r ! with uniqueness and contractivity of operators IA , RA,B .
dt 3 dt Together with the decomposition technique in Theorem 10,
+Yn ασ (ηn + ζn ) + σ 2 ηn ζn ,
8 2 it is used to find infinite horizon value functions with strict
bounds on the error and to study their properties such as
where ξn , ηn , ζn are i.i.d. standard normal random variables. continuity.
Note that the distribution law of Yn+1 , conditional on Yn , is Second, the authors are working on the extension of Propo-
not Gaussian. Two realizations of the processes are plotted sition 3 to the case when the kernels admit densities. For this
in Figure 1. instance the knowledge of the largest absorbing subset of a
given set is extremely useful.
The last direction is to find a way to verify and characterize
the absorbing feature of a given set, which is a problem
related to other areas of mathematical analysis.
R EFERENCES
[1] A. Abate, S. Amin, M. Prandini, J. Lygeros, and S. Sastry. Probabilistic
reachability and safe sets computation for discrete time stochastic
hybrid systems. In Proceedings of the 45th IEEE Conference of
Decision and Control, pages 258–263, 2006.
[2] A. Abate, J.-P. Katoen, J. Lygeros, and M. Prandini. Approximate
model checking of stochastic hybrid systems. European Journal of
Control, 16:624–641, December 2010.
[3] A. Abate, J.-P. Katoen, and A. Mereacre. Quantitative automata model
checking of autonomous stochastic hybrid systems. In Proceedings of
the 14th ACM international conference on Hybrid Systems: Compu-
tation and Control, pages 83–92, 2011.
Fig. 1. Realizations of processes X and Y (resp. in red and blue), starting [4] A. Abate, M. Prandini, J. Lygeros, and S. Sastry. Probabilistic
from the initial condition 0.2. reachability and safety for controlled discrete time stochastic hybrid
systems. Automatica, 44(11):2724–2734, 2008.
[5] M.L. Bujorianu and J. Lygeros. New insights on stochastic reacha-
bility. In Proceedings of the 46th IEEE Conference on Decision and
We select ∆t = 0.01 and sets A = Xgc (0.1) and B = {x ∈ Control, pages 6172–6177, 2007.
X |g(x) < 0.01}. We would like to compare the values of [6] R. Durrett. Probability: Theory and Examples - Third Edition.
the quantity V0.1 (0.15, 0.15) as in (18), obtained by three Duxbury Press, 2004.
[7] M. Fränzle, T. Teige, and A. Eggers. Engineering constraint solvers
different techniques: decomposition, direct reachability com- for automatic analysis of probabilistic hybrid automata. Journal of
putation, and via the smallest SBF (through the Q iterations). Logic and Algebraic Programming, 79(7):436–466, 2010.
For the computation of the reach-avoid problem (decom- [8] O. Hernández-Lerma and J.B. Lasserre. Discrete-time Markov control
processes. Appl. Math, 30, 1996.
position), of probabilistic reachability, and of the SBF, we [9] A.A. Julius and G.J. Pappas. Approximations of stochastic hybrid
have discretized the region A by square cells with diameter systems. Automatic Control, IEEE Transactions on, 54(6):1193–1203,
equal to 0.015. With this discretization we can approximate 2009.
[10] H.J. Kushner. Stochastic stability and control, volume 33. Academic
the original processes by a Markov chain [2]. (This also Press, 1967.
allows computing the associated discretization error, which [11] S.P. Meyn and R.L. Tweedie. Markov chains and stochastic stability.
is not pursued here.) With the Markov chain approximations, Springer Verlag, 1993.
[12] G. Peškir and A. Shiryaev. Optimal stopping and free-boundary
we calculate functions wn , vn and ϕn for n = 2000 steps. In problems. Birkhauser, 2006.
all these cases the calculation time for the value functions is [13] F. Ramponi, D. Chatterjee, S. Summers, and J. Lygeros. On the
approximately 109 sec. connections between PCTL and dynamic programming. In Proceed-
ings of the 13th ACM international conference on Hybrid Systems:
Notice that there is one absorbing set coinciding with Computation and Control, pages 253–262, 2010.
the origin. Hence, the decomposition technique can be used. [14] A. Shiryaev, B. Rozovskii, and G. Grimmett. Optimal stopping rules.
Using local Doob’s inequality we obtain that ψ(A, B) ≥ 0.9. Springer-Verlag Berlin Heidelberg, 2008.

531