Algorithmic Verification of

Recursive Probabilistic State Machines

Kousha Etessami1 and Mihalis Yannakakis2

1
School of Informatics, University of Edinburgh
2
Department of Computer Science, Columbia University

Abstract. Recursive Markov Chains (RMCs) ([EY05]) are a natural

abstract model of procedural probabilistic programs and related systems
involving recursion and probability. They succinctly define a class of de-
numerable Markov chains that generalize multi-type branching (stochas-
tic) processes. In this paper, we study the problem of model checking
an RMC against a given ω-regular specification. Namely, given an RMC
A and a Büchi automaton B, we wish to know the probability that an
execution of A is accepted by B. We establish a number of strong upper
bounds, as well as lower bounds, both for qualitative problems (is the
probability = 1, or = 0?), and for quantitative problems (is the probabil-
ity ≥ p?, or, approximate the probability to within a desired precision).
Among these, we show that qualitative model checking for general RMCs
can be decided in PSPACE in |A| and EXPTIME in |B|, and when A is
either a single-exit RMC or when the total number of entries and exits in
A is bounded, it can be decided in polynomial time in |A|. We then show
that quantitative model checking can also be done in PSPACE in |A|,
and in EXPSPACE in |B|. When B is deterministic, all our complexities
in |B| come down by one exponential.
For lower bounds, we show that the qualitative model checking prob-
lem, even for a fixed RMC, is EXPTIME-complete. On the other hand,
even for reachability analysis, we showed in [EY05] that our PSPACE
upper bounds in A can not be improved upon without a breakthrough on
a well-known open problem in the complexity of numerical computation.

1 Introduction
Recursive Markov Chains (RMCs) are a natural abstract model of procedural
probabilistic programs. They succinctly define a natural class of denumerable
Markov chains that generalize multi-type branching (stochastic) processes. In-
formally, an RMC consists of a collection of finite state component Markov
chains (MC) that can call each other in a potentially recursive manner. Each
component MC has a set of nodes (ordinary states), a set of boxes (each mapped
to a component MC), a well-defined interface consisting of a set of entry and
exit nodes (nodes where it may start and terminate), and a set of probabilistic
transitions connecting the nodes and boxes. A transition to a box specifies the
entry node and models the invocation of the component MC associated with the

N. Halbwachs and L. Zuck (Eds.): TACAS 2005, LNCS 3440, pp. 253–270, 2005.


c Springer-Verlag Berlin Heidelberg 2005
254 K. Etessami and M. Yannakakis

box; when (and if) the component MC terminates at an exit, execution of the
calling MC resumes from the corresponding exit of the box.
RMCs are a probabilistic version of Recursive State Machines (RSMs). RSMs
([AEY01, BGR01]) and closely related models like Pushdown Systems (PDSs)
(see, e.g., [EHRS00, BR00]) have been studied extensively in recent research on
model checking and program analysis, because of their applications to verifica-
tion of sequential programs with procedures. Recursive Markov Chains gener-
alize other well-studied models involving probability and recursion: Stochastic
Context-Free Grammars (SCFGs) have been extensively studied, mainly in nat-
ural language processing (NLP) (see [MS99]). Multi-Type Branching Processes
(MT-BPs), are an important family of stochastic processes with many applica-
tions in a variety of areas (see, e.g., [Har63]). Both SCFG’s and MT-BP’s are
essentially equivalent to single-exit RMC’s: the special case of RMC’s in which
all components have one exit. Probabilistic models of programs and systems are
of interest for several reasons. First, a program may use randomization, in which
case the transition probabilities reflect the random choices of the algorithm. Sec-
ond, we may want to model and analyse a program or system under statistical
conditions on its behaviour (e.g., based on profiling statistics or on statistical
assumptions), and to determine the induced probability of properties of interest
We introduced RMCs in [EY05] and developed some of their basic theory,
focusing on algorithmic reachability analysis: what is the probability of reaching
a given state starting from another? In this paper we study the more general
problem of model checking an RMC against an ω-regular specification: given an
RMC A and a Büchi automaton B, what is the probability that an execution of
A is accepted by B? The techniques we develop in this paper for model checking
go far beyond what was developed in [EY05] for reachability analysis.
General RMCs are intimately related to probabilistic Pushdown Systems
(pPDSs), and there are efficient translations between RMCs and pPDSs. There
has been some recent work on model checking of pPDSs ([EKM04, BKS05]). As
we shall describe shortly, our results yield substantial improvements, when trans-
lated to the setting of pPDSs, on the best algorithmic upper and lower bounds
known for ω-regular model checking of pPDSs.
We now outline the main results in this paper. We are given an RMC A
and a property in the form of a (non-deterministic) Büchi automaton (BA)
B, whose alphabet corresponds to (labels on) the vertices of A. Let PA (L(B))
denote the probability that an execution of A is accepted by B (i.e., satisfies the
property). The qualitative model checking problems are: (1) determine whether
almost all executions of A satisfy the property B (i.e. is PA (L(B)) = 1?, this
corresponds to B being a desirable correctness property), and (2) whether almost
no executions of A satisfy B (i.e. is PA (L(B)) = 0?, corresponding to B being an
undesirable error property). In the quantitative model checking problems we wish
to compare PA (L(B)) to a given rational threshold p, i.e., is PA (L(B)) ≥ p?, or
alternatively, we may wish to approximate PA (L(B)) to within a given number
of bits of precision. Note that in general PA (L(B)) may be irrational or may not
even be expressible by radicals [EY05]. Hence it cannot be computed exactly.
 Algorithmic Verification of Recursive Probabilistic State Machines 255

reachability det. Büchi nondet. Büchi

1-exit P P P in RMC, EXPTIME in Büchi
Qualitative:
Bd P P P in RMC, EXPTIME in Büchi
general PSPACE PSPACE PSPACE in RMC, EXPTIME in Büchi

reachability det. Büchi nondet. Büchi

1-exit PSPACE PSPACE PSPACE in RMC, EXPSPACE in Büchi
Quantitative: Bd P P in RMC P in RMC,
for fixed Büchi for fixed Büchi
general PSPACE PSPACE PSPACE in RMC, EXPSPACE in Büchi

Fig. 1. Complexity of Qualitative and Quantitative problems

We show that the qualitative model checking problems can be solved in

PSPACE in |A| and EXPTIME in |B|. More specifically, in a first phase the
algorithm analyzes the RMC A by itself (using PSPACE only in |A|). In a sec-
ond phase, it further analyses A in conjunction with B, using polynomial time
in A and exponential time in B. If the automaton B is deterministic then the
time is polynomial in B. Furthermore, if A is a single-exit RMC (which cor-
responds to SCFG’s and MT-BP’s), then the first phase, and hence the whole
algorithm, can be done in polynomial time in A. Another such case, where we
can model-check qualitatively in polynomial time in A, is when the total number
of entries and exits in A is bounded (we call them Bd-RMCs). In terms of proba-
bilistic program abstractions, this class of RMC’s corresponds to programs with
a bounded number of distinct procedures, each of which has a bounded number
of input/output parameter values. The internals of the components of the RMCs
(i.e., the procedures) can be arbitrarily large and complex.
For quantitative model checking, we show that deciding whether PA (L(B)) ≥
p, given a rational p ∈ [0, 1], can be decided in PSPACE in |A| and EXPSPACE
in |B|. When B is deterministic, the space is polynomial in both A and B.
Moreover, for A a Bd-RMC, and when B is fixed, there is an algorithm that
runs in P-time in |A|; however, in this case (unlike the others) the exponent of
the polynomial depends on B. Table 1 summarizes our complexity upper bounds
(the “reachability” columns are from [EY05]; all the other results are new).
For lower bounds, we prove that the qualitative model checking problem,
even for a fixed, single entry/exit RMC, is already EXPTIME-complete. On
the other hand, even for reachability analysis, we showed in [EY05] that our
PSPACE upper bounds in A, even for the quantitative 1-exit problem, and the
general qualitative problem, can not be improved without a breakthrough on
the complexity of the square root sum problem, a well-known open problem in
the complexity of exact numerical computation (see Section 2.2).

Related Work. Model checking of flat Markov chains has received extensive
attention both in theory and practice (e.g. [CY95, Kwi03, PZ93, Var85]). It is
known that model checking of a Markov chain A with respect to a Büchi au-
tomaton B is PSPACE-complete, and furthermore the probability PA (L(B)) can
256 K. Etessami and M. Yannakakis

be computed exactly in time polynomial in A and exponential in B. Recursive

Markov chains were introduced recently in [EY05], where we developed some of
their basic theory and investigated the termination and reachability problems;
we summarize the main results in Section 2.2. Recursion introduces a number
of new difficulties that are not present in the flat case. For example, in the flat
case, the qualitative problems depend only on the structure of the Markov chain
(which transitions are present) and not on the precise values of the transition
probabilities; this is not any more the case for RMC’s and numerical issues have
to be dealt with even in the qualitative problem. Furthermore, unlike the flat
case, the desired probabilities cannot be computed exactly.
The closely related model of probabilistic Pushdown Systems (pPDS) was in-
troduced and studied recently in [EKM04, BKS05]. They largely focus on model
checking against branching-time properties, but they also study deterministic
([EKM04]) and non-deterministic ([BKS05]) Büchi automaton specifications.
There are efficient (linear time) translations between RMCs and pPDSs, sim-
ilar to translations between RSMs and PDSs (see [AEY01, BGR01]). Our upper
bounds, translated to pPDSs, improve those obtained in [EKM04, BKS05] by
an exponential factor in the general setting, and by more for specific classes
like single-exit and Bd-RMCs. Specifically, [BKS05], by extending results in
[EKM04], show that qualitative model checking for pPDSs can be done in
PSPACE in the size of the pPDS and 2-EXPSPACE in the size of the Büchi
automaton, while quantitative model checking can be decided in EXPTIME in
the size of the pPDS and in 3-EXPTIME in the size of the Büchi automaton.
They do not obtain stronger complexity results for the class of pBPAs (equiva-
lent to single-exit RMCs). Also, the class of Bd-RMCs has no direct analog in
pPDSs, as the total number of entries and exits of an RMC gets lost in transla-
tion to pPDSs. Reference [EE04] is a survey paper that predates this paper and
summarizes only the results in prior papers [EKM04, EY05, BKS05].
The paper is organized as follows. Section 2 gives necessary definitions and
background on RMC’s from [EY05]. Section 3 shows how to construct from
an RMC, A, a flat Markov chain MA which in some sense “summarizes” A;
this chain plays a central role analogous to the ”summary graph” for RSMs
[AEY01, BGR01]. Section 4 addresses the qualitative model checking problems,
presenting both upper and lower bounds. Section 5 addresses the quantitative
model checking problem; a fundamental “unique fixed point theorem” is proved
for RMC’s, and plays a crucial role in our quantitative algorithms.
Due to space limitations, we have removed almost all proofs from this paper.

2 Definitions and Background

A Recursive Markov Chain (RMC), A, is a tuple A = (A1 , . . . , Ak ), where each
component chain Ai = (Ni , Bi , Yi , Eni , Exi , δi ) consists of:
– A set Ni of nodes
– A subset of entry nodes Eni ⊆ Ni , and a subset of exit nodes Exi ⊆ Ni .
– A set Bi of boxes. Let B = ∪ki=1 Bi be the (disjoint) union of all boxes of A.
 Algorithmic Verification of Recursive Probabilistic State Machines 257

– A mapping Yi : Bi → {1, . . . , k} assigns a component to every box.

Let Y = ∪ki=1 Yi be Y : B → {1, . . . , k} where Y |Bi = Yi , for 1 ≤ i ≤ k.
– To each box b ∈ Bi , we associate a set of call ports, Callb = {(b, en) | en ∈
EnY (b) }, and a set of return ports, Returnb = {(b, ex) | ex ∈ ExY (b) }.
– A transition relation δi , where transitions are of the form (u, pu,v , v) where:
1. the source u is either a non-exit node u ∈ Ni \ Exi , or a return port
u = (b, ex) ∈ Returnb , where b ∈ Bi .
2. The destination v is either a non-entry node v ∈ Ni \ Eni , or a call port
v = (b, en) ∈ Callb , where b ∈ Bi .
3. pu,v ∈ R>0 is the probability of transition from u to v. (We assume pu,v
is rational.)

4. Consistency of probabilities: for each u, {v
|(u,pu,v
,v
)∈δi } pu,v = 1,

unless u is a
call port or exit node; neither have outgoing transitions, in

which case v
pu,v
= 0.
We will use the term vertex of Ai to refer collectively to its set of nodes, k call
ports, and return ports, and we denote this set by Qi , and we let Q = i=1 Qi
be the set of all vertices of the RMC A. That is, the transition relation δi is a
set of probability-weighted directed edges on the set Qi of vertices of Ai . Let
δ = ∪i δi be the set of all transitions of A.
An RMC A defines a global denumerable Markov chain MA = (V, ∆) as
follows. The global states V ⊆ B ∗ × Q are pairs of the form β, u , where β ∈ B ∗
is a (possibly empty) sequence of boxes and u ∈ Q is a vertex of A. More precisely,
the states V ⊆ B ∗ × Q and transitions ∆ are defined inductively as follows:
1. , u ∈ V , for u ∈ Q. ( denotes the empty string.)
2. if β, u ∈ V and (u, pu,v , v) ∈ δ, then β, v ∈ V and (β, u , pu,v , β, v ) ∈ ∆
3. if β, (b, en) ∈ V and (b, en) ∈ Callb , then
βb, en ∈ V , & (β, (b, en) , 1, βb, en ) ∈ ∆.
4. if βb, ex ∈ V and (b, ex) ∈ Returnb , then
β, (b, ex) ∈ V & (βb, ex , 1, β, (b, ex) ) ∈ ∆.
Item 1 corresponds to the possible initial states, 2 corresponds to a transition
within a component, 3 is when a new component is entered via a box, 4 is when
the process exits a component and control returns to the calling component.
Some states of MA are terminating, i.e., have no outgoing transitions. Namely,
states , ex , where ex is an exit. We want MA to be a proper Markov chain, so
we consider terminating states as absorbing, with a self-loop of probability 1.
A trace (or trajectory) t ∈ V ω of MA is an infinite sequence of states t =
s0 s1 s2 . . .. such that for all i ≥ 0, there is a transition (si , psi ,si+1 , si+1 ) ∈ ∆,
with psi ,si+1 > 0. Let Ω ⊆ V ω denote the set of traces of MA . For a state
s = β, v ∈ V , let Q(s) = v denote the vertex at state s. Generalizing this
to traces, for a trace t ∈ Ω, let Q(t) = Q(s0 )Q(s1 )Q(s2 ) . . . ∈ Qω . We will
consider MA with initial states from Init = {, v | v ∈ Q}. More generally
we may have a probability distribution pinit : V → [0, 1] on initial states (we
usually assume pinit has support only in Init, and we always assume it has finite
support). This induces a probability distribution on traces generated by random
walks on MA . Formally, we have a probability space (Ω, F, PrΩ ), parametrized
258 K. Etessami and M. Yannakakis

by pinit , where F = σ(C) ⊆ 2Ω is the σ-field generated by the set of basic

cylinder sets, C = {C(x) ⊆ Ω | x ∈ V ∗ }, where for x ∈ V ∗ the cylinder at x is
C(x) = {t ∈ Ω | t = xw, w ∈ V ω }. The probability distribution PrΩ : F → [0, 1]
is determined uniquely by the probabilities of cylinder sets, which are:

PrΩ (C(s0 s1 . . . sn )) = pinit (s0 )ps0 ,s1 ps1 ,s2 . . . psn−1 ,sn

See, e.g., [Bil95]. RMCs where every component has at most one exit are called
1-exit RMCs. RMCs
where the total number of entries and exits is bounded by
k
a constant c, (i.e., i=1 |Eni | + |Exi | ≤ c) are called Bounded total entry-exit
RMCs (Bd-RMCs, for short).

2.1 The Central Questions for Model Checking of RMCs

We first define reachability probabilities that play an important role in our anal-
ysis. Given a vertex u ∈ Qi and an exit ex ∈ Exi , both in the same com-
∗
ponent Ai , let q(u,ex) denote the probability of eventually reaching the state
, ex , starting at the state , u . Formally, we have pinit (, u ) = 1, and
∗ .
q(u,ex) = PrΩ ({t = s0 s1 . . . ∈ Ω | ∃ i , si = , ex }). As we shall see, the
∗
probabilities q(u,ex) will play an important role in obtaining other probabilities.
Recall that a Büchi automaton B = (Σ, S, q0 , R, F ), has an alphabet Σ, a set
of states S, an initial state q0 ∈ S, a transition relation R ⊆ S×Σ×S, and a set of
accepting states F ⊆ S. A run of B is a sequence π = q0 v0 q1 v1 q2 . . . of alternating
states and letters such that for all i ≥ 0 (qi , vi , qi+1 ) ∈ R. The ω-word associated
with run π is wπ = v0 v1 v2 . . . ∈ Σ ω . The run π is accepting if for infinitely many
i, qi ∈ F . Define the ω-language L(B) = {wπ | π is an accepting run of B}. Note
that L(B) ⊆ Σ ω . Let L : Q → Σ, be a given Σ-labelling of the vertices v of RMC
A. L naturally generalizes to L : Qω → Σ ω : for w = v0 v1 v2 . . . ∈ Qω , L(w) =
L(v0 )L(v1 )L(v2 ) . . .. Given RMC A, with initial state s0 = , u , and given a
BA B over the alphabet Σ, let PA (L(B)) denote the probability that a trace of
.
MA is in L(B). More precisely: PA (L(B)) = PrΩ ({t ∈ Ω | L(Q(t)) ∈ L(B)}).
One needs to show that the sets {t ∈ Ω | L(Q(t)) ∈ L(B)} are measurable (in
F). This is not difficult (see similar proofs in [CY95, Var85]). The model checking
problems for ω-regular properties of RMCs are:

(1) Qualitative model checking problems: Is PA (L(B)) = 1? Is PA (L(B)) = 0?

(2) Quantitative model checking problems: given p ∈ [0, 1], is PA (L(B)) ≥ p?
Also, we may wish to approximate PA (L(B)) to within a given number of
bits of precision.

Note, with a routine for the problem PA (L(B)) ≥ p?, we can approximate
PA (L(B)) to within i bits using binary search with i calls to the routine. Thus,
for quantitative model checking the first problem entails the second. Note that
probabilistic reachability is a special case of model checking: given vertex u of
RMC A and a subset of vertices F , the probability that the RMC starting at u
visits some vertex in F (in some stack context) is equal to PA (L(B)), where we
let the labelling L map vertices in F to 1 and the other vertices to 0, and B is
 Algorithmic Verification of Recursive Probabilistic State Machines 259

the 2-state automaton that accepts strings that contain a 1. Similarly, for the
repeated reachability problem, where we are interested whether a trajectory from
u infinitely often visits a vertex of F , we can let B be the (2-state deterministic)
automaton that accepts strings with an infinite number of 1’s.
To simplify the descriptions of our results, we assume henceforth that Σ = Q,
the vertices of A. This is w.l.o.g. since the problem can be reduced to this case
by relabelling the RMC A and modifying the automaton B (see, e.g., [CY95]),
however care is needed when measuring complexity separately in RMC, A, and
BA, B, since typically B and Σ are small in relation to A. Our complexity results
are all with respect to the standard sizes of A and B.

2.2 Basic RMC Theory and Reachability Analysis (From [EY05])

We recall some of the basic theory of RMCs developed in [EY05], where we
∗
studied reachability analysis. Considering the probabilities q(u,ex) as unknowns,
we can set up a system of (non-linear) polynomial equations, such that the
∗
probabilities q(u,ex) are the Least Fixed Point (LFP) solution of this system.
∗
Use a variable x(u,ex) for each unknown probability q(u,ex) . We will often find it
convenient to index the variables x(u,ex) according to a fixed order, so we can
refer to them also as x1 , . . . , xn , with each x(u,ex) identified with xj for some j.
We thus have a vector of variables: x = (x1 x2 . . . xn )T .

Definition 1. Given RMC A = (A1 , . . . , Ak ), define the system of polynomial

equations, SA , over the variables x(u,ex) , where u ∈ Qi and ex ∈ Exi , for
1 ≤ i ≤ k. The system contains one equation x(u,ex) = P(u,ex) (x), for each
variable x(u,ex) . P(u,ex) (x) denotes a multivariate polynomial with positive ratio-
nal coefficients. There are 3 cases, based on the “type” of vertex u:
1. Type I: u = ex. In this case: x(ex,ex) = 1.

2. Type II: either u ∈ Ni \ {ex} or
u = (b, ex ) is a return port. In these cases:
x(u,ex) = {v|(u,pu,v ,v)∈δ} pu,v · x(v,ex) .
3. Type III: u = (b, en) is a call port.

In this case:
x((b,en),ex) = ex
∈ExY (b) x(en,ex
) · x((b,ex
),ex)

In vector notation, we denote SA = (xj = Pj (x) | j = 1, . . . , n) by: x = P (x).

Given A, we can construct x = P (x) in P-time: P (x) has size O(|A|θ2 ),

where θ denotes the maximum number of exits of any component. For vectors
x, y ∈ Rn , define x  y to mean that xj ≤ yj for every coordinate j. For
D ⊆ Rn , call a mapping H : Rn → Rn monotone on D, if: for all x, y ∈ D,
if x  y then H(x)  H(y). Define P 1 (x) = P (x), and P k (x) = P (P k−1 (x)),
for k > 1. Let q∗ ∈ Rn denote the n-vector of probabilities q(u,ex)
∗
, using the
same indexing as used for x. Let 0 denote the all 0 n-vector. Define x0 = 0, and
xk = P (xk−1 ) = P k (0), for k ≥ 1. The map P : Rn → Rn is monotone on Rn≥0 .

Theorem 1. ([EY05], see also [EKM04]) q∗ ∈ [0, 1]n is the Least Fixed Point
solution, LFP(P ), of x = P (x). Thus, q∗ = P (q∗ ) and q∗ = limk→∞ xk , and
for all k ≥ 0, xk  xk+1  q∗ , and for all q ∈ Rn≥0 , if q = P (q ), then q∗  q .
260 K. Etessami and M. Yannakakis

∗
There are already 1-exit RMCs for which the probability q(en,ex) is irrational
and not “solvable by radicals” ([EY05]). Thus, we can’t compute probabilities
exactly. Given a system x = P (x), and a vector q ∈ [0, 1]n , consider the following
sentence in the Existential Theory of Reals (which we denote by ExTh(R)):

m 
m 
m
ϕ ≡ ∃x1 , . . . , xm Pi (x1 , . . . , xm ) = xi ∧ 0 ≤ xi ∧ xi ≤ qi
i=1 i=1 i=1

ϕ is true precisely when there is some z ∈ Rm , 0  z  q, and z = P (z). Thus,

∗
if we can decide the truth of this sentence, we could tell whether q(u,ex) ≤ p, for
some rational p, by using the vector q = (1, . . . , p, 1, . . .). We will rely on decision
procedures for ExTh(R). It is known that ExTh(R) can be decided in PSPACE
and in exponential time, where the time exponent depends (linearly) only on the
number of variables; thus for a fixed number of variables the algorithm runs in
polynomial time [Can88, Ren92, BPR96]. As a consequence:
Theorem 2. ([EY05]) Given RMC A and rational ρ, there is a PSPACE algo-
∗
rithm to decide whether q(u,ex) ≤ ρ, with running time O(|A|O(1) · 2O(m) ) where
∗
m is the number of variables in the system x = P (x) for A. Moreover q(u,ex) can
be approximated to within j bits of precision within PSPACE and with running
time at most j times the above.
For Bd-RMCs, as shown in [EY05] it is possible to construct efficiently a
system of equations in a bounded number of variables, whose LFP yields the
∗
entry-exit probabilities q(en,ex) . Since ExTh(R) is decidable in P-time when the
number of variables is bounded, this yields:
Theorem 3. ([EY05]) Given Bd-RMC, A & rational p ∈ [0, 1], there is a P-
∗
time algorithm to decide whether, for a vertex u & exit ex, q(u,ex) ≥ p(or < p).
For 1-exit RMCs (SCFGs), the qualitative termination/reachability problem
can be solved efficiently, via an eigenvalue characterization and other techniques.

Theorem 4. ([EY05]) There is a P-time algorithm that for a 1-exit RMC,

∗
vertex u and exit ex, decides which of the following holds:(1) q(u,ex) = 0,(2)
∗ ∗
q(u,ex) = 1,or (3) 0 < q(u,ex) < 1.

Hardness, such as NP-hardness, is not known for RMC reachability. How-

ever, in [EY05] we gave strong evidence of “difficulty”: the square-root sum
∗
problem is P-time reducible to deciding whether q(u,ex) ≥ p, in a 1-exit RMC,
∗
and to deciding whether q(u,ex) = 1 for a 2-exit RMC. Square-root sum is the
following decision problem: given (d1 , . . . , dn ) ∈ Nn and k ∈ N, decide whether

n √
i=1 di ≤ k. It is solvable in PSPACE, but it has been a major open problem
since the 1970’s (see, e.g., [GGJ76, Tiw92]) whether it is solvable even in NP.
As a practical efficient numerical algorithm for computing the probabilities
∗
q(u,ex) , it was proved in [EY05] that a multi-dimensional Newton’s method con-
verges monotonically to the LFP of x = P (x), and constitutes a rapid accelera-
tion of iterating P k (0), k → ∞.
 Algorithmic Verification of Recursive Probabilistic State Machines 261

3 The Conditioned Summary Chain MA
∗
For an RMC A, suppose we somehow have the probabilities q(u,ex) “in hand”.

Based on these, we construct a conditioned summary chain, MA , a finite Markov
chain that will allow us to answer repeated reachability questions. Extensions
of MA will later be a key to model checking RMCs. Since probabilities q(u,ex) ∗

are potentially irrational, we can not compute MA exactly. However, MA will be
important in our correctness arguments, and we will in fact be able to compute
the “structure” of MA , i.e., what transitions have non-zero probability. The
structure of MA will be sufficient for answering various “qualitative” questions.
We will assume, w.l.o.g., that each RMC has one initial state s0 = , eninit ,
with eninit the only entry of a component that does not contain any exits. Any
RMC can readily be converted to an “equivalent” one in this form.
Before describing MA , let us recall from [AEY01], the construction of a “sum-
mary graph”, HA = (Q, EHA ), which ignores probabilities and is based only on
information about reachability in the underlying RSM of A. Let R be the binary
relation between entries and exits of components such that (en, ex) ∈ R precisely
when there exists a path from , en to , ex , in the underlying graph of MA .
The edge set EHA is defined as follows. For u, v ∈ Q, (u, v) ∈ EHA iff one of the
following holds:
1. u is not a call port, and (u, pu,v , v) ∈ δ, for pu,v > 0.
2. u = (b, en) is a call port, and (en, ex) ∈ R, and v = (b, ex) is a return port.
3. u = (b, en) is a call port, and v = en is the corresponding entry.
For
each vertex v ∈ Qi , let us define the probability of never exiting: ne(v) =
∗
1 − ex∈Exi q(v,ex) . Call a vertex v deficient if ne(v) > 0, i.e. there is a nonzero
probability that if the RMC starts at v it will never terminate (reach an exit of
the component).
We define MA = (QMA
, δMA
) as follows. The set of states QMA
of MA is the
set of deficient vertices: QMA
= {v ∈ Q | ne(v) > 0}. For u, v ∈ QMA
, there is a
transition (u, pu,v , v) in δMA
if and only if one of the following conditions holds:
pu,v ·ne(v)
1. u, v ∈ Qi and (u, pu,v , v) ∈ δi , and pu,v = ne(u) .
∗
∗ q(en,ex) ne(v)
2. u = (b, en) ∈ Callb , v = (b, ex) ∈ Returnb , q(en,ex) > 0, & pu,v = ne(u) .
3. u = (b, en) ∈ Callb and v = en, and pu,v = ne(u)
ne(v)
. We call these transitions,
from a call port to corresponding entry, special red transitions.
Note that in all three cases, pu,v is well-defined (the denominator is nonzero)
and it is positive. Recall that we assumed that the initial vertex eninit is the
entry of a component A0 , and A0 has no exits. Thus for all v ∈ Q0 , ne(u) = 1,
and thus Q0 ⊆ QMA
, and if (u, pu,v , v) ∈ δ0 , then (u, pu,v , v) ∈ δMA
.
Proposition 1. Probabilities on transitions out of each state in QMA
sum to 1.
MA is an ordinary (flat) Markov chain. Let (Ω  , F  , PrΩ’ ) denote the probability
space on traces of MA . We now define a mapping ρ : Ω → Ω  ∪ { }, that maps
262 K. Etessami and M. Yannakakis

every trace t of the original (infinite) Markov chain MA , either to a unique

trajectory ρ(t) ∈ Ω  of the MC MA , or to the special symbol . Trajectories
mapped to will be precisely those that go through missing vertices u ∈ Q
that are not in QMA
, i.e., with ne(u) = 0. We show the total probability of
all these trajectories is 0, i.e., PrΩ (ρ−1 ( )) = 0, and moreover, MA preserves
the probability measure of MA : for all D ∈ F  , ρ−1 (D) ∈ F, and PrΩ’ (D) =
PrΩ (ρ−1 (D)). We define ρ in two phases. We first define a map ρH : Ω → Qω ,
where every trajectory t ∈ Ω is mapped to an infinite path ρH (t) in the summary
graph HA . Thereafter, we let ρ(t) = ρH (t) if all vertices of ρH (t) are in MA , and
let ρ(t) = otherwise. We define ρH for a trace t = s0 s1 . . . si . . ., sequentially
based on prefixes of t, as follows. By assumption, s0 = , eninit . ρH maps s0
to eninit . Suppose si = β, u , and, inductively, suppose that ρH maps s0 . . . si
to einit . . . u. First, suppose u is not a call port, and that si+1 = β, v , then
s0 . . . si si+1 maps to einit . . . uv. Next, suppose u = (b, en) is a call port and
si+1 = βb, en . If the trace eventually returns from this call (i.e., there exists
j > i + 1, such that sj = βb, ex and sj+1 = β, (b, ex) , and such that each
of the states si+1 . . . sj , have βb as a prefix of the call stack), then s0 . . . sj is
mapped by ρH to eninit . . . u(b, ex). If the trace never returns from this call, then
s0 . . . si si+1 maps to eninit . . . u en. This concludes the definition of ρH . We show
that the mapping ρ is measure preserving.

Lemma 1. PrΩ (ρ−1 ( )) = 0. Moreover, for all D ∈ F  , ρ−1 (D) ∈ F and

PrΩ (ρ−1 (D)) = PrΩ’ (D).

Let HA = (QHA
, EHA
) be the underlying directed graph of MA . In other words,
QHA = QMA
, and (u, v) ∈ EHA
iff (u, pu,v , u) ∈ δMA
. We show we can compute



HA in P-time for 1-exit RMCs and Bd-RMCs, and in PSPACE for arbitrary
RMCs. The basic observation is: the structure of MA depends only on qualitative
∗
facts about the probabilities q(en,ex) and ne(u), for u ∈ Q.

Proposition 2. For a RMC A (respectively, 1-exit or Bd-RMC), and u ∈ Q,

we can decide whether ne(u) > 0 in PSPACE (respectively, P-time).

Proof. Suppose u is in a component Ai where Exi = {ex1 , . . . , exk }. Clearly,

k ∗
ne(u) > 0 iff j=1 q(u,ex j)
< 1. Consider the following sentence, ϕ, in ExTh(R).


n 
n 
k
ϕ ≡ ∃x1 , . . . , xn Pi (x1 , . . . , xn ) = xi ∧ 0 ≤ xi ∧ x(u,exj ) < 1
i=1 i=1 j=1

Since q∗ is the LFP solution of x = P (x), ϕ is true in the reals if and only

k ∗
if j=1 q(u,ex j)
< 1. This query can be answered in PSPACE. In the special
∗
case of a 1-exit RMC, we have Exi = {ex1 }, and ne(u) > 0 iff q(u,ex 1)
< 1.
As mentioned in section 2.2, this can be answered in P-time for 1-exit RMCs
([EY05]). Similarly, for Bd-RMCs the question can be answered in P-time by
the techniques developed in [EY05]. 

 Algorithmic Verification of Recursive Probabilistic State Machines 263

Corollary 1. For a RMC A (respectively, 1-exit or Bd-RMC), we can compute


HA in PSPACE (respectively, in polynomial time).

Proof. Recall that u ∈ QHA
precisely when u ∈ Q and ne(u) > 0. Thus we can
determine the set of nodes with the said complexities, respectively. The tran-
sitions of type 1 and 3 in the definition of MA are immediately determined.
For the type 2 transitions, where u = (b, en) and v = (b, ex), in order to de-
termine whether to include the corresponding summary edge (u, v) we need to
∗
decide whether q(en,ex) > 0. This can be done in polynomial time by invoking
the reachability algorithm for RSM’s [AEY01, BGR01]. 


4 Qualitative Model Checking

Upper Bounds. Given an RMC A = (A1 , . . . , Ak ) and a (nondeterministic)
Büchi automaton B = (Σ, S, q0 , R, F ) whose alphabet Σ is the vertex set of
A, we wish to determine whether PA (L(B)) = 1, = 0, or is in-between. We

will construct a finite Markov chain MA,B such that PA (L(B)) is equal to the

probability that a trajectory of MA,B starting from a given initial state reaches
one of a designated set of “accepting” bottom SCCs.
First, let B  = (Σ, 2S , {q0 }, R , F  ) be the deterministic automaton obtained
by the usual subset construction on B. In other words, states of B  are subsets
T ⊆ S, and the transition function R : (2S × Σ) → 2S is given by: R (T1 , v) =
{q  ∈ S | ∃q ∈ T1 s.t. (q, v, q  ) ∈ R}. (We make no claim that L(B) = L(B  ).)
Next we define the standard product RMC, A ⊗ B  , of the RMC A, and the
deterministic Büchi automaton B  . A ⊗ B  has the same number of components
as A. Call these A1 , . . . , Ak . The vertices in component Ai are pairs (u, T ), where
u ∈ Qi and T ∈ 2S , and (u, T ) is an entry (exit) iff u is an entry (exit). The
transitions of Ai are as follows: there is a transition ((u, T ), pu,v , (v, R (T, v))) in
Ai iff there is a transition (u, pu,v , v) in Ai .
   
Define MA,B as MA,B = MA⊗B
. Thus MA,B is the conditioned summary
 
chain of RMC A ⊗ B . For qualitative analysis on MA,B , we need the underlying

graph HA,B . Importantly for the complexity of our algorithms, we do not have
to explicitly construct A ⊗ B  to obtain HA,B  
. Observe that states of MA,B =
(Q × 2 , δMA,B
S
) are pairs (v, T ) where v is a state of MA , and T a state of B  .


The initial state of MA,B is (v0 , {q0 }), where v0 is the initial state of MA and q0

of B. The transitions of MA,B from a state (v, T ) are as follows:

– Case 1: v is not a call port. Then for every transition (v, pv,v
, v  ) ∈ δMA
, we
have a corresponding transition ((v, T ), pv,v
, (v  , R (T, v  ))) ∈ δMA,B

.
– Case 2: v is a call port, v = (b, en) where v is vertex in component Ai and box
b is mapped to component Aj . If there is a red transition (v, pv,en , en) ∈ δMA

then there is a red transition ((v, T ), pv,en , (en, R (T, en)) ∈ δMA,B
with the
same probability.
– Case 3: If v has a summary transition (v, pv,v
, v  ) in MA , where v  = (b, ex),
then we have summary transitions of the form ((v, T ), p , (v  , T  )) in MA,B 
264 K. Etessami and M. Yannakakis

to states of the form (v  , T  ) iff there exists a path in MA from , en to

, ex which, viewed as a string, drives B  from T to T  ; the probability p
of the transition is p = p · ne(v  )/ ne(v) where p is the probability of all
such v-v  paths that drive B  from T to T  .

MA,B is a well-defined Markov chain, which is a refinement of MA . That is,

every trajectory of MA,B projected on the first component is a trajectory of MA
and the projection preserves probabilities. We can define a mapping σ from the
trajectories t of the original (infinite) Markov chain MA to the trajectories of

MA,B , or the special symbol , in a similar manner as we defined the mapping
ρ from trajectories of M to MA . For a trajectory t of MA , it is easy to see that
if ρ(t) = then also σ(t) = . Thus, with probability 1 a trajectory of MA

is mapped to one of MA,B . Furthermore, we can show along similar lines the
analogue of Lemma 2, i.e. the mapping σ preserves probabilities.
Consider a product graph (without probabilities) MA ⊗B between the Markov
chain MA and the given nondeterministic BA B (not B  ) as follows: MA ⊗ B has
nodes (v, q), for all vertices v of MA and states q of B, and an edge (v, q) → (v  , q  )
if either (i) v → v  is an ordinary edge or a red edge of MA and q has a transition
to q  on input v  , or (ii) v → v  is a summary edge and the RMC has a path from
v to v  that corresponds to a run of B from q to q  ; if any such run goes through
an accepting state then we mark the edge (v, q) → (v  , q  ) as an accepting edge.
Also, call a node (v, q) accepting if q ∈ F is an accepting state of B.

With every transition (edge) of MA,B and every edge of MA ⊗ B we associate
a string γ over Σ (the vertex set of A) that caused the edge to be included; i.e.,
if edge (v, T ) → (v  , T  ) of MA,B
(respectively, edge (v, q) → (v  , q  ) of MA ⊗ B)
corresponds to an ordinary or red edge of MA then γ = v  . If it corresponds to a
summary edge then we let γ be any string that corresponds to a v − v  path that
drives B  from T to T  (resp., for which B has a path from q to q  ; if the edge
(v, q) → (v  , q  ) is marked as accepting then we pick a path that goes through
an accepting state of B). In the case of a summary edge, there may be many
strings γ as above; we just pick anyone of them.
Let t be any trajectory of MA starting from , v , for some vertex v of MA
and let r be a corresponding run of B starting from a state q. With probability
1, t maps to a trajectory t = ρ(t) of MA . The mapping ρ can be extended to
pairs (t, r), where r is a run of B on t, i.e., the pair (t, r) is mapped to a run
r = ρ(t, r) of MA ⊗ B. If r is an accepting run of B then r goes infinitely often
through an accepting node or an accepting edge. The converse does not hold
necessarily: a non-accepting run r of B corresponding to a trajectory t may be
mapped to a run r of MA ⊗ B that traverses infinitely often an accepting edge.

If B is a deterministic BA, then MA,B and MA ⊗ B are clearly the same

(except that in MA ⊗ B we did not include the probabilities of the edges). In
this case, the analysis is simpler. Let us say that a bottom strongly connected

component (SCC) of MA,B (and MA ⊗B) is accepting iff it contains an accepting
node or an accepting edge.
 Algorithmic Verification of Recursive Probabilistic State Machines 265

Theorem 5. For a RMC A and a deterministic BA B, the probability PA (L(B))

that a trajectory of A is accepted by B is equal to the probability that a trajectory

of MA,B starting from the initial node (v0 , q0 ) reaches an accepting bottom SCC.

Suppose now that B is nondeterministic. We will follow the approach of

[CY95] for flat Markov chains, except that here we have to deal with recursive

calls and with the summary edges of the constructed Markov chain MA,B which
correspond to sets of paths in the original chain MA rather than single steps.
This complicates things considerably.
Let v be a vertex of MA and q ∈ F an accepting state of B. Let D(v, q) be

the subgraph of MA,B induced by the node (v, {q}) and all nodes reachable from
it . We say that the pair (v, q) is special of type 1 if some bottom SCC C of
D(v, q) contains a state (v, T ) with q ∈ T . We associate with such a pair (v, q) a
string γ(v, q) ∈ Σ ∗ that is the concatenation of the strings associated with the
edges of D(v, q) on a path from (v, {q}) to a node of C. (There may be many
such paths; just pick anyone.) Let v = (b, en) be a vertex of MA that is a call
port of a box b of A and let q ∈ F be a non-accepting state of B. Define a graph

D(v, q) as follows. The graph contains a root node vq and a subgraph of MA,B
consisting of the nodes reachable from vq after we add the following edges. We
add an edge from vq to a node (v  , {q  }) of MA,B

, where v  = (b, ex) is a return
port of the same box b as v, iff there is a path γ from , en to , ex such that
B has a run from q to q  on γ that goes through an accepting state; we label
the edge vq → (v  , {q  }) with such a string γ. The graph D(v, q) consists of the

root vq and the subgraph of MA,B induced by all the nodes that are reachable
from vq after adding the above edges. We call the pair (v, q) special of type 2
if some bottom SCC C of D(v, q) contains a state (v, T ) with q ∈ T . As in the
previous case, we associate with the pair (v, q) a string γ(v, q) ∈ Σ ∗ that is the
concatenation of the strings associated with the edges of D(v, q) on a path from
vq to a node of C. Special pairs have the following important properties.

Lemma 2. Suppose (v, q) is special and that RMC A starts at , v and first
performs the transitions in γ(v, q). Then with probability 1 such a trajectory t
of the RMC is accepted by B with initial state q. Specifically, there is a corre-
sponding accepting run r of B such that ρ(t, r) is a run of MA ⊗ B starting from
(v, q) that infinitely repeats node (v, q) if (v, q) is special of type 1, or repeats an
accepting edge out of (v, q) if (v, q) is special of type 2.

Lemma 3. Suppose there is non-zero probability that a trajectory of the RMC

A starting at any vertex u ∈ MA has a corresponding run in MA ⊗ B starting
from any node (u, p) which repeats an accepting state (v, q) infinitely often or
repeats an accepting edge (v, q) → (v  , q  ) infinitely often. Then (v, q) is special.

Proposition 3. PA (L(B)) > 0 iff from (v0 , q0 ) in MA ⊗ B we can reach a

special (v, q).

Call a bottom SCC of the flat Markov chain MA,B accepting if it contains a state
(v, T ), with some q ∈ T such that (v, q) is special; otherwise call it rejecting.
266 K. Etessami and M. Yannakakis


Theorem 6. PA (L(B)) is equal to the probability that a trajectory of MA,B
starting from the initial state (v0 , {q0 }) reaches an accepting bottom SCC.

Thus, PA (L(B)) = 1 iff all bottom SCCs of MA,B reachable from (v0 , {q0 }) are
accepting, and PA (L(B)) = 0 iff no reachable bottom SCC is accepting (i.e., by
Prop. 3, there is no path in MA ⊗ B from (v0 , {q0 }) to a special node (v, q)).
As with MA and HA  
, let HA,B denote the underlying directed graph of MA,B
.

For the qualitative problem, we only need (1) to construct HA,B and thus only
need to know which nodes and edges are present, and (2) to determine which
pairs (v, q) are special, and hence which bottom SCCs are accepting. Thus we
first have to identify the vertices u of the RMC A for which ne(u) > 0, which
can be done in PSPACE for general RMCs and P-time for 1-exit RMCs and

for Bd-RMCs. Then, the edges of HA,B can be determined by the standard
reachability algorithm for RSMs ([AEY01]). This works by first constructing the
genuine product of the underlying RSM of A (ignoring probabilities on transi-
tions) together with the Büchi automaton B  . This defines a new RSM A ⊗ B 
(no probabilities), whose size is polynomial in A and B  , and thus is exponen-
tial in the original non-deterministic Büchi automaton B. The time required
for reachability analysis for RSMs is polynomial ([AEY01]). Thus, once we have

identified the deficient vertices of the RMC, the rest of the construction of HA,B

takes time polynomial in A and B .
To determine which pairs (v, q) are special, we construct for each candidate

(v, q) the graph D(v, q). For (v, q) with q ∈ F , this is immediate from HA,B . For
(v, q) with q ∈/ F and v = (b, en) a call port of a box b, we test for each return
port v  = (b, ex) of the box and each state q  of B whether there should be an
edge vq → (v  , {q  }); this involves a call to the RSM algorithm of [AEY01] to
determine whether there is a path in the RSM A ⊗ B from (en, q) to (ex, q  )
(with empty stack) that goes through a vertex whose second component is an
accepting state of B. Once we determine these edges, we can construct D(v, q).
This takes time polynomial in A and B  . Then compute the SCCs of D(v, q),
examine the bottom SCCs and check if one of them contains (v, T ) with q ∈ T .
Finally, once we have identified the special pairs, we examine the reachable

bottom SCCs of HA,B and determine which ones are accepting and which are
rejecting. The dependence of the time complexity on the size of the given RMC A
is polynomial except for the identification of the vertices u for which ne(u) > 0.
The dependence on |B| is exponential because of the subset construction. If B
is deterministic to begin with, we avoid the exponential blow-up and thus have
polynomial complexity in B. Thus we have:
Theorem 7. Given RMC A & Büchi automaton B, we can decide whether
PA (L(B)) = 0, PA (L(B)) = 1, or 0 < PA (L(B)) < 1 in PSPACE in A, and EX-
PTIME in B. For a 1-exit RMC or Bd-RMC, the time complexity is polynomial
in |A|. And, if B is deterministic, the time complexity in |B| is also polynomial.

Lower Bounds. We show conversely that the exponential time complexity of

qualitative model checking for a nondeterministic BA is in general unavoidable.
 Algorithmic Verification of Recursive Probabilistic State Machines 267

Theorem 8. Deciding whether a given RMC A satisfies a property specified

by a Büchi automaton B with probability = 1, (i.e., whether PA (L(B)) = 1))
is EXPTIME-complete. Furthermore, this holds even if the RMC is fixed and
each component has 1 entry and 1 exit. Moreover, the qualitative “emptiness”
problem, namely deciding whether PA (L(B)) = 0, is also EXPTIME-complete,
again even when the RMC is fixed and each component has 1 entry and 1 exit.

5 Quantitative Model Checking


As mentioned, the transition probabilities of the chain MA,B cannot be computed
exactly, but instead have to be determined implicitly. To do quantitative model
checking in PSPACE in |A|, it will be crucial to use ExTh(R) to uniquely identify
LFP(P ) for the systems x = P (x). The following key theorem enables this.
Theorem 9. (Unique Fixed Point
Theorem) The equations x = P (x) have a
unique solution q ∗ that satisfies ex q(u,ex)
∗
< 1 for every deficient vertex u, and

∗ ∗
ex q(u,ex) ≤ 1 for every other vertex u. (Of course, q = LFP(P ).)

Theorem 10. Given RMC, A, and BA, B, and a rational value p ∈ [0, 1], we
can decide whether PA (L(B)) ≥ p in PSPACE in |A| and in EXPSPACE in B,
specifically in space O(|A|c1 2c2 |B| ) for some constants c1 , c2 . Furthermore, if B
is deterministic we can decide this in PSPACE in both A and B.
Proof. We make crucial use of Theorem 9, and we combine this with use of

the summary chain MA,B , and queries to ExTh(R). Observe that by Theorem

6, all we need to do is “compute” the probability that a trajectory of MA,B ,
starting from the initial state (v0 , {q0 }) reaches an accepting bottom SCC. We

can not compute MA,B exactly, however, we will be able to identify the transition
probabilities uniquely inside a ExTh(R) query, and will, inside the same query
identify the probability of reaching an accepting bottom SCC.
Let q∗ = LFP(P ) be the solution vector of probabilities for the system x =
P (x) associated with RMC A. Recall that by Proposition 2, we can compute in
PSPACE in |A| the set Q = {u ∈ Q | ne(u) > 0} of deficient vertices. We do
this as a first step. Consider next the following quantifier-free formula, where
c(u) is the index of the component of a vertex u:
   
ϕ1 (x) ≡ x = P (x) ∧ 0  x ∧ x(u,ex) < 1 ∧ x(u,ex) = 1
u∈Q
ex∈Exc(u) u∈Q\Q
ex∈Exc(u)

∗
By Theorem 9, the only vector x in Rn for which ϕ1 (x) holds true
is q . In ∗other
words, ϕ1 uniquely identifies LFP(P ). Recall that ne(u) = 1− ex∈Exc(u) q(u,ex) .
Now,
 vector of variables indexed by vertices of A, and let ϕ2 (x, y) ≡
let y be a

u∈Q yu = 1 − ex∈Exc(u) x(u,ex) . The only vector of reals (x, y) that satisfies
∗
ϕ1 ∧ϕ2 is the one where x(u,ex) = q(u,ex) and yu = ne(u). Recall the construction
 
of MA,B . The states of MA,B are pairs (v, T ), where v ∈ Q , and T ⊆ S is a set

of states of B. The transitions of MA,B come in three varieties.
268 K. Etessami and M. Yannakakis

Case 1: v is not a call port, and (v, pv,v
, v  ) ∈ δMA
. Then we have a corresponding
transition ((v, T ), pv,v
, (v  , R (T, v  ))) ∈ δMA,B

, where pv,v
= pv,v
ne(v  )/ ne(v),
and thus pv,v
ne(v) = pv,v
ne(v  ). Associate a variable  zv,v
with each such prob-

ability pv,v
, and define the formula: ϕ3 (y, z) ≡ (v,v
)∈Case1 zv,v
yv = pv,v
yv
.

Case 2: v is a call port, v = (b, en) where v is vertex in component Ai and

box b is mapped to component Aj , and v  = en, and there is a red transition
(v, pv,v
, v  ) ∈ δMA
. Then there is a red transition ((v, T ), pv,v
, (v  , R (T, v  )) ∈
δMA,B

with the same probability. Here pv,v
= ne(v  )/ ne(v), and thus pv,v
ne(v) =

ne(v ). Associate  a variable zv,v
with each such probability pv,v
, and define:
ϕ4 (y, z) ≡ (v,v
)∈Case2 zv,v
yv = yv
.

Case 3: v is a call port that has a summary transition (v, pv,v
, v  ) in MA to a ver-
tex v  = (b, ex), then we have summary transitions of the form ((v, T ), p , (v  , T  ))

in MA,B to the following set of states of the form (v  , T  ): If there exists a
path of MA that starts at the entry en of Aj and ends at the exit ex (with
empty call stack) which, viewed as a string drives B  from T to T  , then we
include the edge ((v, T ), p(v,T ),(v
,T
) , (v  , T  )) in δMA,B

, where p(v,T ),(v
,T
) =
∗  ∗
q((en,T ),(ex,T
)) · ne(v )/ne(v), and where q((en,T ),(ex,T
)) is the probability of
reaching , (ex, T  ) from , (en, T ) in the product RMC A ⊗ B  . First, com-
pute A ⊗ B  and its associated equations w = P ⊗ (w) explicitly. Note that
|A ⊗ B  | = O(|A||B  |). Let Q⊗ be the set of vertices of A ⊗ B  . We can compute
the set Q⊗ of vertices v of A ⊗ B  , for which ne(v) > 0 in PSPACE in |A ⊗ B  |.
Consider now the quantifier-free formula:
   
ϕ5 (w) ≡ w = P ⊗ (w) ∧ 0  w ∧ w(u,ex) < 1 ∧ w(u,ex) =1
u∈Q
⊗ ex∈Exc(u) u∈Q⊗ \Q
⊗ ex∈Exc(u)

By Theorem 9, LFP(P ⊗ ), is the only vector in Rn for which ϕ5 (w) holds

true. In other words, ϕ5 uniquely identifies LFP(P ⊗ ). Now, associate a variable
z(v,T ),(v
,T
) with each probability p(v,T ),(v
,T
) , where v = (b, en) and v  = (b, ex),

and define: ϕ6 (y, w, z) ≡ ((v,T ),(v
,T
))∈Case3 z(v,T ),(v
,T
) yv = w((en,T ),(ex,T
)) yv
.
6
Observe, j=1 ϕj has a unique solution, and the values of variables z in this
solution identify the probabilities p on transitions of MA,B 
. By the methods of
 
section 4, we compute the underlying graph HA,B of MA,B and compute the

SCCs of HA,B that contain either an accepting node or an accepting edge. Let

us define a revised finite Markov chain, MA,B , in which we remove all SCCs

in MA,B that contain an accepting node or edge, and replace them by a new
absorbing node v ∗ , with a probability 1 transition to itself. Furthermore, in MA,B 

we also remove all nodes that can not reach v ∗ , and all transitions into those

nodes. (Technically, some nodes of MA,B may no longer have full probability on
the transitions leaving them, but that is ok for our purposes.)
Now, recall from Markov chain theory (see, e.g., [Bil95]) that for such a

finite (sub-)Markov chain MA,B , there is a linear system of equations t = F (t),

over variables tu,v∗ , where u is any node of MA,B , and where the coefficients in
 Algorithmic Verification of Recursive Probabilistic State Machines 269

the linear system F (t) are the probabilties p on transitions of MA,B 

such that
the least fixed point solution, LFP(F ), of t = F (t) assigns to variable tu,v∗ the
probability that v ∗ is reachable from u. (In particular, one of the linear equations

is tv∗ ,v∗ = 1.) Moreover, because we have eliminated from MA,B all nodes that
∗
can not reach v , LFP(F ) is the unique solution to this system. Thus consider the
7
formula: ϕ7 (w, t) ≡ t = F (t). Thus the formula j=1 ϕj has a unique solution
in the reals, and the values assigned to variables t(u,v∗ ) in this solution identify

the probability of reaching an accepting SCC from node u in MA,B .
∗ 
For initial node u = (v0 , {q0 }) of MA,B , and p ∈ [0, 1], the following sentence,
7
ψ, is true in R iff PA (L(B)) ≥ p: ψ ≡ ∃x, y, z, w, t j=1 ϕj ∧ tu∗ ,v∗ ≥ p. 


Theorem 11. For a fixed BA, B, given a Bd-RMC, A, and a rational value
p ∈ [0, 1], we can decide whether PA (L(B)) ≥ p in time polynomial in |A|.

Proof. (idea) The proof is a modification of Theorem 10. We extend a technique

developed in [EY05]. We use variables only for entry-exit pairs of A and A ⊗
B  , express all other variables as rational functions of those, and transform the
system to one of polynomial constraints in a bounded number of variables.  

References
[AEY01] R. Alur, K. Etessami, and M. Yannakakis. Analysis of recursive state ma-
chines. In Proc. of 13th Int. Conf. on Computer-Aided Verification, pages
304–313, 2001.
[BGR01] M. Benedikt, P. Godefroid, and T. Reps. Model checking of unrestricted
hierarchical state machines. In Proc. of ICALP’01, volume 2076 of LNCS,
pages 652–666, 2001.
[Bil95] P. Billingsley. Probability and Measure. J. Wiley and Sons, 3rd edition,
1995.
[BKS05] T. Brázdil, A. Kučera, and O. Stražovský. Decidability of temporal prop-
erties of probabilistic pushdown automata. In Proc. of 22nd STACS’05.
Springer, 2005.
[BPR96] S. Basu, R. Pollack, and M. F. Roy. On the combinatorial and algebraic
complexity of quantifier elimination. J. of the ACM, 43(6):1002–1045, 1996.
[BR00] T. Ball and S. Rajamani. Bebop: A symbolic model checker for boolean
programs. In SPIN’2000, volume 1885 of LNCS, pages 113–130, 2000.
[Can88] J. Canny. Some algebraic and geometric computations in PSPACE. In Prof.
of 20th ACM STOC, pages 460–467, 1988.
[CY95] C. Courcoubetis and M. Yannakakis. The complexity of probabilistic veri-
fication. Journal of the ACM, 42(4):857–907, 1995.
[EE04] J. Esparza and K. Etessami. Verifying probabilistic procedural programs.
In Proc. FSTTCS’04, 2004. (Invited survey paper).
[EHRS00] J. Esparza, D. Hansel, P. Rossmanith, and S. Schwoon. Efficient algorithms
for model checking pushdown systems. In 12th CAV, volume 1855, pages
232–247. Springer, 2000.
[EKM04] Javier Esparza, Antonı́n Kučera, and Richard Mayr. Model checking prob-
abilistic pushdown automata. In Proc. of 19th IEEE LICS’04, 2004.
270 K. Etessami and M. Yannakakis

[EY05] K. Etessami and M. Yannakakis. Recursive markov chains, stochastic gram-

mars, and monotone systems of non-linear equations. In Proc. of 22nd
STACS’05. Springer, 2005. (Tech. Report, U. Edinburgh, June 2004).
[GGJ76] M. R. Garey, R. L. Graham, and D. S. Johnson. Some NP-complete geo-
metric problems. In 8th ACM STOC, pages 10–22, 1976.
[Har63] T. E. Harris. The Theory of Branching Processes. Springer-Verlag, 1963.
[Kwi03] M. Kwiatkowska. Model checking for probability and time: from theory to
practice. In Proc. 18th IEEE LICS, pages 351–360, 2003.
[MS99] C. Manning and H. Schütze. Foundations of Statistical Natural Language
Processing. MIT Press, 1999.
[PZ93] A. Pnueli and L. D. Zuck. Probabilistic verification. Inf. and Comp.,
103(1):1–29, 1993.
[Ren92] J. Renegar. On the computational complexity and geometry of the first-
order theory of the reals. parts i,ii, iii. J. of Symbolic Computation, pages
255–352, 1992.
[Tiw92] P. Tiwari. A problem that is easier to solve on the unit-cost algebraic ram.
Journal of Complexity, pages 393–397, 1992.
[Var85] M. Vardi. Automatic verification of probabilistic concurrent finite-state
programs. In Proc. of 26th IEEE FOCS, pages 327–338, 1985.