Stat205B: Probability Theory (Spring 2003) Lecture: 5

Asymptotic Behaviour of Markov Chains (continued)

Lecturer: Jim Pitman Scribe: Vinod Prabhakaran [email protected]

Theorem 5.1. If P is

• irreducible
• positive recurrent (i.e., iTi < ∞ for some or all states i. See [1, p. 307, (4.7)])
• aperiodic (i.e., g.c.d{n : Pn (i, i) > 0}=1)

then
lim Pn (i, j) = π j for all i, j
n→∞

where π j is the unique stationary distribution of P.

Proof. The proof will be done in the next lecture. 

Recall the following fact from last lecture:

Fact. If P is irreducible and recurrent, and i is some fixed state, we defined
T 
Xi 
µi ( j) B i  1(Xn = j)

n=0

This is the expected number of j’s in an i-block. We showed that

µi P = µ i
X
i.e., µi ( j)P( j, k) = µi (k)
j

Next we show that such invariant measures have some uniqueness:

Theorem 5.2. If µ and ν are two invariant measures for the above setup (i.e., µ > 0, ν > 0, µP = µ, νP = ν, and P
irreducible and recurrent), then µ = cν, for some c ≥ 0.

Proof. This proof is based on Exercise 4.6 in [1, p. 306]1 . It uses a duality idea which roughly says “P acting on
left versus right is like reversing time”. (The argument will be given just in the positive recurrent case, but can be
generalized to cover the null-recurrent case too.)
We are considering the equation µ = µP, where µ is a row vector. As we will see presently, it easier to deal with
equations of the form h = Ph, where h is a column vector. Since the row sums of P are one, h ≡ constant solves
h = Ph. In fact, for an irreducible and recurrent P, this is the only possibility as shown below:
1 See [1, p. 305, (4.4)] for another proof.

1
Asymptotic Behaviour of Markov Chains (continued) 2

Claim. For an irreducible and recurrent Markov chain, every non-negative harmonic function is a constant.

Proof of claim: (See also [1, p. 299, Exercise 3.9]) Consider (Xn ), a Markov chain with transition probability matrix
P. It is easy to check that h(Xn ) is a non-negative martingale. Non-negativity implies that this martingale converges
(by Martingale convergence theorem [1, p. 236, (2.11)]). But if ∃ i, j such that h(i) , h( j), then h(X n ) = h(i) i.o. and
h(Xn ) = h( j) i.o.. This is because the chain is irreducible and recurrent and therefore hits all the states infinitely often.
But since h(Xn ) converges, this implies that h(i) = h( j).
Now suppose µ is an invariant probability measure and consider the chain (X 0 , X1 , · · · , Xn ) under Pµ . If we define
µ(y)P(y, x)
P̂(x, y) =
µ(x)
then the reversal (Xn , Xn − 1, · · · , X0 ) is a Markov chain with a homogeneous transition probability matrix P̂. Note that
P̂ is a valid transition probabilty matrix since
P
y µ(y)P(y, x)
X
P̂(x, y) = = 1.
y
µ(x)

Also P̂(x, y) ≥ 0. Note that P̂ is well-defined since P irreducible ⇒ µ(x) > 0 as shown below:
P irreducible ⇒ ∀x, y ∃n : Pn (y, x) > 0. But, µ = µP ⇒ µ = µPn ⇒ µ(x) = y µ(y)Pn (y, x) ≥ µ(y)Pn (y, x). So either µ ≡ 0 or
P

µ(x) > 0, ∀x. But µ is assumed to be a probability measure. So it cannot be identically 0.

Continuing to suppose that µ is a probability measure we can easily verify that the reversed chain is irreducible and
recurrent. If ν is anotherinvariant measure, let h(y) = ν(y)/µ(y). Then ν = νP is the same as h = P̂h. Since h has to be
constant by earlier discussion, µ is unique up to constant multiples.


An example on finding invariant measure:

Example. Consider the following population model: Xn =number of individuals at time n, Xn ∈ {0, 1, · · · }. Between
time n and n + 1, each individual dies with probability p independent of others. There is also an immigration of Y n+1
individuals independent of X1 , X2 , · · · , Xn according to Poission(λ). i.e.,
Xn
X
Xn+1 = Zi + Yn+1 (1)
i=1

where Z1 , Z2 , · · · are independent Bernoulli(p) and P(Yn+1 = k) = e−λ λk /k!, k = 0, 1, · · · .

Notice that (Xn ) is a Markov chain with homogeneous probability transition matrix
P( j, k) = P(Xn+1 = k|Xn = j)
and
P( j, .) =Binomial( j, p)∗Poisson(λ). There is no simpler formula for P( j, k) than a convolution, summing over possible
values of the binomial variable. So the system of equations µP = µ for a stationary vector µ is easily handled. To
understand the long-run behaviour of this chain, first recall a key-fact:
Poisson(λ) ∗ Poisson(µ) = Poisson(λ + µ).
P Xn
Also, if in (1), we assume that Xn ∼ Poisson(λ), then i=1 Zi ∼ Poisson(λp), by the thinning property of the Poisson
d
distribution. So if we try X0 ∼ Poisson(ν), we get X1 ∼ Poisson(νp + λ). In order to have X0 = X1 , we just need
ν = λ/(1 − p). We learn that Poisson(λ/(1 − p)) is a stationary distribution. The chain is obviously irreducible since
P( j, k) > 0 ∀ j, k. Using the fact below we can conclude that we have found a unique stationary probability measure
for this Markov chain.
Asymptotic Behaviour of Markov Chains (continued) 3

Fact. If a Markov chain P is irreducible and it has a stationary probability measure then it is recurrent. Then by
previous discussion the stationary measure is unique.

Pn (y, y) = ∞. Recall also that

P
Proof. Recall y is recurrent if n
X X
Pn (x, y) = Py (T y = ∞) Pn (y, y).
n n

P∞
Let Ny B n=0 1(Xn =y) be the number of hits on y. Then
∞ 
X 
Eµ (Ny ) = Eu  1(Xn =y) 
n=0
∞
X
= Pµ (Xn = y)
n=0
=∞ if µ(y) > 0

But X X
Eµ (Ny ) = µ(x)E x (Ny ) ≤ Ey (Ny ) = Pn (y, y)
x n

where the inequality follows from the fact that µ is a probability measure. 

If π is the unique invariant probabaility measure, then

π(x) = 1/E x (T x )

This follows immediately from

X X
E x (T x ) = E x (number of y’s before T x ) = µ x (y)
y y

But µ x (y) is the invariant measure with µ x (x) = 1. If π is the invariant probability measure
X
µ x (y) = π(y)/π(x) ⇒ E x T x = π(y)/π(x) = 1/π(x).
y

References
[1] R. Durrett. Probability: theory and examples. Duxbury Press, Belmont, CA, second edition, 1996.