CHAPTER 4

Markov Chains

4.1 States and transitions

The random walk discussed in the previous chapter is a special case of a more general
Markov process1 . Suppose that a random process passes through a discrete sequence
of steps or trials numbered n = 0, 1, 2, . . ., where the outcome of the n-th trial is the
random variable Xn ; X0 is the initial position of the process. This discrete random
variable can take one of the values i = 1, 2, . . . m. The actual outcomes are called
the states of the system, and are denoted by Ei (i = 1, 2, . . . m) (states can be any
consecutive sequence of integers, say starting with i = 0 or some other integer, but
they can be renumbered to start with i = 1). In most but not quite all cases in this
text, we shall investigate systems with a finite number, m, of states E1 , E2 , . . . Em ,
which are independent and exhaustive.
If the random variables Xn−1 = i and Xn = j, then the system has made a
transition Ei → Ej , that is, a transition from state Ei to state Ej at the n-th trial.
Note that i can equal j, so that transitions within the same state may be possible.
We need to assign probabilities to the transitions Ei → Ej . This discrete process is
known as a chain. Generally in chains the probability that Xn = j will depend on the
whole sequence of random variables starting with the initial value X0 . The Markov
chain has the characteristic property that the probability that Xn = j depends only
on the immediate previous state of the system. Formally this means that we need no
further information at each step other than, for each i and j,

P{Xn = j|Xn−1 = i},

which means the probability that Xn = j given that Xn−1 = i: this probability is
independent of the values of Xn−2 , Xn−3 , . . . , X1 . Put alternatively, conditional on
the present state of the chain, its future and present are independent.
In some chains the probabilities P{Xn = j|Xn−1 = i} are functions of n, the
step or trial number. If this is not the case, so that the probabilities are the same at
every step, then the chain is said to be homogeneous.

1 Andrey Markov (1856–1922), Russian mathematician.

65
66 MARKOV CHAINS
4.2 Transition probabilities
For a finite Markov chain with m states E1 , E2 , . . . , Em , introduce the notation
pij = P{Xn = j|Xn−1 = i}, (4.1)
where i, j = 1, 2, . . . , m to represent the probability of a transition from state Ei
to Ej . The numbers pij are known as the transition probabilities of the chain, and
must satisfy
m
pij ≥ 0, pij = 1
j=1
for each i = 1, 2, . . . , m. If pij > 0, then we say that state Ei can communicate with
Ej : two-way communication is possible if additionally pji > 0 (see Section 4.7(a)
later). Obviously for each fixed i, the list {pij } is a probability distribution, since
at any step one of the outcomes E1 , E2 , . . . , Em must occur: the states Ei , (i =
1, 2, . . . m).
Transition probabilities form an m × m array which can be assembled into a tran-
sition matrix T , where
 
p11 p12 . . . p1m
 p21 p22 . . . p2m 
 
T = [pij ] =  . .. . ..  . (4.2)
 . . . . . . 
pm1 pm2 . . . pmm
Note that each row of T is a probability distribution. Any square matrix for which
pij ≥ 0 and m j=1 pij = 1 is said to be row-stochastic.

Example 4.1. The matrices A = [aij ] and B = [bij ] are m × m row-stochastic matrices.
Show that C = AB is also row-stochastic.

By the multiplication rule for matrices

m

C = AB = [aij ][bij ] = aik bkj .

k=1

Hence cij , the general element of C, is given by

m

cij = aik bkj .

k=1

Since aij ≥ 0 and bij ≥ 0 for all i, j = 1, 2, . . . , m, it follows that cij ≥ 0. Also
m m m m m m

cij = aik bkj = aik bkj = aik · 1 = 1,

j=1 j=1 k=1 k=1 j=1 k=1
m m
since j=1 bkj = 1 and k=1 aik = 1.

It follows from this example that any power T n of the transition matrix T must
also be row-stochastic.
TRANSITION PROBABILITIES 67
(n)
(i) The absolute probability pj
One further probability which will be of interest is the probability of outcome Ej
(0) (0)
after n steps, given an initial probability distribution {pi }. Here pi is the proba-
(0)
bility that initially the system occupies state Ei . Of course we must have mi=1 pi =
(1)
1. Let pj be the probability Ej is occupied after one step. Then, by the law of total
probability (see Section 1.3)
m
(1) (0)
pj = pi pij . (4.3)
i=1

We can express this more conveniently in vector form. Let p(0) and p(1) be the
probability (row) vectors given by
(0) (0) (0)
p(0) = p1 p2 ... pm (4.4)
and
(1) (1) (1)
p(1) = p1 p2 ... pm . (4.5)
Here p(0) is the initial distribution, and the components of p(1) will be the probabil-
ities that each of the states E1 , E2 , . . . , Em is reached after one step. Equation (4.3)
can be represented as a matrix product as follows:
m
(1) (1) (0)
p = pj = pi pij = p(0) T,
i=1

where T is the transition matrix given by (4.2). If p(2) is the distribution after two
steps, then
p(2) = p(1) T = p(0) T T = p(0) T 2 .
Hence after n steps by repeating the process
p(n) = p(n−1) T = p(0) T n , (4.6)
where
(n) (n) (n)
p(n) = p1 p2 ... pm . (4.7)
More generally,
p(n+r) = p(r) T n .
(n)
In (4.7), the component pj is the absolute or unconditional probability of out-
(n)
come Ej at the n-th step given the initial distribution p(0) , that is, P{Xn = j}= pj .
Note that
m
(n)
pj = 1.
j=1

Example 4.2. In a three-state Markov chain with states E1 , E2 , E3 , the chain starts in E2 so
that p(0) = 0 1 0 . Find the absolute probability p(3) if the transition matrix is
1 1 1
2 4 4
1 1
T = 0 2 2
.
3 1
4 4
0
68 MARKOV CHAINS

We require
1 1 1 3
7 5 4
2 4 4 1
T3 = 0 1
2
1
2
= 6 6 4 .
3 1 16
4 4
0 6 5 5
Hence,
7 5 4
1 1
p(3) = p(0) T 3 = 0 1 0 6 6 4 = 6 6 4 .
16 16
6 5 5
This result gives the probabilities that, given that the chain starts in E1 , it is in states E1 , E2 , E3
after 3 steps.
(n)
(ii) The n-step transition probability pij
(n)
We now define pij as the probability that the chain is in state Ej after n steps
(1)
given that the chain started in state Ei . The first step transition probabilities pij = pij
are simply the elements of the transition matrix T . We intend to find a formula for
(n)
pij . Now, by definition,
(n)
pij = P(Xn = j|X0 = i),
and also
m
(n)
pij = P(Xn = j, Xn−1 = k|X0 = i)
k=1
for n ≥ 2, since the chain must have passed through one of all the m possible states
at step n − 1.
For any three events A, B, and C, we have available the identity
P(A ∩ B|C) = P(A|B ∩ C)P(B|C)
(see Example 1.4). Interpreting A as Xn = j, B as Xn−1 = k, and C as X0 = i, it
follows that
(n)
pij = P(A ∩ B|C) = P(Xn = j, Xn−1 = k|X0 = i)
m
= P(Xn = j|Xn−1 = k, X0 = i)P(Xn−1 = k|X0 = i)
k=1
m
= P(Xn = j|Xn−1 = k)P(Xn−1 = k|X0 = i)
k=1
m
(1) (n−1)
= pkj pik , (4.8)
k=1

using the Markov property again. These are known as the Chapman–Kolmogorov
equations 2 . Putting n successively equal to 2, 3, . . ., we find that the matrices with
2 Sydney Chapman (1888–1970), British scientist; Andrey Kolmogorov (1908–1987), Russian mathe-
matician.
TRANSITION PROBABILITIES 69
these elements are, using the product rule for matrices,
m
(2) (1) (1)
pij = pik pkj = T 2,
k=1
m
(3) (2) (1)
pij = pik pkj = T 2T = T 3,
k=1
(2)
since pik are the elements of T 2 , and so on. Generalising this rule,
(n)
[pij ] = T n .

Example 4.3. In a certain region the weather patterns have the following sequence. A day is
described as sunny (S) if the sun shines for more than 50% of daylight hours and cloudy (C)
if the sun shines for less than 50% of daylight hours. Data indicate that if it is cloudy one day
then it is equally likely to be cloudy or sunny on the next day; if it is sunny there is a probability
1
3
that it is cloudy and 23 that it is sunny the next day.
(i) Construct the transition matrix T for this process.
(ii) If it is cloudy today, what are the probabilities that it is (a) cloudy, (b) sunny, in three days’
time?
(iii) Compute T 5 and T 10 . How do you think that T n behaves as n → ∞? How does p(n)
behave as n → ∞? Do you expect the limit to depend on p(0) ?

(i) It is assumed that the process is Markov and homogeneous so that transition probabilities
depend only on the state of the weather on the previous day. This is a two-state Markov chain
with states
E1 = (weather cloudy, C), E2 = (weather sunny, S).
The transition probabilities can be represented by the table below which defines the transition
matrix T :

C S
1 1
1 1 or T = 2 2 .
C 2 2
1 2
1 2 3 3
S 3 3
The actual transition probabilities are
p11 = 12 , p12 = 12 , p21 = 13 , p22 = 23 .
(ii) Measuring steps from today, we define
(0) (0)
p(0) = p1 p2 = 1 0
which means that it is cloudy today. In three days’ time,
1 1 3
(3) (0) 3 2 2
p =p T = 1 0 1 2
3 3

29/72 43/72
= 1 0
43/108 65/108
= 29/72 43/72 = 0.403 0.600 .
Hence the probabilities of cloudy or sunny weather in three days’ time are respectively:
70 MARKOV CHAINS
(3)
(a) p1 = 29/72
(3)
(b) p2 = 43/72.
(iii) The computed values of T 5 and T 10 are (to 6 decimal places):

0.400077 0.599923 0.400000 0.600000

T5 = , T 10 = .
0.399949 0.600051 0.400000 0.600000

Powers of matrices can be easily computed using software such as R or Mathematica. It ap-
pears that
0.4 0.6
Tn → = Q,
0.4 0.6
say, as n → ∞. From Eqn (4.6)
p(n) = p(0) T n .
If T n → Q as n → ∞, then we might expect

(0) (0) 0.4 0.6

p(n) → p(0) Q = p1 p2
0.4 0.6
(0) (0) (0) (0)
= (p1 + p2 )0.4 (p1 + p2 )0.6
= 0.4 0.6
(0) (0)
since p1 + p2 = 1. Note that limn→∞ pn is independent of p(0) . The limit indicates that,
in the long run, 40% of days are cloudy and 60% are sunny.

This example indicates that it would be useful if we had a general algebraic for-
mula for the n-th power of a matrix. The algebra required for this aspect of Markov
chains will be looked at in the next two sections.

4.3 General two-state Markov chains

Consider the two-state chain with transition matrix
1−α α
T = , 0 < α, β < 1.
β 1−β
We want to construct a formula for T n . First find the eigenvalues (λ) of T : they are
given by the solutions of the determinant equation det(T − λI2 ) = 0, that is
1−α−λ α
= 0, or (1 − α − λ)(1 − β − λ) − αβ = 0.
β 1−β−λ
Hence λ satisfies the quadratic equation
λ2 − λ(2 − α − β) + 1 − α − β = 0,
or
(λ − 1)(λ − 1 + α + β) = 0. (4.9)
Denote the eigenvalues of T by λ1 = 1, and λ2 = 1 − α − β = s, say. Note that
stochastic matrices always have a unit eigenvalue. We now find the eigenvectors
GENERAL TWO-STATE MARKOV CHAINS 71
associated with each eigenvalue. Let r1 be the (column) eigenvector of λ1 which is
defined by
1 − α − λ1 α
(T − λ1 I2 )r1 = 0, or r1 = 0,
β 1 − β − λ1
or
−α α
r1 = 0.
β −β
Choose any (nonzero) solution of this equation, say
1
r1 = .
1
Note that the eigenvector associated with λ1 = 1 is always a column of 1’s, the result
following from the fact that T is stochastic.
Similarly the second eigenvector r2 satisfies
1 − α − λ2 α β α
r2 = 0, or r2 = 0.
β 1 − β − λ2 β α
In this case we can choose
−α
r2 = .
β
Now form the matrix C which has the eigenvectors r1 and r2 as columns, so that
1 −α
C= r1 r2 = . (4.10)
1 β
Now find the inverse matrix C −1 of C:
1 β α
C −1 = .
α+β −1 1
If we now expand the matrix product C −1 T C, we find that
1 β α 1−α α 1 −α
C −1 T C =
α+β −1 1 β 1−β 1 β
1 β α 1 −αs
=
α+β −1 1 1 βs
1 0 λ1 0
= = = D, (4.11)
0 s 0 λ2
say. Now D is a diagonal matrix with the eigenvalues of T as its diagonal elements:
this process is known in linear algebra as the diagonalization of a matrix. The result
is significant since diagonal matrices are easy to multiply. From (4.21), if we premul-
tiply D by matrix C and post multiply by C −1 , then we find that T = CDC −1 . Thus
T2 = (CDC −1 )(CDC −1 ) = (CD)(C −1 C)(DC −1 )
= (CD)I2 (DC −1 ) = CDDC −1 = CD 2 C −1 ,
72 MARKOV CHAINS
where
λ21 0 1 0
D2 = = .
0 λ22 0 s2
It can be proved by induction that
T n = CD n C −1 , (4.12)
where
λn1 0 1 0
Dn = = .
0 λn2 0 sn
The product of the matrices in (4.12) can be expanded to give
1 1 −α 1 0 β α
T n = CD n C −1 = . (4.13)
α+β 1 β 0 sn −1 1
Since 0 < α, β < 1, it follows that |s| < 1, and consequently that sn → 0 as
n → ∞. Hence, from (4.13),
1 1 −α 1 0 β α 1 β α
Tn → = .
α+β 1 β 0 0 −1 1 α+β β α
Further, for any initial probability distribution p0 , the distribution over the states after
n steps is given by (see Eqn (4.6))
(0) (0)
p(n) = p(0) T n = p1 p2 Tn

(0) (0) 1 β α β α
→ p1 p2 = , (4.14)
α+β β α α+β α+β
as n → ∞, and the limit is independent of p(0) . The limiting distribution in (4.14) is
usually denoted by π. It satisfies
π = πT,
which is an example of an invariant distribution of the Markov chain, since it is
independent of the initial distribution. The chain is said to be in equilibrium.
If α = β = 1, then result (4.13) still holds with s = −1, but T n no longer has a
limit as n → ∞ but oscillates between two matrices. However, limits for T n exist if
α = 1, 0 < β < 1 or 0 < α < 1, β = 1.
The conditions for an invariant distribution in the two-state chain raises the ques-
tion: what are the conditions for the n-state chain? We will return to this later.

4.4 Powers of the general transition matrix

The method derived for the two-state chain in the previous section can be generalized
to m-state chains. We shall sketch the diagonalization method here. Let T be an
m×m stochastic matrix, in other words, a possible transition matrix. The eigenvalues
of T are given by
|T − λIm | = 0, (4.15)
POWERS OF THE GENERAL TRANSITION MATRIX 73
where Im is the identity matrix of order m. Assume that the eigenvalues are distinct,
and denoted by λ1 , λ2 , . . . , λm . Again note that a stochastic matrix T always has a
unit eigenvalue, say λ1 = 1, with a corresponding unit eigenvector
t
r1 = 1 1 ··· 1 .
This follows since every row in [T − Im ]r1 is zero. 3
The corresponding eigenvectors ri satisfy the equations
[T − λi Im ]ri = 0, (i = 1, 2, . . . , m). (4.16)
Construct the matrix
C= r1 r2 ... rm ,
which has the eigenvectors as columns. By matrix multiplication
TC = T r1 r2 ... rm = T r1 T r2 . . . T rm
= λ1 r1 λ2 r2 ... λm rm (by (4.15))
 
λ1 0 ... 0
 0 λ2 ... 0 
 
= r1 r2 ... rm  .. .. .. .. 
 . . . . 
0 0 ... λm
= CD, (4.17)
where D is the diagonal matrix of eigenvalues defined by
 
λ1 0 . . . 0
 0 λ2 . . . 0 
 
D= . . . . . (4.18)
 .. .. .. .. 

0 0 . . . λm
Hence if (4.17) is multiplied on the right by C −1 , then
T = CDC −1 .
Powers of T can now be easily found since
T2 = (CDC −1 )(CDC −1 ) = (CD)(CC −1 )(DC −1 )
= (CD)Im (DC −1 ) = CD 2 C −1 .
Similarly, for the general power n,
T n = CD n C −1 , (4.19)
where  
λn1 0 ... 0
 0 λn2 ... 0 
Dn = 
 
.. .. .. .. .
 . . . . 
0 0 ... λnm
3 The notation At denotes the transpose of any matrix A in which rows and columns are interchanged.
74 MARKOV CHAINS
Although Eqn (4.19) is a useful formula for T n , the algebra involved can be very
heavy even for quite modest chains with m = 4 or 5, particularly if T contains
unknown constants or parameters. On the other hand, mathematical software is now
very easy to apply for the numerical calculation of eigenvalues, eigenvectors, and
powers of matrices for quite large systems.
As we shall see, the behaviour of T n for large n will depend on the eigenvalues
λ1 , λ2 , . . . , λn : a limit may or may not exist.

Example 4.4 Find the eigenvalues and eigenvectors of the stochastic matrix
1 1 1
4 2 4
1 1 1
T = 2 4 4
.
1 1 1
4 4 2

Construct a formula for T n , and find limn→∞ T n .

The eigenvalues of T are given by

1 1 1
4
−λ 2 4
1 1 1
det(T − λI3 ) = 2 4
−λ 4
1 1 1
4 4 2 −λ
1−λ 1−λ 1−λ
1 1 1
= 2 4
−λ 4
(adding all the rows)
1 1 1
4 4 2
−λ
1 1 1
1 1 1
= (1 − λ) 2 4 −λ 4
1 1 1
4 4 2
−λ
1 1 1
1 1 1 1
= (1 − λ) 2 4
−λ 4
( subtracting 4
of row 1 from row 3)
1
0 0 4
−λ
= (1 − λ)( 41 − λ)(− 14 − λ) = 0.

Define the eigenvalues to be

λ1 = 1, λ2 = 14 , λ3 = − 14 .

The eigenvectors ri (i = 1, 2, 3) satisfy

(T − λi I3 )ri = 0, (i = 1, 2, 3).

Some routine calculations give

1 −1 1
r1 = 1 , r2 = −1 , r3 = −1 .
1 2 0
We now let
1 −1 1
C= r1 r2 r3 = 1 −1 −1 .
1 2 0
POWERS OF THE GENERAL TRANSITION MATRIX 75
Finally, from (4.19)
Tn = CDn C −1
1 −1 1 1 0 0 2 2 2
1
= 1 −1 −1 0 ( 14 )n 0 −1 −1 2 .
6
1 2 0 0 0 (− 41 )n 3 −3 0
In this example two eigenvalues have magnitudes less than one so that T n will approach a
limit in which each row is the stationary distribution.
As n → ∞,
1 −1 1 1 0 0 2 2 2
1
Tn → 1 −1 −1 0 0 0 −1 −1 2
6
1 2 0 0 0 0 3 −3 0
1 1 1
3 3 3
1 1 1
= 3 3 3
= Q, say.
1 1 1
3 3 3

Suppose now that T is the transition matrix of a 3-state Markov chain, and that the
initial probability distribution is p(0) . Then the probability distribution after n steps
is
p(n) = p(0) T n .
The invariant probability distribution p is
p = lim p(n) = lim p(0) T n = p(0) Q = 1
3
1
3
1
3 .
n→∞ n→∞

The vector p gives the long-term probability distribution across the three states. In
other words, if any snapshot of the system is eventually taken for large n, then the
system is equally likely (in this example) to lie in each of the states, and this is
independent of the initial distribution p(0) .
This particular example is covered by the Perron–Frobenius theorem4, which
states: if T or T r , for some r, is a positive stochastic matrix ( that is, every element
of T or T r is strictly positive), then aside from the unit eigenvalue, all other eigen-
values have magnitudes less than one. (A proof is given by Grimmett and Stirzaker
(1982).)
Absorbing barriers were referred to in Section 3.1 in the context of random walks.
Absorbing states are recognizable in Markov chains by a value 1 in a diagonal
element of the transition matrix. Since such matrices are stochastic, then all other
elements in the same row must be zero. This means that once entered, there is no
escape from absorbing state. For example, in the Markov chain with
 1 1 1 
2 4 4
T = 0 1 0 , (4.20)
1 1 1
4 4 2
then the state E2 is absorbing.
As we illustrated in Section 3.1, diagrams showing transitions between states are
4 Oskar Perron (1880–1975); Ferdinand Frobenius (1849–1917), German mathematicians.
76 MARKOV CHAINS
particularly helpful. The states are represented by dots with linking directed curves
or edges if a transition is possible: if no transition is possible then no directed edge
is drawn. Thus the transition diagram for the three-state chain with transition matrix
given by Eqn (4.20) is shown in Figure 4.1. In graph theory terminology, Figure 4.1
shows a directed graph. It can be seen that once entered, there is no escape from the
absorbing state E2 .

1-
2

E1

1-
1- 4
4
1-
4

E2 1-
E3 1-
1 4 2

Figure 4.1 Transition diagram for the transition matrix in Eqn (4.20).

The eigenvalues of T given by Eqn (4.20) are λ1 = 1, λ2 = 14 , λ3 = 34 . (The

eigenvalues of T satisfy the conclusions of the Perron–Frobenius theorem above, but
T in this case is not a positive matrix, which indicates a more general version of the
theorem: see the reference above again.) The corresponding matrix of eigenvectors
is  
1 −1 0
C =  1 0 0 .
1 1 1
Using the method illustrated by Example 4.3, it follows that
n −1
Tn = CD
 C   
1 −1 1 1 0 0 0 1 0
1 n 1 1
=  1 0 0   0 (4) 0   −2 0 2

1 1 1 0 0 ( 34 )n 1
2 −1 1
2
   
1 −1 1 1 0 0 0 1 0
→  1 0 0  0 0 0   − 12 0 1 
2
1 1
1 1 1 0 0 0 2
−1 2
 
0 1 0
=  0 1 0  = Q,
0 1 0
say, as n → ∞. This implies that
p = lim p(0) T n = p(0) Q = 0 1 0
n→∞
POWERS OF THE GENERAL TRANSITION MATRIX 77
for any initial distribution p(0) . This means that the system ultimately ends in E2
with probability 1.

Example 4.5 (An illness–death model) A possible simple illness–death model can be repre-
sented by a four-state Markov chain in which E1 is a state in which an individual is free of a
particular disease, E2 is a state in which the individual has the disease, and E3 and E4 are
respectively death states arising as a consequence of death as a result of the disease, or from
other causes. During some appropriate time interval (perhaps an annual cycle), we assign
probabilities to the transition between the states. Suppose that the transition matrix is (in the
order of the states),
 1 1 
2 4
0 14
1 1 1 1
 4 2 8 8 
T = . (4.21)
0 0 1 0 
0 0 0 1
Find the probabilities that a person ultimately dies after a large number of transitions from
the disease given that he or she did not have the disease initially.

As we might expect, this Markov chain has two absorbing states, E3 and E4 . The individual
probabilities can be interpreted as follows: for example, p11 = 12 means that an individual,
given that s/he is free of the disease in a certain period, has probability 12 of remaining free
of the disease; p24 = 18 means that the probability that an individual has the disease but dies
from other causes is 18 , and so on.
The transition diagram is shown in Figure 4.2.
1 1
2 1
4
2
E1 E2

1
4

1 1
4 1
8 8

E4 E3 1
1

Figure 4.2 Transition diagram for the illness–death model.

In this example it is simpler to partition the matrix T as follows. Let

A B
T = ,
O22 I2

where the submatrices5 A and B are given by

1 1 1
2 4
0 4
0 0 1 0
A= 1 1 , B= 1 1 , O22 = , I2 = .
4 2 8 8
0 0 0 1

5 A submatrix of a matrix is a matrix obtained by deleting any collection of rows and columns in the
original matrix.
78 MARKOV CHAINS
Note that A and B are not stochastic matrices. We then observe that
A B A B A2 (A + I2 )B
T2 = = ,
O22 I2 O22 I2 O22 I2

A3 (A2 + A + I2 )B
T3 = ,
O22 I2
and, in general,
An (I2 + A + A2 + · · · + An−1 )B
Tn = .
O22 I2
Now let
Sn = I2 + A + · · · + An−1 .
It then follows that
(I2 − A)Sn = (I2 + A + · · · + An−1 ) − (A + A2 + · · · + An ) = I2 − An ,
so that
Sn = (I2 − A)−1 (I2 − An ).
Hence,
An (I2 − A)−1 (I2 − An )B
Tn = .
O22 I2
In the matrix T , A is not a stochastic matrix but the method of Section 4.3 to find An still
works. The eigenvalues of A are given by
1 1
−λ
det(A − λI2 ) = 2
1 1
4 = (λ − 14 )(λ − 34 ) = 0.
4 2
−λ

Let λ1 = 14 and λ2 = 34 (note that there is not a unit eigenvalue in this case since A is not
row-stochastic). The matrix C of eigenvectors is
−1 1
C= .
1 1
Hence,
1 −1 1 ( 14 )n 0 −1 1
An = .
2 1 1 0 ( 43 )n 1 1
For the submatrix A, An → 0 as n → ∞, so that
 1 5 
0 0 6 6
1 2
O22 (I2 − A)−1 B  0 0 
Tn → = 3 3 = Q,
O22 I2 0 0 1 0 
0 0 0 1
say. In this case
1 (0)
p= 0 0 p
6 1
+ 13 p(0) (0)
2 + p3
5 (0)
p
6 1
+ 23 p(0) (0)
2 + p4
.
This is a limiting distribution which depends on the initial distribution. For example, if
p(0) = 1 0 0 0 , then
1 5
p= 0 0 6 6
,
from which it can be seen that the probability of an individual not having the disease initially
POWERS OF THE GENERAL TRANSITION MATRIX 79
but ultimately dying as a result of the disease is 16 . On the other hand p(0) = 1
2 0 1
2 0
will give the different limit
7 5
p = 0 0 12 12
.

The examples we have considered so far have stochastic matrices which have real
eigenvalues. How do we proceed if the eigenvalues are complex? Consider the fol-
lowing example.

Example 4.6 Find the eigenvalues and eigenvectors of the stochastic matrix
1 1 3
2 8 8
T = 1 0 0 .
1 1 1
4 2 4

Construct a formula for T n , and find limn→∞ T n .

The eigenvalues of T are given by

1 1 3
2
−λ 8 8
det(T − λI3 ) = 1 −λ 0
1 1 1
4 2 4
−λ
1 3
1 8 8
= (1 − λ) 1 −λ 0 (adding all the columns)
1 1
1 2 4
−λ
1 3
0 8
+ λ 8
= (1 − λ) 1 −λ 0 (subtracting row 2 from rows 1 and 3)
1 1
0 2
+λ 4
−λ
1 3
8
+λ 8
= −(1 − λ) 1 1
2 +λ 4 −λ
1 5
= (1 − λ)(λ2 + 4
λ + 32
).
Hence the eigenvalues are
λ1 = 1, λ2 = 18 (−1 + 3i), λ3 = 18 (−1 − 3i),
two of which are complex conjugates. However, we proceed to find the (complex) eigenvectors
which are given, as before, by
(T − λi I3 )ri = 0, (i = 1, 2, 3).
They can be chosen as
1 −7 − 9i −7 + 9i
r1 = 1 , r2 = −16 + 24i , r3 = −16 − 24i .
1 26 26
We now define C by
1 −7 − 9i −7 + 9i
C= r1 r2 r3 = 1 −16 + 24i −16 − 24i .
1 26 26
80 MARKOV CHAINS
Inevitably the algebra becomes heavier for the case of complex roots. The inverse of C is given
by
416 156 208
1
C −1 = −8 + 14i −3 − 11i 11 − 3i .
780
−8 − 14i −3 − 11i 11 + 3i
The diagonal matrix D of eigenvalues becomes
1 0 0 1 0 0
D= 0 ( −1+3i
8
)n 0 → 0 0 0
0 0 ( −1−3i
8
) n
0 0 0
√
as n → ∞ since |(−1 ± 3i)/8| = 10/8 < 1. Finally, T n can be calculated from the formula

T n = CDn C −1 .

As n → ∞,
 8 1 4 
15 5 15
Tn →  8
15
1
5
4
15
 = Q.
8 1 4
15 5 15
We conclude that complex eigenvalues can be dealt with by the same procedure as real
ones: the resulting formulas for T n and its limit Q will turn out to be real matrices.

This example illustrates the Perron–Frobenius theorem quoted previously in this

section. Although T is not positive, T 2 is a positive matrix. It can be confirmed that,
for the complex eigenvalues
√
10
λ1 = 1 |λ2 | = |λ3 | = < 1.
8
We have not considered the cases of repeated eigenvalues. The matrix algebra
required is beyond the scope of this textbook. However, Problems 4.8 and 4.9 at the
end of this chapter suggest three such cases which are worth investigating.

4.5 Gambler’s ruin as a Markov chain

We start by summarizing the game (see Section 2.1). It is a game of chance between
two players A and B. The gambler A starts with k units (pounds or dollars, etc.) and
the opponent B with a − k units, where a and k are integers. At each play, A either
wins from B one unit with probability p or loses one unit to B with probability
q = 1 − p. The game ends when either player A or B has no stake. What is the
probability that A loses?
The states are E0 , E1 , . . . , Ea (it is convenient in this application to let the list
run from 0 to a), where Er is the state in which the gambler A has r units. This is a
Markov chain, but the transitions are only possible between neighbouring states. This
is also the case for the simple random walk. We interpret E0 and Ea as absorbing
states since the game ends when these states are reached. From the rules of the game,
GAMBLER’S RUIN AS A MARKOV CHAIN 81
the transition matrix is
 
1 0 0 ... 0 0 0
 1 − p 0 p ... 0 0 0
 
T =  ... .. .. . . .. .. ..
 
. . . . .  (an (a + 1) × (a + 1) matrix).
. (4.22)
 
 0 0 0 ... 1−p 0 p 
0 0 0 ... 0 0 1
T is an example of a tridiagonal matrix.
The diagonalization of T is not really a practical proposition for this (a + 1)× (a+
1) matrix except for small a. This is often the case for chains with a large number of
states.
The initial distribution vector p(0) for the gambler’s ruin problem has the elements
(0) 0 i = k,
pi = (0 ≤ i ≤ a, 1 ≤ k ≤ a − 1),
1 i = k,
(n)
assuming an initial stake of k units. In the vector p(n) , the component p0 is the
(n)
probability that the gambler loses the game by the nth play, and pa is the probability
that the gambler has won by the nth play.

Example 4.7 In a gambler’s ruin problem suppose that p is the probability that the gambler
wins at each play, and that a = 5 and that the gambler’s initial stake is k = 3 units. Compute
the probability that the gambler loses/wins by the fourth play. What is the probability that the
gambler actually wins at the fourth play?

In this example, the transition matrix Eqn (4.21) is the 6 × 6 matrix

 
1 0 0 0 0 0
 1−p 0 p 0 0 0 
 
 0 1−p 0 p 0 0 
T = ,
 0 0 1−p 0 p 0 
 0 0 0 1−p 0 p 
0 0 0 0 0 1
and
p(0) = 0 0 0 1 0 0 .
Then
p(4) = p(0) T 4
= (1 − p)3 3(1 − p)3 0 5(1 − p)2 p2 0 p2 + 2(1 − p)p3 .
Symbolic computation is helpful for the matrix powers. Alternatively the distribution can be
obtained using probability arguments. The probability that the gambler loses by the fourth play
is
(4)
p0 = (1 − p)3 ,
which can only occur as the probability of three successive losses. The probability that the
gambler wins at the fourth play is
(4)
p5 = p2 + 2(1 − p)p3 ,
82 MARKOV CHAINS
derived from two successive wins, or from the sequences win/lose/win/win or lose/win/win/win
which together give the term 2(1 − p)p3 . The probability that the gambler actually wins at the
fourth play is 2(1 − p)p3 .

For the general (a + 1) × (a + 1) transition matrix (4.22) for the gambler’s ruin,
whilst T n is difficult to obtain, Q = limn→ T n can be found if we make a reasonable
assumption that the limiting matrix is of the form
 
1 0 ... 0 0
 u1
 0 ... 0 1 − u1  
 u2
 0 . . . 0 1 − u 2


Q= . .. . . .. .. ,
 . . . . . . 
 
 ua−1 0 . . . 0 1 − ua−1 
0 0 ... 0 1
where u1 , u2 , . . . , ua−1 are to be determined. The implication of the zeros in columns
2 to a − 1 is that eventually the chain must end in either state E0 or Ea (see Sec-
tion 2.2). We then observe that, after rearranging indices,
Q = lim T n+1 = ( lim T n )T = QT, (4.23)
n→∞ n→∞

and
Q = lim T n+1 = T ( lim T n) = T Q. (4.24)
n→∞ n→∞
In this case (4.23) turns out to be an identity. However, (4.24) implies
0=T Q − Q = (T − Ia+1 )Q
  
0 0 0 ... 0 0 0 1 0 ... 0 0
 1 − p −1 p ... 0 0  0 u1 0 ... 0 1 − u1 
  
 0
 1 − p −1 . . . 0 0 
 0 u2 0 ... 0 1 − u2 

= . .. .. .. .. .. .. .. .. . . .. .. 
 .. . . . . . . . . . . . 
  
 0 0 0 ... 1 − p −1 p  ua−1 0 ... 0 1 − ua−1 
0 0 0 ... 0 0 0 0 0 ... 0 1
 
0 0 ... 0 0
 (1 − p) − u1 + pu2 0 ... 0 −(1 − p) + u1 − pu2 
 
 (1 − p)u1 − u2 + pu3 0 . . . 0 −(1 − p)u1 + u2 − pu3 
 
= .. .. . . .. .. .

 . . . . . 

 (1 − p)ua−2 − ua−1 0 . . . 0 −(1 − p)ua−2 + ua−1 
0 0 ... 0 0
The right-hand side is zero if the elements in the first and last columns are all zero.
However, note that the elements in the last column duplicate the ones in the first
column. The result is the set of difference equations
pu2 − u1 + (1 − p) = 0,
puk+2 − uk+1 + (1 − p)uk = 0, (k = 1, 2, . . . , a − 3),
CLASSIFICATION OF STATES 83
−ua−1 + (1 − p)ua−2 = 0.
This is equivalent to writing
puk+2 − uk+1 + (1 − p)uk = 0, (k = 0, 1, 2, . . . , a − 2),
u0 = 1, ua = 0,
the latter two equations now defining the boundary conditions. These are the dif-
ference equations and boundary conditions for the gambler’s ruin problem derived
using the law of total probability in Eqns (2.2) and (2.3).

4.6 Classification of states

Let us return to the general m-state chain with states E1 , E2 , . . . , Em and transition
matrix
T = [pij ], (1 ≤ i, j ≤ m).
For a homogeneous chain, recollect that pij is the probability that a transition occurs
between Ei and Ej at any step or change of state in the chain. We intend to investigate
and classify some of the more common types of states which can occur in Markov
chains. This will be a brief treatment, using mainly examples of what is an extensive
algebraic subject.

(a) Absorbing state

We have already met one type of state—namely the absorbing state (see Sec-
tion 3.1). Once entered there is no escape from an absorbing state. An absorbing
state Ei is characterized by the probabilities
pii = 1, pij = 0, (j = i, j = 1, 2, . . . m), (4.25)
in the i-th row of T .

(b) Periodic state

(n)
The probability of a return to Ei at step n is pii . Let t be an integer greater than
1. Suppose that
(n)
pii = 0 for n = t, 2t, 3t, . . .
(n)
pii = 0 for n = t, 2t, 3t, . . . .
In this case the state Ei is said to be periodic with period t. If, for a state, no such t
exists with this property, then the state is described as aperiodic.
Let
(n)
d(i) = gcd{n|pii > 0},
(n)
that is, the greatest common divisor (gcd)6 of the set of integers n for which pii > 0.
Then the state Ei is said to be periodic if d(i) > 1 and aperiodic if d(i) = 1.
This definition, which includes the earlier case, covers the appearance of delayed
periodicity in which a sequence of initial terms are zero (see Problem 4.12 for an

6 (n) (n)
gcd{n|pii > 0} is the largest integer which divides all pii > 0 exactly.
84 MARKOV CHAINS
example of such periodic states which satisfy this general definition but not the one
above) .

Example 4.8. A four-state Markov chain has the transition matrix

 1 1 
0 2
0 2
 0 0 1 0 
T = .
1 0 0 0 
0 0 1 0
Show that all states have period 3.

The transition diagram is shown in Figure 4.3, from which it is clear that all states have
period 3. For example, if the chain starts in E1 , then returns to E1 are only possible at steps

1
E1 2 E2

1
1 1
2

E4 E3
1

Figure 4.3 The transition diagram for Example 4.8.

3, 6, 9, . . . either through E2 or E3 .
The analysis of chains with periodic states can be complicated. However, one can check for
a suspected periodicity as follows. By direct computation
 
1 0 0 0
1 1
 0 0 
S = T3 =  2 2 .
0 0 1 0 
1 1
0 2
0 2

In this example,
S 2 = T 6 = SS = S,
so that
S r = T 3r = S, (r = 1, 2, . . .),
which always has nonzero elements on its diagonal. On the other hand,
 1 1   
0 2
0 2
0 0 1 0
 0 0 1 0   1 0 0 0 
S r+1 = SrS =  , S r+2 = Sr S2 =  1 1 ,
1 0 0 0  0 2
0 2
0 0 1 0 1 0 0 0
CLASSIFICATION OF STATES 85
and both these matrices have zero diagonal elements for r = 1, 2, 3, . . .. Hence, for i =
1, 2, 3, 4,
(n)
pii = 0 for n = 3, 6, 9, . . . ,
(n)
pii = 0 for n = 3, 6, 9, . . . ,
which means that all states are period 3. In this example
d(i) = gcd{3, 6, 9, . . .} = 3, for i = 1, 2, 3, 4.

(c) Persistent state

(n)
Let fj be the probability that the first return or visit7 to Ej occurs at the n-th
(n)
step. This probability is not the same as pjj , which is the probability that a return
occurs at the n-th step, and includes possible returns at steps 1, 2, 3, . . . , n − 1 also.
It follows that
(1) (1)
pjj (= pjj ) = fj , (4.26)
(2) (2) (1) (1)
pjj = fj + fj pjj , (4.27)
(3) (3) (1) (2) (2) (1)
pjj = fj + fj pjj + fj pjj , (4.28)
and, in general,
n−1
(n) (n) (r) (n−r)
pjj = fj + fj pjj (n ≥ 2). (4.29)
r=1

Think about the meaning of these equations: for example, (4.27) states that the prob-
ability of a return to Ej at step 2 is the sum of the probability of a first return at step 2
plus the probability of a first return at step 1. The terms in Eqn (4.28) imply that the
probability of a return at the third step is the probability of a first return at the third
step, or the probability of a first return at the first step and a return two steps later, or
the probability of a first return at the second step and a return one step later.
Equations (4.26) and (4.29) become iterative formulas for the sequence of first
(n)
returns fj , which can be expressed as:
(1)
fj = pjj , (4.30)
n−1
(n) (n) (r) (n−r)
fj = pjj − fj pjj (n ≥ 2). (4.31)
r=1
The probability that a chain returns at some step to the state Ej is
∞
(n)
fj = fj .
n=1

If fj = 1, then a return to Ej is certain, and Ej is called a persistent state.

7 We first met first returns in Section 3.4.
86 MARKOV CHAINS
Example 4.9 A three-state Markov chain has the transition matrix
p 1−p 0
T = 0 0 1 ,
1−q 0 q
where 0 < p < 1, 0 < q < 1. Show that the state E1 is persistent.

For simple chains a direct approach using the transition diagram is often easier than the
(n)
formula (4.29) for fj . For this example the transition diagram is shown in Figure 4.4. If a

p
E1

1- p 1- q

q
E2 1 E3

Figure 4.4 The transition diagram for Example 4.9.

sequence starts in E1 , then it can be seen that first returns to E1 can be made to E1 at every
step except for n = 2, since after two steps the chain must be in state E3 . From the figure it
can be argued that
(1) (2) (3)
f1 = p, f1 = 0, f1 = (1 − p) · 1 · (1 − q),
(n)
f1 = (1 − p).1.q n−3 .(1 − q), (n ≥ 4).
(n)
The last result for f1 for n ≥ 4 follows from the following sequence of transitions:
(n−3) times
E1 E2 E3 E3 · · · E3 E1 .
The probability f1 that the system returns at least once to E1 is
∞ ∞
(n)
f1 = f1 = p+ (1 − p)(1 − q)qn−3 ,
n=1 n=3
∞

= p + (1 − p)(1 − q) qs , (s = n − 3)
s=0
(1 − q)
= p + (1 − p) = 1,
(1 − q)
using the sum formula for the geometric series. Hence f1 = 1, and consequently the state E1
is persistent.
CLASSIFICATION OF STATES 87
In fact all states are persistent. Note also that T 3 is a positive matrix.

∞ (n)
The mean recurrence time µj of a persistent state Ej , for which n=1 fj = 1,
is given by
∞
(n)
µj = nfj . (4.32)
n=1
In Example 4.7 above, the state E1 is persistent and its mean recurrence time is given
by
∞ ∞
(n)
µ1 = nf1 = p + (1 − p)(1 − q) nq n−3
n=1 n=3
3 − 2q 3 − 2p − 2q + pq
= p + (1 − p)(1 − q) 2
= ,
(1 − q) 1−q
which is finite. For some chains, however, the mean recurrence time can be infinite.
A persistent state Ej is said to be null if µj = ∞, and nonnull if µj < ∞. All
states in Example 4.7 are nonnull persistent.
To create a simple example of a finite chain with a persistent null state, we consider
a chain in which the transition probabilities depend on the step number n. Null states
are not possible in finite chains with a constant transition matrix, but a proof will not
be given here.

Example 4.10 A three-state inhomogeneous Markov chain has the transition matrix
1 1
2 2
0
Tn = 0 0 1 ,
1/(n + 1) 0 n/(n + 1)
where Tn is the transition matrix at step n. Show that E1 is a persistent null state.

The transition diagram at a general step n is shown in Figure 4.5. From the figure

1
2

E1

1 1/(n+1)
2

n/(n+1)
E2 1 E3

Figure 4.5 The transition diagram for Example 4.10.

88 MARKOV CHAINS
1
(1) (2) (3) 1 1
f1 , =
f1 = 0, f1 = · 1 · ,
2 2 4
(n) 1 3 4 n−1 1 3
f1 = · 1· · ··· · = , (n ≥ 4).
2 4 5 n n+1 2n(n + 1)
Hence,
∞
1 1 3 1
f1 = + + .
2 8 2 n(n + 1)
n=4
Since
1 1 1
= − ,
n(n + 1) n n+1
it follows that
∞ N
1 1 1 1 1 1
= lim − = lim − = .
n(n + 1) N→∞ n n+1 N→∞ 4 N +1 4
n=4 n=4

Hence,
5 3
f1 =
+ = 1,
8 8
which means that E1 is persistent. On the other hand, the mean recurrence time
∞ ∞
(n) 7 3 n
µ1 = nf1 = + ,
8 2 n(n + 1)
n=1 n=4
∞
7 3 1 1 1 7 3 1
= + + + + ··· = + .
8 2 5 6 7 8 2 n
n=5

The series in the previous equation is the harmonic series

∞
1 1 1
= 1+ + +···,
n 2 3
n=1

minus the first four terms. The harmonic series is a well-known divergent series, which means
that µ1 = ∞. Hence E1 is persistent and null.

States can be both persistent and periodic. In the four-state chain with transition
matrix  
0 1 0 0
 0 0 0 1 
T =  0 1 0
,
0 
1
2
0 12 0
all states are period 3, persistent, and non-null.

(d) Transient state

For a persistent state the probability of a first return at some step in the future is
certain. For some states,
∞
(n)
fj = fj < 1, (4.33)
n=1
which means that the probability of a first return is not certain. Such states are de-
scribed as transient.
CLASSIFICATION OF STATES 89
Example 4.11 A four-state Markov chain has the transition matrix
 1 1 1 
0 2 4 4
1 1
 0 0 
T = 2 2 .
0 0 1 0 
1 1
0 0 2 2

Show that E1 is a transient state.

The transition diagram is shown in Figure 4.6. From the figure

(1) (2) 1 1 (3) (n)
f1 = 0, f1 = 2
· 2
= ( 12 )2 , f1 = ( 12 )3 , f1 = ( 12 )n .

1-
2 1-
2
E1 E2

1-
2

1- 1-
4 4
-8

E4 E3
1- 1- 1
2 2

Figure 4.6 The transition diagram for Example 4.11.

Hence,
∞ ∞
(n)
f1 = f1 = ( 12 )n = 1
2
< 1,
n=1 n=2
implying that E1 is a transient state. The reason for the transience of E1 can be seen from
Figure 4.6, where transitions from E3 or E4 to E1 or E2 are not possible.

(e) Ergodic state

An important state which we will return to in the next section is the state which
is persistent, nonnull, and aperiodic. This state is called ergodic8 . Ergodic states are
important in the classification of chains, and in the existence of limiting probability
distributions, as we shall see in the following section.

Example 4.12 In Example 4.9 we considered the three-state Markov chain with transition
matrix
p 1−p 0
T = 0 0 1
1−q 0 q

8 Generally, the word ergodic means that a state will eventually return to a previous state, or put alterna-
tively a sequence or sample in a state is representative of the whole chain.
90 MARKOV CHAINS
where 0 < p < 1, 0 < q < 1. Show that state E1 is ergodic.

It was shown in Example 4.9 that E1 was persistent with

(1) (2) (n)
f1 = p, f1 = 0, f1 = (1 − p)(1 − q)qn−3 , (n ≥ 3).
It follows that its mean recurrence time is
∞ ∞
(n) 3 − 2q
µ1 = nf1 = p + (1 − p)(1 − q) nq n−3 = < ∞.
(1 − q)2
n=1 n=3

(n)
The convergence of µ1 implies that E1 is nonnull. Also the diagonal elements pii > 0 for
n ≥ 3 and i = 1, 2, 3, which means that E1 is aperiodic. Hence from the definition above, E1
(and E2 and E3 also) is ergodic.

4.7 Classification of chains

In the previous section we considered some defining properties of individual states.
In this section we discuss properties of chains.

(a) Irreducible chains

An irreducible chain is one in which every state can be reached or is accessible
from every other state in the chain in a finite number of steps. That any state Ej can
(n)
be reached from any other state Ei means that pij > 0 for some integer n. This is
also referred to as communicating states.
As defined in Section 4.4, a matrix A = [aij ] is said to be positive if aij > 0
for all i, j. A Markov chain with transition matrix T is said to be regular (to avoid
unnecessary repetition of the term positive) if there exists an integer N such that T N
is positive.
A regular chain is obviously irreducible. However, the converse is not necessarily
true, as can be seen from the simple two-state chain with transition matrix
0 1
T = ,
1 0
since

1 0 0 1
T 2n = = I2 and T 2n+1 = = T,
0 1 1 0
for n = 1, 2, 3 . . .. No power of T is a positive matrix.

Example 4.13 Show that the three-state chain with transition matrix
1 1 1
3 3 3
T = 0 0 1
1 0 0
defines a regular (and hence irreducible) chain.
CLASSIFICATION OF CHAINS 91
For the transition matrix T
4 1 4 16 4 7
9 9 9 27 27 27
2 3 1 1 1
T = 1 0 0 , T = 3 3 3
.
1 1 1 4 1 4
3 3 3 9 9 9

Hence T 3 is a positive matrix, which means that the chain is regular.

An important feature of an irreducible chain is that all its states are of the same
type, that is, either all transient or all persistent (null or nonnull), or all have the
same period. A proof of this is given by Feller (1968, p. 391). This means that the
classification of all states in an irreducible chain can be inferred from the known
classification of one state. It is intuitively reasonable to also infer that the states of a
finite irreducible chain cannot all be transient, since it would mean that a return to any
state would not be certain, even though all states are accessible from all other states
in a finite number of steps. This requires a proof which will not be included here.

(b) Closed sets

A Markov chain may contain some states which are transient, some which are
persistent, absorbing states, and so on. The persistent states can be part of closed
subchains. A set of states C in a Markov chain is said to be closed if any state within
C can be reached from any other state within C, and no state outside C can be
reached from any state inside C. Algebraically, a necessary condition for this to be
the case is that
pij = 0 ∀ Ei ∈ C and ∀ Ej ∈ / C. (4.34)
An absorbing state is closed with just one state. Note also that a closed subset is itself
an irreducible subchain of the full Markov chain.

Example 4.14 Discuss the status of each state in the six-state Markov chain with transition
matrix  1 1 
2 2
0 0 0 0
 14 34 0 0 0 0 
 1 1 1 1 
 4 4 4 4 0 0 
T = 1 .
 4 0 14 14 0 14 
 0 0 0 0 1 1 
2 2
0 0 0 0 12 12

A diagram representing the chain is shown in Figure 4.7. As usual the figure is a great
help in settling questions of which sets of states are closed. It can be seen that {E1 , E2 } is a
closed irreducible subchain since no states outside the states can be reached from E1 and E2 .
Similarly {E5 , E6 } is a closed irreducible subchain. The states E3 and E4 are transient. All
states are aperiodic, which means that E1 , E2 , E5 , and E6 are ergodic.

(c) Ergodic chains

As we have seen, all the states in an irreducible chain belong to the same class.
If all states are ergodic, that is, persistent, nonnull, and aperiodic, then the chain is
described as an ergodic chain.
92 MARKOV CHAINS
_1
_1 2 3_
2 E1 E2 4

1_
4 1_
4
1_
4

_1 E6 1_
4 E3 1_
2 1_ 4
4 1_
_1 4
2 1_
_1
4
2

_1 E5 E4 1_
2 4

Figure 4.7 The transition diagram for Example 4.14.

Example 4.15 Show that all states of the chain with transition matrix
1 1 1
3 3 3
T = 0 0 1
1 0 0
are ergodic.

This is the same chain as in Example 4.13, where it was shown to be irreducible and regular,
which means that all states must be persistent, nonnull, and aperiodic. Hence all states are
ergodic.

Example 4.16 Consider the three-state Markov chain with transition matrix
1 4
5 5
0
T = 0 0 1 .
1 0 0
Show that all states are ergodic. Find the eigenvalues of T and Q = limn→∞ T n . Determine
the mean recurrence times µ1 , µ2 , µ3 for each state, and confirm that the rows of Q all have
the elements 1/µ1 , 1/µ2 , 1/µ3 .

It is easy to check that T 4 is a positive matrix, which implies that the chain is ergodic. The
eigenvalues of T are given by
1 4
5
−λ 5
0
det(T − λI3 ) = 0 −λ 1 = −λ3 + 15 λ2 + 4
5
1 0 −λ
1
= 5
(1 − λ)(5λ2 + 4λ + 4) = 0.
Hence the eigenvalues can be denoted by
λ1 = 1, λ2 = − 25 + 45 i, λ3 = − 25 − 45 i.
The corresponding eigenvectors are
1 − 25 + 45 i − 25 − 45 i
r1 = 1 , r2 = − 12 − i , r3 = − 12 + i .
1 1 1
CLASSIFICATION OF CHAINS 93
We use the method of Section 4.4 to find T n . Let
1 − 25 + 45 i − 25 − 45 i
C= r1 r2 r3 = 1 − 12 − i − 12 + i .
1 1 1
The computed inverse is given by
20 16 16
−1 1
C = −10 − 30i −8 + 14i 18 + i .
52
−10 + 30i −8 − 14i 18 − i
As in Example 4.6, it follows similarly that
1 0 0
T =C n
0 (− 25 + 45 i)n 0 C −1 .
0 0 (− 25 − 45 i)n
Hence,
1 0 0 5 4 4
n −1 1
Q = lim T = C 0 0 0 C = 5 4 4 .
n→∞ 13
0 0 0 5 4 4
5 4 4
The invariant distribution is therefore p = 13 13 13 . Note that the elements in p are
−1
the same as the first row in C . Is this always the case for ergodic chains?
(n)
The first returns fi for each of the states can be easily calculated from the transition
diagram Figure 4.8. Thus,
(1) (2) (3)
f1 = 15 , f1 = 0, f1 = 45 .1.1 = 45 ,
1 n−3
f2(1) = f3(1) = 0, f2(2) = f3(2) = 0, f2(n) = f3(n) = 4
5 5
, (n ≥ 3).

_1
5
E1

_4
5
1

E2 E3
1

Figure 4.8 The transition diagram for Example 4.16.

Hence, by Eqn (4.30),

∞ (n) 1
µ1 = n=1
nf1 = 5
+ 3. 45 = 13
5
.

∞ (n) 4 ∞
µ2 = µ3 = n=1 nf2 = 5 n=3 n( 15 )n−3 = 13
4 .
The vector of reciprocals
1 1 1 5 4 4
µ1 µ2 µ3 = 13 13 13
94 MARKOV CHAINS
agrees with the vector p above calculated by the eigenvalue method.

For ergodic chains this is always the case: the invariant distribution is the vector
of mean recurrence time reciprocals, but we shall not give a proof here.

4.8 A wildlife Markov chain model

Consider the progress of a disease in a closed wildlife population in which births are
excluded. This could be, for example, a bird population not in the breeding season.
We model the progress of the disease as a discrete Markov chain over time steps
given by 0, t, 2t, 3t, . . . which may cover various epochs for t (days, weeks, months,
etc.). We start with a population which is susceptible to the disease. The individuals
may become infected, from which they either die or recover and become immune: in
this model individual birds cannot be re-infected.
The original susceptible population may also die due to other causes. Therefore
the population can be divided into four states: E1 of susceptibles; E2 infected; E3
immune; E4 dead. The proposed transition matrix of the model is
 
p11 p12 0 p14
 0 0 p23 p24 
T =  0
. (4.35)
0 p33 p34 
0 0 0 1
The transition diagram is shown in Figure 4.9. It is assumed that within any time
interval of length t, an infected bird either becomes immune or dies.

E1 E2

E4 E3

Figure 4.9 The transition diagram for wild life model.

The entries in Figure 4.9 are easily explained: p11 is the probability that a sus-
ceptible is not infected, p12 that it is infected, p14 that a susceptible dies, and so on.
Multiple events in any time interval are assumed to be negligible.
If we start with only a susceptible population initially, then we choose
(0) (0) (0) (0)
p(0) = p1 p2 p3 p4 = 1 0 0 0 .
A WILDLIFE MARKOV CHAIN MODEL 95
At time nt the probability that a susceptible is in one of the four states is given by
(n) (n) (n) (n)
p(n) = p1 p2 p3 p4 = p(0) T n .

We could use the eigenvalue method of Section 4.4 to obtain a general solution for
T n . The eigenvalues of T are easy to list, namely {p11 , 0, p33 , 1}, since T is an upper
triangular matrix. However, even in this case, the eigenvectors are complicated. It is
easier to compute powers of T directly.
Some plausible data are applied to the model. An actual wildlife application to
house finches in the USA using Markov chains can be found in the paper by Ziplin et
al (2010) and references cited therein. Here we assume the following representative
data for a bird population with a time step t = 1 week:
p11 = 0.79 p12 = 0.20 p13 =0 p14 = 0.01
p21 =0 p22 =0 p23 = 0.90 p24 = 0.10
.
p31 =0 p32 =0 p33 = 0.99 p34 = 0.01
p41 =0 p42 =0 p43 =0 p44 =1
Hence,
 
0.79 0.20 0 0.01
 0 0 0.90 0.10 
T =
 0
.
0 0.99 0.01 
0 0 0 1
We expect from the effect of the absorbing state E4 (and also from the transition
diagram) that
 
0 0 0 1
 0 0 0 1 
Tn →   0 0

0 1 
0 0 0 1
as n → ∞. To cite a particular case, after 10 weeks,
 
0.09 0.02 0.71 0.17
 0 0 0.82 0.18 
p(10) = p(0) T 10 = 1 0 0 0   0

0 0.90 0.10 
0 0 0 1
= 0.09 0.02 0.71 0.17
(aside from rounding errors).
The probability that a susceptible becomes infected in the time interval (m − 1)t
to mt is pm−1
11 p12 , that is, susceptible for m − 1 weeks followed by the probability of
infection in the following week. Since immunity occurs (that is, no return to suscep-
tibility), then the expected duration d12 for a susceptible to become infected is given
by (the upper limit is not achievable)
∞
p12
d12 = mpm−1
11 p12 = = 4.53 weeks.
m=1
(1 − p11 )2
96 MARKOV CHAINS

(n) (n) (n) (n)

Figure 4.10 The figure shows list plots of the probabilities p1 p2 p3 p4 against the
steps n as given by p(n) = p(0) T n . The corresponding states are E1 of susceptibles; E2
infected; E3 immune; E4 dead.

(n) (n) (n) (n)

Figure 4.10 shows how the individual probabilities p1 , p2 p3 p4 develop
with the time steps.

4.9 Problems

4.1. If T = [pij ], (i, j = 1, 2, 3) and

i+j
pij = ,
6 + 3i
show that T is a row-stochastic matrix. What is the probability that a transition between states
E2 and E3 occurs at any step?
If the initial probability distribution in a Markov chain is
p(0) = 1
2
1
4
1
4
,
what are the probabilities that states E1 , E2 , and E3 are occupied after one step? Explain why
the probability that the chain finishes in state E2 is 13 irrespective of the number of steps.

4.2. If  
1 1 1
2 4 4
1 1 1
T = 3 3 3
,
1 1 1
4 2 4
(2) (2) (2)
calculate p22 , p31 , and p13 .

4.3. For the transition matrix

1 2
3 3
T = 1 3
4 4
PROBLEMS 97
(3) (2) 1 1
calculate p12 , p2 and p(3) given that p(0) = 2 2 . Also find the eigenvalues of T ,
construct a formula for T n , and obtain limn→∞ T n .

4.4. Sketch transition diagrams for each of the following three-state Markov chains.
1 1 1 1 1 1 1 1
3 3 3 2 4 4
0 2 2
(a) A = 0 0 1 ; (b) B = 0 1 0 ; (c) C = 1 0 0 .
1 1 1 1 1
1 0 0 2 2
0 3 3 3

4.5. Find the eigenvalues of the stochastic matrix

a b c
T = c a b , (a > 0, b > 0, c > 0 and a + b + c = 1).
b c a
Show that the eigenvalues are complex if b = c. T is an example of a doubly stochastic
matrix: the elements in each row and column sum to 1.
Find the eigenvalues and eigenvectors in the following cases:
(a) a = 12 , b = 14 , c = 14 ;
(b) a = 12 , b = 18 , c = 38 .

4.6. Find the eigenvalues, eigenvectors, the matrix of eigenvectors C, its inverse C −1 , a for-
mula for T n , and limn→∞ T n for each of the following transition matrices:
(a)
1 7
8 8
T = 1 1
;
2 2
(b)  
1 1 3
2 8 8
1 3 3
T = 4 8 8
.
1 5 1
4 8 8

4.7. The weather in a certain region can be characterized as being sunny (S), cloudy (C), or
rainy (R) on any particular day. The probability of any type of weather on one day depends
only on the state of the weather on the previous day. For example, if it is sunny one day then
sun or clouds are equally likely on the next day with no possibility of rain. Explain what other
day-to-day possibilities are if the weather is represented by the transition matrix

S C R
.
1 1
T = S 2 2
0
1 1 1
C 2 4 4
1 1
R 0 2 2
Find the eigenvalues of T and a formula for T n . In the long run what percentage of the days
are sunny, cloudy, and rainy?

4.8. The eigenvalue method of Section 4.4 for finding general powers of stochastic matrices
is only guaranteed to work if the eigenvalues are distinct. Several possibilities occur if the
stochastic matrix of a Markov chain has a repeated eigenvalue. The following three examples
illustrate these possibilities.
98 MARKOV CHAINS
(a) Let
1 1 1
4 4 2
T = 1 0 0
1 1 1
2 4 4
be the transition matrix of a three-state Markov chain. Show that T has the repeated eigenvalue
λ1 = λ2 = − 14 and λ3 = 1, and two distinct eigenvectors

1 1
r1 = −4 r3 = 1 .
1 1
In this case diagonalization of T is not possible. However, it is possible to find a nonsingular
matrix C such that
T = CJ C −1 ,
where J is the Jordan decomposition matrix9 given by
λ1 1 0 − 14 1 0
J= 0 λ1 0 = 0 − 14 0 ,
0 0 1 0 0 1

C= r1 r2 r3 ,
and r2 satisfies
(T − λ1 I3 )r2 = r1 .
Show that we can choose
−10
r2 = 24 .
0
Find a formula for J n and confirm that, as n → ∞,
 12 1 8 
25 5 25
Tn →  12
25
1
5
8
25
.
12 1 8
25 5 25

(b) A four-state Markov chain has the transition matrix

 
1 0 0 0
3 1
 4
0 4
0 
S= 1 3 .
0 4
0 4
0 0 0 1
Sketch the transition diagram for the chain, and note that the chain has two absorbing states
and is therefore not a regular chain. Show that the eigenvalues of S are − 14 , 14 , and 1 repeated.
Show that there are four distinct eigenvectors. Choose the diagonalizing matrix C as
 
0 0 −4 5
 −1 1 −3 4 
C= .
1 1 0 1 
0 0 1 0

9 Camille Jordan (1838–1922), French mathematician.

PROBLEMS 99
Find its inverse, and show that, as n → ∞,
 
1 0 0 0
4 1
 0 0 
Sn →  5
1
5
4 .
5
0 0 5
0 0 0 1
Note that since the rows are not the same this chain does not have an invariant distribution:
this is caused by the presence of two absorbing states.
(c) Show that the transition matrix
1 1
2
0 2
1 1 1
U= 6 3 2
1 5
6
0 6

has a repeated eigenvalue, but that, in this case, three independent eigenvectors can be associ-
ated with U . Find a diagonalizing matrix C, and find a formula for U n using U n = CDn C −1 ,
where
1
3
0 0
D= 0 13 0 .
0 0 1
Confirm also that this chain has an invariant distribution.

4.9. Miscellaneous problems on transition matrices. In each case find the eigenvalues of T , a
formula for T n , and the limit of T n as n → ∞. The special cases discussed in Problem 4.8
can occur.
 
(a) 1 7 9
2 32 32
T = 1 0 0 ;
1 1 1
2 4 4
 1 1 5

(b)
3 4 12
T = 1 0 0 ;
1 1 1
4 4 2
(c)  1 3 9 
4 16 16
3 1
T = 4
0 4
;
1 1 1
4 4 2
(d)  1 1 1 
4 4 2
5 1 1
T = 12 3 4
;
1 1 1
2 4 4
(e)
 
1 0 0 0
1 1
 0 0 
T = 2 2 .
0 0 1 0 
1 1
0 2 2 0

4.10. A four-state Markov chain has the transition matrix

 1 1 
2 2
0 0
 1 0 0 0 
T = 1 1 1 .
4 2
0 4
3 1
4
0 4
0
100 MARKOV CHAINS
Find fi , the probability that the chain returns to state Ei , for each state. Determine which states
are transient and which are persistent. Which states form a closed subset? Find the eigenvalues
of T , and the limiting behavior of T n as n → ∞.

4.11. A six-state Markov chain has the transition matrix

 1 1 1

4 2
0 0 0 4
 0 0 0 0 0 1 
 
 0 14 0 14 12 0 
T = .
 0 0 0 0 1 0 
 0 0 0 1 1 0 
2 2
0 0 1 0 0 0
Sketch its transition diagram. From the diagram, which states do you think are transient and
which do you think are persistent? Which states form a closed subset? Determine the invariant
distribution in the subset.

4.12. Draw the transition diagram for the seven-state Markov chain with transition matrix
 0 1 0 0 0 0 0

 0 0 1 0 0 0 0 
 1 0 0 1
0 0 0 
 2 2 
 0
T = 0 0 0 1 0 0 .

 0 0 0 0 0 1 0 
 1 1 
2
0 0 0 0 0 2
0 0 0 0 0 0 1
Now discuss the periodicity of the states of the chain. From the transition diagram calculate
(n) (n) (3)
p11 and p44 for n = 2, 3, 4, 5, 6. (In this example you should confirm that p11 = 12 but
(3) (3n)
that p44 = 0; however, p44 = 0 for n = 2, 3, . . . confirming that state E4 is periodic with
period 3.)

4.13. The transition matrix of a three-state Markov chain is given by

3 1
0 4 4
1 1
T = 2
0 2
.
3 1
4 4 0
Show that S = T 2 is the transition matrix of a regular chain. Find its eigenvectors and confirm
that S has an invariant distribution given by 14 37
13
37
10
37 .

4.14. An insect is placed in the maze of cells shown in Figure 4.11. The state Ej is the state
in which the insect is in cell j. A transition occurs when the insect moves from one cell to
another. Assuming that exits are equally likely to be chosen where there is a choice, construct
the transition matrix T for the Markov chain representing the movements of the insect. Show
that all states are periodic with period 2. Show that T 2 has two subchains which are both
regular. Find the invariant distributions of both subchains. Interpret the results.

4.15. The transition matrix of a four-state Markov chain is given by

 
1−a a 0 0
 1−b 0 b 0 
T = , (0 < a, b, c < 1).
1−c 0 0 c 
1 0 0 0
PROBLEMS 101

E4 E2

E1

E5 E3

Figure 4.11 Insect in a maze.

(n)
Draw a transition diagram, and, from the diagram, calculate f1 , (n = 1, 2, . . .), the proba-
bility that a first return to state E1 occurs at the n-th step. Calculate also the mean recurrence
time µ1 . What type of state is E1 ?

4.16. Show that the transition matrix

 
1−a a 0 0
 1−a 0 a 0 
T =
1−a 0 0 a 
1 0 0 0
has two imaginary (conjugate) eigenvalues where 0 < a < 1. If a = 12 , confirm that T has the
8 4 2 1
invariant distribution p = 15 15 15 15
.

4.17. A production line consists of two manufacturing stages. At the end of each manufacturing
stage each item in the line is inspected, where there is a probability p that it will be scrapped,
q that it will be sent back to that stage for reworking, and (1 − p − q) that it will be passed
to the next stage or completed. The production line can be modelled by a Markov chain with
four states: E1 , item scrapped; E2 , item completed; E3 , item in first manufacturing stage; E4 ,
item in second manufacturing stage. We define states E1 and E2 to be absorbing states so that
the transition matrix of the chain is
 
1 0 0 0
 0 1 0 0 
T = .
p 0 q 1−p−q 
p 1−p−q 0 q
An item starts along the production line. What is the probability that it is completed in two
stages? Calculate f3(n) and f4(n) . Assuming that 0 < p + q < 1, what kind of states are E3
and E4 ? What is the probability that an item starting along the production line is ultimately
completed?

4.18. The step-dependent transition matrix of Example 4.10 is

1 1
2 2 0
Tn = 0 0 1 , (n = 1, 2, 3, . . .).
1/(n + 1) 0 n/(n + 1)
Find the mean recurrence time for state E3 , and confirm that E3 is a persistent, nonnull state.

4.19. In Example 4.10, a persistent, null state occurred in a chain with step-dependent transi-
tions: such a state cannot occur in a finite chain with a constant transition matrix. However,
102 MARKOV CHAINS
chains over an infinite number of states can have persistent, null states. Consider the following
chain which has an infinite number of states E1 , E2 , . . . with the transition probabilities
1 1 1 j
p11 = , p12 = , pj1 = , pj,j+1 = , (j ≥ 2).
2 2 j +1 j+1
Find the mean recurrence time for E1 , and confirm that E1 is a persistent, null state.

4.20. A random walk takes place on 1, 2, . . . subject to the following rules. A jump from
position i to position 1 occurs with probability qi , and from position i to i + 1 with probability
1 − qi for i = 1, 2, 3, . . ., where 0 < qi < 1. Sketch the transition diagram for the chain.
Explain why to investigate the persistence of every state, only one state, say state 1, need be
considered. Show that the probability that a first return to state 1 occurs at some step is
∞ j−1

f1 = (1 − qk ) qj .
j=1 k=1

If qj = q (j = 1, 2, . . .), show that every state is persistent.

4.21. A Markov chain maze. Figure 4.12 shows a maze with entrance E1 and further gates E2 ,
E3 , E4 , E5 , and E6 with target E7 . These gates can be represented by states in a Markov

E4
E5

E3 E7 E6 E2

E1

Figure 4.12 The maze in Problem 4.21.

chain. At each E1 , . . . , E6 there are two possible new paths which are assumed equally likely
to be chosen. The target E7 can be considered to be an absorbing state. Construct a 7 × 7
transition matrix for the maze assuming that the walker does not return to a previous state and
does not learn from previous choices: for example, at E1 he or she can still make the mistake
of walking the dead-end again. Find the probabilities that the walker reaches the target in
6, 7, 8, . . . steps.
Suppose now that the walker learns from wrong choices. To accommodate this, let E11 be
the entrance and E12 the return to the entrance after a wrong choice (the dead-end); let E21
and E22 have the same meaning for the second entrance, and so on. Hence the probabilities
are:
P(E12 → E12 ) = 12 ; P(E11 → E21 ) = 12 ; P(E12 → E21 ) = 1;
PROBLEMS 103
P(E21 → E22 ) = 12 ; P(E31 → E21 ) = 12 ; P(E22 → E31 ) = 1;
and similarly for the remaining probabilities. The transition matrix is now 13 × 13. with states
E11 , E12 , E21 , E22 , . . . , E61 , E62 , E7 .
Find the probabilities that the walker reaches the centre in 6,7,8 steps. (One really needs a
computer program to compute the matrix products.)

4.22. In a finite Markov chain a subset C of states is said to be closed if, for states i and j,
i ∈ C, then transition i → j ⇒ j ∈ C.
Find the closed subset in a chain with transition matrix
 1 1

0 0 2 2
0 0
1 1 1
 0 0 2
0 4 4

 1 1 1 
 0 0 3 3
0 3 
T = .
 0 0 0 0 0 1 
 1 1 
2 0 0 0 0 2
0 0 1 0 0 0

4.23. Chess knight moves. A knight moves on a reduced chess board with 4 × 4 = 16 squares.
The knight starts at the bottom left-hand corner and moves according to the usual chess rules.
Treating moves as a Markov chain, construct a 16× 16 transition matrix for the knight moving
from any square (easier if you design a computer program for the matrix). Show that the
knight returns to it, starting corner after 2,4,6 moves (must be even) with probabilities 14 , 16 , 54
7
,
(n)
respectively. Find the corresponding first returns. [Just a reminder: if f11 is the probability
that the first return to corner (1, 1) (say) after n moves, and p11 (m) is the probability that the
knight is at (1, 1) after m moves], then
n−1
(1) (n) (n) (m) (n−m)
f11 = p11 , f11 = p11 − f11 p11 , (n ≥ 2).]
m=1