Chapter 8: Markov Chains1

Yunghsiang S. Han

Graduate Institute of Communication Engineering,

National Taipei University
Taiwan
E-mail: [email protected]

1 Modified from the lecture notes by Prof. Mao-Ching Chiu

Y. S. Han Markov Chains 1

8.1 Markov Processes

• A random process X(t) is a Markov process if the

future of the process given the present is independent
of the past. That is, if for arbitrary times
t1 < t2 < · · · < tk < tk+1 , we have
– For discrete-valued Markov processes
P [X(tk+1 ) = xk+1 |X(tk ) = xk , . . . , X(t1 ) = x1 ]
= P [X(tk+1 ) = xk+1 |X(tk ) = xk ];

– For continuous-valued Markov process

P [a < X(tk+1 ) ≤ b|X(tk ) = xk , . . . , X(t1 ) = x1 ]

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 2

= P [a < X(tk+1 ) ≤ b|X(tk ) = xk ].

• The pdf of a Markov process is given by

fX(tk+1 ) (xk+1 |X(tk ) = xk , . . . , X(t1 ) = x1 )

= fX(tk+1 ) (xk+1 |X(tk ) = xk ).

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 3

Example: Consider the sum process:

Sn = X1 + X2 + · · · + Xn = Sn−1 + Xn ,
where the Xi ’s are an iid sequence. Sn is a Markov process
since
P [Sn+1 = sn+1 |Sn = sn , . . . , S1 = s1 ] = P [Xn+1 = sn+1 − sn ]
= P [Sn+1 = sn+1 |Sn = sn ].

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 4

Example: Consider the moving average of a Bernoulli

sequence:
1
Yn = (Xn + Xn−1 ),
2
where Xi are independent Bernoulli sequence with p = 1/2.
We show that Yn is not a Markov process. The pmf of Yn is

P [Yn = 0] = P [Xn = 0, Xn−1 = 0] = 1/4,

P [Yn = 1/2] = P [Xn = 0, Xn−1 = 1] + P [Xn = 1, Xn−1 = 0]

= 1/2

and
P [Yn = 1] = P [Xn = 1, Xn−1 = 1] = 1/4.

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 5

Now consider
P [Yn = 1, Yn−1 = 1/2]
P [Yn = 1|Yn−1 = 1/2] =
P [Yn−1 = 1/2]
P [Xn = 1, Xn−1 = 1, Xn−2 = 0]
=
1/2
(1/2)3
= = 1/4.
1/2
Suppose that we have additional knowledge about past,
then

P [Yn = 1|Yn−1 = 1/2, Yn−2 = 1]

P [Yn = 1, Yn−1 = 1/2, Yn−2 = 1]
= = 0.
P [Yn−1 = 1/2, Yn−2 = 1]

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 6

Thus

P [Yn = 1|Yn−1 = 1/2] 6= P [Yn = 1|Yn−1 = 1/2, Yn−2 = 1].

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 7

• A integer-valued Markov random process is called a

Markov chain.
• If X(t) is a Markov chain for t3 > t2 > t1 , then we have
P [X(t3 ) = x3 , X(t2 ) = x2 , X(t1 ) = x1 ]
= P [X(t3 ) = x3 |X(t2 ) = x2 ]P [X(t2 ) = x2 |X(t1 ) = x1 ]P [X(t1 ) = x1 ].

• In general,
P [X(tk+1 ) = xk+1 , X(tk ) = xk , . . . , X(t1 ) = x1 ]
= P [X(tk+1 ) = xk+1 |X(tk ) = xk ]P [X(tk ) = xk |X(tk−1 ) = xk−1 ] · · ·
×P [X(t2 ) = x2 |X(t1 ) = x1 ]P [X(t1 ) = x1 ].

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 8

8.2 DISCRETE-TIME MARKOV CHAIN

• Let Xn be a discrete-time integer-valued Markov chain

that starts at n = 0 with pmf
pj (0) = P [X0 = j], j = 0, 1, 2, . . . .
• The joint pmf of the first n + 1 values is
P [Xn = in , . . . , X0 = i0 ]
= P [Xn = in |Xn−1 = in−1 ] · · · P [X1 = i1 |X0 = i0 ]P [X0 = i0 ].

• Assume that the one-step state transition probabilities

are fixed and do not change with time (homogeneous

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 9

transition probability), that is,

P [Xn+1 = j|Xn = i] = pij for all n.

• The joint pmf for Xn , Xn−1 , . . . , X0 is then given by

P [Xn = in , . . . , X0 = i0 ] = pin−1 ,in · · · pi0 ,i1 pi0 (0).

• Xn is completely specified by the initial pmf pi (0) and

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 10

the matrix of one-step transition probabilities P :

 
p00 p01 p02 · · ·
 
 p p p · · · 
 10 11 12 
 . . . .
.. .. .. .. 

P =  
 
 pi0 pi1 pi2 · · · 
.. .. .. ..
 
. . . .

• P is called transition probability matrix.

• Each row of P must add to one since
X X
1= P [Xn+1 = j|Xn = i] = pij .
j j

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 11

Example: A Markov model for speech:

• Two states: silence and speech activity

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 12

Example: Let Sn be the binomial counting process. In

one step, Sn can either stay the same or increase by one.
The transition probability can be given by
 
1−p p 0 0 ···
 
 0 1−p p 0 ··· 
P = .
 
 0 0 1 − p p · · · 
.. .. .. ..
 
. . . .

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 13

The n-step transition probabilities

• Let P (n) = {pij (n)} be the matrix of n-step transition

probabilities, where

pij (n) = P [Xn+k = j|Xk = i] n ≥ 0, i, j ≥ 0.

• Since transition probabilities do not depend on time,

we have

P [Xn+k = j|Xk = i] = P [Xn = j|X0 = i].

• Consider the two-step transition probabilities:

P [X2 = j, X1 = k, X0 = i]
P [X2 = j, X1 = k|X0 = i] =
P [X0 = i]

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 14

P [X2 = j|X1 = k]P [X1 = k|X0 = i]P [X0 = i]

=
P [X0 = i]
= P [X2 = j|X1 = k]P [X1 = k|X0 = i]
= pik (1)pkj (1).

• 2-step transition probabilities are given by

pij (2) = P [X2 = j|X0 = i]

X
= P [X2 = j, X1 = k|X0 = i]
k
X
= pik (1)pkj (1),
k

• Therefore,
P (2) = P (1)P (1) = P 2 .

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 15

• In general, we have

P (n) = P n .

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 16

State Probabilities

• Let p(n) denote the row vector of state probabilities at

time n. The probability pj (n) is related to p(n − 1) by
X
pj (n) = P [Xn = j|Xn−1 = i]P [Xn−1 = i]
i
X
= pij pi (n − 1).
i

• In matrix notation we have

p(n) = p(n − 1)P.

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 17

• pj (n) is related to p(0) by

X
pj (n) = P [Xn = j|X0 = i]P [X0 = i]
i
X
= pij (n)pi (0).
i

• In matrix notation we have

p(n) = p(0)P (n) = p(0)P n .

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 18

Example: Let α = 1/10 and β = 1/5 for the following

Markov chain:

Find P (n) for n = 2 and 4.

Sol: " #2 " #
2 0.9 0.1 0.83 0.17
P = =
0.2 0.8 0.34 0.66

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 19

and
" #2 " #
4 0.83 0.17 0.7467 0.2533
P = = .
0.34 0.66 0.5066 0.4934

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 20

Steady State Probabilities

• As n → ∞, the n-step transition probability matrix

approaches a matrix in which all the rows are equal to
the same pmf
pij (n) → πj for all i.

• As n → ∞
X X
pj (n) = pij pi (0) → πj pi (0) = πj .
i i

• As n becomes large, the probability of state j

approaches a constant independent of time and the
initial state probabilities (equilibrium or steady state).

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 21

• Let the pmf π = {πj }. By noting that as n → ∞,

pj (n) → πj and pi (n − 1) → πi , we have
X
πj = pij πi ,
i

which in matrix notation is

π = πP (n − 1 linearly independent equations).

• The additional equation needed is provided by

X
πi = 1.
i

• π is called the stationary state pmf of the Markov

chain.

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 22

• If we start with p(0) = π, then

p(n) = πP n = π − a stationary process.

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 23

Example: Find the stationary state pmf for the following

Markov chain:

Sol: we have

π0 = (1 − α)π0 + βπi
π1 = απ0 + (1 − β)π1 .

Graduate Institute of Communication Engineering, National Taipei University

Y. S. Han Markov Chains 24

Since π0 + π1 = 1,

απ0 = βπ1 = β(1 − π0 ).

Thus.
β α
π0 = , π1 = .
α+β α+β

Graduate Institute of Communication Engineering, National Taipei University