Statistics 3B (STA03B3)

STOCHASTIC PROCESSES

Lecture 4

Chapter 2 → Markov Chains: First Steps

Dr V. van Appel
Department of Statistics
Faculty of Science, University of Johannesburg

1 / 34
Overview

2 Markov Chains: First Steps

2.3 Basic Computations

2 / 34
2.3 Basic Computations

▶ A powerful feature of Markov chains is the ability to use matrix

algebra for computing probabilities.
▶ To use matrix methods, we consider probability distributions as
vectors.
▶ A probability vector is a row vector of non-negative numbers that
sum to 1.
▶ Bold Greek letters, such as α, λ, and π, are used to denote such
vectors.
▶ Assume that X is a discrete random variable with P (X = j) = αj ,
for j = 1, 2, · · · .
▶ Then, α = (α1 , α2 , · · · ) is a probability vector.
▶ We say that the X is α.

3 / 34
▶ For matrix computations we will identify discrete probability
distributions with row vectors.
▶ For a Markov chain X0 , X1 , · · · , the distribution of X0 is called the
initial distribution of the Markov chain.
▶ If α is the initial distribution, then P (X0 = j) = αj , for all j.
▶ For states i and j, and n ≥ 1, P (Xn = j|X0 = i) is the probability
that the chain started in i hits j in n steps.
▶ The n-step transition probabilities can be arranged in a matrix.
▶ The matrix whose ij th entry is P (Xn = j|X0 = i) is the n-step
transition matrix of the Markov chain.

4 / 34
Definition (n-Step Transition Probabilities)
▶ Of course, for n = 1, this is just the usual transition matrix P.
▶ For n ≥ 1, one of the central computational results for Markov
chains is that the n-step transition matrix is precisely Pn , the nth
matrix power of P.

Proof:
To show that P (Xn = j|X0 = i) = (Pn )ij , condition on Xn−1 , which
gives:
X
P (Xn = j|X0 = i) = P (Xn = j|Xn−1 = k, X0 = i)
k
× P (Xn−1 = k|X0 = i)
X
= P (Xn = j|Xn−1 = k)P (Xn−1 = k|X0 = i)
k
X
= Pkj P (Xn−1 = k|X0 = i),
k

where the second equality is by the Markov property, and the third
equality is by time homogeneity.
5 / 34
For n = 2, this gives
X
P (X2 = j|X0 = i) = Pkj P (X1 = k|X0 = i)
k
X
= Pkj Pik
k
= (P2 )ij .

Hence, the two-step transition matrix is P2 . Similarly, for n = 3,

X
P (X3 = j|X0 = i) = Pkj P (X2 = k|X0 = i)
k
X
2
= Pkj Pik
k
= (P3 )ij .

▶ The three-step transition matrix is P3 .

▶ Induction establishes the general result.
6 / 34
Note the Notation: n-Step Transition Probabilities
▶ Let X0 , X1 , . . . be a Markov chain with transition matrix P.
▶ The matrix Pn is the n-step transition matrix of the chain.
▶ Let Pijn denote the conditional probability that the system will be in
state j after exactly n transitions, given that it is presently in state
i. In other words, the n-step transition probability Pijn is defined as
the probability that the chain is in state i and will go to state j in n
steps.
▶ For n ≥ 0,

Pijn = P (Xn = j|X0 = i), for all i, j.

7 / 34
Note that Pijn = (Pn )ij .
▶ Do not confuse this with (Pij )n , which is the number Pij raised to
the nth power.
▶ Also note that P0 is the identity matrix.
▶ That is,

1, if i = j,
P0ij = P (Xn = j|X0 = i) =
0, if i ̸= j.

▶ P1ij = pij

8 / 34
Example (2.14)
For gambler’s ruin, assume that the gambler’s initial stake is $3 and the
gambler plays until either gaining $8 or going bust. At each play the
gambler wins $1, with probability 0.6, or loses $1, with probability 0.4.
Find the gambler’s expected fortune after four plays.

Solution:
The transition matrix is
0 1 2 3 4 5 6 7 8
0 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1 0.40 0.00 0.60 0.00 0.00 0.00 0.00 0.00 0.00
2 0.00
 0.40 0.00 0.60 0.00 0.00 0.00 0.00 0.00
3 0.00 0.00 0.40 0.00 0.60 0.00 0.00 0.00 0.00
 
P = 4 0.00 0.00 0.00 0.40 0.00 0.60 0.00 0.00 0.00.
 
5 0.00 0.00 0.00 0.00 0.40 0.00 0.60 0.00 0.00
 
6 0.00 0.00 0.00 0.00 0.00 0.40 0.00 0.60 0.00
 
7 0.00 0.00 0.00 0.00 0.00 0.00 0.40 0.00 0.60
 
8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00

9 / 34
▶ The one-step transition matrix
Consider the one-step transition matrix given as follows
 
1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.40 0.00 0.60 0.00 0.00 0.00 0.00 0.00 0.00
 
0.00 0.40 0.00 0.60 0.00 0.00 0.00 0.00 0.00
 
0.00 0.00 0.40 0.00 0.60 0.00 0.00 0.00 0.00
 
P= 0.00 0.00 0.00 0.40 0.00 0.60 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.40 0.00 0.60 0.00 0.00
 
0.00 0.00 0.00 0.00 0.00 0.40 0.00 0.60 0.00
 
0.00 0.00 0.00 0.00 0.00 0.00 0.40 0.00 0.60
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00

10 / 34
▶ The two-step transition matrix
The two-step transition matrix, P2 = P.P, is:

1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

 
0.40 0.24 0.00 0.36 0.00 0.00 0.00 0.00 0.00
0.16 0.00 0.48 0.00 0.36 0.00 0.00 0.00 0.00
 
0.00 0.16 0.00 0.48 0.00 0.36 0.00 0.00 0.00
P2 = P.P = 
 
0.00 0.00 0.16 0.00 0.48 0.00 0.36 0.00 0.00

0.00 0.00 0.00 0.16 0.00 0.48 0.00 0.36 0.00
 
0.00 0.00 0.00 0.00 0.16 0.00 0.48 0.00 0.36
0.00 0.00 0.00 0.00 0.00 0.16 0.00 0.24 0.60

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00

11 / 34
▶ The three-step transition matrix
The three-step transition matrix, P3 = P2 .P, is:

1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

 
0.50 0.00 0.29 0.00 0.22 0.00 0.00 0.00 0.00
0.16 0.19 0.00 0.43 0.00 0.22 0.00 0.00 0.00
 
0.06 0.00 0.29 0.00 0.43 0.00 0.22 0.00 0.00
P3 = P2 .P = 
 
0.00 0.06 0.00 0.29 0.00 0.43 0.00 0.22 0.00

0.00 0.00 0.06 0.00 0.29 0.00 0.43 0.00 0.22
 
0.00 0.00 0.00 0.06 0.00 0.29 0.00 0.29 0.36
0.00 0.00 0.00 0.00 0.06 0.00 0.19 0.00 0.74

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00

12 / 34
▶ The four-step transition matrix
The four-step transition matrix, P4 = P3 .P, is:

P4 = P3 .P =

0 1 2 3 4 5 6 7 8
0 1.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
1 0.496 0.115 0.000 0.259 0.000 0.130 0.000 0.000 0.000
2 0.237
 0.000 0.288 0.000 0.346 0.000 0.130 0.000 0.000
3 0.064 0.115 0.000 0.346 0.000 0.346 0.000 0.130 0.000
P4 =
 
4 0.026 0.000 0.154 0.000 0.346 0.000 0.346 0.000 0.130.
5 0.000 0.026 0.000 0.154 0.000 0.346 0.000 0.259 0.216
 
6 0.000 0.000 0.026 0.000 0.154 0.000 0.288 0.000 0.533
 
7 0.000 0.000 0.000 0.026 0.000 0.115 0.000 0.115 0.744
 
8 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 1.000

13 / 34
The gambler’s expected fortune is
8
X
E(X4 |X0 = 3) = jP (X4 = j|X0 = 3)
j=0
8
X
4
= jP3j
j=0
= 0(0.064) + 1(0.115) + 2(0) + 3(0.346) + 4(0)
+ 5(0.346) + 6(0) + 7(0.130) + 8(0)
= $3.79

14 / 34
Exercise
Find the gambler’s expected fortune after
1. three plays i.e., E(X3 |X0 = 2).
2. six plays i.e., E(X6 |X0 = 5).

15 / 34
The following proposition deals with a class of equations called the
Chapman-Kolmogorov equations, which provide a generalization of the
preceding results obtained for the two-step transition probability.
Proposition (Chapman–Kolmogorov Relationship)
For all 0 < r < n,
(n−r)
X
Pijn = r
Pik Pkj .
k

This proposition states that the probability that the process starts
in state i and finds itself in state j at the end of the nth transition
is the product of the probability that the process starts in state i
and finds itself in an intermediate state k after r transitions and
the probability that it goes from state k to state j after additional
n − r transitions.

16 / 34
Proof.

X
Pijn = P [Xn = j|X0 = i] = P [Xn = j, Xr = k|X0 = i]
k
X
= P [Xn = j|Xr = k, X0 = i]P [Xr = k|X0 = i]
k
X
= P [Xn = j|Xr = k]P [Xr = k|X0 = i]
k
(n−r) (n−r)
X X
r r
= Pkj Pik = Pik Pkj
k k

This completes the proof.

It can be shown that Pijn is the ijth entry (i.e., entry in the ith row and
jth column) in the matrix Pn .

17 / 34
Distribution of Xn
▶ In general, a Markov chain X0 , X1 , . . . is not a sequence of
identically distributed random variables.
▶ For n ≥ 1, the marginal distribution of Xn depends on the n-step
transition matrix Pn , as well as the initial distribution α.
▶ To obtain the probability mass function of Xn , condition on the
initial state X0 .
▶ For all j,
X X
P (Xn = j) = P (Xn = j|X0 = i)P (X0 = i) = Pijn αi (1)
i i

▶ The sum in the last expression of Equation (1) can be interpreted in

terms of matrix operations on vectors.

18 / 34
 ▶ It is the dot product of the initial probability vector α with the jth
column of Pn .
▶ That is, it is the jth component of the vector–matrix product αPn .
(Remember: α is a row vector.)

Definition (Distribution of Xn )
▶ Let X0 , X1 , · · · be a Markov chain with transition matrix P and
initial distribution α.
▶ For all n ≥ 0, the distribution of Xn is αPn .
▶ That is,
P (Xn = j) = (αPn )j , for all j.

19 / 34
Example (2.15)
Consider the weather chain introduced in Example 2.3. For tomorrow,
the meteorologist predicts a 50% chance of snow and a 50% chance of
rain. Find the probability that it will snow 2 days later.

Solution As the ordered states of the chain are rain, snow, and
clear, the initial distribution is α = (0.5, 0.5, 0). We have

r s c r s c
r 0.2 0.6 0.2! r 0.12 0.72 0.16!
P= s 0.1 0.8 0.1 and P2 = s 0.11 0.76 0.13 .
c 0.1 0.6 0.3 c 0.11 0.72 0.17

This gives

20 / 34
  
 0.12 0.72 0.16
αP2 = 0.5
  
0.5 0 0.11 0.76 0.13 = 0.115 0.74 0.145
0.11 0.72 0.17

The desired probability of snow is P (X2 = s) = (αP2 )s = 0.74.

Excercise
1. Find the probability that it will rain 2 days later.
Answer:
The desired probability of rain is P (X2 = r) = (αP2 )r = 0.115.
2. Find the probability that it will snow 3 days later.
Answer:
The desired probability of rain is P (X3 = s) = (αP3 )s =?

21 / 34
Present, Future, and Most Recent Past
The Markov property says that past and future are independent given the
present. It is also true that past and future are independent, given the
most recent past.

Markov Property
Let X0 , X1 , · · · be a Markov chain. Then, for all m < n,

P (Xn+1 = j|X0 = i0 , . . . , Xn−m−1 = in−m−1 , Xn−m = i)

= P (Xn+1 = j|Xn−m = i)
= P (Xm+1 = j|X0 = i) = Pijm+1 , (2)

for all i, j, i0 , . . . , in−m−1 , and n ≥ 0.

22 / 34
Proof.
With m = 0, Equation (2) reduces to the defining Markov relationship as
stated in the Equation given in the previous lecture as follows:

P (Xn+1 = j|X0 = x0 , . . . , Xn−1 = xn−1 , Xn = i)

= P (Xn+1 = j|Xn = i)

Let m = 1. By conditioning on Xn , we have

P (Xn+1 = j|X0 = i0 , · · · , Xn−1 = i)
X
= P (Xn+1 = j|X0 = i0 , · · · , Xn−1 = i, Xn = k)
k
×P (Xn = k|X0 = i0 , · · · , Xn−1 = i)
X
= P (Xn+1 = j|Xn = k)P (Xn = k|Xn−1 = i)
k
X
= P (X1 = j|X0 = k)P (X1 = k|X0 = i)
k
X X
= Pkj Pik = Pik Pkj = Pij2 .
k k

23 / 34
The third equality is by time-homogeneity.

The final equation states that the probability of starting in state

i and being in state j at the end of the second transition is the
probability that we first go immediately from state i to an inter-
mediate state k and then immediately from state k to state j; the
summation is taken over all possible intermediate states k.

For general m, induction gives the result by conditioning on Xn−m+1 .

24 / 34
Joint Distribution
▶ The marginal distributions of a Markov chain are determined by the
initial distribution α and the transition matrix P.
▶ In fact, α and P determine all the joint distributions of a Markov
chain, that is, the joint distribution of any finite subset of
X0 , X1 , X2 , . . ..
▶ In that sense, the initial distribution and transition matrix give a
complete probabilistic description of a Markov chain.
▶ To illustrate, consider an arbitrary joint probability, such as

P (X5 = i, X6 = j, X9 = k, X17 = l) = αP5 i Pij P3jk P8kl .

▶ Indeed, conditional probability, the Markov property, and

time-homogeneity give

25 / 34
Joint Distribution
P (X5 = i, X6 = j, X9 = k, X17 = l)

= P (X17 = l|X5 = i, X6 = j, X9 = k)P (X9 = k|X5 = i, X6 = j)

× P (X6 = j|X5 = i)P (X5 = i) → Conditional Probability
= P (X17 = l|X9 = k)P (X9 = k|X6 = j)P (X6 = j|X5 = i)P (X5 = i)
→ the Markov property
= P (X8 = l|X0 = k)P (X3 = k|X0 = j)P (X1 = j|X0 = i)P (X5 = i)
→ time-homogeneity
P8kl P3jk Pij αP5 i →


=
We know that P (Xn = j) = (αPn )j , for all j.

The joint probability is obtained from just the initial distribution α

and the transition matrix P.

26 / 34
For completeness, here is the general formula.

Joint Distribution
▶ Let X0 , X1 , · · · be a Markov chain with transition matrix P
and initial distribution α.
▶ For all 0 ≤ n1 < n2 < · · · < nk−1 < nk and states
i1 , i2 , · · · , ik−1 , ik ,

P (Xn1 = i1 , Xn2 = i2 , . . . , Xnk−1 = ik−1 , Xnk = ik )

= (αP n1 )i1 (P n2 −n1 )i1 i2 · · · (P nk −nk−1 )ik−1 ik (3)

27 / 34
Example (2.16)
Danny’s daily lunch choices are modelled by a Markov chain with
transition matrix
Burrito Falafel Pizza Sushi
Burrito 0.0 0.5 0.5 0.0
 
Falafel  0.5 0.0 0.5 0.0 
P= .
Pizza  0.4 0.0 0.0 0.6 
Sushi 0.0 0.2 0.6 0.2

On Sunday, Danny chooses lunch uniformly at random. Find the

probability that he chooses sushi on the following Wednesday and Friday,
and pizza on Saturday.

Solution
Let b, f, p, s denote Danny’s lunch choices, respectively. Let X0
denote Danny’s lunch choice on Sunday.

28 / 34
The desired probability is

P (X3 = s, X5 = s, X6 = p) = (αP 3 )s Pss

2
Psp

where α = (1/4, 1/4, 1/4, 1/4). We have

b f p s
b 0.45 0.00 0.25 0.30
 
f 0.20 0.25 0.25 0.30
P2 =
p 0.00 0.32 0.56 0.12
s 0.34 0.04 0.22 0.40

b f p s
b 0.100 0.285 0.405 0.210
 
f 0.225 0.160 0.405 0.210
P3 =
p 0.384 0.024 0.232 0.360
s 0.108 0.250 0.430 0.212

29 / 34
The desired probability is

αP3 s P2ss Psp = (0.248)(0.40)(0.60) = 0.05952

30 / 34
31 / 34
Tutorial Exercises
Do the following exercises from [Dobrow, 2016] :

▶ Chapter 1: 1.1; 1.5; 1.6; 1.9; 1.13; 1.22

▶ Chapter 2: 2.1; 2.4; 2.6; 2.9; 2.10

32 / 34
References I

Dobrow, Robert P. (2016). Introduction to Stochastic

Processes with R, John Wiley & Sons, Incorporated.
Ross, Sheldon M. (2014). Introduction to Probability Models,
Elsevier Science & Technology.

33 / 34
Questions?

34 / 34