1

4 Markov Chains
Introduction Stochastic Processes.
Markov Chains.
Chapman-Kolmogorov Equations
Classification of States
Recurrence and Transience
Limiting Probabilities

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 2

Stochastic Processes

• A stochastic process is a collection of random variables ሼ𝑋 𝑡 , 𝑡 =

0,1, … , 𝑛ሽ. Typically, T is continuous (time) and we have 𝑋 𝑡 , 𝑡 ≥ 0
• Or, T is discrete and we are observing 𝑋𝑛 , 𝑛 = 0,1,2, . . . at discrete
time points n. Refer to X(t) as the state of the process at time t.
• Example: The condition of a machine at the time of the monthly
preventive maintenance is fair, good, or excellent. For month t, the
stochastic process for this situation can be represented as follows:
0 𝑖𝑓 𝑚𝑎𝑐ℎ𝑖𝑛𝑒 𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛 𝑖𝑠 𝑝𝑜𝑜𝑟
𝑋𝑡 = 1 𝑖𝑓 𝑚𝑎𝑐ℎ𝑖𝑛𝑒 𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛 𝑖𝑠 𝑓𝑎𝑖𝑟 𝑡 = 1,2, … , 𝑛
2 𝑖𝑓 𝑚𝑎𝑐ℎ𝑖𝑛𝑒 𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛 𝑖𝑠 𝑔𝑜𝑜𝑑
• The random variable Xt is finite because it represents three states: poor
(0), fair (1), and good (2).

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 3

Markov Chains - Definition

• A Markov chain is a process that present the change from a state
to another state with a given probability.
• A Markov chain is a stochastic process 𝑋𝑡 , 𝑡 = 0,1,2, . . .

• 𝑃 𝑋𝑛+1 = 𝑗ห𝑋𝑛 = 𝑖, 𝑋𝑛−1 = 𝑖𝑛−1 , . . . , 𝑋1 = 𝑖1 , 𝑋0 = 𝑖0

= 𝑃 𝑋𝑛+1 = 𝑗ȁ𝑋𝑛 = 𝑖 is the probbability that state
(n+1) equal j given that previous state n equal to i .
𝑋𝑛+1 depends only on the present state 𝑋𝑛

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 4

Markov Chains - Definition

Denote 𝑃𝑖𝑗 = 𝑃 𝑋𝑛+1 = 𝑗ȁ𝑋𝑛 = 𝑖 , 𝑃𝑖𝑗 ≥ 0 for all 𝑖, 𝑗 as the transition
probability.
Then For any 𝑖, σ𝑎𝑙𝑙𝑗 𝑃𝑖𝑗 = 1 for all states i0, i1,…, in-1 and all n  0 .

Let 𝐏 = 𝑃𝑖𝑗 be the matrix of one-step transition probabilities.

0 1 ... j ... n
0 p00 p01 ... p0j ... p0n
1 p10 p11 ... p1j ... p1n
P= ...
i pi0 pi1 ... pij ... pin
...
n pn0 pn1 ... pnj ... pnn

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 5

Markov Chains - Examples

Example 1: Forecasting the weather if knowing weather one days
𝑋𝑛 is weather of day n =; State space S = 0, 1 ; 𝑃00 = 𝛼, 𝑃10 = 𝛽
𝛼 1−𝛼
Transition matrix is 𝑃 =
𝛽 1−𝛽
Example 2: Estimating machine condition.
Assuming probability to change from one state i to state j is pij
For instance: probability to change from state of fair to state of good is 0.1

State 0 1 2
0 0.5 0.4 0.1
1 0.3 0.6 0.1
2 0.2 0.3 0.5

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 6

Markov Chains - Examples

Example 3: Forecasting the weather if knowing weather two days
• If it rains for the past two days then it will rain tomorrow with probability 0.7
• If it rains today but not yesterday then it will rain tomorrow with probability
0.5
• If it rains yesterday but not today then it will rain tomorrow with probability
0.4
• If it has not rain in the past two days then it will rain tomorrow with
probability 0.2

0 1 2 3
0 0.7 0 0.3 0
States: 0: RR, 1: NR, 2: RN, 3: NN
1 0.5 0 0.5 0
2 0 0.4 0 0.6
3 0 0.2 0 0.8

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 7

Markov Chains - Examples

Example 4: Random Walks

• State Space (S): 0, ±1, ±2, ±3, ±4,….

• Pi, i + 1 = p ; Pi, i - 1 = 1 – p i = 0, 1, …

• At each point of time, either it takes one step to the right with
probability p, or one step to the left with probability 1-p.

… -2 -1 0 1 2 …
S

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 8

Markov Chains - Examples

Example 5: A Gambling Model
wins $1 with 𝑝
Gambler at each play ቊ
loses $1 with 1 − 𝑝

Gambler quits if he goes broke or if he obtains a fortune N.

𝑃𝑖,𝑖+1 = 𝑝; 𝑃𝑖,𝑖−1 = 1 − 𝑝; 𝑖 = 1,2,3, . . . , 𝑁 − 1
𝑃00 = 𝑃𝑁𝑁 = 1: 0 and N are absorbing states

p p p p
1 1
0 1 2 i-1 i i+1 N-1 N

q q q q

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 9

Chapman-Kolmogorov Equations

• n-step Transition Probabilities:

𝑃𝑖𝑗𝑛 = 𝑃ሼ𝑋𝑛+𝑚 = 𝑗ȁ𝑋𝑚 = 𝑖ሽ, 𝑛, 𝑚 ≥ 0, 𝑖, 𝑗 ≥ 0

• Chapman-Kolmogorov Equations
∞

𝑃𝑖𝑗𝑛+𝑚 = ෍ 𝑃𝑖𝑘
𝑛 𝑚
𝑃𝑘𝑗 , 𝑛, 𝑚 ≥ 0, 𝑖, 𝑗 ≥ 0
𝑘=0
• Let P(n) be the matrix of n-step transition probabilities:
• 𝐏 𝑛+𝑚 =𝐏 𝑛 𝐏 𝑚 and 𝐏 𝑛 = 𝐏𝑛

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 10

Chapman-Kolmogorov Equation application

• Given transition matrix P and vector 𝑎(0) = 𝑎𝑗0 , 𝑗 = 1,2, … , 𝑛

Assume after n-step transition, we have 𝑎(𝑛) = 𝑎𝑗𝑛 , 𝑗 = 1,2, … , 𝑛
a(1) = a(0).P;
a(2) = a(1).P = a(0).P.P = a(0).P2
a(3) = a(2).P = a(0).P2.P = a(0).P3
…
a(n) = a(0).Pn
➔ If we want to find a(n) after n steps transition, we should find Pn and
multiply with a(0) .

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 11

Example

.7 .3
• Weather transition probability matrix: 𝑃 =
.4 .6
With: i = 1: it rains; i = 2: it does not rain. Then 4-steps
.5749 .4251
transition matrix is 𝑃4 =
.5668 .4332

• Given Prob. it rains today is α1 = 0.4 and Prob. it does not

rain today is α2 = 0.6. What is probability it will rains after 4
days ? We have 𝛼 0 = (.4, .6)
4 (0) 4 .5749 .4251
• 𝛼 = 𝛼 𝑃 = .4 .6 = .57 .43
.5668 .4332
• What is value of 𝛼 8 ? 𝛼 16 ?

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 12

Classification of States
• State j is accessible from state i if 𝑃𝑖𝑗𝑛 ≥ 0 for some 𝑛 ≥ 0
• If j is accessible from i and i is accessible from j, we say that states i
and j communicate (i  j).
• Communication is a class property:
(i) State i communicates with itself, for all i  0
(ii) If i  j then j  i : communicate is commutative
(iii) If i  j and j  k, then i  k : communicate is transitive
• Therefore, communication divides the state space up into mutually
exclusive classes.
• If all the states communicate, the Markov chain is irreducible.

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 13

Classification of States
An irreducible Markov chain: An reducible Markov chain:

1 2 1 2

0 0

3 4 3 4

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 14

Recurrence vs. Transience

• Let fi be the probability that, starting in state i, the process will ever reenter
state i. If fi = 1, the state is recurrent, otherwise it is transient.
– If state i is recurrent then, starting from state i, the process will reenter
state i infinitely often (w/prob. 1).
– If state i is transient then, starting in state i, the number of periods in
which the process is in state i has a geometric distribution with
parameter 1 – fi.
𝑛 𝑛
• state i is recurrent if σ∞ 𝑃
𝑛=1 𝑖𝑖 = ∞ and transient if σ ∞
𝑛=1 𝑖𝑖 < ∞
𝑃
• Recurrence (transient) is a class property ➔ If i is recurrent (transient) and
i  j then j is recurrent (transient).
• A special case of a recurrent state is if Pii = 1 then i is absorbing.
HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong
 15

Recurrence vs. Transience (2)

• Not all states in a finite Markov chain can be transient.
• All states of a finite irreducible Markov chain are recurrent.
• If state i is recurrent and state i does not communicate with
state j, then 𝑃𝑖𝑗 = 0
– when a process enters a recurrent class of states it can
never leave that class.
– A recurrent class is often called a closed class

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 16

Examples
0 0 0.5 0.5 All states are recurrent
P= 1 0 0 0
0 1 0 0
0 1 0 0

0.5 0.5 0 0 0
Classes: {0,1}, {2,3}- recurrent.
P= 0.5 0.5 0 0 0
0 0 0.5 0.5 0
Class: {4}transient
0 0 0.5 0.5 0
0.25 0.25 0 0 0.5

0 0 0 1
P= 0 0 0 1
0.5 0.5 0 0
Irreducible ➔ All states are recurrent
0 0 1 0

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 17

Limiting Probabilities
• If 𝑃𝑖𝑖𝑛 = 0 whenever n is not divisible by d, and d is the largest integer
with this property, then state i is periodic with period d.
• If a state has period d = 1, then it is aperiodic.
• If state i is recurrent and if, starting in state i, the expected time until
the process returns to state i is finite, it is positive recurrent
(otherwise it is null recurrent).
• A positive recurrent, aperiodic state is called ergodic.

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 18

Limiting Probabilities (2)

Theorem:
• For an irreducible ergodic Markov chain, 𝜋𝑗 = lim 𝑃𝑖𝑗𝑛 exists for all j
𝑛→∞
and is independent of i.
• Furthermore, pj is the unique nonnegative solution of
∞

𝜋𝑗 = ෍ 𝜋𝑖 𝑃𝑖𝑗 , 𝑗 ≥ 0
𝑖=0
∞

෍ 𝜋𝑗 = 1
𝑗=0

• The probability pj also equals the long run proportion of time that the
process is in state j.

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 19

Examples
𝛼 1−𝛼
𝑃=
𝛽 1−𝛽

Limiting probabilities:
𝜋0 = 𝜋0 𝛼 + 𝜋1 𝛽
൞𝜋1 = 𝜋0 1 − 𝛼 + 𝜋1 1 − 𝛽
𝜋0 + 𝜋1 = 1
𝛽 1−𝛼
⇒ 𝜋0 = ; 𝜋1 =
1−𝛼+𝛽 1−𝛼+𝛽

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 20

Limiting Probabilities (3)

• The long run proportions pj are also called stationary probabilities
because if 𝑃 𝑋0 = 𝑗 = 𝜋𝑗 then 𝑃 𝑋𝑛 = 𝑗 = 𝜋𝑗 for all 𝑛, 𝑗 ≥ 0

• Let mjj be the expected number of transitions until the Markov chain,
starting in state j, returns to state j (finite if state j is positive
1
recurrent). Then 𝑚𝑗𝑗 =
𝜋𝑗

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 21

Application: Gambler’s Ruin Problem

• Gambler at each play of the game has prob. p to win one unit and has
prob. q=1-p of losing one unit. Successive plays are independent.
• What is the probability that, starting with i units, the gambler’s
fortune will reach N before going broke?
• Let Xn = player’s fortune at time n:
{Xn ; n = 0,1,2…} is a Markov chain with transition probabilities:
𝑃00 = 𝑃𝑁𝑁 = 1
ቊ
𝑃𝑖,𝑖+1 = 1 − 𝑃𝑖,𝑖−1 = 𝑝𝑖 = 1,2, . . . , 𝑁 − 1

• This Markov chain has three classes:

– {0} and {N} - Recurrent
– {1,2,…,N-1} - Transient
HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong
 22

Application: Gambler’s Ruin Problem (2)

• Let Pi ; 0,1,2,...,N : Prob., starting with i, the gambler reaches N.
• Conditioning on the next game, we have:
𝑞
𝑃𝑖 = 𝑝𝑃𝑖+1 + 𝑞𝑃𝑖−1 𝑜𝑟𝑃𝑖+1 − 𝑃𝑖 = 𝑃𝑖 − 𝑃𝑖−1 𝑓𝑜𝑟𝑖 = 1,2, . . . , 𝑁 − 1
𝑝

• Note that: P0 = 0

⇒ 𝑃𝑖
𝑞 𝑖
1 − ൗ𝑝 𝑞
𝑞 𝑃 1 𝑖𝑓 ൗ𝑝 ≠ 1
= 1 − ൗ𝑝 𝑖 = 2, . . . , 𝑁
𝑞
𝑖𝑃1 𝑖𝑓 ൗ𝑝 = 1

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 23

Application: Gambler’s Ruin Problem (3)

𝑞
1 − ൗ𝑝
𝑁 𝑖𝑓𝑝 ≠ 1ൗ2
𝑞
⇒ 𝑃1 = 1 − ൗ𝑝
• Moreover, PN = 1 1
𝑖𝑓𝑝 = 1ൗ2
𝑁

𝑞 𝑖
1 − ൗ𝑝
𝑖𝑓𝑝 ≠ 1ൗ
𝑞 𝑁 2
⇒ 𝑃𝑖 = 1 − ൗ𝑝 𝑖 = 0,1,2, . . . , 𝑁
𝑖
𝑖𝑓𝑝 = 1ൗ2
𝑁

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 24

Application: Gambler’s Ruin Problem (4)

𝑞 𝑖
1 − ൗ𝑝 𝑖𝑓𝑝 > 1ൗ2
• For N → ∞: 𝑃𝑖 = ൞
0𝑖𝑓𝑝 ≤ 1ൗ2

• For p > 1/2: there is a positive prob. that the gambler’s fortune will
increase indefinitely.
• For p ≤ 1/2: the gambler will “almost certainly” go broke against an
infinitely rich adversary.

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 25

Mean First Passage Time of Recurrent States

• For an ergodic Markov chain, it is possible to go from one state to
another state in a finite number of transitions. Hence:
∞
𝑘
෍ 𝑓𝑖𝑗 =1
𝑘=1
fij(k): Prob. of going from i to j for the first time in exactly k
transitions.
𝑘
• Mean first passage time: 𝜇𝑖𝑗 = σ∞
𝑘=1 𝑘𝑓𝑖𝑗

• Mean first passage time can be found by solving:

∞

𝜇𝑖𝑗 = 1 + ෍ 𝑃𝑖𝑘 𝜇𝑘𝑗

𝑘=0,𝑘≠𝑗

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 26

Example
0 0.5 0.5 0
P= 0 0 1 0
Find the mean first passage time to
0 0.25 0.5 0.25
state 3 from states 0, 1, 2
1 0 0 0

𝜇03 = 1 + 0 × 𝜇03 + 0.5 × 𝜇13 + 0.5 × 𝜇23

ቐ 𝜇13 = 1 + 0 × 𝜇03 + 0 × 𝜇13 + 1 × 𝜇23
𝜇23 = 1 + 0 × 𝜇03 + 0.25 × 𝜇13 + 0.5 × 𝜇23

⇒ 𝜇03 = 6.5; 𝜇13 = 6; 𝜇23 = 5

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 27

• A direct by-product of the steady-state probabilities is the

determination of the expected number of transitions before the
system returns to a state j for the first time.
• This is known as the mean first return time or the mean
recurrence time, and it is computed in an n-state Markov chain
as

1
𝜇𝑖𝑖 = 𝑖 = 1,2, … , 𝑛
𝜋𝑖

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong

 28

Excercises

• Book: Operations Research – An Introduction. Hamdy

Taha, page 642 – 647
• Problems: 1, 2, 4, 9, 10, 11, 13, 18, 23

HCMC University of Technology – Dept. of ISE Assoc. Prof. Ho Thanh Phong