Markov Chain

1 / 68
History

▶ Proposed by the Russian mathematician Markov in 1907

▶ Wide applications
▶ Credit rating
▶ Random walk
▶ Every type of music can be encoded into a Markov chain
▶ Text generation: Markov chains can be used to generate
sentences in a given language → Natural Language
Processing
▶ Logistics, supply chain management, and waiting queues,
▶ Modeling of insurance claims,
▶ Board games, e.g. Snakes and Ladders,
▶ Artificial intelligence, learning theory and machine learning.

2 / 68
Transition Probability

▶ A random process {Xn , n = 0, 1, 2 . . . }

▶ State space = {0, 1, 2, . . . }
▶ If Xn = i say that the process is in state i at time n
▶ Transition probability

X0 → X1 → · · · → Xn−1 → Xn → Xn+1 → . . .

P (Xn+1 = j|Xn = i) = Pij

3 / 68
Example

Binomial asset pricing model p(H) = 2/3 and p(T ) = 1/3

P (S1 = 8|S0 = 4) = P (S2 = 8|S1 = 4) = P48 = 2/3

4 / 68
Question

Compare
P (S3 = 8|S0 = 4, S1 = 8, S2 = 4)
P (S3 = 8|S0 = 4, S1 = 2, S2 = 4)
and
P (S3 = 8|S2 = 4)

5 / 68
Question

Compare
P (S3 = 8|S0 = 4, S1 = 8, S2 = 4)
P (S3 = 8|S0 = 4, S1 = 2, S2 = 4)
and
P (S3 = 8|S2 = 4)
Markov property

5 / 68
Markov Property - Memoryless property

Given current state, the past does not matter

P (Xn = j|X0 = i0 , X1 = i1 , . . . , Xn−1 = i)

=P (Xn = j|Xn−1 = i)

6 / 68
Markov Chain

▶ A Markov chain is a random process with Markov property

7 / 68
Markov Chain

▶ A Markov chain is a random process with Markov property

▶ Model specification
▶ identify all possible states
▶ identify the possible transition
▶ identify the transition probability

7 / 68
Transition matrix
Transition probability

Pij = P (Xn+1 = j|Xn = i)

independent of n

8 / 68
Transition matrix
Transition probability

Pij = P (Xn+1 = j|Xn = i)

independent of n

Transition matrix
To
1 2 ...
 
1 P11 P12 . . .
From

..  .. .. .. 
.  . . . 
i  Pi1 Pi2 . . . 
.. .. .. ..
 
. . . .

Index in row: current state (from)

Index in column: next/future state (to)
8 / 68
Example

A fly moves along a straight line in unit increments. At each

time period, it moves one unit to the left with probability 0.3,
one unit to the right with probability 0.3, and stays in place with
probability 0.4, independently of the past history of movements.
A spider is lurking at positions 1 and 4: if the fly lands there, it is
captured by the spider, and the process terminates. Suppose
that the fly starts at position 2. Construct the Markov chain
model.

9 / 68
 0.4 0.4
0.3
0.3 0.3
1 1 2 3 4 1

0.3

10 / 68
 0.4 0.4
0.3
0.3 0.3
1 1 2 3 4 1

0.3
Sample episode starting from 2:
.3 1 1
▶ 2−
→1→− 1→
− 1
.3 .3 1
▶ 2−
→3−→4→− 4

10 / 68
 0.4 0.4
0.3
0.3 0.3
1 1 2 3 4 1

0.3
Sample episode starting from 2:
.3 1 1
▶ 2−
→1→− 1→
− 1
.3 .3 1
▶ 2−
→3−→4→− 4
.3 .4 .3 .4 .3 .3
▶ 2−
→3−→3−→2−→2−→3−→4

10 / 68
 0.4 0.4
0.3
0.3 0.3
1 1 2 3 4 1

0.3

1 and 4 are absorbing state that once entered, cannot left

11 / 68
Solution
▶ All possible states: 1, 2, 3, 4

12 / 68
Solution
▶ All possible states: 1, 2, 3, 4
▶ Transition probability

12 / 68
Solution
▶ All possible states: 1, 2, 3, 4
▶ Transition probability
▶ p11 = 1 , p44 = 1

12 / 68
Solution
▶ All possible states: 1, 2, 3, 4
▶ Transition probability
▶ p11 = 1 , p44 = 1
▶

0.3
 if j = i + 1
pij = 0.4 if j = i

0.3 if j = i − 1


for i = 2, 3, ..., m − 1

12 / 68
Solution
▶ All possible states: 1, 2, 3, 4
▶ Transition probability
▶ p11 = 1 , p44 = 1
▶

0.3
 if j = i + 1
pij = 0.4 if j = i

0.3 if j = i − 1


for i = 2, 3, ..., m − 1

12 / 68
Example: Weather forecast

Suppose that the chance of rain tomorrow depends on previous

weather conditions only through whether or not it is raining
today and not on past weather conditions. Suppose also that if
it rains today, then it will rain tomorrow with probability .7; and if
it does not rain today, then it will rain tomorrow with probability
.6. Find a Markov chain that modeling the system.

13 / 68
Solution

0.4
0.7

0.6
rain not rain

0.3

▶ State: 1 if rain, 2 if no rain

▶ Transition matrix
Next
Rain No rain
Current

 
Rain .7 .3
No rain .6 .4

14 / 68
Example
Consider a binomial asset pricing model with p(H) = 2/3 and
p(T ) = 1/3, S0 = 4, u = 1/d = 2. Construct the Markov chain
model.

15 / 68
Example
Consider a binomial asset pricing model with p(H) = 2/3 and
p(T ) = 1/3, S0 = 4, u = 1/d = 2. Construct the Markov chain
model.

Solution

P2k ,2k+1 = 2/3

and
P2k ,2k−1 = 1/3
for k = ±1, ±2, . . .

15 / 68
Markov chain for gambler’s ruin

▶ In each round, a player either wins $1, with probability p, or

loses $1, with probability 1 − p.

16 / 68
Markov chain for gambler’s ruin

▶ In each round, a player either wins $1, with probability p, or

loses $1, with probability 1 − p.
▶ The gambler starts with k. The game stops when the
player either loses all their money, or gains a total of n
dollars.

16 / 68
Markov chain for gambler’s ruin

▶ In each round, a player either wins $1, with probability p, or

loses $1, with probability 1 − p.
▶ The gambler starts with k. The game stops when the
player either loses all their money, or gains a total of n
dollars.
▶ Xi : the amount of money at i-th round.

16 / 68
Markov chain for gambler’s ruin

▶ In each round, a player either wins $1, with probability p, or

loses $1, with probability 1 − p.
▶ The gambler starts with k. The game stops when the
player either loses all their money, or gains a total of n
dollars.
▶ Xi : the amount of money at i-th round.
▶ (Xi )i=1,2,.. forms a Markov chain?

16 / 68
Markov chain for gambler’s ruin

▶ In each round, a player either wins $1, with probability p, or

loses $1, with probability 1 − p.
▶ The gambler starts with k. The game stops when the
player either loses all their money, or gains a total of n
dollars.
▶ Xi : the amount of money at i-th round.
▶ (Xi )i=1,2,.. forms a Markov chain?



 p if j = i + 1, 0 < i < n,

1 − p if j = i − 1, 0 < i < n,
▶ Transition matrix: Pij =
1 if i = j = 0, or i = j = n,



0, otherwise.

16 / 68
Example - Credit rating

Transition from Rating at the start of a year to Rating at the end

of the year

17 / 68
n-steps transition probability

Given that the Markov chain (Xn ) starts at initial state i, want to
know probability that it will be in state j after n steps

rij (n) = P (Xn = j|X0 = i)

Remark

rij (1) = pij

18 / 68
Chapman-Kolmogorov Equation for n-step transition probability
Key recursion
X
rij (n) = rik (n − 1)pkj
k

starting with
rij (1) = pij

Time 0 Time n-1 Time n

1
ri1(n-1) p 1j

...
i k
rik(n-1)
p kj j
...

rim(n-1)
p mj
m
19 / 68
Proof

Time 0 Time n-1 Time n

All the path that start from i and
1 visit j after n steps can be
ri1(n-1) p 1j
...

divided into some subsets

i k based on the state that it visits
rik(n-1)
p kj j
at time n − 1
...

rim(n-1)
p mj
m

20 / 68
Time 0 Time n-1 Time n

1
Case 1: Starting from state i, it
ri1(n-1) p 1j
...
visits state 1 at time n − 1 and
k
in the last transition, it moves
i
rik(n-1)
p kj j from state 1 to state j at time n
...

rim(n-1)
p mj
m

21 / 68
Case 2:
(n−1)steps last step
state i −−−−−−−→ state 2 −−−−−→ state j

...
Case k:
(n−1)steps last step
state i −−−−−−−→ state k −−−−−→ state j

...

22 / 68
Thanks to total probability rule

rij (n) = P (Xn = j|X0 = i)

= P (Xn = j, Xn−1 = 1|X0 = i) + · · · + P (Xn = j, Xn−1 = m|X0 = i)
Xm
= P (Xn = j, Xn−1 = k|X0 = i)
k=1

By multiple law

P (Xn = j, Xn−1 = k|X0 = i)

= P (Xn = j|Xn−1 = k, X0 = i) P (Xn−1 = k|X0 = i)
| {z }
memoryless property

= P (Xn = j|Xn−1 = k)P (Xn−1 = k|X0 = i)

= rkj (1)rik (n − 1) = pkj rik (n − 1)

Hence
m
X
rij (n) = rik (n − 1)pkj
k=1

23 / 68
Matrix representation

Let  
r11 (n) ... r1m (n)
 .. .. .. 
P (n) = . . . 
rm1 (n) . . . rmm (n)
with P (1) = P then

P (2) = P (1) P (1) = P 2

P (3) = P (2) P (1) = P 3

24 / 68
n-step transition matrix

P (n) = P n = P.P
| {z. . . P}
n times

▶ Pijn = P (Xn = j|X0 = i)

▶ Row i of P n : conditional distribution of Xn given X0 = i

25 / 68
Example - Weather forecast

0.4
0.7

0.6
rain not rain

0.3
If it rains today, calculate the probability that it will rain 4 days
from now.

26 / 68
Solution

▶ Transition matrix  
.7 .3
P =
.6 .4
▶ Want to find r00 (4)
▶ Need to calculate P 4
After 4 days
Rain No rain
Current

 
Rain .5749 .4251
= P4
No rain .5668 .4332

4 = 0.5749
▶ So P00

27 / 68
Unconditional distribution of Xn

▶ Distribution of random initial state X0

π (0) (i) = P (X0 = i)

▶ Distribution of Xn

π (n) (i) = P (Xn = i)

Information about state Xn of Markov chain after n steps when

you don’t know the starting point of the process at initial time 0

28 / 68
Unconditional distribution of Xn

π (n) = π (0) P n
where  
π (0) = π1(0) π2(0) . . . πm
(0)

and  
π (n) = π1(n) π2(n) . . . πm
(n)

X0 = 1

n
(0) P1j
π1

start X0 = k n
Xn = j after n steps
πk
(0) Pkj

(0) n
πm Pmj

X0 = m

29 / 68
Proof

Thanks to Total law probability

X
P (Xn = j) = P (Xn = j|X0 = i)P (X0 = i)
i
X
= Pijn P (X0 = i)
i
X
= P (X0 = i)Pijn
i
(0)
X
= πi Pijn
i

30 / 68
Example - Weather forecast

0.4
0.7

0.6
rain not rain

0.3

Suppose probability rain today is .4, what is the probability that

it will rain 4 days from now

31 / 68
Solution

▶ State: 1 = rain, 2 = not rain

▶ Initial probability for weather today

π (0) = .4 .6

▶ Transition matrix  
.7 .3
P =
.6 .4

32 / 68
▶ 4-step transition matrix

P 4 = P.P.P.P = (P.P ).(P.P ) = ...

▶ Distribution for weather 4 days from now

π (4) = π (0) P 4 = ... ...

▶ Probability that it will rain 4 days from now

P (X4 = 1) = π (4) (1) = ...

33 / 68
Practice - Simulate Markov Chain by Monte Carlo

Simulate a sample path which represents state of channel that

a customer views in 30 years given that she watch Channel 1 at
the beginning.

34 / 68
Practice - Simulate Markov Chain by Monte Carlo

Simulate a sample path which represents state of channel that

a customer views in 30 years given that she watch Channel 1 at
the beginning.
▶ Initialization X0 = 1
▶ For i from 1 to 30 do
▶ If Xi−1 = 1, simulate the channel in the next year with pmf
[ 0.8 0.2 ]
▶ If Xi−1 = 2, simulate the channel in the next year with pmf
[ 0.1 0.9 ]

34 / 68
Long term behavior of Markov chain

▶ Does rij (n) converge to something?

▶ Does the limit depend on initial state?
Applications: Google Page’s rank problem . . .

35 / 68
 0.4
0.7

0.6
rain not rain

0.3

   
.7 .3 .57 .43
rij (1) = P = , rij (∞) =
.6 .4 .57 .43

In long term, it will rain with probability .57 whatever the

weather today is

36 / 68
37 / 68
After a lot of transition, the fly is at position 4 with probability
▶ 1/3 if it starts at position 2
▶ 2/3 if it starts at state 3
▶ 0 if it starts at other state
Probability that the fly is at position j after long time depends
on initial state

38 / 68
 1 0.5
1 2 3
0.5 1

▶ n odd then r22 (n) = 0

▶ n even then r22 (n) = 1

39 / 68
 1 0.5
1 2 3
0.5 1

▶ n odd then r22 (n) = 0

▶ n even then r22 (n) = 1
▶ rij (n) diverges

39 / 68
Question

Do rij (n) converge to πj which is independent of the initial state

i?
1. Under which condition?
2. How to find πj if it exists?

40 / 68
Answer for question 2

▶ Start from key recursion rij (n) = k rik (n − 1)pkj

P

▶ let n → ∞ X
πj = πk pkj for all j
k
▶ Addition equation j πj = 1
P

▶ (πj ) is called the stationary distribution of the Markov

chain

41 / 68
Interpretation

After some steps, the distribution of Xn is approximately {πj }

and will not change much

P (Xn = j) ≈ πj for n large enough

πj : steady - state probability

42 / 68
Find stationary distribution

Solve
(
πP = π
P
πi = 1

43 / 68
Example

0.8 0.9

0.1

Channel 1 Channel 2
0.2

Initial market share of each channel is 50%. What will be the

market share after a long time?

44 / 68
Solution

 
.8 .2
▶ Transition matrix P =
.1 .9


▶ Stationary distribution π = π1 π2 satisfies

.8π1 + .1π2 = π1
( 
πP = π
or .2π1 + .9π2 = π2
π1 + π2 = 1 
π1 + π2 = 1


▶ Result π1 = 1/3, π2 = 2/3

45 / 68
After a long time, the market is stable. Each year, there is about

▶ 33% of customers watch channel 1

▶ 67% of customers watch channel 2

46 / 68
Practice

Find stationary distribution of the Markov chain with transition

probability
 
0.8 0.2
P =
0.6 0.4

47 / 68
Answer for question 1

If the Markov chain has the following properties

▶ recurrent states are all in a single class
▶ single recurrent class is not periodic
then the limit of rij (n) exists and independent of initial state

48 / 68
Classification of states

49 / 68
Types of state

▶ State j is accessible from state i if Pijn > 0 for some n ≥ 0

▶ Two states that are accessible from each other are said to
communicate
▶ If i communicates with j and j communicates with k then i
communicates with k.
▶ Markov chain is irreducible if all states communicate with
each other.

50 / 68
Example

51 / 68
Recurrent and Transient State

▶ State i is recurrent if: starting from i, and from wherever

you can go, there is a way of returning to i
▶ If not recurrent, called transient

52 / 68
▶ If a recurrent state is visited once, it will be visited infinitely
numbers of time
▶ a transient state will only be visited a finite number of times.

53 / 68
Return time

▶ Return time

τii = min{n ≥ 1 : Xn = i|X0 = i}

and
τii = ∞ if Xn ̸= i ∀n ≥ 1
▶ Probability of return to state i given starting at i

fi = P (τii < ∞)

▶ If i is recurrent then fi = 1
▶ If i is transient then fi < 1

54 / 68
Total number of visits a state

▶ Total number of visits to state i given starting at i is

∞
X
N= I{Xn =i|X0 =i}
i=0

▶
P (N = n) = fin−1 (1 − fi )
▶ N is geometric distributed with parameter 1 − fi
▶ (
∞, if fi = 1
E(N ) = 1
1−fi , , if fi < 1

55 / 68
By linear property of expectation
∞
X
E(N ) = E(I{Xn =i|X0 =i} )
n=0
X∞
= P (Xn = i|X0 = i)
n=0
X∞
= Piin
n=0

Proposition
State i is recurrent if and only if
∞
X
Piin = ∞
n=0

56 / 68
Reccurent Class

collection of recurrent states that “communicate” to each other

and to no other state

57 / 68
Example

58 / 68
Example

59 / 68
Example

60 / 68
Practice

Determine classes of recurrent states of the Markov chain

61 / 68
Markov chain decomposition

▶ Transient states
▶ Recurrent classes

62 / 68
▶ once the state enters (or starts in) a class of recurrent
states, it stays within that class; since all states in the class
are accessible from each other, all states in the class will
be visited an infinite number of times;
▶ if the initial state is transient, then the state trajectory
contains an initial portion consisting of transient states and
a final portion consisting of recurrent states from the same
class

63 / 68
Analyze long - term behavior

▶ The Markov chain stays forever at a recurrent class that it

visits first

64 / 68
Analyze long - term behavior

▶ The Markov chain stays forever at a recurrent class that it

visits first
▶ Need to analyze chains that consist of a single recurrent
class

64 / 68
Periodicity

Consider a reccurrent class R

1. R is said to be periodic if its states can be grouped in d > 1 disjoint
subsets S1 , ..., Sd so that all transitions from one subset lead to the next
subset
(
j ∈ Sk+1 if k ≤ d − 1
If i ∈ Sk and pij > 0 then
j ∈ S1 if k = d

2. R is aperiodic if not periodic, i.e there exist a state s and a number n

such that ris (n) > 0 for all i ∈ R

65 / 68
Structure of a periodic reccurrent class

66 / 68
▶ a periodic recurrent class, a positive time n, and a state j in
the class, there must exist some state i such that rij (n) = 0
because he subset to which j belongs can be reached at
time n from the states in only one of the subsets.

67 / 68
▶ a periodic recurrent class, a positive time n, and a state j in
the class, there must exist some state i such that rij (n) = 0
because he subset to which j belongs can be reached at
time n from the states in only one of the subsets.
▶ thus a way to verify aperiodicity of a given recurrent class
R, is to check whether there is a special time n ≥ 1 and a
special state s ∈ R that can be reached at time n from all
initial states in R, i.e., ris (n) > 0 for all i ∈ R

67 / 68
Theorem

Let {Xn } be a Markov chain with a single reccurent class and aperiodic. The
steady-state probability πj associated with the state j satisfies the following
properites
1.
(n)
lim Pij = πj
n→∞

2. πj are the unique nonnegative solution of the balance equation

∞
X ∞
X
πj = πi pij , πj = 1
i=1 j=1

{πj } is called the stationary distribution

68 / 68