Markov Chains

Introduction
◼ Markov Process (MP) and Markov Chain (MC)
◼ Discrete Time Markov Chains (DTMC)
◼ Homogeneous and non-homogeneous Markov chains
◼ Transient and steady state Markov chains
◼ Continuous Time Markov Chains (CTMC)
◼ Homogeneous and non-homogeneous Markov chains
◼ Transient and steady state Markov chains

2
 Relationship among Random
Processes
SMP
𝑝𝑖𝑗 arbitrary
𝑓𝜏 arbitrary RW
MP 𝑝𝑖𝑗 = 𝑑(𝑗 − 𝑖)
𝑝𝑖𝑗 arbitrary 𝑓𝜏 arbitrary
𝑓𝜏 memoryless BD
Poisson
𝑝𝑖𝑗 = 0 for 𝑗 − 𝑖 > 1 RP
𝜆𝑖 = 𝜆
𝑓𝜏 memoryless 𝑓𝜏 memoryless 𝑝𝑖𝑗 = 𝑑(1)
𝑓𝜏 arbitrary
PB
𝜆𝑖 > 0, 𝜇𝑖 = 0
𝑓𝜏 memoryless

𝑝𝑖𝑗 : transition probability from state 𝑖 to state 𝑗

𝑓𝜏 : distribution of time between transitions (state holding time)
𝑑(𝑘) : dependency on the difference in indices 𝑘 3
 Markov Process
◼ The definition of Markov process 𝑋(𝑡)
◼ The future of the process does not depend on its past, only on its
present, which is called Markovian property
𝐏 𝑋 𝑡𝑘+1 = 𝑥𝑘+1 𝑋 𝑡𝑘 = 𝑥𝑘 , ⋯ , 𝑋 𝑡0 = 𝑥0
= 𝐏 𝑋 𝑡𝑘+1 = 𝑥𝑘+1 𝑋 𝑡𝑘 = 𝑥𝑘 , 𝑥𝑘 ∈ ℝ, 𝑘 = 0,1, …
◼ The set of possible values 𝑋(𝑡) may take on is called the
state space. There are two possible cases: discrete-state
process, often called as a chain, and continuous-state
process.
◼ Depending on whether the permitted times of changes
are fixed at predetermined times or at anytime, there are
two cases: discrete-time process and continuous-time
process. In the former case we write 𝑋𝑘 rather than 𝑋(𝑡).
4
 Discrete-Time Markov Chain
◼ When we are dealing with “chains”, 𝑋(𝑡) can only take
discrete values from a countably infinite set. We usually
assume the state space is the set of nonnegative integers,
𝑥𝑘 ∈ ℕ = {0,1,2, … }.
◼ For a DTMC, the state change is permitted at certain
discrete time instants, indexed by an integer 𝑘,
𝐏 𝑋𝑘+1 = 𝑥𝑘+1 𝑋𝑘 = 𝑥𝑘 , ⋯ , 𝑋0 = 𝑥0
= 𝐏 𝑋𝑘+1 = 𝑥𝑘+1 𝑋𝑘 = 𝑥𝑘 , 𝑘 = 0,1, …

5
 Chapman-Kolmogorov Equations (1)
◼ A Markov chain is specified by identifying the set of its
states, i.e., state space and the set of its transition
probabilities.
◼ Define the one-step transition probabilities
𝑝𝑖𝑗 𝑘 ≡ 𝐏{𝑋𝑘+1 = 𝑗|𝑋𝑘 = 𝑖}
▪ Let 𝐴(𝑖) be the set of states that are accessible from
state 𝑖. Clearly, for all time instants 𝑘, and for all states 𝑖
෍ 𝑝𝑖𝑗 𝑘 = 1
𝑗∈𝐴(𝑖)
▪ Also, define the 𝑛-step transition probabilities
𝑝𝑖𝑗 𝑘, 𝑘 + 𝑛 ≡ 𝐏{𝑋𝑘+𝑛 = 𝑗|𝑋𝑘 = 𝑖}

6
 Chapman-Kolmogorov Equations (2)
1
𝑖 𝑗

…
◼ Using the total probability rule,
𝑅
𝑝𝑖𝑗 𝑘, 𝑘 + 𝑛
𝑅 𝑘 𝑢 𝑘+𝑛
= ෍ 𝐏 𝑋𝑘+𝑛 = 𝑗 𝑋𝑢 = 𝑟, 𝑋𝑘 = 𝑖 𝐏{𝑋𝑢 = 𝑟|𝑋𝑘 = 𝑖}
𝑟=1
◼ Using the memoryless property of MC, we obtain the
Chapman-Kolmogorov equation, for 𝑘 ≤ 𝑢 ≤ 𝑘 + 𝑛,
𝑝𝑖𝑗 𝑘, 𝑘 + 𝑛
𝑅

= ෍ 𝐏{𝑋𝑘+𝑛 = 𝑗|𝑋𝑢 = 𝑟}𝐏{𝑋𝑢 = 𝑟|𝑋𝑘 = 𝑖}

𝑟=1

7
 CKE in Matrix Form (1)
◼ Define the one-step and 𝑛-step transition probability
matrices, respectively,
𝐏 𝑘 ≡ 𝑝𝑖𝑗 𝑘 ,
𝐇 𝑘, 𝑘 + 𝑛 ≡ 𝑝𝑖𝑗 𝑘, 𝑘 + 𝑛
◼ Then, we can re-write the Chapman-Kolmogorov
Equation as
𝐇 𝑘, 𝑘 + 𝑛 = 𝐇 𝑘, 𝑢 𝐇(𝑢, 𝑘 + 𝑛)
◼ Choose 𝑢 = 𝑘 + 𝑛 − 1, then
𝐇 𝑘, 𝑘 + 𝑛 = 𝐇 𝑘, 𝑘 + 𝑛 − 1 𝐇 𝑘 + 𝑛 − 1, 𝑘 + 𝑛
= 𝐇 𝑘, 𝑘 + 𝑛 − 1 𝐏 𝑘 + 𝑛 − 1
Forward Chapman- One step transition
Kolmogorov Equation probability matrix
8
 CKE in Matrix Form (2)
▪ Choose, 𝑢 = 𝑘 + 1, then
𝐇 𝑘, 𝑘 + 𝑛 = 𝐇 𝑘, 𝑘 + 1 𝐇 𝑘 + 1, 𝑘 + 𝑛
= 𝐏 𝑘 𝐇 𝑘 + 1, 𝑘 + 𝑛

Backward Chapman- One step transition

Kolmogorov Equation probability matrix

9
 State Probabilities
◼ We define the state probability as
𝜋𝑖 (𝑘) ≡ 𝐏{𝑋𝑘 = 𝑖}
◼ We also define the state probability vector as
𝝅 𝑘 ≡ 𝜋0 𝑘 𝜋1 𝑘 𝜋2 𝑘 …
◼ Using the total probability rule, we can write
𝜋𝑖 𝑘 = 𝐏 𝑋𝑘 = 𝑖 = ෍ 𝐏 𝑋𝑘−1 = 𝑗 𝐏{𝑋𝑘 = 𝑖|𝑋𝑘−1 = 𝑗}
𝑗

= ෍ 𝜋𝑗 (𝑘 − 1)𝑝𝑗𝑖 (𝑘 − 1)
𝑗
▪ In vector form, one can write
𝝅 𝑘 = 𝝅 𝑘 − 1 𝐏(𝑘 − 1)

10
 Homogeneous Markov Chains (1)
◼ Markov Chain is called homogeneous when the
transition probabilities are independent of time 𝑘. In that
case, we have the one-step transition matrix as
𝑝𝑖𝑗 𝑘 = 𝑝𝑖𝑗 or 𝐏 𝑘 = 𝐏
◼ Also, the 𝑛-step transition matrix is
(𝑛)
𝑝𝑖𝑗 𝑘, 𝑘 + 𝑛 = 𝑝𝑖𝑗 or 𝐇 𝑘, 𝑘 + 𝑛 = 𝐏 (𝑛) = 𝐏 𝑛
◼ The forward and backward CKEs are, respectively, given
by
(𝑛) (𝑛−1)
𝑝𝑖𝑗 = ෍ 𝑝𝑖𝑟 𝑝𝑟𝑗 ,
𝑟
(𝑛) (𝑛−1)
𝑝𝑖𝑗 =෍ 𝑝𝑖𝑟 𝑝𝑟𝑗 .
𝑟
11
 Homogeneous Markov Chains (2)
◼ For each 𝑗, the steady-state probability is defined as
𝜋𝑗 ≡ lim 𝜋𝑗 (𝑘).
𝑘→∞
Also, the stationary probability vector can be defined as
𝝅 ≡ lim 𝝅(𝑘) ,
𝑘→∞
assuming the limit exists.
▪ The {𝜋𝑗 } are unique solution to the system of equation
below:
𝜋𝑗 = ෍ 𝜋𝑖 𝑝𝑖𝑗 ,
𝑖
෍ 𝜋𝑗 = 1
𝑗

12
 Homogeneous Markov Chains (3)
▪ Since lim 𝛑(𝑘) = lim 𝝅(𝑘 − 1) = 𝝅 and lim 𝐏(𝑘 −
𝑘→∞ 𝑘→∞ 𝑘→∞
1) = 𝐏, for homogeneous MC we have the following
well-known relation:

𝝅 𝑘 = 𝝅 𝑘 − 1 𝐏(𝑘 − 1)
⟹ 𝝅 = 𝝅𝐏

13
Homogeneous Markov Chains (4)
Example: One-Step Transition Matrix
p11

p01 p12 p22

p00 0 1 2
p10 p21
p20

◼ For this MC, the one-step transition matrix is given by

𝑝00 𝑝01 0
𝐏 = 𝑝10 𝑝11 𝑝12
𝑝20 𝑝21 𝑝22

14
Homogeneous Markov Chains (5)
Example: Transient State Vector
◼ Suppose that 𝝅(0) = [1 0 0] and
0.5 0.5 0
𝐏 = 0.35 0.5 0.15
0.245 0.455 0.3
◼ Find 𝝅 𝑘 for 𝑘 = 1,2, ….
0.5 0.5 0
𝝅 1 = 1 0 0 0.35 0.5 0.15
0.245 0.455 0.3
= 0.5 0.5 0
◼ In general, the transient behavior is obtained by solving
the difference equation
𝝅 𝑘 = 𝝅 𝑘 − 1 𝐏 = 𝝅(0)𝐏 𝑘

15
 State Holding Times (1)
◼ Suppose that at point 𝑘, the Markov Chain has changed
into state 𝑖, i.e., 𝑋𝑘 = 𝑖. An interesting question is how
long it will stay at state 𝑖.
◼ Let 𝑇𝑖 be a state holding time that represents the number
of time slots during which the MC stays at state 𝑖.
◼ We are interested on the quantity 𝐏{𝑇𝑖 = 𝑛}
𝐏 𝑇𝑖 = 𝑛 = 𝐏 𝑋𝑘+𝑛 ≠ 𝑖, 𝑋𝑘+𝑛−1 = 𝑖, … , 𝑋𝑘+1 = 𝑖 𝑋𝑘 = 𝑖
= 𝐏{𝑋𝑘+𝑛 ≠ 𝑖|𝑋𝑘+𝑛−1 = 𝑖, … , 𝑋𝑘+1 = 𝑖, 𝑋𝑘 = 𝑖}
× 𝐏 𝑋𝑘+𝑛−1 = 𝑖, … , 𝑋𝑘+1 = 𝑖 𝑋𝑘 = 𝑖 Due to the Markovian
= 𝐏 𝑋𝑘+𝑛 ≠ 𝑖 𝑋𝑘+𝑛−1 = 𝑖 property

× 𝐏{𝑋𝑘+𝑛−1 = 𝑖|𝑋𝑘+𝑛−2 = 𝑖, … , 𝑋𝑘+1 = 𝑖, 𝑋𝑘 = 𝑖}

× 𝐏 𝑋𝑘+𝑛−2 = 𝑖, … , 𝑋𝑘+1 = 𝑖 𝑋𝑘 = 𝑖
𝐏(𝐴, 𝐵|𝐶) = 𝐏(𝐴|𝐵, 𝐶)𝐏(𝐵|𝐶)
𝐴 = 𝑋𝑘+𝑛 ≠ 𝑖
𝐵 = {𝑋𝑘+𝑛−1 = 𝑖, … , 𝑋𝑘+1 = 𝑖}
𝐶 = {𝑋𝑘 = 𝑖} 16
 State Holding Times (2)
= (1 − 𝑝𝑖𝑖 ) Due to the Markovian property,
these become 𝑝𝑖𝑖2
× 𝐏{𝑋𝑘+𝑛−1 = 𝑖|𝑋𝑘+𝑛−2 = 𝑖}
× 𝐏 𝑋𝑘+𝑛−2 = 𝑖 𝑋𝑘+𝑛−3 = 𝑖, … , 𝑋𝑘+1 = 𝑖, 𝑋𝑘 = 𝑖
× 𝐏 𝑋𝑘+𝑛−3 = 𝑖, … , 𝑋𝑘+1 = 𝑖 𝑋𝑘 = 𝑖
= ⋯ = (1 − 𝑝𝑖𝑖 )𝑝𝑖𝑖𝑛−1 × 𝐏{𝑋𝑘+𝑛−𝑛 = 𝑖|𝑋𝑘 = 𝑖}
= (1 − 𝑝𝑖𝑖 )𝑝𝑖𝑖𝑛−1
▪ This is the Geometric distribution with parameter 1 − 𝑝𝑖𝑖 .
▪ Clearly, 𝑇𝑖 has the memoryless property.

17
 Classification of States
◼ Definitions
◼ State 𝑗 is reachable from state 𝑖 if the probability to go
from 𝑖 to 𝑗 in 𝑛 > 0 steps is greater than zero (Two
states 𝑖 and 𝑗 accessible to each other are said to
communicate, and we write 𝑖 𝑗).
◼ A subset 𝑆 of the state space is closed if 𝑝𝑖𝑗 = 0 for

every 𝑖 ∈ 𝑆 and 𝑗 ∉ 𝑆.
◼ A state 𝑖 is said to be absorbing if it is a single element
closed set.
◼ A closed set (class) 𝑆 of states is irreducible if any
state 𝑗 ∈ 𝑆 is reachable from every state 𝑖 ∈ 𝑆.
◼ A Markov chain is said to be irreducible if whose state
space is irreducible (or one communicating class).
18
 Example: Reducible/Irreducible MC
◼ Irreducible Markov Chain
p01 p12 p22
p00 0 1 2
p10 p21

◼ Reducible Markov Chain Not closed set

p01 p12 p23
0 1 2 p32 3
p00 p10
p14
Absorbing State p22 p33
4
Closed irreducible set 19
 Transient and Recurrent States (1)
▪ A state 𝑖 is recurrent if for every 𝑗 that is accessible from
𝑖, 𝑖 is also accessible from 𝑗.
▪ If 𝑖 is recurrent, for all 𝑗 that belong to 𝐴(𝑖) we induce that 𝑖
belongs to 𝐴(𝑗).
▪ If a recurrent state is visited once, it is revisited an infinite
number of times.
▪ A state 𝑖 is transient if there is a state 𝑗 ∈ 𝐴 𝑖 such that 𝑖
is not accessible from 𝑗.
▪ A transient state will only be visited a finite number of
times.
Recurrent Recurrent 𝐴(1) = {1},
1 𝐴(2) = {1,2,3},
2 3 4 𝐴(3) = {3,4},
𝐴 4 = 3,4 .
Transient 20
 Transient and Recurrent States (2)
▪ If 𝑖 is recurrent, 𝐴(𝑖) forms a recurrent class (or simply
class), states in 𝐴(𝑖) communicate each other, and are
closed to outside.
▪ For a recurrent state 𝑖, we have 𝐴(𝑖) = 𝐴(𝑗) for all 𝑗 ∈ 𝐴(𝑖).
▪ MC Decomposition: A Markov chain can be decomposed
into one or more recurrent classes.
▪ Decomposition provides a powerful conceptual tool for
MC. We only concern chains of a single recurrent class.
▪ A recurrent state is accessible from all states in its own
class, but inaccessible from recurrent states in other
classes.
▪ A transient state is inaccessible from any recurrent state.
▪ At least one, possibly more, recurrent states are
accessible from a given transient state.
21
 Transient and Recurrent States (3)
▪ Examples of Markov chain decomposition:
MC has a single class
of recurrent states,
{1, 2, 3, 4}

Single class of recurrent

states {1, 2} and one transient
state 3

Two classes of recurrent

states {1} and {4, 5}, and
two transient states 2, 3
22
 Transient and Recurrent States (4)
◼ Hitting (First Passage) Time 𝑇𝑖𝑗 ≡ min{𝑛 ≥ 0: 𝑋𝑛 = 𝑗|𝑋0 =
𝑖}
◼ Recurrence Time 𝑇𝑖𝑖 is the first time that the MC returns
to state 𝑖.
(𝑛)
◼ Denoting by 𝑓𝑖𝑖 ≡ 𝐏{𝑇𝑖𝑖 = 𝑛}, the probability, 𝑓𝑖𝑖 , that the
state will ever return to 𝑖 given it starts from 𝑖 is given by
∞
(𝑛)
𝑓𝑖𝑖 ≡ ෍ 𝑓𝑖𝑖 = 𝐏{ever returing to state 𝑖}
𝑛=1
◼ The event that the MC will ever return to state 𝑖 given it
started from 𝑖 is equivalent to 𝑇𝑖𝑖 < ∞, therefore
𝑓𝑖𝑖 = 𝐏{ever returing to state 𝑖} = 𝐏{𝑇𝑖𝑖 < ∞}
▪ A state is recurrent if 𝑓𝑖𝑖 = 1 and transient if 𝑓𝑖𝑖 < 1

23
 Positive/Negative Recurrent States
◼ Let 𝑀𝑖𝑖 be the mean recurrence∞time of state 𝑖
(𝑛)
𝑀𝑖𝑖 ≡ 𝐄 𝑇𝑖𝑖 = ෍ 𝑛𝑓𝑖𝑖
𝑛=1
◼ A state is said to be positive (or nonnull) recurrent if 𝑀𝑖𝑖 <
∞. If 𝑀𝑖𝑖 = ∞ then the state is said to be negative (or null)
recurrent.
◼ Notes for recurrence (transience) property
◼ If state 𝑖 is positive (null) recurrent and 𝑖 𝑗, then state
𝑗 is also positive (null) recurrent. Positive (or Null)
recurrence is class property.
◼ If state 𝑖 is transient and 𝑖 𝑗, then state 𝑗 is also
transient. Transience is also class property.

24
 Example: Positive Recurrence

p01 p12 p23

0 1 2 p32 3
p00 p10
p14
p22 p33
Transient 4
States Positive
Recurrent
States
Positive Recurrent
State
25
 Theorems about Transience and
Recurrence
∞ (𝑛)
◼ σ
State 𝑗 is recurrent if and only if 𝑛=1 𝑝𝑖𝑗 = ∞ for all 𝑖.
∞ (𝑛)
◼ σ
For 𝑖 = 𝑗, 𝑛=1 𝑝𝑗𝑗 = ∞
∞ (𝑛)
◼ σ
State 𝑗 is transient if and only if 𝑛=1 𝑝𝑖𝑗 < ∞ for all 𝑖,
meaning that starting in 𝑖, the number of transitions into
state 𝑗 is finite.
(𝑛)
◼ For 𝑖 = 𝑗, σ∞
𝑛=1 𝑗𝑗 < ∞
𝑝
◼ (Solidarity Theorem) In an irreducible MC, all states are
of the same type, i.e., states are all positive recurrent or
all null recurrent or all transient.

26
 Properties about Finite MC
◼ In an MC with a finite number of states, the following
properties hold:
◼ There should be no null recurrent state

◼ Not all states can be transient

◼ At least one state should be recurrent

◼ If the MC is finite and irreducible, all states are

positive recurrent

◼ Whereas, if there are infinite number states, different

types of states (recurrent null, recurrent nonnull or
transient) can coexist.

27
 Periodic and Aperiodic States
◼ Suppose that the structure of the MC is such that state 𝑖
is visited repeatedly after a number of steps that are an
integer multiple of an integer 𝑑 > 1. Then the state is
called periodic with period 𝑑.
◼ If no such integer exists (i.e., 𝑑 = 1) then the state is
called aperiodic. A Markov chain is aperiodic if every
state is aperiodic.
◼ Example: state 1 is periodic
1 0.5
0 1 2
0.5 1
Periodic State 𝑑 = 2
28
Class Periodicity (1)
◼ Consider a recurrent class 𝑅.
◼ The recurrent class 𝑅 is called periodic if its states can
be grouped in 𝑚 > 1 disjoint subsets 𝑆1 , … , 𝑆𝑚 , so that
all transitions from 𝑆𝑘 lead to 𝑆𝑘+1 (or to 𝑆1 if 𝑘 = 𝑚)
◼ Example:

◼ Periodicity is a class property. This means that for any

two periodic states 𝑖 and 𝑗, if 𝑖 𝑗, then 𝑑(𝑖) = 𝑑(𝑗).
29
Class Periodicity (2)
◼ The recurrent class 𝑅 is aperiodic if and only if there
exists a time 𝑛 such that 𝐏 𝑇𝑖𝑗 = 𝑛 > 0, for all 𝑖, 𝑗 ∈ 𝑅.
◼ Example: From state 1, every state is reachable at time
𝑛 = 3, so the unique recurrent class of the chain is
aperiodic.

◼ A converse statement also turns out to be true: if a

recurrent class 𝑅 is aperiodic, then there exists a time
𝑛 that 𝐏 𝑇𝑖𝑗 = 𝑛 > 0, for every 𝑖, 𝑗 ∈ 𝑅. 30
 Limit Theorem
◼ Let 𝑗 be any state in a Markov chain. As 𝑛 → ∞,
◼ if 𝑗 is transient or null recurrent, then

(𝑛)
𝑝𝑖𝑗 → 0
◼ if 𝑗 is positive recurrent and aperiodic, then
(𝑛) 1
𝑝𝑖𝑗 →
𝑀𝑗𝑗
◼ if 𝑗 is positive recurrent and periodic with period 𝑑(𝑗),
then
(𝑛𝑑(𝑗)) 𝑑(𝑗)
𝑝𝑖𝑗 →
𝑀𝑗𝑗

31
 Example of Limit Theorem
◼ Consider an irreducible the Markov chain with
0 1 0 0.5 0 0.5
𝐏 = 0.5 0 0.5 , 𝐏 2 = 0 1 0 , 𝐏 3 = 𝐏, 𝐏 4 = 𝐏 2 , … ,
0 1 0 0.5 0 0.5
⟹ 𝐏 (2𝑛) = 𝐏 2 , 𝐏 (2𝑛+1) = 𝐏
(2𝑛) (2𝑛+1)
We note 𝑝𝑖𝑖 > 0, 𝑝𝑖𝑖 = 0 for each 𝑖 and all the states
are periodic with 𝑑 = 2.
(1) (2) (𝑛)
◼ Now 𝑓11 = 0, 𝑓11 = 1 so that 𝑓11 = σ∞ 1 11 = 1, i.e., state
𝑓
1 is recurrent and by the Solidarity Theorem, states 0 and 2
are also recurrent.
(𝑛)
◼ 𝑀11 = σ∞1 𝑛𝑓11 = 1 × 0 + 2 × 1 = 2. State 1 is positive
recurrent, and states 0 and 2 are also positive recurrent.
(2𝑛) 𝑑 2
◼ By the Limit Theorem, 𝑝11 → = = 1.
𝑀11 2 32
Ergodicity
◼ A state 𝑖 is said to be ergodic if it is aperiodic and
positive recurrent. If all states in an irreducible Markov
chain are ergodic, then the chain is said to be ergodic.
◼ Ergodicity is a class property.
◼ It can be shown that a finite state irreducible Markov
chain is ergodic if it has at least one aperiodic state.
Why? Aperiodicity is also a class property!
◼ More generally, a Markov chain is ergodic if there is a
number 𝑁 such that any state can be reached from any
other state in any number of steps less or equal to a
number 𝑁.

33
 Steady State Analysis (1)
◼ Recall that the probability of finding the DTMC at state 𝑗
after the 𝑘th step is given by 𝜋𝑗 𝑘 = 𝐏{𝑋𝑘 = 𝑗}
◼ An interesting question is what happens in the “long run”,
i.e., 𝜋𝑗 ≡ lim 𝜋𝑗 (𝑘) ?
𝑘→∞
◼ This is referred to as steady (stationary) state or
equilibrium state probability
◼ Questions:
◼ Do these limits exist ?

◼ If they exist, do they converge to a legitimate

probability distribution, i.e., σ 𝜋𝑗 = 1 ?

◼ How do we evaluate 𝜋𝑗 , for all 𝑗 ?
34
 Steady State Analysis (2)
◼ Recall the recursive probability
𝝅 𝑘 = 𝝅 𝑘 − 1 𝐏 = 𝝅(0)𝐏 𝑘
▪ If steady state exists, then 𝝅 𝑘 = 𝝅(𝑘 − 1), and
therefore the steady state probabilities are given by the
solution to the equations
𝝅 = 𝝅𝐏 and ෍ 𝜋𝑖 = 1
𝑖
◼ In an irreducible Markov Chain, the presence of periodic
states prevents the existence of a steady state probability
◼ Example: State vector alternates two values!
0 1 0
𝐏 = 0.5 0 0.5 , 𝝅 0 = [1 0 0]
0 1 0
⇒ For odd 𝑘, 𝝅 𝑘 = 0 1 0 ,
For even 𝑘, 𝝅 𝑘 = [0.5 0 0.5] 35
 Stationary Distribution for DTMC
◼ In an irreducible aperiodic homogeneous Markov chain
consisting of positive recurrent states, a unique
stationary state probability vector π exists such that
𝜋𝑗 > 0 and
1
𝜋𝑗 = lim 𝜋𝑗 (𝑘) =
𝑘→∞ 𝑀𝑗𝑗
where 𝑀𝑗𝑗 is the mean recurrence time of state 𝑗. The
interpretation is that 𝜋𝑗 is the long-run fraction of time in
state 𝑗.
◼ The steady state vector π is determined by solving
𝝅 = 𝝅𝐏 and σ𝑗 𝜋𝑗 = 1. Any solution {𝜋𝑗 } to these
equations is called a stationary distribution.
36
 Limiting Distribution for DTMC
◼ The limiting probabilities of a DTMC is defined as
𝜋𝑗∗ ≡ lim 𝑝𝑖𝑗 (𝑘, 𝑘 + 𝑛)
𝑛→∞
◼ This definition assumes that the limit exists and that the
limit is the same for all values of 𝑖, independent of 𝑘.
That is, lim 𝐏 𝑛 exists and the rows of this matrix are
𝑛→∞
all the same. Then 𝜋𝑗∗ is an element from the 𝑗th column
of this limiting matrix.
◼ 𝜋𝑗∗ is called a limiting distribution if σ𝑗 𝜋𝑗∗ = 1 holds.
◼ It is possible that lim 𝑝𝑖𝑗 (𝑘, 𝑘 + 𝑛) = 0 for all 𝑗, in which
𝑛→∞
case the limiting probabilities exist but 𝜋𝑗∗ would not
be a limiting distribution.
37
 Example: Limiting Distribution (1)
◼ Consider a DTMC that has a stationary distribution but
no limiting distribution. This periodic chain alternates
between two states, 0 and 1, with
0 1
𝐏= .
1 0
◼ The chain is irreducible and positive recurrent, so it has
1 1
a unique solution of 𝝅 = 𝝅𝐏, that is, 𝝅 = [ , ].
2 2
◼ Is this the limiting distribution? The answer is no. The
chain has period 2. In particular, 𝐏 (𝑛) does not converge:
1 0
, 𝑛 even,
0 1
𝐏 (𝑛) =
0 1
, 𝑛 odd.
1 0 38
 Example: Limiting Distribution (2)
1 1
◼ Suppose we choose 𝝅 0 = [ , ]. From 𝝅 𝑘 =
2 2
𝝅 𝑘 − 1 𝐏, we have
1 1 0 1 1 1
𝝅 1 =[ , ] =[ , ],
2 2 1 0 2 2
1 1
and similarly, 𝝅 𝑛 = [ , ] for all 𝑛.
2 2
◼ Even though lim 𝐏
(𝑛) does not exist (i.e., limiting
𝑛→∞
distribution does not exist), it is possible for lim 𝝅 𝑛 to
𝑛→∞
exist (i.e., stationary distribution exists), but only if the
starting state probability is chosen randomly according
to the stationary distribution.

39
Relationship between Stationary and
Limiting distributions for DTMC
Condition Resulting Properties Comment

Irreducible, positive Unique stationary 𝜋𝑗 is the long-run fraction

recurrent distribution exists of time in state 𝑗

Irreducible, positive Unique stationary Ensemble averages =

recurrent, 𝝅 0 = 𝝅 distribution exists, time average
process is stationary and
ergodic

Irreducible, positive Unique stationary 𝜋𝑗 = 𝜋𝑗∗ ;

recurrent, aperiodic distribution exists, Process becomes
process is ergodic, independent of starting
limiting distribution also state in the limit
exists

40
 Summary about State Distributions
◼ An irreducible and positive recurrent DTMC has a unique
solution to the stationary equations
𝝅 = 𝝅𝐏 and σ𝑗 𝜋𝑗 = 1,
namely, 𝜋𝑗 = 1Τ𝑀𝑗𝑗 . Furthermore, if the chain is aperiodic,
the limiting distribution exists and is equal to the
stationary distribution.
◼ By this, there are two main ways to interpret 𝜋𝑗 . The first
interpretation is that 𝜋𝑗 is the long run fraction of time
spent in state 𝑗. The second interpretation is that 𝜋𝑗 is the
probability of being in state 𝑗 a long time from now.
◼ If the chain has a limiting distribution, then it becomes the
only stationary distribution. However, the converse is not
necessarily true. 41
 Birth-Death Example (1)
1−𝑝 1−𝑝 1−𝑝
0 1 ⋯ 𝑖
𝑝 𝑝 𝑝 𝑝

𝑝 1−𝑝 0 ⋯
𝑝 0 1−𝑝 ⋯
𝐏=
0 𝑝 0 ⋯
⋮ ⋮ ⋮ ⋱
◼ Thus, to find the steady state vector π we need to solve
𝝅 = 𝝅𝐏 and σ𝑗 𝜋𝑗 = 1

42
 Birth-Death Example (2)
◼ In other words,
𝜋0 = 𝜋0 𝑝 + 𝜋1 𝑝,
𝜋𝑗 = 𝜋𝑗−1 1 − 𝑝 + 𝜋𝑗+1 𝑝, 𝑗 = 1,2, …
◼ Solving these equations, we get
2
1−𝑝 1−𝑝
𝜋1 = 𝜋0 , 𝜋2 = 𝜋0
𝑝 𝑝
1−𝑝 𝑗
◼ In general, 𝜋𝑗 = 𝜋0
𝑝
◼ Summing all terms and using σ𝑗 𝜋𝑗 = 1, we get
∞ 𝑗
1−𝑝 1
𝜋0 ෍ = 1 ⟹ 𝜋0 = 𝑗
𝑝 1−𝑝
𝑗=0 σ∞
𝑗=0 𝑝
43
 Birth-Death Example (3)
◼ Therefore, for all states 𝑗 we get
𝑗 ∞ 𝑖
1 1−𝑝 1−𝑝
𝜋𝑗 = with 𝐶 = ෍
𝐶 𝑝 𝑝
𝑖=0
◼ If 𝑝 < 1/2, then 𝐶 = ∞
⇒ 𝜋𝑗 = 0, for all 𝑗, all states are transient
𝑝
◼ If 𝑝 > 1/2, then 𝐶 = >0
2𝑝−1

2𝑝−1 1−𝑝 𝑗
⇒ 𝜋𝑗 = for all 𝑗, all positive recurrent
𝑝 𝑝
◼ If 𝑝 = 1/2, then 𝐶 = ∞
⇒ 𝜋𝑗 = 0, for all 𝑗, all null recurrent

44
 Reducible Markov Chains (1)
Transient Irreducible
Set 𝑇 Set 𝑆1

Irreducible
Set 𝑆2
◼ In steady state, we know that the Markov chain will
eventually end in an irreducible set and the previous
analysis still holds.
◼ The only question that arises, in case there are two or
more irreducible sets, is the probability it will end in each
set. 45
Reducible Markov Chains (2)
Transient Irreducible
Set 𝑇 Set 𝑆
s1
r
sn
i

◼ Suppose we start from state 𝑖 ∈ 𝑇. Then, there are two

ways to go to 𝑆.
◼ Directly in one step go to 𝑆 or
◼ Indirectly, first go to 𝑟 ∈ 𝑇, and then go to 𝑆 after 𝑘 − 1
steps.
◼ Define
𝜌𝑖 𝑆 = 𝐏 𝑋𝑘 ∈ 𝑆 𝑋0 = 𝑖 , 𝑘 = 2,3, …
46
 Reducible Markov Chains (3)
◼ First consider the direct case by one-step transition
𝐏 𝑋1 ∈ 𝑆 𝑋0 = 𝑖 = ෍ 𝑝𝑖𝑗
𝑗∈𝑆
◼ Next consider the indirect case for 𝑘 = 2, 3, …
𝐏 𝑋𝑘 ∈ 𝑆, 𝑋𝑘−1 = 𝑟𝑘−1 ∈ 𝑇, … , 𝑋1 = 𝑟 ∈ 𝑇 𝑋0 = 𝑖
= 𝐏 𝑋𝑘 ∈ 𝑆, 𝑋𝑘−1 = 𝑟𝑘−1 ∈ 𝑇, … 𝑋1 = 𝑟 ∈ 𝑇, 𝑋0 = 𝑖
× 𝐏 𝑋1 = 𝑟 ∈ 𝑇 𝑋0 = 𝑖
= 𝐏 𝑋𝑘 ∈ 𝑆, 𝑋𝑘−1 = 𝑟𝑘−1 ∈ 𝑇, … 𝑋1 = 𝑟 ∈ 𝑇 𝑝𝑖𝑟
= 𝜌𝑟 (𝑆)𝑝𝑖𝑟
◼ Summing the two cases, we have
𝜌𝑖 𝑆 = ෍ 𝑝𝑖𝑗 + ෍ 𝜌𝑟 (𝑆)𝑝𝑖𝑟
𝑗∈𝑆 𝑟∈𝑇
47
 Absorption Probability Equation
◼ Consider a homogeneous MC where each state is either
transient or absorbing and fix a particular absorbing state
𝑠. Then, starting from any initial state 𝑖, 𝑎𝑖 = 𝐏{𝑋𝑛 =
𝑠|𝑋0 = 𝑖} is the unique solution to the equations:
𝑎𝑠 = 1, 𝑎𝑠′ = 0, for other absorbing states 𝑠′ ≠ 𝑠
𝑎𝑖 = ෍ 𝑝𝑖𝑗 𝑎𝑗 for all transient states 𝑖
𝑗
◼ Why? Let us consider a transient state 𝑖. Then
𝑎𝑖 = 𝐏{𝑋𝑛 = 𝑠|𝑋0 = 𝑖}
= ෍ 𝐏 𝑋𝑛 = 𝑠 𝑋0 = 𝑖, 𝑋1 = 𝑗 𝐏{𝑋1 = 𝑗|𝑋0 = 𝑖}
𝑗

= ෍ 𝐏{𝑋𝑛 = 𝑠|𝑋1 = 𝑗}𝑝𝑖𝑗

𝑗

= ෍ 𝑝𝑖𝑗 𝑎𝑗
48
𝑗
 Example: Absorption Probability
◼ Note that there are two recurrent classes in Fig(a), {1} and
{4,5}. We lump {4,5} into state 6 as in Fig(b).
➢ What is the prob. of eventually reaching state 6?
◼ Starting from transient states 2 and 3, we have

𝑎2 = 0.2𝑎1 + 0.3𝑎2 + 0.4𝑎3 + 0.1𝑎6 ,

𝑎3 = 0.2𝑎2 + 0.8𝑎6
◼ Using 𝑎1 = 0 and 𝑎6 = 1,

we obtain
𝑎2 = 21/31,
𝑎3 = 29/31.

49
 Example: Gambler’s Ruin
◼ A gambler wins $1 at each round with probability 𝑝 and
loses $1 with probability 1 − 𝑝. He plays repeatedly until
either wins a target of $𝑚 or loses all his money.
➢ What is the probability of winning the game?
◼ In the figure, for 𝑚 = 4, states {0, 4} are absorbing, and

other states are all transient.

◼ In general, the absorption probability 𝑎𝑖 is the probability

of winning, starting from transient state 𝑖,

𝑎0 = 0, 𝑎𝑚 = 1,
𝑎𝑖 = 1 − 𝑝 𝑎𝑖−1 + 𝑝𝑎𝑖+1 , 𝑖 = 1, … , 𝑚 − 1,
𝑚=4

50
 Expected Time to Absorption
◼ Consider the expected number of steps until a recurrent
state is entered, starting from a particular transient state 𝑖,
𝜇𝑖 ≡ 𝐄 min 𝑛 ≥ 0: 𝑋𝑛 is recurrent 𝑋0 = 𝑖}
◼ If 𝑖 is recurrent, 𝜇𝑖 = 0 from the definition.
◼ Otherwise, for all transient states 𝑖, we have
𝜇𝑖 = 𝐄[min{𝑛 ≥ 0: 𝑋𝑛 is recurrent | 𝑋0 = 𝑖}]
= ෍ 𝐄[min{𝑛 ≥ 0: 𝑋𝑛 is recurrent | 𝑋0 = 𝑖, 𝑋1 = 𝑗}]
𝑗
∙ 𝐏{𝑋1 = 𝑗|𝑋0 = 𝑖}
= ෍ (1 + 𝜇𝑗 )𝑝𝑖𝑗
𝑗

= 1 + ෍ 𝜇𝑗 𝑝𝑖𝑗
𝑗
51
 Example: Spider-and-Fly Model
◼ Consider the spider-and-fly model. The states
correspond to possible fly positions, and the absorbing
states 1 and 𝑚 correspond to capture by a spider.
◼ Let us calculate the expected number of steps until the
fly is captured. We have
𝜇𝑖 = 1 + 0.3𝜇𝑖−1 + 0.4𝜇𝑖 + 0.3𝜇𝑖+1 , for 𝑖 = 2, … , 𝑚 − 1
𝜇1 = 𝜇𝑚 = 0
◼ For 𝑚 = 4, we obtain 𝜇2 = 𝜇3 = 10/3.

52
 Mean First Passage and Recurrence
Times
◼ Let us consider the mean first passage time to reach a
particular recurrent state 𝑠, starting from any other state 𝑖,
𝑡𝑖 = 𝐄[min{𝑛 ≥ 0: 𝑋𝑛 = 𝑠| 𝑋0 = 𝑖}]
Using the similar approach given earlier, we have
𝑡𝑖 = 1 + ෍ 𝑡𝑗 𝑝𝑖𝑗 , for all 𝑖 ≠ 𝑠,
𝑗
𝑡𝑠 = 0.
◼ We also want to calculate the mean recurrence time of
the special state 𝑠, which is given by
𝑡𝑠∗ = 𝐄[number of transitions up to the first return to 𝑠, from 𝑠]
= 𝐄[min{𝑛 ≥ 1: 𝑋𝑛 = 𝑠| 𝑋0 = 𝑖}]
= 1 + ෍ 𝑡𝑗 𝑝𝑠𝑗
𝑗
53
 Example: “Up-to-Date”-”Behind” Model
◼ Consider the ‘up-to-date’ and ‘behind’ model. Alice is
taking an on-line probability class, and in each week, she
can be up-to-date or may have fallen behind, attending
the class.

Let us focus on state 𝑠 = 1 and calculate the mean first

passage time to state 1, starting from state 2. We have
𝑡1 = 0 and 𝑡2 = 1 + 0.6𝑡1 + 0.4𝑡2 = 1 + 0.4𝑡2 . So 𝑡2 = 5/3.
◼ The mean recurrence time to state 1 is given by
𝑡1∗ = 1 + 0.8𝑡1 + 0.2𝑡2 = 4/3.
54
 Continuous-Time Markov Chains (1)
◼ The process {𝑋 𝑡 , 𝑡 ≥ 0} is a CTMC if for all 𝑠, 𝑡 ≥ 0,
𝐏 𝑋 𝑡 + 𝑠 = 𝑗 𝑋 𝑠 = 𝑖, 𝑋 𝑢 = 𝑘, 0 ≤ 𝑢 ≤ 𝑠
= 𝐏{𝑋 𝑡 + 𝑠 = 𝑗|𝑋 𝑠 = 𝑖}
◼ In this case, transitions are permitted to occur at any time

◼ Recall that the Markovian property implies that

▪ 𝑋(𝑡 + 𝑠) depends only on 𝑋(𝑠) (state memory)

▪ It does not matter how long the amount of time that the MC
stays in a state before leaving the state (age memory).
▪ Why? Suppose that a CTMC enters state 𝑖 at some time,
say time 0, and that the process does not leave state 𝑖
during the next 𝑠 time units. Then the probability it remains
still there during [𝑠, 𝑠 + 𝜏] is just the probability that it stays in
state 𝑖 for at least 𝜏 additional time units.
55
 Continuous-Time Markov Chains (2)
◼ That is, if we let 𝑇𝑖 denote the amount of time that the MC
stays in state 𝑖 before making a transition into a different
state, then
𝐏 𝑇𝑖 > 𝑠 + 𝜏 𝑇𝑖 > 𝑠 = 𝐏 𝑇𝑖 > 𝜏 for all 𝑠, 𝜏 ≥ 0.
◼ Hence, the random variable 𝑇𝑖 is memoryless and must thus
be exponentially distributed.
◼ In summary, a CTMC has the following properties that each
time it enters state 𝑖:
▪ The amount of time 𝑇𝑖 it spends in state 𝑖 before making a
transition into a different state is exponentially distributed
with rate, say, 𝜆𝑖 .
▪ When the MC leaves state 𝑖, it will next enter state 𝑗 with
some probability, call it 𝑃𝑖𝑗 , where σ𝑗≠𝑖 𝑃𝑖𝑗 = 1 (i. e., 𝑃𝑖𝑖 = 0).
56
 Transition Function
◼ Define the two-argument transition function
𝑝𝑖𝑗 𝑠, 𝑡 ≡ 𝐏 𝑋 𝑡 = 𝑗 𝑋 𝑠 = 𝑖 , 𝑠≤𝑡
◼ The continuous-time analogue of the Chapman-
Kolmogorov equation is, for 𝑠 ≤ 𝑢 ≤ 𝑡
𝑝𝑖𝑗 𝑠, 𝑡
= ෍ 𝐏 𝑋 𝑡 = 𝑗 𝑋 𝑢 = 𝑟, 𝑋 𝑠 = 𝑖 𝐏{𝑋 𝑢 = 𝑟|𝑋 𝑠 = 𝑖}
𝑟
◼ Using the memoryless property
𝑝𝑖𝑗 𝑠, 𝑡 = ෍ 𝐏 𝑋 𝑡 = 𝑗 𝑋 𝑢 = 𝑟 𝐏{𝑋 𝑢 = 𝑟|𝑋 𝑠 = 𝑖}
𝑟
◼ Define 𝐇 𝑠, 𝑡 = 𝑝𝑖𝑗 𝑠, 𝑡 , 𝑖, 𝑗 ∈ ℕ. Then
𝐇 𝑠, 𝑡 = 𝐇 𝑠, 𝑢 𝐇 𝑢, 𝑡 , 𝑠≤𝑢≤𝑡
▪ Note that 𝐇(𝑠, 𝑠) = 𝐈.
57
 Transition Rate Matrix (1)
◼ Consider the forward CKE for 𝑠 ≤ 𝑡 ≤ 𝑡 + Δ𝑡
𝐇 𝑠, 𝑡 + ∆𝑡 = 𝐇 𝑠, 𝑡 𝐇(𝑡, 𝑡 + ∆𝑡)
◼ Subtracting 𝐇(𝑠, 𝑡) from both sides and dividing by Δ𝑡
𝐇 𝑠, 𝑡 + ∆𝑡 − 𝐇(𝑠, 𝑡) 𝐇 𝑠, 𝑡 𝐇 𝑡, 𝑡 + ∆𝑡 − 𝐈
=
∆𝑡 ∆𝑡
◼ Taking the limit as Δ𝑡 → 0
𝜕𝐇(𝑠, 𝑡)
= 𝐇 𝑠, 𝑡 𝐐(𝑡)
𝜕𝑡
where the transition rate matrix 𝐐(𝑡) is given by
𝐇 𝑡, 𝑡 + ∆𝑡 − 𝐈
𝐐 𝑡 = lim
∆𝑡→0 ∆𝑡
◼ This is nothing more than the forward CKE for the CTMC.
58
 Transition Rate Matrix (2)
◼ The elements of 𝐐 𝑡 are transition rate defined as
1 − 𝑝𝑖𝑖 𝑡, 𝑡 + ∆𝑡
−𝑞𝑖𝑖 𝑡 = lim
∆𝑡→0 ∆𝑡
𝑝𝑖𝑗 𝑡, 𝑡 + ∆𝑡
𝑞𝑖𝑗 𝑡 = lim ,𝑖 ≠ 𝑗
∆𝑡→0 ∆𝑡
◼ If in state 𝑖 at time 𝑡, then the probability that a transition
occurs during the interval 𝑡, 𝑡 + ∆𝑡 is given by −𝑞𝑖𝑖 𝑡 ∆𝑡 +
𝑜 ∆𝑡 . Thus −𝑞𝑖𝑖 𝑡 is the total rate at which the process
departs state 𝑖.
◼ Similarly, given that in state 𝑖 at time 𝑡, the probability that a
transition to state 𝑗 (𝑖 ≠ 𝑗) occurs in the interval 𝑡, 𝑡 + ∆𝑡 is
given by 𝑞𝑖𝑗 𝑡 ∆𝑡 + 𝑜 ∆𝑡 .

59
 Homogeneous Case (1)
◼ In the homogeneous case, the transition functions do not
depend on 𝑠 and 𝑡, but only on the difference τ = 𝑡 − 𝑠,
thus 𝑝𝑖𝑗 𝑠, 𝑡 becomes the one-argument transition
function 𝑝𝑖𝑗 (𝜏 = 𝑡 − 𝑠) defined as
𝑝𝑖𝑗 𝜏 ≡ 𝐏 𝑋 𝑠 + 𝜏 = 𝑗 𝑋 𝑠 = 𝑖
▪ Note that the transition function holds the following
identity
෍ 𝑝𝑖𝑗 (𝜏) = 1 for all 𝑖
𝑗
▪ The one-argument transition function 𝑝𝑖𝑗 (𝜏) now means
the probability that the CTMC, presently in state 𝑖, will be
in state 𝑗 after an additional time 𝜏, or means the
probability that there is a transition in an interval of 𝜏.
60
 Homogeneous Case (2)
▪ Also, 𝐇 𝑠, 𝑡 becomes a one-argument type that depends
only on the difference between the two instances,
𝐇 𝜏 = 𝑡 − 𝑠 , and let us denote by 𝐏(𝜏) this one-argument
function as
𝐇 𝜏 ≡ 𝐏 𝜏 ≡ 𝑝𝑖𝑗 𝜏 , 𝑖, 𝑗 ∈ ℕ.
◼ It follows that the transition rate matrix becomes constant
𝐇 𝑡, 𝑡 + ∆𝑡 − 𝐈 𝐏 ∆𝑡 − 𝐈
𝐐 𝑡 = lim = lim =𝐐
∆𝑡→0 ∆𝑡 ∆𝑡→0 ∆𝑡
▪ Thus, the forward CKE becomes
𝑑𝐏(𝜏) 1 if 𝑖 = 𝑗
= 𝐏 𝜏 𝐐 with 𝑝𝑖𝑗 0 = ቊ
𝑑𝜏 0 if 𝑖 ≠ 𝑗
⇒ 𝐏 𝜏 = 𝑒 𝐐𝜏 (This implies 𝐏 0 = 𝐈)
61
 State Holding Time
◼ We know that the time the CTMC will spend at state 𝑖, 𝑇𝑖 , is
memoryless and thus exponentially distributed by
𝐏{𝑇𝑖 ≤ 𝑡} = 1 − 𝑒 −𝜆𝑖𝑡
where 𝜆𝑖 is called the transition rate out of state 𝑖 to any
states 𝑗 that is reachable from state 𝑖, 𝑗 ≠ 𝑖, and is
interpreted as the average number of transitions out of
state 𝑖 per unit time when the CTMC stays at state 𝑖.
◼ Since only a fraction 𝑃𝑖𝑗 of the transitions out of state 𝑖 will
lead to state 𝑗, 𝑗 ≠ 𝑖, 𝜆𝑖𝑗 = 𝜆𝑖 𝑃𝑖𝑗 is considered as the
average number of transitions from 𝑖 to 𝑗 per unit time
spent at 𝑖.
◼ We call 𝜆𝑖𝑗 the transition rate from 𝑖 to 𝑗. We obtain 𝜆𝑖 as
𝜆𝑖 = ෍ 𝜆𝑖𝑗 62
𝑗≠𝑖
 Transition Rate Matrix Q (1)
◼ Recall that
𝑑𝐏(𝑡) 𝑑𝑝𝑖𝑗 (𝑡)
=𝐏 𝑡 𝐐⇒ = ෍ 𝑝𝑖𝑘 (𝑡)𝑞𝑘𝑗
𝑑𝑡 𝑑𝑡 𝑘
➢ Case 1) First consider the 𝑞𝑖𝑗 , 𝑗 ≠ 𝑖. Then the above becomes
𝑑𝑝𝑖𝑗 (𝑡)
= 𝑝𝑖𝑖 𝑡 𝑞𝑖𝑗 + ෍ 𝑝𝑖𝑘 (𝑡)𝑞𝑘𝑗
𝑑𝑡 𝑘≠𝑖
◼ Evaluating this at 𝑡 = 0, we have
𝑑𝑝𝑖𝑗 (𝑡)
ቤ = 𝑞𝑖𝑗 𝑝𝑖𝑖 0 = 1, 𝑝𝑖𝑘 0 = 0 for all 𝑘 ≠ 𝑖
𝑑𝑡 𝑡=0
▪ The event that will take the state from 𝑖 to j has exponential
lifetime with rate 𝜆𝑖𝑗 . Therefore, given that such a transition
has occurred in the interval of length 𝑡, the probability of this
transition, 𝑝𝑖𝑗 (𝑡), is given by 𝑝𝑖𝑗 𝑡 = 1 − 𝑒 −𝜆𝑖𝑗 𝑡 . 63
 Transition Rate Matrix Q (2)
◼ Thus,
𝑑𝑝𝑖𝑗 (𝑡) 𝑑(1 − 𝑒 −𝜆𝑖𝑗 𝑡 )
ቤ = ቤ = 𝜆𝑖𝑗 𝑒 𝜆𝑖𝑗 𝑡 ቚ = 𝜆𝑖𝑗
𝑑𝑡 𝑡=0 𝑑𝑡 𝑡=0 (1)
𝑡=0

◼ In summary, for 𝑗 ≠ 𝑖, 𝑞𝑖𝑗 equals to the Poisson rate that

activates a transition to state 𝑗 from state 𝑖, 𝑞𝑖𝑗 = 𝜆𝑖𝑗 (2)
➢ Case 2) Next, consider the 𝑞𝑖𝑖 . Then
𝑑𝑝𝑖𝑖 (𝑡)
= 𝑝𝑖𝑖 𝑡 𝑞𝑖𝑖 + ෍ 𝑝𝑖𝑘 (𝑡)𝑞𝑘𝑖
𝑑𝑡 𝑘≠𝑖
𝑝𝑖𝑖 0 = 1,
◼ Evaluating this at 𝑡 = 0, we have
𝑝𝑖𝑘 0 = 0 for all 𝑘 ≠ 𝑖
𝑑𝑝𝑖𝑖 (𝑡)
ቤ = 𝑞𝑖𝑖
𝑑𝑡 𝑡=0

64
 Transition Rate Matrix Q (3)
෍ 𝑝𝑖𝑗 (𝑡) = 1
𝑑[1 − σ𝑗≠𝑖 𝑝𝑖𝑗 𝑡 ] 𝑗
⇒ อ = 𝑞𝑖𝑖
𝑑𝑡
𝑡=0
𝑑[1 − σ𝑗≠𝑖 𝑝𝑖𝑗 𝑡 ] 𝑑𝑝𝑖𝑗 𝑡
⇒ อ = −෍ ቤ Using (1)
𝑑𝑡 𝑗≠𝑖 𝑑𝑡 𝑡=0
𝑡=0
= −෍ 𝜆𝑖𝑗 = −𝜆𝑖 = 𝑞𝑖𝑖 (3)
𝑗≠𝑖
◼ In summary, − 𝑞𝑖𝑖 equals to the transition rate out of state 𝑖,
that is, − 𝑞𝑖𝑖 = 𝜆𝑖 .
◼ Note that for each row 𝑖 of 𝐐, the sum is from (2) and (3)
෍ 𝑞𝑖𝑗 = ෍ 𝑞𝑖𝑗 + 𝑞𝑖𝑖 = ෍ 𝜆𝑖𝑗 − ෍ 𝜆𝑖𝑗 = 0.
𝑗 𝑗≠𝑖 𝑗≠𝑖 𝑗≠𝑖

65
 Transition Probabilities 𝑃𝑖𝑗
◼ After deriving the transition rates, it is easy to calculate the
transition probabilities. State transitions occur at random
points in time 𝑌1 < ⋯ < 𝑌𝑘 < ⋯. Let 𝑋𝑘 be the state right
after the transition at 𝑌𝑘 , which we call the embedded state.
◼ Define the transition probability of this embedded MC as
𝑃𝑖𝑗 = 𝐏{𝑋𝑘+1 = 𝑗|𝑋𝑘 = 𝑖} with ෍ 𝑃𝑖𝑗 = 1
𝑗≠𝑖
◼ Starting from state 𝑖, 𝑃𝑖𝑗 corresponds to the fraction 𝜆𝑖𝑗 of
the transitions to state 𝑗 to the total transitions 𝜆𝑖 out of
state 𝑖.
◼ Then, we have
𝜆𝑖𝑗 𝑞𝑖𝑗
𝑃𝑖𝑗 = = , 𝑗 ≠ 𝑖, and 𝑃𝑖𝑖 = 0.
𝜆𝑖 −𝑞𝑖𝑖
66
 Example: Computer System (1)
◼ Assume a computer system where jobs arrive according to
a Poisson process with rate 𝜆. Each job is processed
using a First-In-First-Out (FIFO) policy. The processing
time of each job is exponential with rate 𝜇. The computer
has buffer to store up to two jobs that wait for being
processed. Jobs that find the buffer full are lost.
◼ Draw the transition rate diagram.
◼ Find the transition rate matrix 𝐐.
◼ Find the transition probability matrix 𝐏.

67
 Example: Computer System (2)
𝜆 𝜆 𝜆

0 1 2 3 𝜆

𝜇 𝜇 𝜇
◼ The transition rate matrix is given by
−𝜆 𝜆 0 0
𝜇 −(𝜆 + 𝜇) 𝜆 0
𝐐=
0 𝜇 −(𝜆 + 𝜇) 𝜆
0 0 𝜇 −𝜇
◼ The transition probability matrix is given by
0 𝜆/𝜆 0 0
𝜇/(𝜆 + 𝜇) 0 𝜆/(𝜆 + 𝜇) 0
𝐏=
0 𝜇/(𝜆 + 𝜇) 0 𝜆/(𝜆 + 𝜇)
0 0 𝜇/𝜇 0
68
 Transient State Analysis
▪ Similar to the discrete-time case, we define
𝜋𝑗 𝑡 ≡ 𝐏{𝑋 𝑡 = 𝑗}
▪ In vector form, For the non-
𝝅 𝑡 = [𝜋0 𝑡 𝜋1 𝑡 𝜋2 𝑡 … ] homogeneous case,
▪ With initial probabilities obtaining a general
𝝅 0 = [𝜋0 0 𝜋1 0 𝜋2 0 … ] solution is not easy!

▪ Using our previous notation (for the homogeneous case)

𝝅 𝑡 = 𝝅 0 𝐏 𝑡 = 𝝅 0 𝑒 𝐐𝑡
▪ Differentiating with respect to 𝑡 gives us more “inside”
𝑑𝝅(𝑡) 𝑑𝜋𝑗 (𝑡)
=𝝅 𝑡 𝐐⇔ = 𝑞𝑗𝑗 𝜋𝑗 𝑡 + ෍ 𝑞𝑖𝑗 𝜋𝑖 (𝑡)
𝑑𝑡 𝑑𝑡
𝑖≠𝑗

69
 “Probability Fluid” view
◼ We view 𝜋𝑗 (𝑡) as the level of a “probability fluid” that is
stored at each node 𝑗 (0 = empty, 1 = full).
𝑑𝜋𝑗 (𝑡)
= ෍ 𝑞𝑖𝑗 𝜋𝑖 (𝑡) − ෍ 𝑞𝑗𝑘 𝜋𝑗 𝑡
𝑑𝑡 𝑘≠𝑗
𝑖≠𝑗
change in the
probability fluid inflow outflow

i 𝑞𝑖𝑗 𝑞𝑗𝑘 k
…

…
j ෍ 𝑞𝑗𝑘 𝜋𝑗 𝑡
𝑘≠𝑗
= −𝑞𝑗𝑗 𝜋𝑗 𝑡
Inflow Outflow
70
 Steady State Analysis (1)

◼ We are interested in the “long-run” probabilistic behavior of

the CTMC, i.e., 𝜋𝑗 = lim 𝐏{𝑋 𝑡 = 𝑗} (or 𝝅 = lim 𝝅(𝑡))
𝑡→∞ 𝑡→∞
◼ These are referred to as steady state or equilibrium state or
stationary state probabilities.
◼ Meanwhile, the limiting probabilities are defined by
𝜋𝑗∗ = lim 𝑝𝑖𝑗 (𝑡)
𝑡→∞
◼ As with the discrete-time case, we need to address the
following questions:
▪ Under what conditions do the limits exist?
▪ If they exist, do they form legitimate probabilities?
▪ How can we evaluate these limits?
71
 Steady State Analysis (2)
◼ Theorem: In an irreducible CTMC consisting of positive
recurrent states, a unique stationary state probability
vector π exists with 𝝅 = lim 𝝅 𝑡 .
𝑡→∞
◼ These vectors are independent of the initial state
probability and can be obtained by solving
𝝅𝐐 = 𝟎 and ෍ 𝜋𝑗 = 1.
𝑗
◼ Using the “probability fluid” view
inflow i 𝑑𝜋𝑗 (𝑡) k
𝑞𝑖𝑗 =0
𝑑𝑡 𝑞𝑗𝑘
0 = ෍ 𝑞𝑖𝑗 𝜋𝑖 (𝑡) − ෍ 𝑞𝑗𝑘 𝜋𝑗 𝑡 …

…
𝑘≠𝑗 j
𝑖≠𝑗

0 change outflow Inflow Outflow

72
 Example: Computer System (1)
𝜆 𝜆 𝜆

0 1 2 3 𝜆

𝜇 𝜇 𝜇
◼ For the previous computer system example, what are the
steady state probabilities?
◼ Solve
−𝜆 𝜆 0 0
𝜇 − 𝜆+𝜇 𝜆 0
𝝅𝐐 = 𝜋0 𝜋1 𝜋2 𝜋3 =𝟎
0 𝜇 − 𝜆+𝜇 𝜆
0 0 𝜇 −𝜇
𝜋0 + 𝜋1 + 𝜋2 + 𝜋3 = 1

73
 Example: Computer System (2)
◼ The solution is obtained as
𝜆
−𝜆𝜋0 + 𝜇𝜋1 = 0 ⇒ 𝜋1 = 𝜋0
𝜇
2
𝜆
𝜆𝜋0 − 𝜆 + 𝜇 𝜋1 + 𝜇𝜋2 = 0 ⇒ 𝜋2 = 𝜋0
𝜇
3
𝜆
𝜆𝜋1 − 𝜆 + 𝜇 𝜋2 + 𝜇𝜋3 = 0 ⇒ 𝜋3 = 𝜋0
𝜇
1
𝜋0 + 𝜋1 + 𝜋2 + 𝜋3 = 1 ⇒ 𝜋0 = 2 3
𝜆 𝜆 𝜆
1+ + +
𝜇 𝜇 𝜇

74
 Example: Birth-Death Chain (1)
λ0 λi-1 λi
0 1 … μi …
μ1 i μi+1

◼ Find the steady state probabilities (birth-and-death chain)

◼ Similar to the previous example,
−𝜆0 𝜆0 0 ⋯
𝜇1 − 𝜆1 + 𝜇1 𝜆1 ⋯
𝐐=
0 𝜇2 − 𝜆2 + 𝜇2 ⋯
⋮ ⋮ ⋮ ⋱
◼ And we solve
∞

𝝅𝐐 = 𝟎 and ෍ 𝜋𝑖 = 1
𝑖=0
75
 Example: Birth-Death Chain (2)
◼ The solution is obtained
𝜆0
−𝜆0 𝜋0 + 𝜇1 𝜋1 = 0 ⇒ 𝜋1 = 𝜋0
𝜇1
𝜆0 𝜆1
𝜆0 𝜋0 − 𝜆1 + 𝜇1 𝜋1 + 𝜇2 𝜋2 = 0 ⇒ 𝜋2 = 𝜋0
𝜇1 𝜇2
◼ In general,
𝜆0 ⋯ 𝜆𝑗−1
𝜆𝑗−1 𝜋𝑗−1 − 𝜆𝑗 + 𝜇𝑗 𝜋𝑗 + 𝜇𝑗+1 𝜋𝑗+1 = 0 ⇒ 𝜋𝑗 = 𝜋0
𝜇1 ⋯ 𝜇𝑗
◼ Making the sum equal to 1
∞
Solution exists if
∞
𝜆0 ⋯ 𝜆𝑗−1 𝜆0 ⋯ 𝜆𝑗−1
𝜋0 1+෍ =1 𝑆 =1+෍ <∞
𝜇1 ⋯ 𝜇𝑗 𝜇1 ⋯ 𝜇𝑗
𝑗=1 𝑗=1

76
Homework #8
Textbook “Introduction to Probability”, 2nd Edition, D. Bertsekas and J. Tsitsiklis
Chapter7 p.380-p.405, Problems 1, 4, 5, 10, 11, 13, 18
Due date:아주BB 과제출제 확인

Homework #9
Textbook “Introduction to Probability”, 2nd Edition, D. Bertsekas and J. Tsitsiklis
Chapter7 p.380-p.405, Problems 28, 29, 30, 32, 36, 37, 38
Due date: 아주BB 과제출제 확인

77