ESI 4313

Operations Research 2

Markov Chains Basics

Part 2
Example 3
 Recall the 1st Smalltown example:
 90% of all sunny days are followed by a sunny day
 80% of all cloudy days are followed by a cloudy day

 0.9 0.1 
P  
 0.2 0.8 

 0.9 0.1   0.9 0.1   0.83 0.17 

2
P    
 0.2 0.8   0.2 0.8   0.34 0.66 
Example 3
 Some n-step transition probability matrices are:

 0.83 0.17   0.9 0.1   0.78 0.22 

3
P    
 0.34 0.66   0.2 0.8   0.44 0.56 

 0.68 0.32  20  0.67 0.33 

P 10
  P  
 0.65 0.35   0.67 0.33 
 Long-run behavior
 Note that, as n grows large,
 the matrix Pn hardly changes
 the rows of the matrix are (approximately) the same
 Can you explain this?
 Let’s see what happens to Pn for the gambling
example
Example 2
 For simplicity, let us assume that x0=2
 The state space is then {0,1,2,3,4}
 Let’s also assume that p=½
 The transition probability matrix is then:

1 0 0 0 0
 
 ½ 0 ½ 0 0 
P 0 ½ 0 ½ 0
 
 0 0 ½ 0 ½
0 0 0 0 1
 
Example 2
 Some n-step transition probability matrices are:

1 0 0 0 0 1 0 0 0 0
1  5 1 
 2
1
4 0 1
4 0  8 0 1
4 0 8

P2   14 0 1
2 0 1 
4 P3   14 1
4 0 1
4
1 
4
  1 
0
1
4 0 1
4
1
2  8 0 1
4 0 5
8
0 0 0 0 1  0 0 0 0 1 
 
 Example 2
 Some more n-step transition probability
matrices are:

 1 0 0 0 0   1 0 0 0 0 
   
 .73 .02 0 .02 .23   .75 0 0 0 .25 
P 10   .485 0 .03 0 .485  P 15   .50 0 0 0 .50 
   
 .23 .02 0 .02 .73   .25 0 0 0 .75 
 0 0 0 0 1   0 0 0 0 1 
  
 Long-run behavior
 Again, it seems that, as n grows large,
 the matrix Pn hardly changes
 However, in this case
 the rows of the matrix do not approach the same
values
 Can you explain this?
Example 2
 Next, let us look at the case p=¼
 The transition probability matrix is then:

1 0 0 0 0
3 
 4 0 1
4 0 0
P  0 3
4 0 1
4 0
 
0 0 0
3 1
4 4
0 0 0 0 1 

 Example 2
 Some n-step transition probability matrices are:

 1 0 0 0 0   1 0 0 0 0 
   
 .75 .19 0 .06 0   .89 0 .09 0 .02 
P   .56 0 .38 0 .06  P   .56 .28 0 .10 .06 
2 3

   
 0 .56 0 .19 .25   .42 0 .28 0 .30 
 0 0 0 0 1   0 0 0 0 1 
 
 Example 2
 Some more n-step transition probability
matrices are:

 1 0 0 0 0   1 0 0 0 0 
   
.97 .005 0 0 .025  .975 0 0 0 .025 
 
.10  P   .90 0 .10 
15
P 10   .89 0 .01 0 0 0
   
 .66 .01 0 .01 .32   .675 0 0 0 .325 
 0 1   0
 0 0 0  0 0 0 1 
 Classification of states and Markov
chains
 In terms of their limiting behavior, there seem to
be different types of Markov chains
 We will next discuss a useful way to classify
different states and Markov chains
 Communicating states
 We say that two states communicate if they are
reachable from each other

1-p 1-p 1-p

1 0 1 2 3 4 1
p p p

 States 1, 2, and 3 communicate with each other

 States 0 and 4 do not communicate with any other
state
 Closed sets
 We say that a set of states is closed if the
Markov chain cannot leave that set of states

1-p 1-p 1-p

1 0 1 2 3 4 1
p p p

 The sets {0} and {4} are closed sets

 Closed sets – another example
 A closed set can contain more than one state
 Example:

½ 1-p 1-p 1-p

-1 0 1 2 3 4 1
½ p p p

 The sets {-1,0} and {4} are closed sets

 Absorbing states
 Any state that we can never leave is called an
absorbing state

1-p 1-p 1-p

1 0 1 2 3 4 1
p p p

 The states 0 and 4 are absorbing states

 Transient states
 If it is possible to leave a state and never return
to it, that state is called a transient state

1-p 1-p 1-p

1 0 1 2 3 4 1
p p p

 The states 1, 2, and 3 are transient states

 Recurrent states
 Any state that is not transient is called a
recurrent state

½ 1-p 1-p 1-p

-1 0 1 2 3 4 1
½ p p p

 The states -1, 0 and 4 are recurrent states

 (A)Periodic states
 If all paths leading from a state back to that
state have a length that is a multiple of some
number k>1 that state is called a periodic state
 The smallest such k is called the period of the state.
1-p
0 1
p All states are periodic
1-p p p
1-p with period 2.
p
3 2
1-p
 Ergodic Markov chains
 If all states in a Markov chain communicate with
each other and are
 Recurrent
 Aperiodic
then the chain is called ergodic
 Example:
 Smallville’s weather Markov chain
but not:
 Gambling Markov chain
 Limiting behavior
 If a Markov chain is ergodic, then there exists
some vector
   1  2   s 
such that
 1 2   s 
 
n   1 2   s 
lim P 
n     
 
 1 2   s 
 Limiting behavior
 Interpretation:
 After many transitions, the probability that we are in
state i is approximately equal to i, independent of
the starting state
 The values i are called the steady-state probabilities,
and the vector  is called the steady-state
(probability) distribution or equilibrium distribution
 Limiting behavior
 Can you explain why a steady-state distribution
exists for ergodic Markov chains, but not for
other chains?
 Consider the case of
 Periodic chains

 non-communicating pairs of states

 In the following, we will assume we have an

ergodic Markov chain
 Limiting behavior
 How do we compute the steady-state
probabilities ?
 Recall the following relationship between the
multi-step transition probabilities:
s
Pij (n  1)  P
k 1
ik (n)  pkj

 The theoretical result says that, for large n

Pij (n  1)   j
Pik (n)   k
 Limiting behavior
 So, for large n,
s
Pij (n  1)  P
k 1
ik (n)  pkj

becomes
s
j  
k 1
k pkj

 In matrix notation,   P
 Limiting behavior
 But…    P
 Has   0 as a solution…
 In fact, it has an infinite number of solutions
(because P is singular – why?)
 Fortunately, we know that  should be a probability
distribution, so not all solutions are meaningful!
 We should ensure that


i 1
i 1
Example 3
 Recall the 1st Smalltown example:

 90% of all sunny days are followed by a sunny day

 80% of all cloudy days are followed by a cloudy day

 Find the steady-state probabilities

 0.9 0.1 
P  
 0.2 0.8 
Example 3
 We need to solve the system
 0.9 0.1 
 1 2  1 2 
     
 0.2 0.8 
1   2  1
or
 1  0.9 1  0.2 2
 2  0.1 1  0.8 2
1   2  1
 We can ignore one of the first 2 equations!
(Why?)
Example 3
 Solving
 1  0.9 1  0.2 2
1   2  1

gives   1  2    2 3 1
3 
 Compare with

20  0.67 0.33 
P  
 0.67 0.33 
 Mean first passage time
 Consider an ergodic Markov chain
 Suppose we are currently in state i
 What is the expected number of transitions until we reach state
j?
 This is called the mean first passage time from state i to
state j and is denoted by mij
 For example, in Smallville’s weather example, m12 would be the
expected number of days until the first cloudy day, given that it
is currently sunny
 How can we compute these quantities?
 Mean first passage time
 We are currently in state i
 In the next transition, we will go to some state k
 If k=j, the first passage time from i to j is 1
 If kj, the mean first passage time from i to j is 1+mkj
 So: s
mij  pij  1  p
k 1
ik (1  mkj )
k j s s
s
1 p ik mkj  p
k 1
ik  p
k 1
ik mkj
k 1
k j k j
 Mean first passage time
 We can thus find all mean first passage times by
solving the following system of equations:
s s
mij  p
k 1
ij  p
k 1
ik mkj i  1,..., s; j  1,..., s
k j
 What is mii?
 The mean number of transitions until we return to
state i
 This is equal to 1/i !
Example 3
 Recall the 1st Smalltown example:

 90% of all sunny days are followed by a sunny day

 80% of all cloudy days are followed by a cloudy day

 0.9 0.1 
P  
 0.2 0.8 
 The steady-state probabilities are 1=2/3 and
2=1/3
Example 3
 Thus:
 m11 =1/1 = 1/(2/3) = 1½
 m22 =1/2 = 1/(1/3) = 3
 And m12 and m21 satisfy:

m12  1  p11m12 m12  1  0.9m12

m21  1  p22 m21 m21  1  0.8m21
m12  10
m21  5
Absorbing chains
 While many practical Markov chains are
ergodic, another common type of Markov chain
is one in which
 some states are absorbing
 the others are transient
 Examples:
 Gambling Markov chain
 Work-force planning
Example
 State College admissions office has modeled
the path of a student through State College as
a Markov chain
 States:
 1=Freshman, 2=Sophomore, 3=Junior, 4=Senior,
5=Quits, 6=Graduates
 Based on past data, the transition probabilities have
been estimated
Example
 Transition probability matrix:
1 2 3 4 5 6
1 .10 .80 0 0 .10 0

 
2 0 .10 .85 0 .05 0

3 0 0 .15 .80 .05 0 
 
4 0 0 0 .10 .05 .85 
5 0 0 0 0 1 0 
 
6 0 0 0 0 0 1 
Example
 Clearly, states 5 and 6 are absorbing states,
and states 1-4 are transient states
 Given that a student enters State College as a
freshman, how many years will be spent as
freshman, sophomore, junior, senior before
entering one of the absorbing states?
Example
 Consider the expected number of years spent
as a freshman before absorption (quitting or
graduating)
 Definitely count the first year
 If the student is still a freshman in the 2nd year,
count it also
 If the student is still a freshman in the 3rd year,
count it also
 Etc.
 Example
 Recall the transition probability matrix:
matrix Q matrix R
.10 .80 0 0 .10 0
 
0 .10 .85 0 .05 0
0 0 .15 .80 .05 0 
P  
0 0 0 .10 .05 .85 
0 0 0 0 1 0 
 
0 0 0 0 0 1  matrix I
Example
 The number of years spent in states 1-4 when
entering State College as a freshman can now
be found as the first row of the matrix (I-Q)-1
 If a student enters State College as a
freshman, how many years can the student
expect to spend there?
 Sum the elements in the first row of the matrix (I-
Q)-1
Example
 In this example:
1.11 0.99 0.99 0.88 
 
 0 1.11 1.11 0.99 
(I  Q)1 
 0 0 1.18 1.05 
 
 0 0 0 1.11 
 The expected (average) amount of time spent
at State College is: 1.11 + 0.99 + 0.99 + 0.88
= 3.97 years
 Example
 What is the probability that a freshman eventually
graduates?
 We know the probability that a starting freshman will
have graduated after n years from the n-step transition
probability matrix
 It’s the (1,6) element of Pn !
 The probability that the student will eventually
graduate is thus
 the (1,6) element of limnPn !
Example

  n 1
 
n  lim Q  lim  Q  R 
n

lim P   n   n 
 0  
n 
 0 I 
 
 0  I  Q  1 R 
 
0 I
 
Example
 So the probability that a student that starts as
a freshman will eventually graduate is
 the (1,6-4=2) element of (I-Q)-1R
 In the example:
 0.25 0.75 
 
1  0.16 0.84 
(I  Q) R 
 0.11 0.89 
 
 0.06 0.94 