Math 106

Lecture 19
Long Range Predictions with Markov Chains

1
© m j winter, 2002

Transition Matrix; Transition Diagram

Recall the cereal Jaz.

Two states: buys Jaz (state 1), and buys not-Jaz
(state 2)

To 0.2
1 2 1 2
1 0.8 0.2  0.8 0.4 0.6
From  
2 0.4 0.6 
.

from state 2
2
to state 1

1
 Markov System (Chain)

• A system that can be in one of several (numbered)

states, and can pass from one state to another each
time step according to fixed probabilities.

• We use T for the transition matrix, and p for the

probability matrix (row matrix). The entries in p
represent the probabilities of finding the system in
each of the states.The sum of these probabilities is 1.

• We also regard p as describing the percentage

distribution in each state of the system.
3

probability (distribution) after n steps

If p is the initial
probability matrix, then
•pT is the probability
matrix after one step

•pT2 is the probability

matrix after 2 steps

•pTn is the probability

matrix after n steps
4

2
 (Jaz) probability (distribution) after n steps

If p is the initial In Jaz example

probability matrix, then
p = [ 0.25 0.75 ]
pT is the probability
matrix after one step pT = [ 0.5 0.5 ]
pT2= [ 0.6 0.4 ]

pT2 is the probability pT3= [ 0.64 0.36]

matrix after 2 steps pT4= [ 0.656 0.344]

pTn is the probability

matrix after n steps
5

Steady State

It appears that, if we had used pT4= [ 0.656 0.344]

as the initial probability matrix, the probability matrix after one
step would be (about) the same.

Define the equilibrium matrix, L, as the probability

matrix which is the solution to

LT = L.

Example: find the equilibrium matrix for the cereal system

3
 Systems with non-unique solutions

Example 1 Example 2
in matrix form
3x + y = 5
AX = X
6x + 2y = 10

solution set is every clearly X = O, the

point on the line zero matrix is a
y = 5 – 3x solution

Singular Matrix

Look at the system: If we try to find A–1 on

the calculator, we get the
 2 1 1   x  0 
1 −2 2   y  = 0  message
    
 3 −1 3   z  0 

One element of the solution

set is x = 0, y = 0, z = 0 If the matrix of coefficients is
singular, either the system
has NO solutions or it has
other solutions include infinitely many
x=4, y = – 3, z = – 5
8
x = 8, y = – 6, z = –10

4
 A closer look at this singular matrix

Look at the system: with augmented matrix

 2 1 1   x  0  2 1 1 0
1 −2 2   y  = 0  1 −2 2 0 
      
 3 −1 3   z  0   3 −1 3 0 

The third row/equation is

the sum of the other two. It
is not independent.

We might be able to add another

equation and still have a solution. 9

Equilibrium Matrix for Jaz Example

Let L = [ x y ] ; know x + y = 1
(0.8 − 1) x + 0.4 y = 0
0.2 x + (0.6 − 1) y = 0
Then LT=L becomes
.8 .2 
[x y]   = [ x y] −0.2 x + 0.4 y = 0
.4 .6 
0.2 x − 0.4 y = 0
or
0.8x + 0.4y = x
This is really only one equation.
0.2x + 0.6y = y
Use the fact that x + y = 1

10

5
 Solving for L
The system
L is sometimes called the (a)
x+y=1 steady-state probability matrix.
2x - 4y = 0
Technique: we used the fact that P
becomes
is a probability matrix to replace
x+y=1 one of the equations.
-6y = -2
or
x = 2/3; y = 1/3
L = [ 2/3 1/3 ] = [ .6666… .3333…. ]
11

Bookbuying Problem

Professor X is an avid reader of history books,

biographies, and mysteries. She buys and reads a
book every week.
*If she is presently reading a history book, there is a
50% chance that she will switch to a mystery the next
week.
*She never reads two biographies in a row; after a
week of reading a biography, there is a 75% chance
that she will switch to a history book the next week.
*She always reads history after reading a mystery.
*She never reads two history books in a row.
Assuming this trend continues indefinitely, what fraction
of her library will consist of biographies 1,000 weeks
from now?

12

6
 Transition Diagram & Matrix
States
1 History

2 Biography
 0 .50 .50 
3 Mystery
.75 0 .25 
 
 1 0 0 
0 .75 0
1 2
.50
1.0 .25
.50
3

0
13

Solution - by solving LT = L
L = [x y z] Calculate solution:
LT = L
use calculator
 0 .50 .50 
[x y z] .75 0 .25  = [x y z]
x = .47… History
 
 1 0 0  y = .23… Biography
z = .29… Mystery

-1x +.75y + 1z = 0
.50x - 1y + 0z = 0
.5x +.25y -1z = 0
x + y + z =1
14

7
 Long Term Transition Matrix
Assume
We can use T to find the
(1) that some power of the steady-state probability
transition matrix T has no zero
entries matrix.
(2) The powers of T approach a
fixed matrix T.

T is called the
steady-state transition matrix
or the
long-term transition matrix.

15

Show: For large values of n, Tn is, essentially, T

Tn *T = Tn+1 is also essentially T. TT=T

Write this equation in matrix form:

 [first row of T ]   [first row of T ] 

[second row of T ] [second row of T ]
 T =  
 :   : 
 [last row of T ]   [last row of T ] 
   

Each row of T is the steady state probability matrix L

where LT = L.
16

8
 Solution- bookbuying - by taking powers of T

screen showing A^8 and Here is A^64 stored as

A^64 matrix B

17