Slides by

John
Loucks
St. Edward’s
University

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
1
 Chapter 16: Markov Processes

 Market Share Analysis

 Transition Probabilities
 Steady-State Probabilities
 Absorbing States
 Transition Matrix with Submatrices
 Fundamental Matrix

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
2
 Markov Processes

 Markov process models are useful in studying the

evolution of systems over repeated trials or sequential
time periods or stages.
• the promotion of managers to various positions
within an organization
• the migration of people into and out of various
regions of the country
• the progression of students through the years of
college, including eventually dropping out or
graduating

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
3
 Markov Processes

 Markov processes have been used to describe the

probability that:
• a machine that is functioning in one period will
function or break down in the next period.
• a consumer purchasing brand A in one period will
purchase brand B in the next period.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
4
 Example: Market Share Analysis

Suppose we are interested in analyzing the market

share and customer loyalty for Murphy’s Foodliner and
Ashley’s Supermarket, the only two grocery stores in a
small town. We focus on the sequence of shopping trips
of one customer and assume that the customer makes
one shopping trip each week to either Murphy’s
Foodliner or Ashley’s Supermarket, but not both.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
5
 Example: Market Share Analysis

We refer to the weekly periods or shopping trips as

the trials of the process. Thus, at each trial, the
customer will shop at either Murphy’s Foodliner or
Ashley’s Supermarket. The particular store selected in a
given week is referred to as the state of the system in
that period. Because the customer has two shopping
alternatives at each trial, we say the system has two
states.
State 1. The customer shops at Murphy’s Foodliner.
State 2. The customer shops at Ashley’s Supermarket.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
6
 Example: Market Share Analysis

Suppose that, as part of a market research study, we

collect data from 100 shoppers over a 10-week period.
In reviewing the data, suppose that we find that of all
customers who shopped at Murphy’s in a given week,
90% shopped at Murphy’s the following week while 10%
switched to Ashley’s.
Suppose that similar data for the customers who
shopped at Ashley’s in a given week show that 80%
shopped at Ashley’s the following week while 20%
switched to Murphy’s.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
7
 Transition Probabilities

 Transition probabilities govern the manner in which

the state of the system changes from one stage to the
next. These are often represented in a transition
matrix.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
8
 Transition Probabilities

 A system has a finite Markov chain with stationary

transition probabilities if:
• there are a finite number of states,
• the transition probabilities remain constant from
stage to stage, and
• the probability of the process being in a particular
state at stage n+1 is completely determined by the
state of the process at stage n (and not the state at
stage n-1). This is referred to as the memory-less
property.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
9
 Example: Market Share Analysis

 Transition Probabilities

pij  probability of making a transition from state i

in a given period to state j in the next period

p11 p12 0.9 0.1

P= =
p21 p22 0.2 0.8

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
10
 Example: Market Share Analysis

 State Probabilities

Murphy’s
.9 P = .9(.9) = .81
Murphy’s
Ashley’s
.9
Murphy’s .1 P = .9(.1) = .09
Murphy’s P = .1(.2) = .02
Ashley’s .2
.1
Ashley’s P = .1(.8) = .08
.8

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
11
 Example: Market Share Analysis

 State Probabilities for Future Periods

Beginning Initially with a Murphy’s Customer

 State Probabilities for Future Periods

Beginning Initially with an Ashley’s Customer

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
12
 Steady-State Probabilities

 The state probabilities at any stage of the process can

be recursively calculated by multiplying the initial state
probabilities by the state of the process at stage n.
 The probability of the system being in a particular
state after a large number of stages is called a steady-
state probability.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
13
 Steady-State Probabilities

 Steady state probabilities can be found by solving the

system of equations P =  together with the
condition for probabilities that i = 1.
• Matrix P is the transition probability matrix
• Vector  is the vector of steady state probabilities.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
14
 Example: Market Share Analysis

 Steady-State Probabilities

Let 1 = long run proportion of Murphy’s visits

2 = long run proportion of Ashley’s visits
Then,

.9 .1
[1 2] = [1 2]
.2 .8

continued . . .

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
15
 Example: Market Share Analysis

 Steady-State Probabilities

.91 + 2 = 1 (1)

11 + 2 = 2 (2)
1 + 2 = 1 (3)

Substitute 2 = 1 - 1 into (1) to give:

1 = .91 + .2(1 - 1) = 2/3 = .667

Substituting back into (3) gives:
2 = 1/3 = .333

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
16
 Example: Market Share Analysis

 Steady-State Probabilities

Thus, if we have 1000 customers in the system, the

Markov process model tells us that in the long run,
with steady-state probabilities 1  .667 and 2  .333,
21
3 667 customers will be Murphy’s and 333 customers
will be Ashley’s.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
17
 Example: Market Share Analysis

Suppose Ashley’s Supermarket is contemplating

an advertising campaign to attract more of Murphy’s
customers to its store. Let us suppose further that
Ashley’s believes this promotional strategy will
increase the probability of a Murphy’s customer
switching to Ashley’s from 0.10 to 0.15.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
18
 Example: Market Share Analysis

 Revised Transition Probabilities

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
19
 Example: Market Share Analysis

 Revised Steady-State Probabilities

.851 + .202 = 1 (1)

.151 + .802 = 2 (2)
1 + 2 = 1 (3)

Substitute 2 = 1 - 1 into (1) to give:

1 = .851 + .20(1 - 1) = .57

Substituting back into (3) gives:
2 = .43

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
20
 Example: Market Share Analysis

Suppose that the total market consists of 6000

customers per week. The new promotional strategy will
increase the number of customers doing their weekly
shopping at Ashley’s from 2000 to 2580.
If the average weekly profit per customer is $10, the
proposed promotional strategy can be expected to
increase Ashley’s profits by $5800 per week. If the cost
of the promotional campaign is less than $5800 per
week, Ashley should consider implementing the
strategy.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
21
 Example: North’s Hardware

Henry, a persistent salesman, calls North's

Hardware Store once a week hoping to speak with
the store's buying agent, Shirley. If Shirley does not
accept Henry's call this week, the probability she
will do the same next week is .35. On the other
hand, if she accepts Henry's call this week, the
probability she will not do so next week is .20.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
22
 Example: North’s Hardware

 Transition Matrix

Next Week's Call

Refuses Accepts
This Refuses .35 .65
Week's
Call Accepts .20 .80

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
23
 Example: North’s Hardware

 Steady-State Probabilities
Question
How many times per year can Henry expect to
talk to Shirley?
Answer
To find the expected number of accepted calls per
year, find the long-run proportion (probability) of a
call being accepted and multiply it by 52 weeks.
continued . . .

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
24
 Example: North’s Hardware

 Steady-State Probabilities
Answer (continued)
Let 1 = long run proportion of refused calls
2 = long run proportion of accepted calls
Then,

.35 .65
[ ] = [ ]
.20 .80

continued . . .

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
25
 Example: North’s Hardware

 Steady-State Probabilities
Answer (continued)

.351 + .202 = 1 (1)

.651 + .802 = 2 (2)
1 + 2 = 1 (3)

Solve for 1 and 2.

continued . . .

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
26
 Example: North’s Hardware

 Steady-State Probabilities
Answer (continued)
Solving using equations (2) and (3). (Equation 1 is
redundant.) Substitute 1 = 1 - 2 into (2) to give:

.65(1 - 2) + .802 = 2

This gives 2 = .76471. Substituting back into

equation (3) gives 1 = .23529.
Thus the expected number of accepted calls per
year is:
(.76471)(52) = 39.76 or about 40

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
27
 Example: North’s Hardware

 State Probability
Question
What is the probability Shirley will accept
Henry's next two calls if she does not accept his
call this week?

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
28
 Example: North’s Hardware

 State Probability

Answer Refuses
.35 P = .35(.35) = .1225
Refuses
Accepts
.35
Refuses .65 P = .35(.65) = .2275
Refuses P = .65(.20) = .1300
Accepts .20
.65
Accepts P = .65(.80) = .5200
.80

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
29
 Example: North’s Hardware

 State Probability
Question
What is the probability of Shirley accepting
exactly one of Henry's next two calls if she accepts
his call this week?

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
30
 Example: North’s Hardware

 State Probability
Answer
The probability of exactly one of the next two calls
being accepted if this week's call is accepted can be
found by adding the probabilities of (accept next week
and refuse the following week) and (refuse next week
and accept the following week) =

.13 + .16 = .29

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
31
 Absorbing States

 An absorbing state is one in which the probability that

the process remains in that state once it enters the
state is 1.
 If there is more than one absorbing state, then a
steady-state condition independent of initial state
conditions does not exist.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
32
 Transition Matrix with Submatrices

 If a Markov chain has both absorbing and

nonabsorbing states, the states may be rearranged
so that the transition matrix can be written as the
following composition of four submatrices: I, 0, R, and
Q:

I 0

R Q

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
33
 Transition Matrix with Submatrices

I = an identity matrix indicating one always

remains in an absorbing state once it is reached
0 = a zero matrix representing 0 probability of
transitioning from the absorbing states to the
nonabsorbing states
R = the transition probabilities from the
nonabsorbing states to the absorbing states
Q = the transition probabilities between the
nonabsorbing states

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
34
 Example: Jetair Aerospace

The vice president of personnel at Jetair Aerospace

has noticed that yearly shifts in personnel can be
modeled by a Markov process. The transition matrix is:
Next Year
Same Pos. Promotion Retire Quit Fired
Current Year
Same Position .55 .10 .05 .20 .10
Promotion .70 .20 0 .10 0
Retire 0 0 1 0 0
Quit 0 0 0 1 0
Fired 0 0 0 0 1

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
35
 Example: Jetair Aerospace

 Transition Matrix

Next Year
Retire Quit Fired Same Promotion
Current Year
Retire 1 0 0 0 0
Quit 0 1 0 0 0
Fired 0 0 1 0 0
Same .05 .20 .10 .55 .10
Promotion 0 .10 0 .70 .20

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
36
 Fundamental Matrix

 The fundamental matrix, N, is the inverse of the

difference between the identity matrix I and the Q
matrix.
N = (I - Q )-1

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
37
 Example: Jetair Aerospace

 Fundamental Matrix

-1 -1
1 0 .55 .10 .45 -.10
N = (I - Q ) -1 = - =
0 1 .70 .20 -.70 .80

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
38
 Example: Jetair Aerospace

 Fundamental Matrix
The determinant, d = a11a22 – a21a12
= (.45)(.80) - (-.70)(-.10) = .29
Thus,

.80/.29 .10/.29 2.76 .34

N= =
.70/.29 .45/.29 2.41 1.55

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
39
 NR Matrix

 The NR matrix is the product of the fundamental (N)

matrix and the R matrix.
 It gives the probabilities of eventually moving from
each nonabsorbing state to each absorbing state.
 Multiplying any vector of initial nonabsorbing state
probabilities by NR gives the vector of probabilities
for the process eventually reaching each of the
absorbing states. Such computations enable
economic analyses of systems and policies.

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
40
 Example: Jetair Aerospace

 NR Matrix
The probabilities of eventually moving to the
absorbing states from the nonabsorbing states are
given by:

2.76 .34 .05 .20 .10

NR = x
2.41 1.55 0 .10 0

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
41
 Example: Jetair Aerospace

 NR Matrix (continued)

Retire Quit Fired

Same .14 .59 .28

NR =
Promotion .12 .64 .24

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
42
 Example: Jetair Aerospace

 Absorbing States
Question
What is the probability of someone who was just
promoted eventually retiring? . . . quitting? . . .
being fired?

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
43
 Example: Jetair Aerospace

 Absorbing States (continued)

Answer
The answers are given by the bottom row of the
NR matrix. The answers are therefore:

Eventually Retiring = .12

Eventually Quitting = .64
Eventually Being Fired = .24

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
44
 End of Chapter 16

© 2016 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted
in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
45