Lecture Notes on Operations Research

LECTURE 9
MARKOV ANALYSIS
Markov Concept

The Markovian theory holds that random natural or artificial events may exhibit a short
memory; that is to say a random event is influenced by the immediately preceding event.
Consequently, a process or feature, natural or artificial that has an element of randomness or
unpredictability, but in which the past event has influence on a subsequent event is said to have
Markovian properties.

In a Markov process, the probability of being in a given state at a given time or place is based
on the preceding state or states. If the frequencies or probabilities attached to the transitions
from one state to the next are established, it is possible to predict the times or places of
occurrences of other states.

A common natural process that has Markovian properties is the weather. The changes or
transitions of the different states of the weather (sunny, cloudy, rain) form a chain of events in
which the state of today’s weather has some influence on tomorrow’s weather. Another
example is times of economic expansion, economic stagnation and economic contraction; these
form a chain of events with Markovian properties. In a typical business environment, Markov
analysis is used to predict the likely future demand of commodities because they are greatly
influenced by current supply and demand (Thierauf et al. (1985)). These examples are
characterised by chains or sequences of discrete states in time or place that are essentially ‘one
dimensional’. The Markov concept has also been applied in geology. For example, the
successive occurrences of different minerals (states) in a column in rock represents a one-
dimensional sequence of events that exhibit strong Markovian properties. Davis (1986)
demonstrates the use of Markovian concept in studying natural stratigraphic sequence.

In mining, areas where Markov analysis is used include finding optimal maintenance policy,
determining future manpower needs, inventory control and marketing.

Terminologies
• State
• Transition probability (from transition frequency)
• Transition matrix
• Markov chains
• Memory (1st, 2nd, 3rd, … higher order Markov frequency)

These terminologies are best explained with an example. Suppose we are studying a
stratigraphic sequence and we observe that there occur lithologic transitions from bed to bed,
moving downwards in the stratigraphic sequence.

The lithologic types will be the states. The frequency of moving from one lithologic type to
another will be transition frequency from which the transition probability can be calculated.

Markov Analysis by V. A. Temeng 1

 Lecture Notes on Operations Research

If, for example, there are only three lithologies (states): sandstone (s1), shale (s2) and limestone
(s3), then the probabilities of moving from one lithology to another can be written in the form
of a 3 by 3 transition matrix as:

To
s1 s2 s3
s1  p11 p12 p13 
 
From s2  p21 p22 p23 
s3  p31 p32 p33 

If we have the following hypothetical transition matrix pertaining to the lithological transitions
in a descending sequence:

s1 s2 s3  ROM
s1 0 1 0  1
 1 2
s2 0 3 3  1
s3 1 0 1 1
2 2

This matrix says in effect that the only possible transition directly from sandstone (s1) is to
shale (s2). Thus, there is absolute certainty (ie, probability p12 = 1) that sandstone will be
succeeded immediately by shale. Also, there is absolute certainty that sandstone will not be
succeeded immediately by sandstone (ie., probability p11 = 0) or by limestone (ie., probability
p13 = 0). Note that each row sums to 1 reflecting the fact that there is absolute certainty of a
transition from any lithology to some lithology. (Ergodic Markov Chains are formed when it is
possible to go from one state to any other one).

The lithologies, sandstone, shale, and limestone are said to have Markovian properties because
the occurrence of a particular lithology is influenced by the preceding lithology. The lithologies
constitute a Markovian chain. In this situation where the pattern of occurrences of the
lithologies is limited to bed-bed observation, the Markov chain is said to be of the 1st order.

It is important to note that if we consider a succession of transitions, we can compute the

probabilities of moving from one state to another in 2, 3 or more steps. These probabilities can
be calculated by raising the matrix to successive powers (ie. 2nd, 3rd or higher orders). For
example,

s1 s2 s3 s1 s2 s3
2
s1 0 1 0  s1 0 1 2
  3
2
3
s2 1
0 3 3  = s 2 1 1 5

 13 9 9
s3 1 0 1 s3 
1 1

2 2 4 2 4

This matrix indicates that it is possible to go from sandstone (s1) to limestone (s3) in 2 steps. If
we raise the transition matrix to progressively higher powers, we will find that very soon the
values in each of the rows are identical. When this point is reached, there is no longer any
memory effect.

Markov Analysis by V. A. Temeng 2

 Lecture Notes on Operations Research

Process Modelling with Markovian Chains

First-order Markovian chains can be used to model a mining process, being it physical or
economic, if it has the following properties:

(i) The set of possible outcomes is finite;

(ii) The probability of the next outcome depends on the immediately preceding
outcome; and
(iii) The probabilities associated with the outcomes are constant over time.

Each individual outcome is a state. Thus, there will be as many states as there are possible
outcomes. For the purpose of notation, we shall let si denote the ith state out of a total m possible
states, where 2  m < . Each time a new result or outcome occurs, the process is said to have
stepped or incremented one step. This can be repeated over and over as many times as desired.
Any step may represent a time period or any condition which would result in another possible
outcome. The symbol n will be used to indicate the number of steps or increments. If n = 1, this
represent the possible outcome next time; and if n = 2, this represents the outcome time after
next. If the probabilities of going from every state to the same state and all other states are
known, then the transition probability matrix can be written to be the model of the Markov
chain.

In the example of the stratigraphic sequence of sandstone, shale and limestone that was used in
explaining the terminologies, the transition probability matrix (P) was as follows:

s1 s2 s3
s1 0 1 0 
 
P = s2 0 1 2 
 1 3 13 
s3 2 0 2
 

In this transition matrix, each element represents a probability of going from one state to
another. For notational purposes, an element in a transition matrix will be called P ij. This is
conditional probability that if the process is now in state i, it will be in state j in the next step.
For example, p23 = 2 3 means if the current bed in the stratigraphic sequence (s2 at n = 0) is
shale, then the probability that the next bed will be limestone (s3 at n = 1) is 2 3 .

A transition matrix must meet the following conditions:

(i) Each element must be a probability, its value being between 0 and 1. This simply
means it is impossible to have a negative probability greater than 1.

(ii) Each row must sum to exactly 1. If one sums the probabilities of all outcomes,
obviously the sum must be equal to 1.

The general mathematical model of the transition matrix (P) that represents a process with
Markovian properties can be written as follows:

Markov Analysis by V. A. Temeng 3

 Lecture Notes on Operations Research

s1 s2 s3  sm
s1  p11 p12 p13  p1m 
s2  p21 p22 p23  p2 m 
 
P = s3  p31 p32 p33  p 3m 
 
      
sm  pm1 pm 2 pm 3  pmm 

We note that for a given state at n = 0, a row exhaustively enumerates all possible states that
the process can take. Thus a row is a probability vector. This is to be expected since a vector is
simply a 1m matrix. For the purpose of notation, a row vector will be labelled as vi to represent
 
the ith row. For example, v2 = 0 13 2 3 . A probability vector must meet requirements similar
to those for a transition matrix:

0  pij  1

where:
m

p
j =1
ij i = 1, 2, 3 … m

We can thus say that a transition matrix P is composed of rows of probability vector vi.

Note: For those who may have forgotten matrix application, let us remind ourselves that given:

a11 a12 a13  b11 b12 b13 

   
A = a 21 a 22 a 23  B = b21 b22 b23  and C = c11 c12 
c13 , then
a 31 a 32 a 33  b31 b32 b33 


CA = c11a11 + c12 a 21 + c13 a 31 c11a12 + c12 a 22 + c13 a 32 c11a13 + c12 a 23 + c13 a 33 
a11b11 + a12 b21 + a13b31 a11b12 + a12 b22 + a13b32 a11b13 + a12 b23 + a13b33 
 
AB = a 21b11 + a 22 b21 + a 23b31 a 21b12 + a 22 b22 + a 23b32 a 21b13 + a 22 b23 + a 23b33 
a 31b11 + a 32 b21 + a 33b31 a 31b12 + a 32 b22 + a 33b32 a 31b13 + a 32 b23 + a 33b33 

Illustration of the Development of Transition Matrix

Assuming that three lithologies of sandstone (s1), shale (s2) and limestone (s3) are identified in
a geological field exercise. Figure 9.1 illustrates the sequence of occurrence of the three
lithologies in the exercise.

Looking at the bed-to-bed transitions from top to down, the following scenarios result:

s1 to s1 = 0/3 s2 to s1 = 2/3 s3 to s1 = 0/3

s1 to s2 = 1/3 s2 to s2 = 1/3 s3 to s2 = 2/3
s1 to s3 = 2/3 s2 to s3 = 0/3 s3 to s3 = 1/3

Markov Analysis by V. A. Temeng 4

 Lecture Notes on Operations Research

s1
s2
s1
s3
s3
s2
s1
s3
s2
s2
Figure 9.1

The transition matrix that results from the above lithologic transitions is:

s1 s2 s3
1 2
𝑠1 0 3 3
𝑠2 ⌊ 23 13 0⌋
𝑠3 0 2 1
3 3

The development of a transition matrix may also be illustrated with the following example.
Assuming Tarkwa-Nsuaem Municipality has four mobile service providers namely, MTN,
Vodafone (VOD), Tigo (TIG) and GLO. The current month’s subscribers are given as: 1000
for MTN, 500 for VOD, 400 for TIG and 100 for GLO.

A study on the movement of subscribers from one mobile service provider to the other from
the current month to the next month is given as: 50 subscribers will move from MTN to VOD,
50 from MTN to TIG and 100 from MTN to GLO. 50 subscribers will move from VOD to
MTN, 25 from VOD to TIG, and 25 from VOD to GLO. 50 subscribers will move from TIG
to MTN, 50 from TIG to GLO and no subscriber will move from TIG to VOD. 10 subscribers
will from GLO to MTN, 10 from GLO to VOD and none from GLO to TIG.

Based on the movement of subscribers the transition matrix on the monthly number of
subscribers for the mobile service providers can be developed as follows:

The monthly movement of subscribers from the current to the next month can be represented
as follows:

Current Month’s Next Month’s Subscribers

Subscribers 𝑀𝑇𝑁 𝑉𝑂𝐷 𝑇𝐼𝐺 𝐺𝐿𝑂
1000 𝑀𝑇𝑁 800 50 50 100
500 𝑉𝑂𝐷
[ 50 400 25 25 ]
400 𝑇𝐼𝐺 50 0 300 50
100 𝐺𝐿𝑂 10 10 0 80
̅̅̅̅̅
910 ̅̅̅̅̅
460 ̅̅̅̅̅
375 ̅̅̅̅̅
255

We can say from the matrix above that MTN lost a total of (50+50+100 = 200) subscribers to
other mobile service providers and only 800 of the original subscribers were maintained.
However, MTN also gained some new subscribers from other service providers: 50 from VOD,

Markov Analysis by V. A. Temeng 5

 Lecture Notes on Operations Research

50 from TIG and 10 from GLO. So, a total of (50+50+10 = 110) subscribers were gained from
the other service providers in the next month, which makes the next month’s total subscribers
for MTN (800+110 = 910) as shown above.

We can determine the total net loses/gains of the various service providers from the current to
the next month as follows. From the above matrix, a total net (1000-910 = 90) subscribers were
lost by MTN from the current to the next month to various service providers. A total net (500-
460 = 40) were lost by VOD to other service providers from the current month to the next
month, etc.

The monthly transition probabilities can be obtained by looking at the current monthly
subscribers as against the movements to various service providers. For instance, MTN has 1000
subscribers in the current month and in the next month lost 200 of them, remaining 800. That
means 800/1000 = 0.8 of the original MTN subscribers remained, giving a transition probability
of 0.8 from MTN to MTN in the next month; 50/1000 = 0.05 customers of MTN were lost to
VOD, giving a transition probability of 0.05 from MTN currently to VOD in the next month;
50/1000 = 0.05 of the current month’s MTN subscribers were lost to TIG, giving a transition
probability of 0.05 from MTN currently to TIG in the next month; and 100/1000 = 0.1 of the
current month’s MTN subscribers were lost to GLO, giving a transition probability of 0.1.

The monthly subscriber transition matrix can therefore be obtained by dividing mobile service
providers subscriber movement by the current month’s subscribers:

𝑀𝑇𝑁 𝑉𝑂𝐷 𝑇𝐼𝐺 𝐺𝐿𝑂

𝑀𝑇𝑁 0.8 0.05 0.05 0.1
𝑉𝑂𝐷 0.1 0.8 0.05 0.05 ]
[
𝑇𝐼𝐺 0.125 0 0.75 0.125
𝐺𝐿𝑂 0.1 0.1 0 0.8

Probability Analysis by Markov Chains

The transition probability can be used to determine the probability of an outcome after n steps,
in a Markov chain, given some specified starting state si. We shall demonstrate this with the
following illustrative example:

Question 1
The BGL exploits an alluvial deposit which has an erratic grade distribution. For mine planning
purposes, BGL has categorised the mining blocks into low grade, average grade and high grade.
A mining block lasts for one month. After mining several blocks, BGL has compiled the
following data which shows the number of transitions from one grade to another.

Mining Current Block Next Mining Block

(n = 0) (n = 1)
% Low grade (s1) % Average grade (s2) % High grade (s3)
Low grade (s1) 40 30 30
Average grade (s2) 20 50 30
High grade (s3) 25 25 50

Markov Analysis by V. A. Temeng 6

 Lecture Notes on Operations Research

(a) If the current block being mined by BGL is low grade, what is the probability that the block
to mine in 2 months time will be low grade?
(b) If the current block is high grade (s3 at n = 0),

(i) what are the probabilities of the block to be mined next month being high grade,
average grade and low grade (s1, s2 and s3 at n = 1)?
(ii) what are the probabilities of next 2 month’s mining block being low grade, average
grade and high grade (s1, s2 and s3 at n = 2)?

(c) What is the probability that in 4 months to come, the mining block will be high grade if the
current mining block is average grade?

Solution
The table can be transformed into probabilities as follows:

Mining Current Block Next Mining Block

(n = 0) (n = 1)
% Low grade (s1) % Average grade (s2) % High grade (s3)
Low grade (s1) 0.40 0.30 0.30
Average grade (s2) 0.20 0.50 0.30
High grade (s3) 0.25 0.25 0.50

The resulting table is simply a transition probability table which can be written out as:

s1 s2 s3
s1 0.40 0.30 0.30
 
s2 0.20 0.50 0.30
s3 0.25 0.25 0.50

 
From state si, v11 = 0.40 0.30 0.30 . This says if the current mining block is low grade (s1),
the probability that the next mining block will be low grade (s1) is p11 = 0.40, average grade
(s2) is p12 = 0.30 and high grade (s3) is p13 = 0.30. Thus the row vector v i1 gives the probabilities
of outcomes for the next step, n = 1, given a specified current state at n = 0. For the purpose of
notation, let vin be the probability vector which describes the probabilities of possible outcomes
in n steps, if the current state is si. The question (a) being considered will be answered if v 12
can be obtained. This gives the probabilities of all grades two steps (two months) from now
given that the current state (mining block) is low grade, s1 at n = 0. This information is obtained
from the product of v 11 and P:

0.40 0.30 0.30

 
2
1 1
1
 
v = V ( P) = 0.40 0.30 0.30 0.20 0.50 0.30
0.25 0.25 0.50


= 0 .40( 0.40) + 0.30( 0.20) + 0.30( 0.25) 0.40( 0.30) + 0.30( 0.50) + 0.30( 0.25)
0.40(0.30) + 0.30(0.30) + 0.30(0.50)

Markov Analysis by V. A. Temeng 7

 Lecture Notes on Operations Research


= 0.295 0.345 0.360 
This says that if the current states is s1 at n = 0, then two steps latter (n = 2) the probability of
being in state s1 is p11 = 0.295, in state s2 is p12 = 0.345 and in state s3 is p13 = 0.360.

(b) Given s3 at n = 0,


(i) V31 = 0.25 0.25 0.50 
This means given s3 at n = 0,
probability of being s1 at n = 1 is p31 = 0.25;
probability of being in s2 at n = 1 is p32 = 0.250; and
probability of being in s3 at n = 1 is p33 = 0.50.

(ii)
0.40 0.30 0.30
V = V (P ) = 0.25 0.25 0.500.20 0.50 0.30
2
3
1
3

0.25 0.25 0.50

= 0.275 0.325 0.400

This means given s3 at n = 0,

probability of being s1 at n = 2 is p31 = 0.275;
probability of being in s2 at n = 2 is p32 = 0.325; and
probability of being in s3 at n = 2 is p33 =0.400.

(c) Given s2 at n = 0, what is p23 at n = 4?

V21 = 0.20 0.50 0.30

V24 ( )
= V21 P 3

3
0.40 0.30 0.30 0.277 0.351 0.372
 
= 0.20 0.50 0.30 = 0.269 0.359 0.372
P3  
0.25 0.25 0.50 0.275 0.345 0.380

 V24 ( )
= V21 P 3

0.277 0.351 0.372

= 0.20 0.50 0.30 0.269 0.359 0.372
0.275 0.345 0.380

= 0.272 0.353 0.374

Markov Analysis by V. A. Temeng 8

 Lecture Notes on Operations Research

This means if the current mining block is average grade then the probability that in the next
four months the block to be mined will be high grade, p23 = 0.374.

Actually, if the results after n steps is desired, Pn gives even more complete information since
it is composed of all individual vectors Vin . Thus, Pn gives the probabilities of being in any
given state for all possible conditions or states. For example, if we are to find the probabilities
of moving from any mining block of any grade currently being mined into any mining block in
the next four months, then we have to evaluate P4.

s1 s2 s3
4
0.40 0.30 0.30 s1 0.274 0.355 0.374
P = 0.20 0.50 0.30
4
= s 2 0.272 0.353 0.374
 
0.25 0.25 0.50 s3 0.274 0.350 0.376

The first row gives the probabilities associated with starting state s1. The second row gives the
probabilities associated with starting state s2. The third row gives the probabilities associated
with starting state s3.

• We note that the second row is equal to V24 calculated in question (c).

The Generalised Solution (Summary):

In terms of the defined notation, the following has been demonstrated to be true:

(i) At time zero (right now) we know exactly what the state is; thus there is a probability
of 1 that state i exists.

Vi 0 =  pi1 pi 2 ... pii ... pim  and pii = 1

(ii) At the next step, n=1, the probability vector Vi n is the ith row in the transition
probability matrix P:

Vi1 = Vi 0 (P ) =  pi1 pi 2 ... pim 

The probability for results or outcomes in two steps 2 (n=2) is the product of the
probability vector Vi1 and the transition matrix P:

Vi 2 = Vi1 (P )
The following analysis is also true:

( )
Vi 3 = Vi 2 (P ) = Vi1 P (P ) = Vi1 P 2

( )
Vi 4 = Vi 3 (P ) = Vi1 P 2 (P ) = Vi1 P 3

( ) (
Vi n = Vi n −1 (P ) = Vi1 P n − 2 (P ) = Vi1 P n −1 )

Markov Analysis by V. A. Temeng 9

 Lecture Notes on Operations Research

Thus the probabilities of outcomes n steps from now can be determined using the probability
vector Vi1 and some power of the transition matrix P. Finally, the probabilities associated with
all outcomes in n steps from now can be determined when the transition probability matrix P is
raised to the power n (Pn).

Ergodic Markov Chains

An ergodic Markov chain describes a process in which it is possible to go from one state to any
other state. it is not necessary for this to be possible in just one step, but it must be possible for
any outcome to be reached regardless of the present state.

If it can be shown that, in a transition probability matrix, the process is ergodic, then it is
possible to determine what happens to the process in the long run. This means it is possible to
determine the probabilities of outcomes after steady-state conditions have been reached after
the process has been in operation for a long time (i.e. after the process has gone through several
steps). When a steady state condition has been reached, the probabilities in the transition matrix
become constant and the matrix can be described as being in equilibrium or being irreducible.
A transition probability matrix in steady-state condition permits us to ask the following
questions about the process:

(i) In the long run, what is the overall percentage of occurrence of a particular state?
(ii) What is the overall probability that a particular state will occur in the long run?

Answers to these questions enable us to estimate expected outcomes of given states in a process
that has Markovian properties.

Our first assignment then is to ensure that a given Markov chain is ergodic. To do this, the
transition probability matrix must be inspected to see whether it describes an ergodic chain.
This means we must check to see if it possible to get from every starting state or current state
to all other states (note that one may go directly from s1 to s2 but not directly from s2 to s1). As
an illustration, let us consider the following transition probability matrix in which X is some
positive pij value.

s1 s 2 s3 s 4 s5
s1  x x 0 x x
s2 0 x x 0 x 
 
P1 = s3 0 0 0 x x 
 
s4  x 0 x 0 x
s5  x x 0 0 0 

From state s1, it is possible to go directly to every state except s3. However, it is possible to go
from s1 to s2 then to s3. Thus it is possible to go from s1 to any other state. Now, we check to
all other states; it is sufficient to show that it is possible to get to state s1 from all other states
since that implies that it is possible to go to all other state:

From s2: go to s5, then from s5 to s1.

From s3: go to s5, then from s5 to s1.
From s4: go to s1.

Markov Analysis by V. A. Temeng 10

 Lecture Notes on Operations Research

From s5: go to s1.

The transition probability matrix (P1) is therefore for an ergodic Markov chain.
A more restricted case of an ergodic chain is a regular chain. a regular chain is defined as a
chain having a transition matrix P for which some power of P has only positive probability
elements (no zeros). The rule is that all regular Markovian chains are ergodic. However, not all
ergodic chains are regular. Therefore, if we can demonstrate that a given transition matrix is a
regular chain, then we can also say that the transition matrix is an ergodic chain and we can
find the steady-state conditions.

The easiest way to check if a chain is regular is to keep squaring the matrix until all zeros are
removed. If we take the following transition matrix P2 as an example:

s1 s 2 s3 s 4 s5 s1 s 2 s3 s 4 s5
s1  x x 0 x x s1 x x x x x
0 x x 0 x  s2 x x x x x 
s2
  
P1 = s3 0 0 0 x x  P12 = s 3 x x x 0 x
   
s4  x 0 x 0 x s4 x x 0 x x
s5  x x 0 0 0  s5  x x x x x 

s1 s 2 s3 s 4 s5
s1 x x x x x
s2 x x x x x 

P = s3
1
4
x x x x x
 
s4 x x x x x
s5  x x x x x 

since P14 has all its elements to be positive non-zero, we conclude that P14 represent a regular
Markovian chain and that P1 is an ergodic Markovian chain.

Let’s look at another example:

s1 s2 s3 s4 s1 s2 s3 s4 s1 s2 s3 s4
s1 x 0 x 0 s1 x 0 x 0 s1 x 0 x 0
0 x  0 x  0 x 
P = s2
1
2  x 0 P = s2
2
2
 x 0 P = s2
2
4
 x 0
s3 x 0 x 0 s3 x 0 x 0 s3 x 0 x 0
     
s4 0 x 0 x s4 0 x 0 x s4 0 x 0 x

This transition matrix P2 will continually repeat this pattern for all real powers of P2. Therefore
it is not regular and it can be concluded that the original transition matrix P2 is not ergodic.
True, if we check up the original transition matrix P2, we find that:

From s1: go to s1 or s3 and

From s3: go to s3 or s1.

Markov Analysis by V. A. Temeng 11

 Lecture Notes on Operations Research

Thus the original transition matrix P2 is not ergodic.

Determination of Steady-State Condition

The existence of steady-state conditions in a regular ergodic chain can be demonstrated most
easily by computing Pin for various values of n. As an example, if we take the grade transition
matrix of Bonte Mining Company,

s1 s2 s3 s1 s2 s3
s1 0.40 0.30 0.30 0.295 0.345 0.360
PB = s 2 0.20 0.50 0.30 P = 0.255 0.385 0.360
B
2

s3 0.25 0.25 0.50 0.275 0.325 0.400

s1 s2 s3
s1 0.2740 0.3516 0.3744
P = s 2 0.2724 0.3532 0.3744
B
4

s3 0.2740 0.3500 0.3760

s1 s2 s3
s1 0.273448 0.351576 0.374976
PB6 = s 2 0.273384 0.351640 0.374976
s3 0.273480 0.351400 0.375040

s1 s2 s3
s1 0.27343744 0.35156352 0.37499904
PB8 = s 2 0.27343448 0.35156608 0.37499904
s3 0.27344000 0.35155840 0.37500160

From the successive powers (n) of PB it can be seen that as n becomes larger, the probability
values Pij tend to a limit and each probability vector tends to become equal for all values of i.

Deductions:

The following deductions can be made:

(i) For a sufficiently large value of n, the probability vector Vin becomes equal for
all i and does not change for larger values of n.

(ii) Since Vi n +1 = Vi n P and Vi n +1 = Vi n , there exists at steady-state conditions, a

vector V* such that V*=V*P.

Markov Analysis by V. A. Temeng 12

 Lecture Notes on Operations Research

The Vector V* contains the probabilities which exist at steady-state conditions. The second
statement provides an analytical method for obtaining the probability values under steady-state
conditions. Since V* is still a probability vector, the following conditions must exist:

V
j =1
j =1

and

V1 ,V2 ,V3 ,...,Vm P = V1 ,V2 ,V3 ,...,Vm 

If this matrix product is expanded, there will be m equations. When added to the requirement
that the sum of probabilities in the probability vector must be equal to 1, there will be m+1
equations with m unknowns which can be solved. Normally, one of the expanded equations
can be discarded so that we have m equations with m unknowns for quick solution. Care must
be taken not to discard the equation:

V
j =1
j =1

Otherwise the remaining equations become meaningless when Vj=0. We shall demonstrate how
to find the steady-state probabilities with an example.

Question:
Given the grade transition matrix P3 of Bonte Mining Company (BMC), determine the steady-
state probabilities and hence predict the number of mining blocks expected to be low grade,
average grade and high grade if there are 100 mining blocks in all.

s1 s2 s3
s1 0.40 0.30 0.30
P3 = s 2 0.20 0.50 0.30
s3 0.25 0.25 0.50

Solution:

The following two conditions apply:

V
j =1
j =1

V1 V2 V3 P3 = V1 V2 V3 

This means:

v1 + v2 + v3 = 1

Markov Analysis by V. A. Temeng 13

 Lecture Notes on Operations Research

0.40 0.30 0.30

v1 v 2 v3  0.20 0.50 0.30 = v1 v 2 v3 
0.25 0.25 0.50

0.4v1 + 0.2v2 + 0.25v3 0.3v1 + 0.5v2 + 0.25v3 0.3v1 + 0.3v 2 + 0.5v3  = v1 , v 2 , v3 

which means:

v1 + v 2 + v3 = 1
0.4v1 + 0.2v 2 + 0.25v3 = v1
0.3v1 + 0.5v 2 + 0.25v3 = v 2
0.3v1 + 0.3v 2 + 0.50v3 = v3

Discarding the last equation and making algebraic manipulations gives:

v1 + v 2 + v3 = 1
− 0.6v1 + 0.2v 2 + 0.25v3 = 0
0.3v1 − 0.5v 2 + 0.25v3 = 0

 1 1 1  v1  1 
− 0.6 0.2 0.25 v  = 0
  2   
 0.3 − 0.5 0.25 v3  0

which gives the steady state probabilities to be:

v1 = 0.273
v 2 = 0.352
v3 = 0.375
We can now answer the question: out of 100 mining blocks, the expected number of blocks
with low, average and high grade are as follows:

(i) For low grade, the expected number of blocks is:

100(v1 ) = 100(0.273) = 27.3

(ii) For average grade, the expected number of blocks is:

100(v2 ) = 100(0.352) = 35.2

(iii) For high grade, the expected number of blocks is:

100(v3 ) = 100(0.375) = 37.5

Markov Analysis by V. A. Temeng 14

 Lecture Notes on Operations Research

Lecture Assignment

A manufacturing firm wishes to predict the status of machine operations in the future. Through
historical records, the firm has determined that if a machine breaks down in a particular week,
the probability of breakdown in the following week, after repair, is 0.2. However if the machine
has not broken down in a particular week, the probability of breakdown in the following week
is 0.6. The firm has therefore developed the following probability table of machine
breakdowns:

Week n + 1 status
Week n status No Breakdown Breakdown
No Breakdown 0.4 0.6
Breakdown 0.8 0.2

a. Assuming that the machine is not broken down in week n, what are the probabilities that
the machine will break down in week n+1, n+2, n+3, n+4 and n+5?
b. Determine the steady state condition (i.e. forecast the percentage of future weeks the
machine will breakdown).

Markov Analysis by V. A. Temeng 15