Example: (s,S) Inventory Control System with Periodic Review

Suppose that the inventory of a certain item is counted at the end

of each business day, after a random demand has occurred.
If the number of units is ≤ s=2, the reorder point, enough is
ordered to bring the inventory level up to S=6. (Assume that the
replenishment is instantaneous.)
The daily demand has probability distribution:
Demand d 0 1 2 3 4 5 ≥6
P{D=d} 0.04979 0.1494 0.224 0.224 0.168 0.1008 0.05041

(which is Poisson with expected value 3).

DTMC/s,S) inventory system page 1 ©D.Bricker, U. of Iowa, 2001

Define
♦ the state of the system to be the inventory level, i.e.,
I= {0, 1, 2, …6} before any replenishment, and
♦ the stages to be the days.
Transition probabilities are:
 P { D = i − j} if i>s and i ≥ j

pij =  P { D = S − j} if i ≤ s
0 otherwise


DTMC/s,S) inventory system page 2 ©D.Bricker, U. of Iowa, 2001

Examples of system characteristics that may be of interest:

♦ if the system begins in state S=6, what is the probability of the

first stockout occurring 3 days hence?
Answer: f S(,0)
3

♦ if the system begins in state S=6, what is the average length of

time until a stockout (zero inventory) occurs?
Answer: mS ,0

DTMC/s,S) inventory system page 3 ©D.Bricker, U. of Iowa, 2001

 ♦ if the system begins in state S, what is the expected number of
stockouts during the next n days?
Answer: ∑ p( )
k ≤n
n
S, j

♦ over a sufficiently long period of time, what is the fraction of the

days that the end-of-day inventory level is i?
Answer: π i

DTMC/s,S) inventory system page 4 ©D.Bricker, U. of Iowa, 2001

Transition Probabilities
from \to 0 1 2 3 4 5 6
0 0.08392 0.1008 0.168 0.224 0.224 0.1494 0.04979
1 0.08392 0.1008 0.168 0.224 0.224 0.1494 0.04979
2 0.08392 0.1008 0.168 0.224 0.224 0.1494 0.04979
3 0.5768 0.224 0.1494 0.04979 0 0 0
4 0.3528 0.224 0.224 0.1494 0.04979 0 0
5 0.1847 0.168 0.224 0.224 0.1494 0.04979 0
6 0.08392 0.1008 0.168 0.224 0.224 0.1494 0.04979

Note that because the state is the inventory level before

replenishments, the transition probabilities out of states 0, 1,
and 2=s are identical to those for state S=6.
Thus, p64 = P{Demand = 6 − 4 = 2} = 0.224, and
p30 = P{Demand ≥ 3} = 0.5768

DTMC/s,S) inventory system page 5 ©D.Bricker, U. of Iowa, 2001

Two simulations of 20 stages of the system,
beginning with full inventory:

In both of these realizations of the process, four stockouts have

occurred during the first 20 days of operation.

DTMC/s,S) inventory system page 6 ©D.Bricker, U. of Iowa, 2001

The expected number of visits to state j during the first n stages, if
the system begins in state i, may be found by summing the first n
powers of P. In this particular case,
20

∑P
k =1
k
=

from \to 0 1 2 3 4 5 6
0 4.35285 2.92231 3.55777 3.65758 3.06261 1.85714 0.589734
1 4.35285 2.92231 3.55777 3.65758 3.06261 1.85714 0.589734
2 4.35285 2.92231 3.55777 3.65758 3.06261 1.85714 0.589734
3 4.72536 3.00256 3.52746 3.51926 2.90429 1.7616 0.559473
4 4.54813 3.01517 3.60129 3.6023 2.9318 1.74659 0.554716
5 4.43127 2.97713 3.60575 3.66134 3.00358 1.77409 0.546833
6 4.35285 2.92231 3.55777 3.65758 3.06261 1.85714 0.589734

20

∑ p6,0( ) = 4.35 stockouts during

k
Thus, we would expect approximately
k =1
the first twenty days of operation, if the inventory is initially full.

DTMC/s,S) inventory system page 7 ©D.Bricker, U. of Iowa, 2001

P 2 , the second power of the transition probability matrix, is

from \to 0 1 2 3 4 5 6
0 0.2696 0.1661 0.1848 0.1683 0.1237 0.06756 0.02004
1 0.2696 0.1661 0.1848 0.1683 0.1237 0.06756 0.02004
2 0.2696 0.1661 0.1848 0.1683 0.1237 0.06756 0.02004
3 0.1085 0.107 0.1671 0.2154 0.2129 0.1419 0.04731
4 0.1709 0.1254 0.168 0.1943 0.1819 0.1196 0.03987
5 0.2395 0.1502 0.175 0.1738 0.1441 0.08863 0.02872
6 0.2696 0.1661 0.1848 0.1683 0.1237 0.06756 0.02004

and the third power is

from \to 0 1 2 3 4 5 6
0 0.2069 0.1413 0.1756 0.1855 0.1597 0.09903 0.03189
1 0.2069 0.1413 0.1756 0.1855 0.1597 0.09903 0.03189
2 0.2069 0.1413 0.1756 0.1855 0.1597 0.09903 0.03189
3 0.2616 0.1631 0.1839 0.1706 0.1281 0.07126 0.0214
4 0.2406 0.1552 0.1813 0.1766 0.1399 0.08126 0.0251
5 0.2173 0.146 0.1778 0.183 0.1534 0.09305 0.02954
6 0.2069 0.1413 0.1756 0.1855 0.1597 0.09903 0.03189

DTMC/s,S) inventory system page 8 ©D.Bricker, U. of Iowa, 2001

 ( )
n
Consider the behavior of P6,0 as n increases:

n ( )
n
p6,0
1 0.0839179
2 0.269638
3 0.206912
4 0.228275
5 0.220989
6 0.223475
7 0.222627
8 0.222916
9 0.222818
10 0.222851
11 0.22284
12 0.222844
13 0.222842
14 0.222843
15 0.222843
16 0.222843

DTMC/s,S) inventory system page 9 ©D.Bricker, U. of Iowa, 2001

 ( )
n
Compare to the behavior of P4,0 as n increases:

n ( )
n
p4,0
1 0.57681
2 0.108458
3 0.261614
4 0.209636
5 0.227347
6 0.221306
7 0.223367
8 0.222664
9 0.222904
10 0.222822
11 0.22285
12 0.22284
13 0.222844
14 0.222842
15 0.222843
16 0.222843

DTMC/s,S) inventory system page 10 ©D.Bricker, U. of Iowa, 2001

The limiting distribution of the state of the system (independent of
the initial state) is
Steadystate
State Probability
i πi
0 0.222843
1 0.14779
2 0.178159
3 0.181227
4 0.150444
5 0.0907828
6 0.0287543

This implies that 22.2843% of the days a stockout occurs, 14.779%

of the days the ending inventory is 1, etc.

DTMC/s,S) inventory system page 11 ©D.Bricker, U. of Iowa, 2001

Recursive Computation of First-Passage Probabilities

fij( ) = pij( ) − ∑ p (jj

n−k )
f ij( ) where f ij( ) ≡ pij
n n k 1

k <n
(1)
f 6,0 = p6,0 = 0.08392
( 2) ( 2 )  (1) (1) 
f 6,0 = p6,0 − p0,0 f 6,0 = 0.2696 − 0.08392 × 0.08392=0.26256
 
( 3)
f 6,0 ( 3)  (1) ( 2 )
= p6,0 f 6,0 = 0.2069 − [ 0.08392 × 0.26256 + 0.2696 × 0.08392]
( 2 ) (1) 
− p0,0 f 6,0 + p0,0
 
= 0.1622
etc.

Thus, if the inventory is full on Monday morning (or equivalently,

Sunday evening),
the probability that the first stockout occurs Wednesday evening (n=3) is
16.22%.

DTMC/s,S) inventory system page 12 ©D.Bricker, U. of Iowa, 2001

 ( ) n
The probability distribution f 6,0 of the first-passage times are

( )
n n
f 6,0
1 0.0839179
2 0.262596
3 0.162248
4 0.12649
5 0.0931353
6 0.0694925
7 0.0516882
8 0.0384743
9 0.0286333
10 0.0213104
11 0.0158602
12 0.0118039
13 0.00878497
14 0.00653818
15 0.00486601
16 0.00362151

DTMC/s,S) inventory system page 13 ©D.Bricker, U. of Iowa, 2001

 n ( )
n
f 6,0 ( )
n
n × f 6,0 ∑k × f ( )
k ≤n
k
6,0

1 0.0839179 0.0839179 0.0839179

2 0.262596 0.525192 0.60911
3 0.162248 0.486745 1.09586
4 0.12649 0.50596 1.60182
5 0.0931353 0.465677 2.06749
6 0.0694925 0.416955 2.48445
7 0.0516882 0.361818 2.84627
8 0.0384743 0.307794 3.15406
9 0.0286333 0.2577 3.41176
10 0.0213104 0.213104 3.62486
11 0.0158602 0.174462 3.79933
12 0.0118039 0.141646 3.94097
13 0.00878497 0.114205 4.05518
14 0.00653818 0.0915345 4.14671
15 0.00486601 0.0729902 4.2197
16 0.00362151 0.0579441 4.27765

DTMC/s,S) inventory system page 14 ©D.Bricker, U. of Iowa, 2001

The partial sums in the last column are successive approximations
of the mean first passage time m6,0 , but convergence is somewhat
slow:

DTMC/s,S) inventory system page 15 ©D.Bricker, U. of Iowa, 2001

Generally, it is more computationally efficient to compute the mean
first passage times by solving a (7×7) system of linear equations for
mi ,0 , i = 0,1,...6 :

mij = 1 + ∑ pik mkj , ∀ i ∈ I & fixed j

k≠ j

m0,0 = 1 + p0,1m1,0 + p0,2 m2,0 + p0,3 m3,0 + " + p0,6 m6,0


m1,0 = 1 + p1,1m1,0 + p1,2 m2,0 + p1,3 m3,1 + " + p1,6 m6,1
Fix j=0: 
etc.
m6,0 = 1 + p6,1m1,0 + p6,2 m2,0 + p6,3 m3,1 + " + p6,6 m6,1

That is, we solve the 7×7 linear system for m0,0 , m1,0 ," , m6,0 :
1 -0.100819 -0.168031 -0.224042 -0.224042 -0.149361 -0.0497871 1
0 0.899181 -0.168031 -0.224042 -0.224042 -0.149361 -0.0497871 1
0 -0.100819 0.831969 -0.224042 -0.224042 -0.149361 -0.0497871 1
0 -0.224042 -0.149361 0.950213 0 0 0 1
0 -0.224042 -0.224042 -0.149361 0.950213 0 0 1
0 -0.168031 -0.224042 -0.224042 -0.149361 0.950213 0 1
0 -0.100819 -0.168031 -0.224042 -0.224042 -0.149361 0.950213 1

DTMC/s,S) inventory system page 16 ©D.Bricker, U. of Iowa, 2001

 The complete mean first passage matrix is found by solving 7 such
linear systems (one per column):

from \to 0 1 2 3 4 5 6
0 4.48747 6.76636 5.61296 4.75467 5.77744 10.1005 34.7774
1 4.48747 6.76636 5.61296 4.75467 5.77744 10.1005 34.7774
2 4.48747 6.76636 5.61296 4.75467 5.77744 10.1005 34.7774
3 2.81583 6.22338 5.78307 5.51794 6.82984 11.1529 35.8298
4 3.61112 6.13803 5.36867 5.05969 6.64699 11.3183 35.9952
5 4.13554 6.39543 5.34364 4.73395 6.16983 11.0153 36.2693
6 4.48747 6.76636 5.61296 4.75467 5.77744 10.1005 34.7774

Thus, starting with a full inventory (state 6), the expected number
of days until a stockout occurs (state 0) is 4.48747.

DTMC/s,S) inventory system page 17 ©D.Bricker, U. of Iowa, 2001

The mean recurrence times mii on the diagonal are the reciprocals

of the steadystate probabilities, i.e., mii = 1 .

πi
For example, one should expect a stockout once every
1
m00 = 1
π 0 = 0.222843 = 4.48747 (days)
and to find the inventory full (S=6) at the end of the day once every
1
m6,6 = 1 = = 34.7774 (days)
π6 0.0287543

DTMC/s,S) inventory system page 18 ©D.Bricker, U. of Iowa, 2001