Output analysis

1
 Outlines
 Types of output statistics

discrete-time và continuous-time
statistics
 transient state and steady state of
the system
 Simulation types
 Analysis of terminating simulation
 Analysis of steady state simulation

2
 Output statistics
 Output data:
 Average throughput in 1 hour or 1 day
 The average length of the queue
 The average waiting time
 Discrete-time statistics:
 Change at discrete time
 The average waiting time, the smallest waiting
time
 Continuous-time (time average) statistics:
 Integrated by time
 The time-average of the length of queue,
machine utilization
3
 Output statistics
 The performance measure used
depends on the objectives of the
simulation
 The output data is random in
nature
 Single simulation run is inadequate
for characterizing the system
 Multiple simulation runs are required,
hence need to estimate the average
and confidence level
4
 Random nature of output
data
 Consider a manufacturing system
 The concerned data is the throughput in
the 1st hr Y1, 2nd hr Y2, 3rd hr Y3, ..., mth hr Ym
 Replicate the simulation n times with
the same initial condition
 For 1st replication, we have
 y11, y12, y13,..., y1m
 For nth replication, we have
 yn1, yn2, yn3,..., ynm
5
 Random nature of output
data
 The observations in one replication
are not identical independent
distribution
 Nonstationary and autocorrelated
 Throughput of the previous hour will
affect the next hour
 How can we estimate the average
hourly throughput?
 Note that many statistical procedures
are based on IID assumption
6
 Random nature of output
data
 The observations across replications
are IID (independent across runs)
 Hence, to estimate the statistics, we
need to replicate a number of
simulation runs, and apply the
statistical procedures across the
replication
 Example: E(Yi) = sum(yji)/n

7
 Transient state and steady
state
 Assembly line

Startup: hourly throughput is small due to no WIP

after warmup: hourly throughput is large and
constant

The hourly throughput at the startup depends on
WIP at the startup
 Banking office

Opening: almost no queue

After a few hours: queue length is longer and
become almost unchanged

The length of the queue at the start depends on
the number of customers at the opening
8
Transient state và steady
state
B a ét ñaàu traïn g tha ùi o ån
ñònh ô ûqua n s a ùt k + 1
N hö õn g ha øm m a ät ño äc huye ån tie áp

v = E (Y )

Y i4 . . . Y in
Y i3
Y i2
K ho ân g nhadistribution
Non-normal át thie át la ø
Y i1
ha øm m aät ño äc hua ån
E (Y ) o

i1 i2 i3 i4 i

9
 Transient state và steady
state
 Let Fi(y|I) is the distribution of variable Y
at time i.
 With I represents the initial conditions used
to start the simulation
 For a given I, Fi(y|I) changes with I

Fi(y|I) is called transient distribution, and
represent a transient state of the system
 when i∞, Fi(y|I)  F(y) does not depend
on I and I
 F(y) is called a steady-state distribution, and
represents a steady state of the system
10
 Transient state and steady
state
 Instead of i∞, we can identify the
moment i=k+1 at which Fi(y) is
closed to constant
 How long it takes Fi(y|I) to
converge (k small or large) to F(y)
depends on the initial condition I
 Hourly throughput
 Queue length

11
 Transient state and steady
state
E (D i )  Example: queueing
15
system M/M/1
 Output: waiting
time of customer Di
10
 s is the number of
customer in the
8,1 d
system at the start
 With different
5
values of s, the
speeds of
converge of E(Di)
0 i
vary
80 160 240 320 400 480
12
H ì n h 6.2 H aøm E (D i ) ñ oái vôùi h eäth oán g
 Simulation type
 Basing on the output analysis
 Terminating simulation (mô phỏng có sự
kiện kết thúc)
 Nonterminating simulation or steady-state
simulation(mô phỏng có sự kiện kết thúc)
 Terminating simulation: there is a
natural event that determines the
length of the simulated time
 nonterminating simulation: there is no
natural event to determine the length of
the simulated time

13
 Simulation type
 Examples for terminating
simulation
 Banks: open at 8 am and close at 5
pm
 Shipyard: completes building a ship
 It is important to determine the
initial condition as the output
statistics will probably include the
transient state and affected by the
initial condition
14
 Simulation type
 Examples for nonterminating simulation
 Hospital emergency room
 Manufacturing plant
 We are interested in the steady-state of
the system, and the steady-state
distribution of output
 Choose the initial condition so that the
steady-state is converged fastest

15
 Simulation type
 A study is nonterminating or
terminitaing also depends on the
objective of the simulation
 Example
 Nhà máy sản xuất, ngày làm việc 8
tiếng, ta quan tâm tới WIP

16
 Simulation type
 Some nonterminating systems do not
have a steady-state distribution
 Example:
 a plant has 3 8-hour shifts with a rest of 1 hr
between shifts
 Hourly throughput is not steady
 Example:
 A call center
 the number of calls changes by weekday
then the waiting time for the call ith Di does
not have steady-state distribution
 Cycle-state
17
 Analysis of terminating
simulation
 Run n replications with different
streams of random numbers
 To ensure independent random
numbers
 Analysis of the output
 Mean
 Confidence level
 The number of replications n could
affect the precision of the analysis
18
 Estimate of mean
 xj is the value of the random
variable X (represent the
performance measure that is
interested) from j=1…n
replications
 Estimate the expected value of X
as usually
  = E(X) = sum(xj)/n

19
 Estimate of confidence
level
 xj is the value of the random variable X
(represent the performance measure
that is interested) from j=1…n
replications
 Confidence range 100(1 – )% for  as
follows
S 2 (n)
X (n)  tn  1,1  / 2
n

20
 Note on normality and
independence
 The formula is correct if

xi are independent
 X is normally distributed
 Independence: by different streams
of random numbers
 Normality
 Central theorem: n10
 No less than 5 replications
 If X is the average statistics: the
average waiting time of a customer
21
 The number of replications
 Confidence level half-length (half-width)
depends on the number of replications n
 Determine n so as to achieve the error
(sai số) or precision (độ chính xác) of
the estimation as desired
 2 types of error:
 Absolute error
 Relative error

22
 Absolute error
 X  : absolute error
 but X   tn  1,1  / 2
S 2 ( n)
n

 If we want to achieve the absolute

error less than  then the number
of replications should be
 S 2 ( n) 
n   min i n : ti  1,1  / 2  
 i 

 With n is a initial number of 23

 Absolute error
 The procedure
 Start with n (n=10 or higher)
 Estimate the mean and variable
 If the half-width is less than the error then
stop
 Otherwise increase i until achieving the
desired error
 Approximate t distribution 2
with z
distribution  z1  / 2  2
n     S ( n)
  

24
 Relative error
X /
 : relative error
 If we want to achieve the absolute error
less than  then the number of
replications should be 
ti  1,1  / 2 S 2 (n)
n   min i n :  
  i 
 With n is a initial number of replications
 or
 ti  1,1  / 2 S 2 ( n)  
n   min i n :   '
 X i 1  
25
 Relative error
 The procedure
 Start with n (n=10 or higher)
 Estimate the mean and variable
 If the half-width is less than the error then
stop
 Otherwise increase i until achieving the
desired error
 Approximate t distribution with z
distribution  z 2
1  / 2 
n     S 2
( n)
  ' X ( n) 
26
 Sequential procedure
 It is probably that with the initial n
replications, the estimations of
mean and variation are not correct
 Then the estimation of n() or n()
could not be correct
 Too big: waste of resources
 Too small: wrong estimation of mean
and variation
 Sequential procedure
 Sequential procedure
 (1) implement initial n0 replication and let
n = n0
S 2 ( n)
 (2) estimate X and  (n) tn 1,1  / 2 n
 (3) if the stop criterion is satisfied then
we have an estimation of mean as follows

X   ( n )   X   ( n )
 Otherwise, implement one more replication,
and repeat step (2) with n=n+1
 Sequential procedure
 It is recommended to use with small
error and with n010
 Small relative error 0,15
 Absolute error:?
 Alternative
 (0)Implement the initial n replications
 (1) Calculate n() or n()
 (2) Implement more [n()-n]/2 or [n()-n]/2
replications
 (3) Estimate mean and variation, calculate
the error. If being unsatisfied, repeat from
step (1)
 Estimate the probability and
the proportion
 Some measures of performance could be
 Probability: the probability that the customer
has to wait more than 30’
 Proportion: the proportion that the products
have weight in the range 50-100 gram
(probability?)
 Let X is the random variable that
underlies the interested measure of
performance
 Waiting time of the customer
 Weight of the product
 Estimate the probability and
the proportion
 Let p = P(XB) is the probability that X
lies in the range of interested values
 Implement n replications, we have n
values of X: x1, x2,…xn
 If S is the number of xj falling in the set
B and n is large, then

S pˆ (1  pˆ ) pˆ (1  pˆ )
pˆ  pˆ  z1  / 2
n
 p  pˆ  z1  / 2
n
n
 Initial condition
 The initial condition will strongly affect the
result of terminating simulation
 The initial condition should reflect the real initial
condition of the system
 Example: simulate to determine the waiting
time of the customer from 10:00 AM-
12:00PM
 Simulate the activities from 8:00AM with I=0
(no customer), delete data of 8h-10h from
analysis
 Simulate the activities from 10:00AM, I=random
number of customers from a pre-determined
distribution
 Steady-state simulation
 Suppose we study a manufacturing system
 Steady state hourly throughput: Y

At various t, we have random variables : Y1 at
t=1, Y2 at t=2, ..., Ym at t=m
 Steady state: Fi(y)  F(y) when i  
 We can estimate Y from Y1, Y2, ..., Ym
 But as F1(y), F2(y),…, using it can cause
bias to the estimation
 It is important to determine the point i=l at
which Fi(y)  F(y) to delete all Yi l off the
data used
33
Steady-state simulation
B a ét ñaàu traïn g tha ùi o ån
ñònh ô ûqua n s a ùt k + 1
N hö õn g ha øm m a ät ño äc huye ån tie áp

v = E (Y )

Y i4 . . . Y in
Y i3
Y i2
K ho ân g nhadistribution
Non-normal át thie át la ø
Y i1
ha øm m aät ño äc hua ån
E (Y ) o

i1 i2 i3 i4 i

34
 Initial transient
 Welch method
 Find l such that E[Y(ti)] with i>l

=lim E[Y(ti)]
 (1) Implement n replications (n=5
or n=10) with large m
 m is 11 times the expected length of
the transient
 yji is the ith observation (i=1…m) of
the jth replication (j=1…n)
35
Initial transient
 (2) Calculate the average of yji

y11 y12 y13  y1, m  2 y1, m  1 y1, m

y21 y22 y23  y2, m  2 y2, m  1 y2, m
      
yn1 yn 2 y n 3  y n , m  3 y n , m  2 yn , m
Y1 Y2 Y3 Ym  3 Ym  2 Ym

36
 Initial transient
 w
 SmoothYi with width w  Yi  s 
 s w
 2w  1 , i  w  1,...,m  w
Yi (w)   i  1

  Yi  s
 s (i  1) , i 1...w
 2i  1

Y1 Y2 Y3  Ym  2 Ym  1 Ym

Y1 ( w) Y2 ( w) Y3 ( w) Ym  2 ( w) Ym  1 ( w) Ym ( w)
 Half-width w should be the smallest integer so
that the plot is smooth

37
 Initial transient
 The right value of w requires trial and error
 Small w causes ragged plot, large w causes too smooth plot
 Initial w=m/4
 If can not choose the right w, implement 5 or 10 more
replications
 Rather run long replication then many replication (n<=25)
 Choosing the point I from the plot where the plot
converges
N i (30) N i (30)
70 70

60 60

l = 24

50 50
1 160 i 1 160 i 38
a) b)
 Replication/deletion
approach
 Suppose we want to estimate steady-state
=E(Y) of the random process Y1, Y2,…
 Replication/ deletion approach
 Similar to replication approach in terminating
simulation
 Use only the observations after the warmup
period L for the analysis
 Suppose we implement n’ replications
with m’ > L (m’=11L)
 Yji is the ith observation là quan sát thứ i
from replication j
39
 Replication/deletion
approach
 We have point estimator
m'

Y
i l 1
ji

Xj  j 1, 2,..., n '

m' l
 Xj : independent identical random
variables (IID) with E[Xj]   hence

S 2 (n ')
X (n ') tn ' 1,1  / 2
n'

40
 Replication/deletion
approach
 Widely,
 we run n replications with the run length
m to determine L
 and then run n’ replication with the
length m’ to estimate 
 If the initial n replications have the
run length m large ( in comparison to
L),
 we can use these n replications to find 
 The method to decide a sample size
is similar to terminating simulation
41
 Batch method
 Disadvantage of replication method

With each replication we delete L data
points => with m’ replications we delete
m’L => waste
 Batch method

Replicate once to avoid many deletions
 Batch method

Divide the replication into batches of size k

Each is considered as a separate
independent replication
 Batches are not independent

Can cause bias to the estimators

If k is large then the estimators are
relatively uncorrelated 42
 Batch method
 Run one replication with the observations
Y1, Y2, …, Ym (warmup period is deleted)
 Divide the replication into n batches with
the length k, we havekj


i k ( j  1) 1
Yi
Xj  j 1, 2,..., n
k
 Xj can be considered IID with E[Xj]  
then
S 2 ( n)
X (n, k ) tn  1,1  / 2
n
43
 Batch method
 It is difficult to decide the size of k
 A simple procedure:
 (1) run one replication and delete
the warmup period, the remaining
length should be at least 10 times
the deleted
 (2) divide the replication into at
least n=100 batches but not more
than 400
44
 Batch method
 Estimate the correlation coefficient at lag 1
n 1

 X j  X  X j 1  X 
1  j 1 n

 X  X
2
j
j 1


If ρ1 0,2 then divide into 30-40 batches for
estimating 
 Otherwise lengthen the replication from 50%
to 100% and repeat step (2)
 If lengthening the replication is not possible
the divide into 10 batches for estimating 
45