5/4/2024

EMÜ322
Simulation Modeling and Analysis
Output Data Analysis for a Single
System
Banu Yuksel Ozkaya
Hacettepe University
Department of Industrial Engineering
1

Output Analysis
• Output data analysis is the examination of
data generated by a simulation.
– To predict the performance of a system
– Compare the performance of two or more
alternative system designs

1
 5/4/2024

Output Analysis
• In simulation studies, a great deal of money
and time is spent on model development and
programming, but little effort is made to
analyze the simulation output data
appropriately.
• A common practice is to make a single
simulation run of somewhat arbitrary length
and then to treat the resulting simulation
estimates as the true characteristics.
3

Output Analysis
• Since random samples are used to derive a
simulation model through time, they are
particular realizations of random variables
that may have large variances.
• The estimates could differ greatly from the
corresponding true characteristics for the
model.
• Therefore, there could be a significant
probability of making erroneous inference
about the system under study.
4

2
 5/4/2024

Output Analysis
• Reasons for why output data analyses have
not been conducted appropriately:
– There is an unfortunate impression that
simulation is largely an exercise in computer
programming.
• Simulation is a computer-based statistical sampling
experiment.
• If the results have any meaning, appropriate statistical
techniques must be used to design and analyze
simulation experiments.

Output Analysis
• Reasons for why output data analyses have
not been conducted appropriately:
– The output processes of virtually all simulations
are non-stationary and auto-correlated.
– Classical statistical techniques based on IID
observations are not directly applicable.
• There still exist problems about output analysis
• Available methods are often complicated to apply.

3
 5/4/2024

Output Analysis
• Reasons for why output data analyses have
not been conducted appropriately:
– The specification of the initial conditions of the
system at time 0 can pose a problem about the
output data.
– Because of autocorrelation, the initial conditions
will influence the performance measures.

Output Analysis
• Nature of simulation output:
– Y1,Y2,…. Yi …..be an output stochastic process from
a single simulation run.
• Example: waiting time of ith job, utilization of a
machine at ith day, throughput in ith hour.
– Yi’s will in general be neither independent nor
identically distributed.

4
 5/4/2024

Output Analysis
• Nature of simulation output:
– y11, y12, y13…. y1m be a realization of the random
variables Y1,Y2,…. Ym from making a simulation run of
length of m observations. (1st replication)
– y21, y22, y23…. y2m be another realization of the
random variables Y1,Y2,…. Ym from making another
simulation run of length of m observations. (2nd
replication)
– ……..
– yn1, yn2, yn3…. ynm be another realization of the
random variables Y1,Y2,…. Ym from making another
simulation run of length m observations. (nth
replication)
9

Output Analysis
• Nature of simulation output:
– y11, y12, y13….y1i……y1m
y21, y22, y23….y2i …….y2m
……………………………………………………
yn1, yn2, yn3…. yni…...ynm
– The observations from a particular replication (y11,
y12, y13….y1i……y1m) (rows) are clearly not IID.
– y1i, y2i, y3i….……yni (columns) are IID observations
of the random variable Yi.
10

5
 5/4/2024

Output Analysis
• Nature of simulation output:
– The independence across runs is the key to output
analysis methods.
– The goal of output analysis is to use the
observations yji (i=1,2,….m, j=1,2,….n) to draw
inferences about random variables Y1,Y2,….Ym.

11

Output Analysis
• Example: A bank with five tellers and one queue – output
statistics from 5 independent replications of a simulation of
the bank assuming no customers initially.

Proportion of
Average Average
Number Finish Customers
Replication Delay in Queue
Served Time Delayed <
Queue Length
5 minutes
1 484 8.12 1.53 1.52 0.917
2 475 8.14 1.66 1.62 0.916
3 484 8.19 1.24 1.23 0.952
4 483 8.03 2.34 2.34 0.822
5 455 8.03 2.00 1.89 0.840

12

6
 5/4/2024

Output Analysis
• Transient and steady state behavior of a
stochastic process: • Steady-state seems to start at
time k+1.
• Steady-state does not mean that
Yk+1, Yk+2,… will all take on the
same value, rather it means they will
have approximately the same
distribution

13

Output Analysis
• Transient and steady state behavior of a
stochastic process:
Di: waiting time of ith customer in an
M/M/1 queuing system
s: number of customers in the system
at time 0

14

7
 5/4/2024

Output Analysis
• Transient behavior:
– Behavior of the system a short time after the system starts operating
– Distribution and expected values depend on the initial conditions
– Distribution and expected values depend on the time index

• Steady state behavior:

– Behavior of the system a long time after the system starts operating
– Distribution and expected values don’t depend on the initial
conditions
– Distribution and expected values don’t depend on the time index

15

Output Analysis
• Transient and steady state behavior of a
stochastic process:
Ci: cost of ith month in an (s,S)
inventory system

16

8
 5/4/2024

Output Analysis
• Types of simulations with regard to output
analysis:

Steady-state cycle parameters

17

Output Analysis
• Types of simulations with regard to output
analysis:
– Terminating simulation: There is a natural event E
that specifies the length of each run. Simulated
system opens at time 0 under well-specified initial
conditions and close at stopping time TE. Event E
might correspond to a time point
• At which the system is cleaned out.
• Beyond which no additional information is obtained.
• Which is specified by the management.
18

9
 5/4/2024

Output Analysis
• Types of simulations with regard to output
analysis:
– Terminating simulation: There is a natural event
and a stopping time TE.
• Example 1: A bank opens at 8:30am with no customers
present and 8 of the 11 tellers working and closes at
4:30pm. The goal is to model the interaction between
customers and tellers over the entire day. (system is
cleaned out)

19

Output Analysis
• Types of simulations with regard to output
analysis:
– Terminating simulation: There is a natural event
and a stopping time TE.
• Example 2: A communication system consists of several
components plus several backup components. Consider
the system until the system fails. (no additional info)

B
A D
C
20

10
 5/4/2024

Output Analysis
• Types of simulations with regard to output
analysis:
– Terminating simulation: There is a natural event
and a stopping time TE.
• Example 3: An aerospace manufacturer receives a
contract to produce 100 airplanes, which must be
delivered within 18 months. The company would like to
simulate various manufacturing configurations to see
which one can meet the delivery deadline with the
least cost.

21

Output Analysis
• Types of simulations with regard to output
analysis:
– Terminating simulation: Event E is specified
before any runs are made.
– Since initial conditions of a terminating simulation
generally affect the desired performance
measures, the initial conditions should be
representative of those for the actual system.

22

11
 5/4/2024

Output Analysis
• Types of simulations with regard to output
analysis:
– Non-Terminating simulation: There is no natural
event E to specify the length of a simulation run.
– This often occurs when we are designing a new
system in the long run when it is operating
normally.
– “In the long run” does not translate into a
terminating event E.

23

Output Analysis
• Types of simulations with regard to output
analysis:
– Non-Terminating simulation: There is no natural
event E to specify the length of a simulation run.
• Example 1: Consider a company that is going to build a
manufacturing system and would like to determine the
long run mean hourly throughput of their system after
it has been running long enough for the workers to
know their jobs.

24

12
 5/4/2024

Output Analysis
• Terminating Simulations:
– Recall that in a terminating simulation, there is a
natural event E that specifies the length of each
run.
– Simulated system opens at time 0 under well-
specified initial conditions and close at stopping
time TE.
– Examples: Simulation of a bank that opens and
closes at constant times over time; simulation of a
communication network until it fails.
25

Output Analysis
• Terminating Simulations:

TE TE TE TE TE

– End of one epoch is the beginning of a new epoch.

– Ending state of the previous epoch does not affect
the starting state of a new epoch.
– TE might be deterministic or stochastic
26

13
 5/4/2024

Output Analysis
• Non-Terminating Simulations:
– Recall that a non-terminating simulation is the one
with no such terminating event to define the end of
system operation and no starting condition.
– There is no natural event E to specify the length of
the run.
0
– There is a single epoch of the system that
continues indefinitely.
– Example: Simulation of hospital emergency rooms,
simulation of fire departments.
27

Output Analysis
• Types of simulations with regard to output
analysis:
– Example: Consider a manufacturing company that
operates 16 hours a day (two shifts) with WIP
carrying from one day to the next. Terminating or
non-terminating simulation?

28

14
 5/4/2024

Output Analysis
• Terminating and Non-Terminating Simulations:
– Some systems are difficult to classify.
• Production systems working in shifts
• Inventory of the markets
• Flow of money at a bank
– Questions to ask:
• Do they close at some fixed points?
• Do they have a fixed starting condition?
• Does the ending state of one epoch affect the starting
state of the next epoch?
29

Output Analysis
• Transient vs Steady-state:
– Transient Behavior: If the distributions of the random
variables {Yn: n=1,2,……..} depend upon
• Initial conditions
• The point at which they occur either in sequence or time
then, these random variables are transient characteristics.
– Example: The waiting time of the 5th customer in a
queuing system.

30

15
 5/4/2024

Output Analysis
• Transient vs Steady-state:
– Steady-state Behavior: If the distributions of the
random variables {Yn: n=1,2,……..} converge to what
we call a steady-state (or a limiting) distribution for
any set of initial conditions, then these random
variables are steady-state characteristics.
– The steady-state response is the behavior of the
system after a long period of time and independent
of the starting state of the system.
– Example: Waiting time of the 1000th customer
31

Output Analysis
• Output Analysis of Terminating Simulations:
– Interested in determining the transient response
of the system subject to its natural starting
conditions.
– The run length of simulation (TE) can not be
manipulated. The only way to increase the sample
size is to replicate the runs ➔ your precision for
estimation will increase.

32

16
 5/4/2024

Output Analysis
• Output Analysis of Terminating Simulations:
– Initial conditions are very important.
• Start with natural initial conditions
– Start with empty state if the system is empty initially.
• Start with empty state but collect data during a certain
amount of time.
– Discard the initial observations and collect statistics for the
required time period.

33

Output Analysis
• Output Analysis of Terminating Simulations:
– Suppose that we make n independent replications
of a terminating simulation, where each
replication is terminated by the event E.
– Assume for simplicity that there is a single
measure of performance of interest, X.
– Let Xj be a random variable on the jth replication
for j=1,2,….,n➔ Xj’s are iid random variables.

34

17
 5/4/2024

Output Analysis
• Output Analysis of Terminating Simulations:
Run 1 X1

Run 2 X2
Simulation
Model

Run n Xn

– Xj is the average of observations from the simulation

output at replication (run) j.

35

Output Analysis
• Output Analysis of Terminating Simulations:
– Examples:
• For a queuing system, Xj might be the average delay
(waiting time) of all customers over a day on the jth
replication.
• For an inventory system, Xj might be the total
average cost on the jth replication.

36

18
 5/4/2024

Output Analysis
• Output Analysis of Terminating Simulations:
– Suppose that we would like to obtain a point
estimate and confidence interval for the mean
m=E[X].
– Make n independent replications of the simulation.
Then, X1,X2,…..Xn be the resulting IID random
variables.
1 n 1 n
X ( n ) =  X j , S ( n) =
2
 ( X j − X ( n )) 2
n j =1 n − 1 j =1

37

Output Analysis
• Output Analysis of Terminating Simulations:
– Confidence interval on the mean of a normal
distribution with unknown variance
– 100(1-a)% Confidence Interval:
S 2 ( n)
X (n)  ta / 2,n −1
n

38

19
 5/4/2024

Output Analysis
• Output Analysis of Terminating Simulations:
– Confidence interval on the mean of a normal
distribution with unknown variance
– Half length:
S 2 ( n)
ta / 2,n −1
n

– Smaller the half length, higher the precision. 39

Output Analysis
• Output Analysis of Terminating Simulations:
– How to achieve the desired accuracy?

S 2 ( n)
hl = ta / 2,n −1  (desired accuracy)
n

40

20
 5/4/2024

Output Analysis
• Output Analysis of Terminating Simulations:
– Required sample size is:
2
 S 2 ( n) 
nr  ta / 2, nr −1  (*)
  
 

– Since this requires an iterative procedure, it can

also be calculated as follows – this only gives a
lower bound!
2
 S 2 ( n) 
nr   za / 2 
   41

Output Analysis
• Nature of simulation output:
– y11, y12, y13….y1i……y1m
y21, y22, y23….y2i …….y2m
……………………………………………………
yn1, yn2, yn3…. yni…...ynm
– The observations from a particular replication (y11,
y12, y13….y1i……y1m) (rows) are clearly not IID.
– y1i, y2i, y3i….……yni (columns) are IID observations
of the random variable Yi.
42

21
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– We are interested in defining the steady-state
performance of the system.
– The steady-state response is the behavior of the
system after a long period of time and is independent
of the starting state of the system.
– Let Y1,Y2,….. be the output process of a system.
– We want to estimate the steady-state mean

 = lim E[Yi ]
i →
43

Output Analysis
• Output Analysis of non-Terminating Simulations:
– A natural way to estimate for  is to take the average
of all response variables in the simulation.
– If transient state values are also included in the
computation, estimator will be biased.
– Therefore, we have to determine the initial transient
state and discard the observations during this initial
state.
– This initial transient period is also called warm-up
period.
44

22
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
E[Yi]

Is this the end of the warm-up period?

i
What is the warm up period? 45

Output Analysis
• Output Analysis of non-Terminating Simulations:
E[Yi]

End of the warm-up period

i
Discard the observations in this part
46

23
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– In order to eliminate the initialization bias, we may
want to delete some number of observations at the
beginning of a run and use only the remaining
observations to estimate .
– Given the observations, Yi1,Yi2,Yi3,……. Yim in the ith
simulation run, it is suggested to use the following to
find the estimate in the ith run:
m
Length of the
warm-up period
Y
j =l +1
ij

(in terms of the number of Y i (m, l ) =

observations or time units) m−l 47

Output Analysis
• Output Analysis of non-Terminating Simulations:
– The suggested estimator Y i (m, l ) is less biased than
Y i (m) .

48

24
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:

Di: waiting time of ith customer in an

M/M/1 queuing system
s: number of customers in the system
at time 0

l=~450 customers

49

Output Analysis
• Output Analysis of non-Terminating Simulations:

Ci: cost of ith month in an (s,S)

inventory system

l=~75 months

50

25
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Running Average: For the ith replication, calculate the
average of first N output variables Yij where N=1,2,…..m.
N

Y
j =1
ij Nth running average

N
1. Compute the running averages 2. Plot all these running averages with respect to
N=1➔ Yi1/1 N values
N=2➔ (Yi1+Yi2)/2 3. Length of the warm-up period is the smallest N
N=3➔ (Yi1+Yi2+Yi3)/3 point beyond which we don’t observe fluctuating/
changing running averages
N=m➔ (Yi1+Yi2+…..Yim)/m

51

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Running Average:
l=700

52

26
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Running Average:
• Below are two running averages of the same output variable
in two different simulation runs. What is the problem?

l=700
l=500

53

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Running Average:
• Due to inherent variability, different warm-up periods may
exist in different simulation runs.
• Not desired to use different warm-up periods across
different replications.
• We need a more robust procedure ➔ Welch Procedure.

54

27
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
• A graphical procedure to determine a time index l such that
E[Yi]= for i>l where l is the warm-up period.
• Based on n independent replications of the simulation

55

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
• Step 1: Make n replications of the simulation (n>=5), each of
length m. Let Yij be the jth observation from the ith
replication (i=1,2,…n; j=1,2,….m)
n
• Step 2: Let Y j =  Yij / n for i=1,2,….m. The averaged process
i =1
has means E[Y j ] = E[Y j ] and variances var[Y j ] = var[Y j ] / n

56

28
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
– y11 y12 y13 …… y1i …… y1m-2 y1m-1 y1m
y21 y22 y23 …… y2i …… y2m-2 y2m-1 y2m
Step 1
………………………………………………………………………………………….
yn1 yn2 yn3 …… yni …… ynm-2 ynm-1 ynm

Y1 Y 2 Y 3 Yi Y m − 2 Y m −1 Y m Step 2 (Averaged
Process for j=1,2,…m
57

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
• Step 3: Define the following moving average Y j (w) (where w
is the window and is a positive integer such that w<=m/4) as
follows:  j+w
 
Y s 2w + 1 j = w + 1, w + 2,....m − w

w: window (Parameter Y j ( w) = 2s =j −j 1− w

of the procedure)

 Y s 2 j −1 j = 1,2,...., w
 s =1
• If j is not too close to the beginning of the replications, Y j (w) is the simple average
of 2w+1 observations of the averaged process centered at observation j
• If j is close to the beginning of the replications, Y j (w) is the simple average of 2j-1
observations of the averaged process centered at observation j (j<=w)
58

29
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
– y11 y12 y13 …… y1i …… y1m-2 y1m-1 y1m
y21 y22 y23 …… y2i …… y2m-2 y2m-1 y2m
Step 1
………………………………………………………………………………………….
yn1 yn2 yn3 …… yni …… ynm-2 ynm-1 ynm

Y1 Y 2 Y 3 Yi Y m − 2 Y m −1 Y m Step 2 (Averaged
process)

Step 3 (Moving
Y 1 (1) Y 2 (1) Y 3 (1) Y i (1) Y m −1 (1) Average with w=1)
59

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
• Step 4: Plot Y j (w) for j=1,2,3,……m-w and choose l to be that
value beyond which Y 1 ( w), Y 2 ( w)..... appears to have
converged.

60

30
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
• Example: For simplicity, assume that m=10, w=2, Y j = j for
j=1,2,…..5 and Y j = 6 for j=6,7,…..10. Calculate Y j (w) for
j=1,2,……8 and plot the Y j (w) values for j=1,2,…8.

61

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Welch Procedure - Example

62

31
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Welch Procedure – Example

j≥w+1
j≤m-w

63

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Welch Procedure – Example

64

32
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
• Recommendations:
– Initially make n=5 or 10 replications (depending on model
execution time) with m as large as practical. m should be
larger than the anticipated value of l and also large enough
to allow infrequent events.
– Plot Y j (w) for several values of the window w and choose
the smallest value of w for which the corresponding plot is
reasonably smooth. Use this plot to determine l. (If w is too
small, Y j (w) will be ragged. If w is too large, Y j
observations will be over aggregated. 65

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
• Recommendations:
– If no value of w is satisfactory, make 5 or 10 additional
replications of length m. Then, repeat the procedure once
again.

66

33
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
• Example: Consider the N1,N2,….. Where Ni is the number of
parts produced in the ith hour. Suppose we want to
determine the warm-up period l so that we can eventually
estimate the steady-state throughput =E[N]
0.9 good
Machining Inspection
center Station

0.1 bad
67

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
• Example: Suppose we made 10 independent replications of
each of length m=800 hours (100 days). The averaged
process is given in
the following figure.
F Further smoothing
is necessary

68

34
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– How to determine the warm-up period?
• Welch Procedure
• Example: w=10,w=20

69

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Suppose we want to estimate the steady-state mean
=E[Y] of the process Y1,Y2,….Desired properties for a
method to estimate =E[Y]:
• Good statistical performance
• Easy to understand and implement
• Applicable to all types of output parameters
• Applicable to estimate several different parameters for the
same simulation model

70

35
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach
– Batch Means Approach
– Autoregressive Approach
– Spectral approach
– Regenerative approach
– Standardized Time Series

71

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Suppose we want to estimate the steady-state mean
=E[Y] of the process Y1,Y2,….Desired properties for a
method to estimate =E[Y]:
• Good statistical performance
• Easy to understand and implement
• Applicable to all types of output parameters
• Applicable to estimate several different parameters for the
same simulation model
– We will assume that we know the length of the
warm-up period, l.
72

36
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach
– Batch Means Approach
– Autoregressive Approach
– Spectral approach
– Regenerative approach
– Standardized Time Series

73

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach:
• Once the warm-up period l is determined, discard the first l
observations from each replication and use the remaining
observations to estimate .
• y11 y12 y13 …… y1l y1l+1 …… y1m-1 y1m ➔ m-l observations
y21 y22 y23 …… y2l y2l+1 …… y2m-1 y2m ➔ m-l observations
………………………………………………………………………………………….
yn1 yn2 yn3 …… ynl ynl+1 …… ynm-1 ynm ➔ m-l observations

74

37
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach:
• When using the replication/deletion approach, each
replication is regarded as a single sample for the purpose of
estimating .
• For replication i, define:
1 m Average of Yij values for
Y i. (m, l ) =
m−l
Y
j =l +1
ij j=l+1,l+2,….m (steady state observations)
in the ith replication
• The overall point estimator is given by:
1 n 1 n 1 m
Y .. (m, l ) = 
n i =1
Y i. (m, l ) = 
n i =1 m − l
Y
j =l +1
ij
Average over all
replications
75

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach:
• For simplicity, abbreviate Y .. (m, l ) and Y i. (m, l ) by Y .. and Y i.
• What is find the standard error of Y .. ?
Sample variance
of the averages
obtained from s.e.(Y .. ) = S / n where
2
the replications 1 n 1 n
S =  (Y i. − Y .. ) = ( Y i. − nY .. )
2 2 2

n − 1 i =1 n − 1 i =1
• Then, 100(1-a)% Confidence Interval for  is
S S
Y .. − ta / 2, n −1    Y .. + ta / 2, n −1
Averages obtained
n n
from the replications are
Normally distributed and their variance is unknown. 76

38
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach:
• The length of each replication beyond the deletion point
should be at least ten times the amount of data deleted.
– m-l>=10l➔ m>=11l
• Given this run length, the number of replications should be
up to 25 replications.

77

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach:
• Sample size in non-Terminating Simulations:
• In order to estimate a long-run performance measure, 
within ± with confidence 100(1-a)% may be achieved by:
– Increase the number of replications, n.
Focus on this option
– Increase the run length, m.

Know that this is also another option

You want to find the number of observations for which the half length
(at 100(1-a)% significance level) is at most .
78

39
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach:
• Sample size determination in non-Terminating Simulations:
• In order to estimate a long-run performance measure,  within
± with confidence 100(1-a)%, the required sample size is the
minimum value of n that will satisfy the following inequality
2
t S  S0 is the sample variance of the averages obtained
n   a / 2,n−1 0  from the initial n0 replications, and  is the desired
   precision (maximum error)
• Since this requires an iterative procedure, an initial estimate
for n is given by:
2
 za / 2 S0 
n  or n≥n0
   79

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach:
• Sample size determination in non-Terminating Simulations:
• Example: In a M/G/1 queuing simulation, an initial sample
size n0=10 is taken with the initial estimate of variance
S02=25.30. Suppose, it is desired to estimate the long run
mean queue length within =2 customers with 90%
confidence.
2
 z S  1.645  25.30
2
n   a/2 0  = = 17.1 or n≥10
   22
The initial value of n will be taken as 18 since for 10≤n ≤17, the desired precision
cannot be achieved half-length would be larger than the desired precision 80

40
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach:
• Sample size determination in non-Terminating Simulations:
• Example: Hence, at least 18 replications are necessary.
2
n 18 19 .  ta / 2,n−1S0 
n   
t0.05,n-1 1.74 1.73   
2
 t0.05,n −1S0  desired inequality
  19.15 18.93
  

19 replications are necessary indicating that you should make 9 additional

replications in addition to n0=10 initial replications
81

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach:
– Advantages:
• Simple and easy to implement
• Generates independent sequences via multiple independent
replications of the simulation

82

41
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach:
– Disadvantages:
• Waste computer time associated with transient period
each run (each replication)
• It is critical that number of observations to be deleted (l)
is chosen correctly.
– If l is too small, then the estimator will be biased.
– If l is too large, CI will be wider than necessary.

83

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Replication/Deletion Approach:
– Disadvantages:
• Example: 10 replications with m=25000 minutes
l=0 minutes l=1000 minutes l=2000 minutes
Y .. 7.94 8.21 8.43
s.e.(Y .. ) 1.47 1.57 1.59

As fewer data are deleted;

– The standard error decreases.
– The confidence interval shifts downward, reflecting the greater
downward bias.
84

42
 5/4/2024

Output Analysis
• ARENA

l=100 hours
m>=1100 hours

ARENA will give an aggregate

Summary and also gives what
is happening in each replication
under Reports

85

Output Analysis
• ARENA Create 1 Process 1
Dispose 1
0
0 0
Expo(1) triangular(0.5,0.7,1.5)

Main report will give the overall/aggregated results.

86

43
 5/4/2024

Output Analysis
• ARENA

87

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
• There is a merit in using an experiment design that is
based on a single long replication.
• The disadvantage of a single replication design arises
when we try to compute the standard error of the sample
mean; the data are dependent and the usual estimator is
biased.

88

44
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
• The method of batch means attempts to solve the problem
by dividing the output data from one replication (after
appropriate deletion) into a few large batches and then
treating the means of these batches as if they were
independent. Suppose m=l+nd observations (or time units)

Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations Batch 1 of Batch 2 of Batch 3 of Batch n of

deleted d observations d observations d observations d observations
89

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Batches will be treated as independent replications
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn

Yi : sample mean of the observations (Y1j‘s) in the ith batch

What is the estimate for ?
What is the estimate for the variance of the sample mean?

90

45
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
n

Y i

Y = i =1
is the estimate for 
n
S2 1 n (Yj - Y) 2 n (Yj - Y) 2
=  = is the variance of the sample mean
n n i =1 n -1 i =1 n(n - 1)
91

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
What might be the underlying assumption to calculate the variance
of the sample mean? ➔ Batches should be independent
Y 1 , Y 2 ...... are not independent.
If the batch size, d is sufficiently large, successive batch means will
be approximately independent.
92

46
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
Suppose you have a fixed simulation run (m=l+nd fixed).
• What happens if d is very large?
–The number of batches (n) will decrease
• What happens if d is very small?
–The batches will not be independent of each other
93

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
• For a fixed total simulation run, there is little benefit from dividing
it into more than n<=30 batches.
• There is no reason to consider numbers of batches much greater
than 30, no matter how much raw data are available.
• The performance of the confidence interval, in terms of the width
and variability, is poor for fewer than 10 batches. 94

47
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
• Although there is theoretically autocorrelation between batch
means at all lags, the lag-1 autocorrelation 1 = corr (Y j , Y j +1 ) is
studied to assess the independence between batch means.
• When lag-1 autocorrelation is near zero, the batch means are said
to be independent. This is based on the observation that the
autocorrelation of many stochastic process decreases as the lag
increases
95

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
• The lag-1 autocorrelation should not be estimated from a small
number of batch means (such as 10<=n<=30 given above). There is
a bias with small number of batch means.
•It is suggested that lag-1 autocorrelation should be estimated from
a larger number of batch means based on a smaller batch size
(perhaps 100<=n<=400).
•When the autocorrelation between these batches is approximately
zero, the autocorrelation will be even smaller if we rebatch the data
to between 10 and 30 batch means based on a larger batch size. 96

48
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
• If the total simulation run is chosen sequentially to attain a
specified precision, it is helpful to allow the batch size and number
of batches to grow as the run length increases.

97

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
• The following strategy is recommended:
1. Obtain output data from a single replication and delete l
observations (or time periods) as appropriate (using an
appropriate and robust method). Recall that you should collect
at least 10 times as much data as is deleted.

98

49
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
• The following strategy is recommended:
2. Form up to 100<=n<=400 batches with retained data
(observations that correspond to the steady state period), and
compute the batch means. Estimate the sample lag-1
autocorrelation of the batch means as:
 (Y )( )
n −1
j − Y Y j +1 − Y
j =1
̂1 =
 (Y )
n
2
j −Y 99
j =1

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
• The following strategy is recommended:
3. Check the sample lag-1 autocorrelation (computed in Step 2) to
see whether it is sufficiently small:

 (Y )( )
n −1
j − Y Y j +1 − Y
j =1
̂1 =
 (Y )
n
2
j −Y
j =1
100

50
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
• The following strategy is recommended:
3. a. If the sample lag-1 autocorrelation <= 0.2, rebatch the data to
10<=n<=30 batches, and calculate the estimate and variance of
the sample mean with current 10<=n<=30 value.
 (Y )( )
n −1
j − Y Y j +1 − Y
j =1
̂1 =
 (Y )
n
2
j −Y
101
j =1

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
• The following strategy is recommended:
3. b. If the sample lag-1 autocorrelation > 0.2, extend the replication
by 50% to 100% and go to step 2.

 (Y )( )
n −1
j − Y Y j +1 − Y
j =1
̂1 =
 (Y )
n
2
j −Y
102
j =1

51
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
Y1 Y2 Yl Yl+1 Yl+2 Yl+d Yl+d+1 Yl+(n-1)d+1 Yl+nd

l observations
deleted Y1 Y2 Y3 Yn
• The following strategy is recommended:
3. b. If the sample lag-1 autocorrelation > 0.2 and it is not possible to
extend the replication, rebatch the data into approximately n=10
batches and calculate the estimate and variance of the sample mean
with current n~10.

103

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
– Example: Simulation of an M/G/1 queuing system.
Estimate the steady-state expected delay in queue,
wQ with 95% confidence interval. Assume that one
run of the model has been made, simulating 3000
customers after the deletion point.

104

52
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
– Example:
– We form n=100 batches of batch size d=30 (a total
of 3000 observations) and estimate the lag-1
autocorrelation to be 1=0.346
– 1=0.346>0.2

105

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
– Example:
– Decide to extend the simulation to 6000 customers
after the deletion point and estimate the lag-1
autocorrelation.
– Based on n=100 batches of batch size d=60 and
estimate the lag-1 autocorrelation to be 1=0.004.
– Autocorrelation check is passed.

106

53
 5/4/2024

Output Analysis
• Output Analysis of non-Terminating Simulations:
– Batch Means Approach:
– Example:
– Rebatch the data n=30 batches of size d=200. Then,
the following results are obtained:
Y = 9.04, S 2 = 0.604  30 .
– Then, 95% confidence interval is given by:
Y − t 0.025, 29 0.604  wQ  Y +t 0.025, 29 0.604
9.04 − 2.04  0.777  wQ  9.04 + 2.04  0.777
7.45  wQ  10.63
107

Output Analysis
• ARENA:
• How to write the output data to a text/csv file?
• M/G/1 Queuing System.
– Exponential inter-arrival times with mean 1 hour
– Triangular service time with parameters (0.2,0.6,1) hour
– Number of replications: 1
– Run length is 3000 hours.

C reate 1 P rocess 1 ReadWrite 1 D ispose 1

0 0
0

108

54
 5/4/2024

Output Analysis
• ARENA:
• How to write the output data to a text/csv file?
– Readwrite (Advanced Process Panel) module

Entity number (like 2,3,…)

Waiting time of each entity

109

Output Analysis
• ARENA:
• How to write the output data to a text/csv file?
– File (Advanced Process Panel) data module

Arena File Name Address of the file

in Readwrite module

110

55
 5/4/2024

Output Analysis
• ARENA:
• Output Analyzer
• Write the output data in a format that only Output Analyzer
can read (in Arena model)

• Export the files in the Output Analyzer to a readable format

(.dax file) waiting time.dat

waiting time.dax
111

Output Analysis
waiting time.dat

• ARENA:
• Output Analyzer
• Batch Means

waiting time batches.flt

112

56
 5/4/2024

Output Analysis
• ARENA:
• Output Analyzer waiting time batches.flt
• Batch Means

waiting time batches.dax

(Readable file )

113

57