5/4/2024

EMÜ322
Comparing Alternative System
Configurations
Banu Yuksel Ozkaya
Hacettepe University
Department of Industrial Engineering
1

Introduction
• In the “Output Analysis for a Single System”
chapter, we have seen how to employ
different statistical analyses to the output
from a simulation model.
• Now, we will discuss statistical analyses of the
output from several different simulation
models that might represent competing
designs or alternative operating policies.

1
 5/4/2024

Introduction
• The real utility of simulation lies in comparing
such alternatives before implementation.
• Since the observations regarding performance
measures contain random variation, statistical
analysis is needed to discover whether any
observed differences are due to differences in
design or merely to the random fluctuation
inherent in the models.

Comparison of Two System Designs

• Compare two possible configurations of a system
based on a pre-specified performance measure.
• The mean performance measure for system i is
denoted by qi (i=1,2).
• The goal of the simulation experiments is to
obtain point and interval estimates of the
difference in mean performance, namely q1 -q2
(or qi –qj where i≠j and i,j=1,2…K with K being the
total number of alternative designs)
4

2
 5/4/2024

Comparison of Two System Designs

• Example: A vehicle safety inspection station
performs three jobs: (1) brake check, (2)
headlight check, (3) steering check. The
present system has three stalls in parallel, that
is a vehicle enters a stall, where one attendant
makes all three inspections.

Cars arrive Cars depart

Three attendants 5

Comparison of Two System Designs

• Example: Arrivals occur completely at random
at an average rate of 9.5 per hour and that the
times for a brake check, a headlight check, and
a steering check are normally distributed with
means of 6.5, 6 and 5.5 minutes, all having
standard deviation 0.5 minute. There is no
limit on the queue of the waiting vehicles.

3
 5/4/2024

Comparison of Two System Designs

• Example: In an alternative system design,
each attendant will specialize in a single task,
and each vehicle will pass through three
workstations in series. No space is allowed
between workstations. The increased
specialization suggest that mean inspection
times will decrease by 10%: to 5.85, 5.4, 4.95
minutes for brake, headlight and steering
inspections.
Cars arrive Cars depart

7
Brake Inspection Headlight check Steering check

Comparison of Two System Designs

• Example: A council has decided to compare
two systems on the basis of mean response
time per vehicle.

Cars arrive Cars depart

Three attendants

Cars arrive Cars depart

Brake Inspection Headlight check Steering check 8

4
 5/4/2024

Comparison of Two System Designs

• A terminating or non-terminating simulation?
• Decide on a warm-up period and run length,
T(i) as well as the number of replications, ni for
each system i.
• From replication k of system i, the simulation
analyst obtains an estimate Yki of the mean
performance qi

Comparison of Two System Designs

Sample Sample
System Replication Mean Variance
1 Y11 Y21 …… Yn11 Y •1 S12
2 Y12 Y22 ……. Yn22 Y •2 S 22

• Y •1 − Y •2 is an estimator for q1-q2.The aim is to

compute a confidence interval for q1-q2. The
confidence interval is used to answer two
questions:
1. How large is the mean difference and how precise is
the estimator of mean difference?
2. Is there a significant difference between two systems?
10

5
 5/4/2024

Comparison of Two System Designs

1. x x
Y •1 − Y •2 0

If the 100(1-a)% confidence interval for q1-q2 is

totally to the left of zero, there is a strong
evidence for the hypothesis that q1-q2 <0 at a
significance level.
In our example, the mean response time for
system 1 (original system) is smaller than for
system 2 (alternative system) at a significance
level.
11

Comparison of Two System Designs

2. x x
0 Y •1 − Y •2

If the 100(1-a)% confidence interval for q1-q2 is

totally to the right of zero, there is a strong
evidence for the hypothesis that q1-q2 >0 at a
significance level.
In our example, the mean response time for
system 2 (alternative system) is smaller than for
system 1 (original system) at a significance level.
12

6
 5/4/2024

Comparison of Two System Designs

3. x x
0Y •1 − Y • 2

If the 100(1-a)% confidence interval for q1-q2

contains zero, then, in the data at hand, there is
no strong statistical evidence that one system
design is better than the other at a significance
level.

13

Comparison of Two System Designs

• A two-sided 100(1-a)% confidence interval
for q1-q2 is always of the form:
(
Y •1 − Y •2  ta / 2,v s.e. Y •1 − Y •2 )
where Y •i is the sample mean performance
measure for system i over
n
all ni replications.
1
Y
i

Y •i = ji
ni j =1
and v is the degrees of freedom associated
with the variance estimator and s.e.()
represents the standard error
14

7
 5/4/2024

Comparison of Two System Designs

• Independent sampling:
• Independent sampling means that different and
independent random number streams will be
used to simulate the system.
• All observations of simulated system 1 are
statistically independent of all observations of
simulated system 2.
V (Yri )  i2
V (Y •i ) = = ,
ni ni
 12  22
V (Y •1 − Y •2 ) = +
n1 n2 15

Comparison of Two System Designs

• Independent sampling with Equal Variances:
• In some cases, it is reasonable to assume that
two variances are equal to each other (but
unknown in value) . How can you achieve such
equality?  1 =  2 = 
2 2 2

• If it is reasonable to assume equal variances, use

the pooled estimator, Sp2 to estimate  2
(n1 − 1) S12 + (n2 − 1) S 22
S =
2

n1 + n2 − 2
p

16

8
 5/4/2024

Comparison of Two System Designs

• Independent sampling with Equal Variances:
• Then, the pooled estimator has  = n1 + n2 − 2
degrees of freedom.
( )
s.e. Y •1 − Y •2 = S p
1 1
+
n1 n2

Standard error of a point estimator

is the square root of the variance of (
Y •1 − Y •2  ta / 2,v s.e. Y •1 − Y •2 )
that estimator. Variance 2 on slide
#15, is replaced by the pooled
estimator, SP2
17

Comparison of Two System Designs

• Independent sampling with Unequal Variances:
• If the assumption of equal variances cannot safely be
made, the standard error of the point estimate is given
by:
( S2 S2
s.e. Y •1 − Y •2 = ) 1
n1
+ 2
n2
with v degrees of freedom where  is approximated by
(rounded to integer):
(S12 n1 + S 22 n2 )
2

Variance i2 on slide #15, is

replaced by the sample
= 2
( )(
(S1 n1 )2 (n1 − 1) + (S22 n2 )2 (n2 − 1) )
( )
variance, Si2
Y •1 − Y •2  ta / 2,v s.e. Y •1 − Y •2 18

9
 5/4/2024

Comparison of Two System Designs

• Common Random Numbers (CRN)
• CRN means that, for each replication, the same
random numbers are used to simulate both systems.
Therefore, n=n1=n2and simulation run lengths are
equal to each other as well as the warm-up periods.
• For a replication r, are Yr1 and Yr2 independent of each
other?
Sample Sample
System Replication
Mean Variance
1 Y11 Y21 …… Yn1 Y •1 S12
2 Y12 Y22 ……. Yn2 Y •2 S 22
19

Comparison of Two System Designs

• Common Random Numbers (CRN)
• The variance of the 2mean difference is given by:
2
1 2
V (Y •1 − Y •2 ) = + − 2 cov(Y •1 , Y •2 )
n n
• To compute a 100(1-a)% confidence interval
with correlated data, compute the differences
Sample Sample
System Replication Mean Variance
1 Y11 Y21 …… Yn1 Y •1 S12
2 Y12 Y22 ……. Yn2 Y •2 S 22
Difference D1=Y11-Y12 D2=Y21-Y22 …….. Dn=Yn1-Yn2 D = 1 n D 1  n 2 2
 i
n i =1
S D2 =   Di − n D 
n − 1  i =1 
20

10
 5/4/2024

Comparison of Two System Designs

• Common Random Numbers (CRN)
• The standard error of mean difference is given by:
( ) ( )
s.e. Y •1 − Y •2 = s.e. D =
SD
n
with =n-1 degrees of freedom.

D
(
Y •1 − Y •2  ta / 2,v s.e. Y •1 − Y •2 )

21

Comparison of Two System Designs

• Example: The vehicle safety inspection station.
Replication 1 2 – ind. 2- CRN Difference
(1- 2-CRN)
1 29.59 51.62 29.55 0.04
2 23.49 51.91 24.26 -0.77
3 25.68 45.27 26.03 -0.35
4 41.09 30.85 42.64 -1.55
5 33.84 56.15 32.45 1.39
6 39.57 28.82 37.91 1.66
7 37.04 41.30 36.48 0.56
8 40.20 73.06 41.24 -1.04
9 61.82 23.00 60.59 1.23
10 44.00 28.44 41.49 2.51
Sample mean 37.63 43.04 37.26 0.37
Sample variance 118.90 244.33 109.52 1.74 22

11
 5/4/2024

Comparison of Two System Designs

• Example: The vehicle safety inspection station.
• Independent sampling with ????? Variances
( ) (
Y •1 − Y •2 − ta / 2,v s.e. Y •1 − Y •2  q1 − q 2  Y •1 − Y •2 + ta / 2,v s.e. Y •1 − Y •2 )
− 5.41 − t0.025,17 6.03  q1 − q 2  −5.41 + t0.025,17 6.03
− 18.1  q1 − q 2  7.3 95% Confidence interval for q1−q2

• Common random numbers

( ) (
D − ta / 2,v s.e. Y •1 − Y •2  q1 − q 2  D + ta / 2,v s.e. Y •1 − Y •2 )
− 0.04 − t0.025,9 0.41  q1 − q 2  −0.04 + t0.025,9 0.41
− 0.50≤ qq1 1−−q
-0.58 q 22≤1.30
1.31 95% Confidence interval for q231−q2

Comparison of Several System Designs

• We want to compare K alternative system designs.
• The comparison will be made on the basis of some
specified performance measure, qi of system i for
i=1,2,….K.
• Possible Goals:
• Estimation of each parameter, qi
• Comparison of each performance measure qi to a
control measure q1, e.g. existing system.
• All pair-wise comparisons, qi−qj for i≠j
• Selection of the best qi
24

12
 5/4/2024

Comparison of Several System Designs

• The first three goals will be achieved by the
construction of (C) confidence intervals.
• The number of confidence intervals is C=K, C=K-1,
C=K(K-1)/2 for goal 1,2, and 3, respectively.

25

Comparison of Several System Designs

• Bonferroni Approach for Multiple Comparisons:
• Suppose that C confidence intervals are computed
and that the ith interval has confidence coefficient
(level) 100(1-ai)% where i=1,2,…C.
• Let Si be the statement that the ith confidence
interval contains the parameter being estimated.
➔ Statement Si will be true with probability 1-ai
for i=1,2,…C.
• In goals 1,2 and 3, we would like to have high
confidence that all statements are true
simultaneously. 26

13
 5/4/2024

Comparison of Several System Designs

• Bonferroni Approach for Multiple Comparisons:
• The Bonferroni inequality states that
C
P(all Si are true, i = 1,2,...C )  1 −  a j = 1 − a E
j =1

where aE is called the overall error probability.

P(one or more Si is false, i = 1,2 ,...C )  a E =

aE is an upper bound on the probability of a false

conclusion.
27

Comparison of Several System Designs

• Bonferroni Approach for Multiple Comparisons:
• Set aE to be 0.05 or 0.10. (Don’t try smaller values)
• The individual aj may be chosen to be aE /C or
unequal.
• An advantage of Bonferroni approach is that it
holds whether the models for the alternative
designs are run with independent sampling or
with common random numbers.
• The major disadvantage is the increased width of
each individual interval.
• Use a maximum of 20 comparisons (C≤20) 28

14
 5/4/2024

Comparison of Several System Designs

• Bonferroni Approach for Multiple Comparisons:
• Goal 1:Estimation of each parameter, qi.
• Construct a 100(1-ai)% confidence interval for
parameter qi with C=K.
• This type of procedure is most often used to
estimate multiple parameters of a single
system, rather than to compare systems.

29

Comparison of Several System Designs

• Bonferroni Approach for Multiple Comparisons:
• Goal 2: Comparison of each performance measure
qi to a control measure q1, e.g. existing system.
• Compare all designs to one specific design-
usually to an existing system: construct a
100(1-ai)% confidence interval for parameter
qi −q1 (or q1 −qi) with C=K-1.

30

15
 5/4/2024

Comparison of Several System Designs

• Bonferroni Approach for Multiple Comparisons:
• Goal 3: All pair-wise comparisons, qi−qj for i≠j
• Compare all designs to each other - for any two
system designs i≠j, construct a 100(1-aij)%
confidence interval for parameter qi −qj with
C=K(K-1)/2. (The overall error probability is
equal to the sum of individual aij values for i≠j)

31

Comparison of Several System Designs

• Bonferroni Approach for Multiple Comparisons:
• Example: Consider the vehicle inspection station.
Alternative system designs are the following:
1. Existing system (parallel stations)
2. No space between stations in series
3. One space between brake and headlight
inspection only
4. One space between headlight and steering
inspection only

32

16
 5/4/2024

Comparison of Several System Designs

• Bonferroni Approach for Multiple Comparisons:
• Example: Comparison with an existing system. Set
aE =0.05 and ai=0.05/3=0.0167.
• Common random numbers are used so we
calculate Dri=Yr1-Yri
• -1.19<=q1-q2<=2.79
0.67<=q1-q3<=10.71
-0.94<=q1-q4<=3.46
With an overall 95% confidence level,
1. q1 −qi values are in the corresponding confidence intervals
2. q1 > q3 and there is no definite comparison between q1 and q2 and q1 and q4 33

17