Simulation

Chapter 6

Verification and Validation

Objective
 Knowing when to use terminating vs. non terminating
simulation

 The student must be able to calculate the needed number of

replications

 Understand the comparison of two scenarios with the

statistically significant method

 Be able to determine the warm-up period

Plan
 Verification and validation process
 Terminating versus non-terminating simulation
 Statistical analysis of output from terminating simulations
 Confidence Intervals and Number of Replications
 Validation with real system data, comparing two Scenarios
 Statistical analysis of output from non-terminating simulations
 Warm-up period and run length in steady-state simulation
 Truncated replications method
Verification and Validation
 Verification is the process of ensuring that the model design
(conceptual model) has been transformed into a computer model
with sufficient accuracy.
 Validation is the process of ensuring that the model represents
reality at a given confidence level.
 Verification is concerned with building the model right, which is
used in comparison of conceptual model to the computer
representation.
 Validation is concerned with building the right model, which is used
to determine that a model is an accurate representation of a real
system. It is achieved through comparing the model to actual
system behavior. This process is repeated until model accuracy is
judged to be acceptable.
Verification
The goal of Verification process is to ensure that the model:
 Include all of the components specified under the system definition phase
 Actually be able to run without any errors or warnings

 To effectively include all of the components specified under the system

definition phase, the practitioner should use the divide-and-conquer
approach.
 Break the larger, more detailed system model into a smaller, simpler. The
smaller, simpler model will be correspondingly easier to debug.
 Once the small, simple model includes all of the basic desired
components, run the model. Any errors in syntax or variable naming can
be more easily addressed.

 The same approach we used in Laboratory 2 for the Call Center Problem
Validation
The goal of validation is a two fold process:
 To produce a model that represents true system behavior, this can be
used as a substitute for the actual system, for the purpose of
experimenting.
 To increase the acceptance, credibility level of model, so that the
model will be used by managers and other decision makers.
There are two major types of validation
 Face validity means that the model, at least on the surface, represents
reality (flow processing correctly, etc.).
 Statistical validity : a quantitative comparison between the output
performance of the actual system and the model. Statistical validation
technique differs between terminating and non-terminating simulation
model (Law and Kelton, 2000).
Terminating versus non-terminating simulations
 Terminating simulation have a naturally occurring event that ends
the period of interest. In contrast, nonterminating simulation we do
not have a naturally occurring terminating event.
 Terminating simulation:
 Runs for some duration of time TE, where E is a specified event that stops
the simulation.
 Starts at time 0 under well-specified initial conditions.
 Ends at the stopping time TE.
 Bank example: Opens at 8:30 am (time 0) with no customers present and 8
of the 11 teller working (initial conditions), and closes at 4:30 pm (Time TE =
480 minutes).
 The simulation analyst chooses to consider it a terminating system because
the object of interest is one day’s operation.
Terminating versus non-terminating simulations
 Non-terminating simulation:
 Runs continuously, or at least over a very long period of time.
 Examples: assembly lines that shut down infrequently, telephone
systems, hospital emergency rooms.
 Initial conditions defined by the analyst.
 Runs for some analyst-specified period of time TE.
 Study the steady-state (long-run) properties of the system: properties
that are not influenced by the initial conditions of the model.
 In non-terminating or steady-state simulation the steady-
state behavior of the system is to be analyzed. The modeler
must determine a suitable length of time to run the model.
Terminating versus non-terminating simulations
 Terminating simulation
Terminating versus non-terminating simulations
 Non-terminating simulation
Terminating versus non-terminating simulations
 Whether a simulation is considered to be terminating or non-terminating
depends on both
 The objectives of the simulation study and
 The nature of the system.

 Output analysis (examination of the data generated by a simulation)

depends on the chooses simulation

 For terminating simulations statistics are gathered on performance measures

for several simulation runs, or replications, of the period of interest using a
different random seed for each run. This procedure enables statistically
independent and unbiased observations to be made on the system
response over the period simulated.
Statistical analysis of output from terminating
simulations
For terminating simulations, the questions to answer in running the experiment is:
How many replications to make?

 The number of replications should be determined by the precision required for

the output : the desired width of the confidence interval.

 A confidence interval is a statistical means for showing how accurately the

mean average of a value is being estimated. The narrower the interval the
more accurate the estimate.

 When applying confidence intervals to simulation output, more replications

(samples) are performed until the interval becomes sufficiently narrow to satisfy
the desired precision.
Statistical analysis of terminating simulations

Steps to determine the needed number of replications to make:

 Step 1 : Make 5 (to 10) replications and collect statistics on the desired
performance measures (average time in system of part or call, lead-time, etc. )
 Step 2 : Calculate the sample mean, standard deviation and the confidence
interval for the performance measure across replications
 Step 3: Selecting a level of Precision
 Step 4: Compute the needed number of replications
Statistical analysis of terminating simulations
Step 1: Run “Model 3. doe” for 5 replications collect the Performance measure :
Percent Rejected

Application see: NumberReplication.xls

Replication Percent Rejected (adding 2 new

Number Percent Rejected (Base case ) Sales)
1 8,29 11,05
2 13,03 7,99
3 11,6 11,63
4 12,04 6,66
5 15,12 11,88
Statistical analysis of terminating simulations
 Step 2 : Calculate the sample mean, standard deviation and the confidence
interval (CI) for the performance measure across n replications:
1 n
Sample mean: X ( n)   X j
n j 1
n

 (x i  X ( n)) 2
Value from Student’s t-distribution
Sample variance: S ( n) 
2 i 1
with n-1 degree of freedom
n 1
S ( n)
X ( n)  t n 1,1 / 2
Confidence interval for the mean: n
Half-width of CI
A significance level (α) of 5% is often selected. This gives a 95% probability that the
value of the true mean (obtained if the model is run for an infinite period) lies within
the confidence interval (this is known as a 95% confidence interval).
Statistical analysis of terminating simulations

CRITICAL VALUESFORTHESTUDENT’St-DISTRIBUTION
Statistical analysis of terminating simulations
Step 3: Selecting a level of Precision:
 Application to Model 3.doe, from initial 5 replications the Percent Rejected is
with 95% confidence :

 We generally seek a Half width of less than 10% of the mean value: target
decrease the Half-width from 3,08 to 1,2.
Statistical analysis of terminating simulations
Step 4: Compute the needed number of replications

 Set half-width = h, solve for

 Not really solved for n (t, s depend on n)

n grows
h0 = half width from quadratically
 Approximation: “initial” number n0 as h
(here 5) of replications decreases

Solution with 33 replications we find : 10,65± 0.88

Validation with real system data
 System validation data collection varies significantly from input data collection
processes.

 In system validation data collection, the major concern is collecting data

which reflects the overall performance of the system.

 A common method is to collect system or flow time. This is the time that it takes
an entity to be processed or flow through the entire system under study.

 Example : in a manufacturing order-type model, this would involve recording

the time that the order was received and completed in the actual system. This
type of data may be collected for 50 actual orders in order to make a
comparison with the orders from the simulation model.
Simple method but problem of statistic robustness: existence of autocorrelation
Validation with real system data
 Robust Statistical validation (Source: Simulation Modelling Handbook)
Comparing Two Scenarios
 Usually compare alternative system scenarios, configurations,
layouts, etc.
 Reasonable but not-quite-right idea
 Make confidence intervals on expected outputs from each
scenario, see if they overlap.
Application : Look at Percent Rejected
 Base case:
10.65 ± 0.88, or [9.77, 11.53]
 More-resources case (Adding 2 New sales):
Overlap

9.77 ± 0.73, or [9.04, 10.5]

 This doesn’t allow for a precise efficient statistical conclusion
Comparing Two Scenarios: Output Analyzer
Using Arena Output Analyzer
 Output Analyzer is a separate application that operates on .dat files produced by
Arena
 Launch separately from Windows, not from Arena
 Start Output Analyzer, open a new data group
 Basically, a list of .dat files of current interest
 Can save data group for later use – .dgr file extension
 Application to Model 3
 Add file Percent Rejected.dat to Statistic module, Output column
 Saves output statistics to this file:
Comparing Two Scenarios: Output Analyzer
Simulate the Base case (with n =33 Replications) and gather
PercentRejected.dat, then simulate the Scenario with adding 2 new sales (n=45)
and gather PercentRejecteds1.dat.
Comparing Two Scenarios: Output Analyzer
 Analyze > Compare Means menu option
 Add data files … “A” and “B” for two scenarios
 Select “Lumped” for Replications field
 Title, confidence level, accept Paired-t Test.
Comparing Two Scenarios: Output Analyzer
 Results:

Confidence interval on difference miss 0

 Conclude that there is a (statistically) significant difference on output
performance measures (Base Case-Scenario adding 2 new sales)
Statistical analysis of output from non-terminating
simulations
• In non-terminating steady-state simulation model has no natural termination
time the modeler is interested in long-term dynamics and statistics.

• The model starts empty, it takes some time before it reaches this steady-state.
The initial conditions tend to bias the desired long-term statistics

Solution : start statistics collection after an initial period of system warm-up to

eliminate bias due to observations taken during the transient state of the
model.

Objective: determining the initial warm-up period and the run length.
Analysis of non-terminating simulations

 In steady-state the performance measures of the system (e.g., processing rates,

utilization, etc.) exhibit statistical regularity (i.e., the distribution of these
variables are approximately the same from one time period to the next).
 The time that it takes to reach steady-state is a function of the amount of
activity and the stochasticity in the model. For some models it can be reached
in few hours of simulation time. For other models it may take several hundred
hours to reach steady-state.
Analysis of non-terminating simulations
 The Issue : determine Warm-up and run length times?
 Run a preliminary simulation of the system, preferably with (3 to 5)
replications, and observe at what time the system reaches statistical
stability. The length of each replication should be relatively long (Law
and Kelton, 1991).

 Ifdifferent Warm-up Periods for different output performances ?

Solution : take maximum (must specify a single Warm-up Period for
whole model)
Analysis of non-terminating simulations
 Application to A Wokstation (Model NTS.doe)
 In a workstation jobs arrive with exponential interarrival times of
mean 1 hour, and a fixed unit processing time of 0.75 hours.
 Simulate 3 replications during 700 hours
 Gather statistics on the average waiting time in queue
 Plot statistics with Output Anlyzer
Truncated replications method
Warm-u p
period

The transient state up until time 250 hours after this warm-up period the simulation model
goes into the steady state.
Truncated replications method
 Ifyou can identify appropriate warm-up and run-length times, use
the Truncated Replication method with just make replications as
for terminating simulations
 Only difference: Specify Warm-up Period in
Run > Setup > Replication Parameters
 Proceed with confidence intervals, comparisons, all statistical analysis as in
terminating case
Batching method
 If
model warms up very slowly, truncated replications can be
costly
 Have to “pay” warm-up on each replication
 Batching Method is an alternative