Verification and

Validation of
Simulation
Models
14th Lecture

Slides based on:

1) “Discrete-Event System Simulation”, by Banks et al.
2) “Discrete-Event Simulation: A First Course”, by Leemis and Park
Prepared by: Ahmed Raoof
 Overview
• Why
➢ To produce a useful model that represents true behavior closely enough for
decision-making purposes.
➢ To increase credibility to an acceptable level.

• What is the process

➢ Verification
❖ Building the model correctly (correctly implemented with good input and
structure)
❖ Conceptual model to computer representation.
❖ Input parameters and logical structure.
❖ Verification is an integral part of model development.
➢ Validation
❖ Building the correct model (an accurate representation of the real system).
❖ Model vs. actual system behavior.
❖ An iterative calibration process.

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 2

Verification of
Simulation Model
Chapter #10 (textbook)

Mar 9th, 2021

SYSC 4005/5001 (By Ahmed Raoof) 3
 Common-sense Suggestions
• The purpose of model verification is to assure that the conceptual model (Assumptions,
abstractions, simplifications, parameters) is reflected accurately in the operational model.
• In this section, we will review the following concepts:
1) Have someone else check the model, preferably an expert in the simulation software being used.
2) Make a flow diagram that includes each logically possible action a system can take when an event
occurs, and follow the model logic for each action for each event type.
3) Closely examine the model output for reasonableness under a variety of settings of the input
parameters. Have the implemented model display a wide variety of output statistics, and examine all
of them closely. (Often Overlooked!)
4) Print the input parameters at the end of the simulation, to be sure that these parameter values have
not been changed inadvertently.
5) Self-documenting the model as possible.
6) Use of Interactive Run Controller (IRC) or a debugger to assists in finding and correcting logical errors
7) The use of graphical interfaces.

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 4

 Importance of Examining the model output
for reasonableness
• Example: A model of a complex network of queues consisting of many service centers.
➢ Response time is the primary interest, however, it is important to collect and print out many statistics in addition to
response time.
➢ Two statistics that give a quick indication of model reasonableness are: Current Contents (e.g. instantaneous queue
size) and Total Counts (e.g. number of arrivals in a period). For example:
❖ If the current content grows in a more or less linear fashion as the simulation run time increases, it is likely that a queue is
unstable (we will talk about this later in chapter #6).
❖ If the total count for some subsystem is zero (or a constant), indicates no items or the same number of items entered that
subsystem, a highly suspect occurrence.

• Other methods of verification:

➢ Compute certain long-run measures of performance, e.g. compute the long-run server utilization and compare to
simulation results.
➢ Use of a Trace
❖ A detailed printout of the state of the simulation model over time.

• Experience and personal motivation is needed!

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 5

 Verification Example (very simple one!)
• Using example 5 from Ch.2 (single server queue). an analyst
made a run over 16 units of time and observed that the time-
average length of the waiting line was 𝑳෡𝒒 = 0.4375 customer,
which is certainly reasonable for a short run of only 16 time
units. Nevertheless, the analyst decided that a more detailed
verification would be of value.
➢ Using equations from Ch.6 (I know we still not there yet!), the
estimated time-average length of the waiting line is is also
0.4375, which may indicate no errors.
➢ Using a trace (see the figure beside), we can see that in CLOCK
= 3 and 11, the system is idle and a customer comes in, but the
simulation does not change the STATUS to 1. This actually
causes wrong calculations of the time-average length.

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 6

Calibration and
Validation of
Simulation Models
Chapter #10 (textbook)

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 7

 Overview
• Usually conducted simultaneously.

• Validation: the overall process of comparing the model

and its behavior to the real system and its behavior.

• Calibration: the iterative process of comparing the model

to the real system, making adjustments (or even major
changes) to the model, comparing the revised model to
reality, making additional adjustments, comparing again,
and so on.

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 8

 Overview (Continue)
• No model is ever a perfect representation of the system
➢ The modeler must weigh the possible, but not guaranteed, increase in model accuracy versus the cost
of increased validation effort.

• Three-step approach:
➢ Build a model that has high face validity (high degree of realism, sensitivity analysis etc.)
➢ Validate model assumptions.
➢ Compare the model input-output transformations with the real system’s data.

• Statistical techniques such as hypothesis testing are used during the validation process!!

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 9

 Step ONE: Face Validity
• A model that appears reasonable.

• Involve potential users:

➢ Reasonableness.
➢ Credibility.

• Sensitivity Analysis
➢ Direction of change in output: whether the model behaves in the expected way when one or more
input variables is changed.

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 10

 Step TWO: Validate Model Assumptions
• General classes of model assumptions:
➢ Structural assumptions: how the system operates.

➢ Data assumptions: reliability of data and its statistical analysis.

❖ Test correlation and goodness of fit for data (what we did in Input Modeling).

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 11

 Step THREE: Validating Input-Output Transformation
• Goal: Validate the model’s ability to predict future behavior

• The model is viewed as an input–output transformation, accepting values of

input parameters and transforming these into output measures of performance.
It is this correspondence that is validated.

• One possible approach: use historical data that have been reserved for validation
purposes only.

• This is the ultimate test for the model!

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 12

 The Bank Example
• One drive-in window served by one teller
• Data collection
➢ 90 customers during 11 AM to 1 PM on a Friday.
❖ A typical rush hour.
❖ Observed service times {Si, i = 1,2,…,90}.
❖ Observed interarrival times {Ai, i = 1,2,…,90}.
➢ Input modeling (chapter 9, again!)
❖ Interarrival times: exponentially distributed with rate λ = 45 ➔ Poisson arrival process.
❖ Service times: N(1.1, 0.22)

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 13

 The Bank Example (Continue)
• A model was developed in close consultation with bank management and employees
• Model assumptions were validated
• Resulting model is now viewed as a “black box”

0.22

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 14

 The Bank Example (Continue)
• Real system data are necessary for validation.
➢ System responses should have been collected during the same time period (from 11 AM to
1 PM on the same Friday.)

• Compare the average delay from the model Y2 with the actual delay Z2:
➢ Average delay observed, Z2 = 4.3 minutes, consider this to be the true mean value µ0= 4.3.
➢ When the model is run with generated random variates X1n (generated arrival times) and
X2n (generated service times), Y2 should be close to Z2.
➢ Six statistically independent replications of the model, each of 2-hour duration, are run.

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 15

 The Bank Example (Continue)
• Simulation Results (of the 6 replications of the model):

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 16

 The Bank Example (still, Continue!)
• Compare the average delay from the model Y2 with the actual delay Z2
• Null hypothesis testing: evaluate whether the simulation and the real system
are the same (with regard to output measures):

𝐻0 : 𝐸 𝑌2 = 4.3 minutes
𝐻1 : 𝐸 𝑌2 ≠ 4.3 minutes
1
➢ In general, 𝑌ത2 = σ𝑛𝑖=1 𝑌2𝑖 , and 𝜇0 = 𝑍2 = 4.3. Then, 𝐻0 : 𝐸 𝑌ത2 = 4.3
𝑛
➢ If H0 is not rejected, then, there is no reason to consider the model invalid.
➢ If H0 is rejected, the current version of the model is rejected, and the modeler needs to
improve the model.

• Which test should we use?

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 17

 The Bank Example (the very long continue!)
• Conduct the t–test (See the last slides after the “What Next” section for more details, and this link):
➢ Choose level of significance (α = 0.05) and sample size (n = 6).
➢ Compute the same mean and sample standard deviation over the n replications:

𝑛
1 σ𝑛𝑖=1 𝑌2𝑖 − 𝑌ത2 2
ത
𝑌2 = ෍ 𝑌2𝑖 = 2.51 minutes, 𝑆= = 0.82 minutes
𝑛 𝑛−1
𝑖=1
➢ Compute test statistics and compare to the critical values (from Table A.5):

𝑌ത2 −𝜇0 2.51−3.4 𝛼

𝑡0 = 𝑆 = 0.82 = 5.24, 𝑡0.025,5 = 2.57 (two-sided test➔use 2 )
𝑛 6

➢ ∵ 𝑡0 > 𝑡0.025,5 , reject H0. Concludes that the model is inadequate.

• Check: the assumptions justifying a t–test, that the observations (Y2i) are normally and
independently distributed.
• Similarly, we can compare the model output with the observed output for the other measures:
➢ Y4 ↔ Z4, Y5 ↔ Z5, and Y6 ↔ Z6

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 18

 Error Types
• Type I error (α):
➢ Error of rejecting a valid model.
➢ Controlled by specifying a small level of significance, α.
𝑃 𝐻0 rejected 𝐻0 is valid = 𝛼
➢ Rejection of H0 is a strong conclusion. Hence, error probability α is chosen small.
• Type II error (β):
➢ Error of accepting a model as valid when it is invalid.
𝑃 𝐻0 failed to reject 𝐻1 is valid = 𝛽
➢ Controlled by specifying critical difference between assumed and real parameters and the sample size n.
➢ Failure to reject H0 is a weak conclusion.

• For a fixed sample size n, increasing α will decrease β.

• What can we do about the bank model that was rejected? Modeling errors were identified and
improved arrival/service models were developed. Read example in page 404-408 of the text for
more details. We will “re-continue” the bank example very soon!!!

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 19

 Error Types (Continue)
• For validation, the power of the test is defined as:
➢ 𝑃 detecting an invalid model = 1 − 𝛽
❖ Detecting a departure from 𝐻0 : 𝜇 = 𝜇0 when, in fact, such a departure exists (i.e. H1 is true!)
i.e 𝑃 reject 𝐻0 𝐻1 is true = 1 − 𝛽 = Power of the test (we want it close to 1!)
𝛽 = 𝑃 Type II error = 𝑃 failing to reject 𝐻0 𝐻1 is true (we want it small!)
❖ Note: 𝛼 = 𝑃 Type I error = 𝑃[reject 𝐻0 |𝐻0 is true)
➢ Consider failure to reject H0 as a strong conclusion, the modeler would want β to be small.
➢ Value of β depends on:
❖ Sample size, n
𝐸 𝑌 −𝜇
❖ The true difference, δ, between E(Y) and µ: 𝛿 =
𝜎
• In general, the best approach to control β error is:
➢ Specify the critical difference, δ
➢ Choose a sample size, n, by making use of the operating characteristics curve (OC curves – see figures
A.10 and A.11 in the textbook).

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 20

 Back to the Bank Example (Really?!)
• Original assumption: When a car arrives to find the window immediately available,
the teller begins service immediately.
• The real situation:
➢ Teller have other duties:
❖ Serving walk-in customers if no cars were present.
❖ Tellers always finishes with previous customers before serving new customers.
❖ When a car arrives at the window and the teller is busy, it has to wait until he is done.
• Modeler had to correct the model’s structure. New results are as follows:

• 𝑌෠2 = 4.78, 𝑠 = 1.66, 𝑡0 = 0.71 and 𝑡0.025,5 = 2.57 ⇒ 𝐻0 is not rejected

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 21

 The Bank Example (Ahhh! Continue)
• We want to test the following hypothesis:
H0: E[Y2]=4.3, H1: E[Y2] ≠4.3
• Following the methodology from the previous slide, we see that the null hypothesis is not
rejected.
• Suppose the modeler would like to reject H0 with probability of at least 0.9 (β ≤ 0.1) if the
sample mean delay E[Y2] differed from the real one (4.3) by
1 minute → 𝐸 𝑌 − 𝜇 = 1
0.75
➢ Then
𝐸 𝑌 −𝜇 1
𝛿= = = 0.60
𝜎 1.66
➢ and therefore 𝛽 𝛿 = 𝛽 0.6 = 0.75 for n = 6 (Figure A.10). That is
bad because it is far from 0.1!!
• How can we make β ≤ 0.1? 0. 1
➢ We should increase our sample size (i.e. replications) to approx. 30
0.6
(given that σ would be the same, we need to recalculate it!)

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 22

 Confidence Interval Testing
• If Y is simulation output, and 𝜇 = 𝐸(𝑌), evaluate whether simulation and real system are
close enough ➔ 𝜇 − 𝜇0 ≤ 𝜖 and ε is decided by the analyst.

𝑆
• Confidence interval (C.I.) for µ is 𝑌ത ± (𝑡𝛼,𝑛−1 × )
2 𝑛

• Validating the model:

➢ Suppose the C.I. does not contain µ0
❖ If the best-case error is > 𝜀, model needs to be revised and improved.
❖ If the worst-case error is ≤ 𝜀, accept the model.
❖ If best-case error is ≤ 𝜀, additional replications are necessary.

➢ Suppose the C.I. contains µ0

❖ If either the best-case or worst-case error is > 𝜀, additional replications are necessary.
❖ If the worst-case error is ≤ 𝜀 , accept the model.

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 23

 The Bank Example (Dude, Seriously?!)
• Bank example: 𝝁𝟎 = 𝟒. 𝟑, and “close enough” is 𝝐 = 𝟏 minute of expected
customer delay.
➢ A 95% confidence interval, based on the 6 replications is
𝑆 1.66
𝑌ത ± (𝑡𝛼,𝑛−1 × ) = 2.51 ± 2.57 × = 2.51 ± 1.74
2 𝑛 6
Yielding the C.I. = [0.77, 4.25]

➢ So, 𝝁𝟎 falls outside the confidence interval → the best case |4.25 – 4.3| = 0.05 < 1, but the
worst case |0.77 – 4.3| = 3.53 > 1

➔ Additional replications are needed to reach a decision.

The textbook used the non-revised simulation results, but got the same conclusion.

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 24

 Using Historical Input Data
• An alternative to generating input data:
➢ Use the actual historical record.
➢ Drive the simulation model with the historical record and then compare model output to
system data.
➢ In the bank example, use the recorded interarrival and service times for the customers
{An, Sn, n = 1,2,…}.

• Validation
➢ Need to collect independent replications for output data.
➢ Similar approach to generated input data.

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 25

What’s Next?

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 26

 Do It Yourself
• READ: Examples 10.3 and 10.4

• Assignment #3: Try to solve these exercises from the textbook

(P.413-414): 1 to 3.

• Solution will be posted after 10 days (on Mar 19, 2021) on CULearn.

• Might come in the midterm!

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 27

 Things To Do Before The Next Lecture

READ:
• Textbook, chapter ELEVEN, Estimation of Absolute Performance

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 28

 What is t-distribution?
◼ Let Z1, Z2, …, Zk be normally and independently distributed random
variables with mean µ=0 and variance σ2=1. Then the random
variable 𝒳 2 = 𝑍12 + 𝑍22 + ⋯ + 𝑍𝑘2 is said to follow the chi-squared
distribution with k degrees of freedom, abbreviated 𝒳𝑘2 .

◼ Let 𝑍~𝑁(0,1) (i.e. Z follows standard Normal distribution) and V be a

chi-squared random variable with k degrees of freedom. If Z and V
are independent, then the random variable
𝑍
𝑇=
𝑉
𝑘
is said to follow the t distribution with k degrees of freedom,
abbreviated tk.

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 29

 Why t-distribution?
◼ Suppose
◼ It can be shown that

◼ We know
therefore

◼ By definition, we have follow t distribution with n-1

degree of freedom
therefore

Mar 9th, 2021 SYSC 4005/5001 (By Ahmed Raoof) 30