Module 7B PDF
Module 7B PDF
MODULE 7B
DISCRETE EVENT SIMULATION: VERIFICATION & VALIDATION
In this module, we are going to verify and validate the simulation output after we build the
simulation from previous module, that is Module 6B DISCRETE EVENT SIMULATION:
BUILD THE MODEL 2. This material will be discussed in Module 7B DISCRETE EVENT
SIMULATION: VERIFICATION & VALIDATION.
Define the
Collect and
Objective: Build the Validate the Conduct
Analyze
Scope and Model Model Experiment
System Data
Requirement
1
7.4 Theoritical Basic
1. Importance of Model Verification and Validation
At the verification and validation stage, there are two models examined, the name
are conceptual model and simulation model. A modeler is tasked to translate or
compile such a model in a real system into a conceptual model. Furthermore, the
model in conceptual form translate to simulation model. Modelers in translating the
system, have the opportunity for translation errors. This process does have a high
error rate. Therefore, to reduce the error rate must be checked again by validation
and verification (Harrell, Ghosh, & Bowden, 2004). Despite their importance,
seldom is adequate attention given to verification and validation when doing
simulation. The primary reasons for neglecting this important activity are:
a. Time and Budget Pressures
b. Laziness
c. Overconfidence
d. Ignorance
2. Verification
Verification is the process of determining whether the simulation model correctly
reflects the conceptual model (Harrell, Ghosh, & Bowden, 2004).
Once everything has been done at the outset to ensure that the model has been built
correctly and a standard for comparison has been established, several techniques can
be used to verify the model. Some of the more common ways include:
a. Conduct Model Code Reviews
A model code review can be conducted by either the modeler or someone else
familiar with both modelling and the system being modelled. The purpose of the
model code review is to check for errors and inconsistencies.
b. Check the Output for Reasonableness
One way to help determine reasonableness of the output is to replace random
times and probabilistic outcomes with constant times and deterministic
outcomes in the model. This allows you to predict precisely what the results will
be because the analytically determined results should match the results of the
simulation.
c. Watch the Animation for Correct Behavior
Animation can be used to visually verify whether the simulation operates the
way you think it should. Errors are detected visually that can otherwise go
2
unnoticed. Animation is usually more helpful in identifying a problem than in
discovering the cause of a problem.
d. Use the Trace and Debug Facilities Provided with the Software
Trace and debugging information provide detailed textual feedback of what is
happening during a simulation. This allows the modeler to look under the hood
and see exactly what is going on inside the simulation.
3. Validation
Validation is the process of determining whether the conceptual model correctly
reflects the real system (Harrell, Ghosh, & Bowden, 2004). Here are several
techniques that can be used to validating a model:
a. Watching the Animation
The visual animation of the operational behavior of the model is compared with
one’s knowledge about how the actual system behaves. This can include
dynamic plots and counters that provide dynamic visual feedback.
b. Comparing with The Actual System
Both the model and the system are run under the same conditions and using the
same inputs to see if the results match.
c. Comparing with Other Models
If other valid models have been built of the process such as analytic models,
spreadsheet models, and even other valid simulation models, the output of the
simulation can be compared to these known results. For example, a simple
simulation model might be compared to a validated queuing model.
d. Conducting Degeneracy and Extreme Condition Tests
There are known situations for which model behavior degenerates, causing a
particular response variable to grow infinitely large. An example is the number
of items in a queue prior to a server where the service rate is less than the arrival
rate. These situations can be forced in the model (by increasing the number of
arrivals) to see if the model degenerates as expected. The model can also be run
under extreme conditions to see if it behaves as would be expected. For example,
if arrivals are cut off, the model should eventually run dry of entities.
e. Checking for Face Validity
Face validity is checked by asking people who are knowledgeable about the
system whether the model and its behavior appear reasonable. This technique
can be used in determining if the logic in the conceptual model is correct and if
3
a model’s input–output relationships are reasonable.
f. Testing Against Historical Data
If historical information exists for both operating and performance data, the
model can be tested using the same operating data and comparing the results
with the historical performance data. This is sort of an “as-once-was” model.
g. Performing Sensitivity Analysis
This technique consists of changing model input values to determine the effect
on the model’s behavior and its output. The same relationships should occur in
the model as in the real system. The modeler should have at least an intuitive
idea of how the model will react to a given change. It should be obvious, for
example, that doubling the number of resources for a bottleneck operation
should increase, though not necessarily double, throughput.
h. Running Traces
An entity or sequence of events can be traced through the model processing logic
to see if it follows the behavior that would occur in the actual system.
i. Conducting Turing Tests
People who are knowledgeable about the operations of a system are asked if they
can discriminate between system and model outputs. If they are unable to detect
which outputs are the model outputs and which are the actual system outputs,
this is another piece of evidence to use in favor of the model being valid.
4. The Difference between Verification and Validation
4
model in a real system. While verification is the stage of checking whether the
simulation model has represented a conceptual model. Based on this information,
the two things are different but have the same goal of ensuring that the model made
is largely correct and in accordance with the reference model (Harrell, Ghosh, &
Bowden, 2004).
5. Normality Test
A normality test is used to determine whether sample data has been drawn from a
normally distributed population. Normal distribution data needed in order to find
comparison for the data using Welch test. Normality test can be done using
Kolmogorov-Smirnov test.
6. Welch Test
Welch test is the parametric test for comparing means between two independent
groups without assuming equal population variances. Welch test requires that the
observation drawn from each population (simulated system) be normally distributed
and independent within a population and between population (Harrell, Ghosh, &
Bowden, 2004). Welch test will be used to compare alternatives system designs
based on the estimates of the expected value of the performance measure that have
been derive from simulating the alternative system design. T-test method and the
Welch confidence interval method for testing hypotheses about two alternatives.
The null hypothesis and the alternate hypotheses are:
H0: µ1 - µ2 = 0
H1: µ1 - µ2 ≠ 0
Where:
H0: null hyphothesis µ1: average sample 1
H1: alternative hypothesis µ2: average sample 2
hw denotes the half-width of the confidence interval and is computed by:
tdf,α/2 is a factor obtained based on the value of α/2 and the estimated degrees of
freedom. Note that the degrees of freedom term in the Student’s t table is an integer
value. Given that the estimated degrees of freedom will seldom be an integer value,
5
this will have to use interpolation to compute the tdf,α/2 value.
7. Number of Replication (Sample Size)
Simulation that only run once do not represent the actual state of a system.
Replication is used to find out the number of samples needed, and that sample has
represented the population. Number of replications needed to establish a particular
confidence interval for a specified amount of absolute error (denoted by e) between
the point estimate of the mean x and the unknown true mean µ. e is equal to the hw
of a confidence interval. The equation to do replication is:
Where:
8. Dashboard
Dashboard are a blank window pane that can you can customize with anything that
you need to access during a simulation run. Dashboard in FlexSim can be used to
view charts and the statistics of the running model. In FlexSim, there are several
categories of dashboard such as Charts, WIP Templates, Throughput Template,
Staytime Template, State Templates, Zone Template, General, and Model Input.
6
Figure 7.4 2 Categories of Dashboard
7
7.5 Study Case
SIMBI Corporation is an assembly company that sell a stirling machine. The company do
the assembly process of the components separately, then after several processes it will
turned out to be a stirling machine for the finish product. SIMBI Corporation has 25
factories in Indonesia. The products are consist of four main components that are Lower
Platform, Main Machine, Piston Tube, and Upper Platform. As a product variation,
SIMBI Corporation produce 2 color variation in the stirling machine, and the variation are
green stirling machine and yellow stirling machine, and the variation are 60% green
stirling machine and 40% yellow stirling machine.
After running for some time SIMBI Corporation found that the demand forecasting for the
13th month is 80.526. Therefore with the 23 working days in a month, one factory should
produce at least 140 stirlings/day.
With the existing system, the company will have a loss profit around Rp 5.100.000,00/day.
In order to fulfill the demand forecasting, SIMBI Corporation need your help as an industry
analyst to find and give a solution that will be used to decide what will be done to face the
problems that occur in the company with hard Operation Research Methodology, and also
you need to do a production system simulation with a consideration of working hour per
shift or amount of machine.
Every arrival of lower platform package consists with 70 unit components. After its arrival,
The Lower Platform Package then enters the Separator Machine with distribution time U
(204, 215) minutes.
8
Every arrival of Piston Tube Package consist of 65-unit components. After its arrival,
Piston Tube Package then enters the separator machine with distribution time U (185, 216)
minutes. After processed by separator machine with distribution time N (1.02, 0.268),
piston tube unit and lower platform unit are being processed by installation machine with
distribution time N (1.62, 0.405) to be half-finished stirling machine through the queue
lower platform package and the queue of piston tube package.
Every arrival of the Upper Platform Package consists of 75-unit components. Upper
platform is divided by two color, that are green and yellow with the percentage of production
is respectively 60% and 40%. After its arrival, upper platform unit enters the separation
machine with distribution time U (225, 246) minutes.
Every arrival of Main Machine Package consists of 75-unit components, with distribution
time N (249, 26.8) minutes to enters the separator machine. After its processed by separator
machine, the half-finished product, upper platform unit and main machine unit are being
processed again in assembly machine with distribution time N (3.28, 0.888) minute to be
Finished Stirling Machine in green color and yellow color. Then the finished product of
green stirling machine and yellow stirling machine will be brought to the inspection area.
In the Inspection Area, Finished Stirling Machine in Green and yellow color will be
inspected by one operator of quality control with distribution time N (2.21, 0.457) minute.
The operator of quality control will get salary Rp 20.000,00/hour. Inspected Stirling
Machine in Green and Yellow color then moved to the warehouse area through the queue
conveyor for Green stirling machine and Yellow stirling machine.
Based on the calculation from 3rd module this following below is table of arrival time
distribution for every package:
9
The following is table of process time distribution in every workstation:
Table 7.5 2 Process Time Distribution
The operation process chart of stirling machine is as shown in the figure below:
10
The working hours of the company last for 9 hours, starting from 08.00 AM – 05.00 PM for
5 working days (Monday – Friday) with one hour for rest time between 12.00 PM – 01.00
PM. Throughout the production processes that has been carried out by the operators, Sales
Manager of SIMBI Corporation records the unfulfilled order which is as many as 6 stirlings
on average.
The CEO of SIMBI Corporation need your help as an industry analyst to find and give a
solution that will be used to decide what will be done to face the problems that occur in the
company with hard Operational Research Methodology, and also you need to do a
production system simulation with a consideration of working shift or amount of machine
to maximize the total output and company’s profit.
11
7.6 Lab Work
7.6.1 Verification
A. Checking Reasonable Output
1. Run the simulation until 12.00 PM at the same day as the warm-up date.
2. Based on the study case, the separator of lower platform package, piston tube
package, upper platform package, and stirling package separates each package
into 70 unit lower platform, 65 unit piston tube, 75 unit upper platform, and 75
unit main machine. Look the output of arrival at lower platform package by
clicking the Arrival_of_Lower_Platform then take a screenshoot.
3. Do the same thing for another platform such as Separator_Lower_Platform and
Queue_Lower_Platform. If the output on each platforms are same as in the study
case, it will be shown below:
12
6. For combiners, we can took the Installation_Machine as an example for the
combiner output. Click on Installation_Machine and take a screenshoot.
7. Click the Queue_Half_Finished_Product then look at the input in Quick
Properties. If the number of inputs are same as the output of
Installation_Machine, the model is running correctly.
13
2. Select Add a Dashboard, then the display will be appear as shown below:
3. Click WIP vs Time in the Library bar and drag it to Dashboard empty space.
14
4. Double click at WIP vs Time → Click on icon → Click Select Object →
Choose Separator_Lower_Platform → Click Apply → Click OK.
5. Run the simulation with the set time start from 07.00 AM until 12.00 PM. Before
run the simulation, click Reset.
6. The graphic will be shown as below:
7. Right-click on the WIP Vs Time graphic, then click on Export to CSV, save with
format WIP Lower Platform_FRI-XXX.
8. Open the CSV file and make sure that the Time cell column is on the time format,
it will shown as below:
15
Time Object WIP
4/20/2020 10:27 Separator_Lower Platform 1
4/20/2020 10:28 Separator_Lower Platform 0
10. Make a new column with name Time Difference and the format is time.
Time Object WIP Time Difference
10:27:35 Separator_Lower Platform 1
10:28:25 Separator_Lower Platform 0 0:00:50
13. Based on study case, the distribution time is Normal (1.02, 0.268) minutes.
Because it is in Normal distribution, change the distribution time to seconds
((1.02*60)-(0.268*60)) seconds = 45.12 seconds, and ((1.02*60)+(0.268*60)
16
seconds = 77.28 seconds) with 45.12 seconds as lower limit and 77.28 seconds
as upper limit.
14. Next step is for Installation_Machine.
15. Click Dashboard, then click on Add a Dashboard. Then the display will be
shown as below:
16. Click WIP vs Time and drag it to Dashboard empty space. It will show as below:
17. Double click at WIP vs Time → Click on icon → Click Select object →
Choose Installation_Machine → Click Select → Click Apply → Click OK.
17
Figure 7.6.2 13 Time Plot
18. Run the simulation with the set time start from 07.00 AM until 12.00 PM. Before
run the simulation, click Reset.
20. Right-click on the WIP Vs Time graphic, then click Export to CSV, save with
format WIP_Installation Machine_FRI-XXX.
18
Figure 7.6.2 16 Export to CSV
21. Open the csv file and make sure that the Time cell column is on the time format,
it will shown as below:
Time Object WIP Time Difference
10:28:25 Installation_Machine
1
10:28:25 Installation_Machine
2
10:29:45 Installation_Machine
0 0:01:20
10:29:45 Installation_Machine
1
10:29:45 Installation_Machine
2
10:31:31 Installation_Machine
0 0:01:46
10:31:31 Installation_Machine
1
10:31:31 Installation_Machine
2
10:34:09 Installation_Machine
0 0:02:38
10:34:09 Installation_Machine
1
10:34:09 Installation_Machine
2
10:36:12 Installation_Machine
0 0:02:03
10:36:12 Installation_Machine
1
10:36:12 Installation_Machine
2
10:37:31 Installation_Machine
0 0:01:19
10:37:31 Installation_Machine
1
Figure 7.6.2 17 WIP
22. Make a new column with name Time Difference and the format is time.
Time Object WIP Time Difference
10:28:25 Installation_Machine
1
10:28:25 Installation_Machine
2
10:29:45 Installation_Machine
0 0:01:20
10:29:45 Installation_Machine
1
10:29:45 Installation_Machine
2
10:31:31 Installation_Machine
0 0:01:46
10:31:31 Installation_Machine
1
10:31:31 Installation_Machine
2
10:34:09 Installation_Machine
0 0:02:38
10:34:09 Installation_Machine
1
10:34:09 Installation_Machine
2
10:36:12 Installation_Machine
0 0:02:03
10:36:12 Installation_Machine
1
10:36:12 Installation_Machine
2
10:37:31 Installation_Machine
0 0:01:19
10:37:31 Installation_Machine
1
19
23. In this column, substract time which has 0 WIP with 2 WIP. Number 2 in WIP
indicates that every 1x process at the Installation_Machine, it needs 2 part that
are Lower Platform and Piston Tube.
24. Then, average the time at the Time Difference column and turn it in to seconds.
The value of 97 (in second, as shown in Figure 7.6.9 19 Average Time
Difference) in average of Time Difference is the value of Average of Dashboard
that must be filled in Excel file for Labwork.
11:52:45 Installation_Machine
2
11:54:14 Installation_Machine
0 0:01:29
11:54:14 Installation_Machine
1
11:54:14 Installation_Machine
2
11:55:19 Installation_Machine
0 0:01:05
11:55:19 Installation_Machine
1
11:55:19 Installation_Machine
2
11:56:46 Installation_Machine
0 0:01:27
11:56:46 Installation_Machine
1
11:56:46 Installation_Machine
2
11:58:47 Installation_Machine
0 0:02:01
11:58:47 Installation_Machine
1
11:58:47 Installation_Machine
2
0:01:37 97
Figure 7.6.2 19 Average Time Difference
25. Based on study case, the distribution time is Normal (1.62, 0.405) minutes.
Because it is in Normal distribution, change the distribution time to seconds
(1.62*60)-(0.405*60) seconds = 72.9 seconds, (1.62*60)+(0.405*60) = 121.5
seconds, with 72.9 seconds as lower limit and 121.5 seconds as upper limit.
26. Do the same for other machines.
Note: If the distribution time is in Uniform, for example U (2.67, 3.8) minutes. Then, lower
limit for the distribution is (2.67*60) = 160.2 in seconds, while the upper limit is (3.8*60) =
228 in seconds.
7.6.2 Validation
A. Create Composite Throughput Per Hour on Dashboard
20
2. Select Add a Dashboard, then the display will be appear as shown below:
3. Click Throughput Per Hour in the Library bar and drag it to Dashboard empty
space.
21
4. Click icon at the Objects → Click Select Objects → Choose
Inspection_Area → Click Select → Change interval into 1 Hour → Click
Apply → Click OK.
5. Run the simulation with the set time for Warm-up Time on 07.00 AM and Stop
Time on 06.00 PM and run for fifteen work days from Monday, 20 April 2020
until Saturday, 9 May 2020. Before run the simulation, click Reset.
22
B. Determine Number of Replication
1. On Throughput Per Hour graphic, right-click then click export to CSV.
3. Create 3 new table columns and name it with Day, Total Day, and Total Output.
4. At the Day column, enter this formula =TEXT (the cell of Time; “dd”) in order
to enter the date when we run automatically.
5. At the Total Day column, enter the date when the simulation is running, which
is April 20 to May 9.
23
6. At the Total Output column, enter this formula =SUMIF(DAY;TOTAL
DAY;THROUGHPUT), as shown in Figure 7.6.2. 28 Total Output.. Lock the
formula except Total Day column. Then drag it down to add the output
according to the day automatically.
7. Copy the data in your excel in the table name Determine Number of Replication,
put it in order without including the one that has throughput 0 (zero).
Determine Number of
Replication
Replication Output
1 86
2 137
3 131
4 138
5 139
6 135
7 133
8 135
9 135
10 138
11 132
12 138
13 137
14 134
15 133
24
C. Calculate Validation
1. Insert the output from replication and output of the real system (real system
data can be seen in Table 7.5 3 Target and Realization) into the table. Insert the
data according to the number of replications obtained. Because 12 was the result,
insert 12 data and exclude the warm-up output.
1 137 136
2 131 135
3 138 136
4 139 135
5 135 133
6 133 132
7 135 131
8 135 134
9 138 136
10 132 134
11 138 135
12 137 134
Average 135.6666667 134.25
2. Open the IBM SPSS application and insert the simulation model data and real
system data, then change the variable name and the measure.
25
4. Put in all the variables into the dependent list then select plots and checklist on
normality plots with tests. After that, click on continue and ok.
5. In the normalization calculation results, right-click and select special copy, then
paste in analysis of normalization box. It can be seen that the standard deviation
of the calculation results is 0.200 which is more than an alpha value 0.050, it can
be said that the simulation results are valid.
26
Figure 7.6.2 38 Analysis of Welch Test
27
RERERENCE
Harrell, Ghosh, & Bowden. (2004). Simulation Using ProModel, Second Edition. The
McGraw-Hill Companies.
28