Proceedings of the 2004 Winter Simulation Conference

R .G. Ingalls, M. D. Rossetti, J. S. Smith, and B. A. Peters, eds.

INTRODUCTION TO MODELING AND SIMULATION

John S. Carson II

AutoMod Group
Brooks Automation
140 Dickerson Road
Marietta, GA 30067, U.S.A.

ABSTRACT nent. A discrete-event model is one whose state changes

only at discrete times called event times. When an event oc-
Simulation is a powerful tool for the analysis of new sys- curs, it may trigger new events, activities and processes.
tem designs, retrofits to existing systems and proposed In concept, a model’s state is a (long) vector, that is, a
changes to operating rules. Conducting a valid simulation list of values that are sufficient to define the complete state
is both an art and a science. This paper provides an intro- of the system at any point in time. In practice, a model’s
duction to simulation and modeling and the main concepts state is defined implicitly by the internal status of all the
underlying simulation. It discusses a number of key issues entities used in the simulation software package.
regarding a simulation team, how to conduct a simulation An event is an instantaneous occurrence that changes
study, the skills required and the steps involved. It also the model’s state. Examples include an arrival event for a
provides project management guidelines and outlines pit- customer at a bank, and a service completion event for the
falls to avoid. same customer. An activity is a duration of time, such as a
service time or interarrival time, that is initiated by an event
1 DEFINITIONS AND CONCEPTS in conjunction with the model being in a certain state. For
example, when arrivals are defined by a probability distribu-
There are many types and kinds of simulation. In this tuto- tion of interarrival times, then when one arrival occurs (an
rial we limit ourselves to discrete, stochastic process- event), the model generates a new interarrival time (an activ-
oriented simulation. This covers almost all simulations ity) which in turn will cause the next arrival event.
discussed at the Winter Simulation Conference. It ex- Primary events are those driven by data. Examples in-
cludes Monte Carlo-type simulations in a spreadsheet clude arrival times and service completion times. In simula-
(sampling studies, financial and risk analyses, and so on). tion terms, the primary events are scheduled to occur at some
It also excludes equation-based numerical solvers, for ex- future time, calculated from data and statistical assumptions.
ample, differential equation solvers and other equation- For example, if we assume that inter-arrival times are expo-
based models. Although not explicitly discussed, it may nentially distributed with a mean of 10 minutes, then at the
include training simulations and man-in-the-loop simula- time of an arrival, the model draws a new exponential sam-
tions such as many conducted by the military. ple, adds it to the current time, and schedules the resulting
future time as the time of the next arrival event.
1.1 Simulation Concepts Secondary events are those generated internally by
model logic. For example, in a waiting line model, when a
A model is a representation of a system or process. A server becomes available and there is a waiting entity (per-
simulation model is a representation that incorporates time son or product) at the front of the queue, then a “service
and the changes that occur over time. A discrete model is begin” event is scheduled to occur immediately.
one that changes only at discrete points in time, not con- An entity is an object in the model. Dynamic entities
tinuously. are created at time zero or at other times by an arrival event.
A model may incorporate logical, mathematical and Dynamic entities usually represent some real-world object
structural aspects of the system or process. A discrete-event that is flowing through a system. Examples include auto-
model, the type discussed in this paper and the type repre- mobiles in a manufacturing model, pallets or cases in a
sented by the great majority of papers at the Winter Simula- warehouse model, passengers in an airport model, and tele-
tion Conference, is one based on the concepts of state, phone calls in a communications model. Entities have stan-
events, activities and processes. Time is a critical compo- dard and customized attributes that individualize the entity.

9
 Carson

A resource is an entity that provides a service to dy- model to be configurable, that is, to represent a number of
namic entities. A resource usually has a finite capacity somewhat different system or process configurations.
representing some system constraint. Examples include a Simple examples include parameters that allow a user to
worker or a team of workers doing a task, a machine, or a vary the number of workers at a workstation, the speed of a
vehicle. In some models, resources may have user-defined machine or vehicle, the timing characteristics of a con-
resource states and special characteristics such as down- veyor control system, and so on.
times and availability schedules. As a descriptive model, you can use a simulation
Almost all discrete-event models are stochastic. That model to experiment with, and evaluate and compare, any
is, they contain some components that are modeled as a number of system alternatives. Evaluation, comparison
statistical distribution. This introduces random variation and analysis are the key reasons for doing simulation. Pre-
into a model, making it into a statistical or sampling ex- diction of system performance and identification of system
periment. More precisely, when one or more components problems and their causes are the key results.
are stochastic (for example, interarrival or service times),
then model outputs are stochastic, necessitating some kind 3 WHEN SHOULD SIMULATION BE USED?
of statistical analysis to draw valid conclusions.
Virtually all simulation software packages include Simulation is most useful in the following situations:
automatic collection of performance statistics, and easy
collection of custom statistics. 1. There is no simple analytic model, spreadsheet
model or “back of the envelope” calculation that
1.2 Simulation Worldviews is sufficiently accurate to analyze the situation.
2. The real system is regularized; that is, it is not
Simulation modeling software usually takes one of three chaotic and out of control. System components
worldviews: event scheduling, process interaction, or ac- can be defined and characterized and their interac-
tivity scanning. As their names suggest, each puts its main tion defined.
focus on the events, the processes, or the activities in a 3. The real system has some level of complexity, in-
simulation, respectively. When following an event sched- teraction or interdependence between various com-
uling perspective, a model developer must define the ponents, or pure size that makes it difficult to grasp
model logic and system state changes that occur whenever in its entirety. In particular, it is difficult or impos-
any event occurs. sible to predict the effect of proposed changes.
A process is a sequence of events, activities and other 4. You are designing a new system, considering ma-
time delays associated with one entity as it flows through a jor changes in physical layout or operating rules
system. For example, a customer process at a bank consists in an existing system, or being faced with new
of an arrival event (to the lobby, perhaps), joining and wait- and different demand.
ing in a queue (a delay), a service time by a teller , and fi- 5. You are considering a large investment in a new
nally a service completion event. In terms of concepts dis- or existing system, and it represents a system
cussed earlier, the service time is an activity and the teller is modification of a type for which you have little or
a resource. Simulation software based on the process inter- no experience and hence face considerable risk.
action perspective, or worldview, provide a way for a user to 6. You need a tool where all the people involved can
define a process for each entity in the system. agree on a set of assumptions, and then see (both
Activity scanning provides a way to define model statistically and with animation) the results and ef-
logic by focusing on activities from the point of view of a fects of those assumptions. That is, the simulation
resource, defining resource state changes depending on process as well as the simulation model can be
various events. For example, in the bank, the teller serves used to get all members of a team onto a (more)
one customer until completion, then looks at the queue. If common understanding.
the queue is not empty, the teller “takes” the first entity out 7. Simulation with animation is an excellent training
of the queue, changes its own state to “busy” and begins a and educational device, for managers, supervisors,
new service activity. If the queue is empty, the teller engineers and labor. (Don’t tell me, show me.) In
changes its own state to idle. fact, in systems of large physical scale, the simu-
For a more in-depth look at simulation concepts and lation animation may be the only way in which
worldviews, see Carson (1993) and Banks et al. (2000). most participants can visualize how their work
contributes to overall system success or creates
2 HOW IS SIMULATION USED? problems for others.

A simulation model is a descriptive model of a process or

system, and usually includes parameters that allow the

10
 Carson

4 THE ADVANTAGES AND VALUE 5.1 The Team

OF SIMULATION, AND THE
DISADVANTAGES Simulations are almost always conducted by a simulation
team, not an isolated individual. Sometimes one individual
Simulation allows experimentation with a model of a sys- plays several roles. The various roles include:
tem. Without a model, you either experiment with a real
system (if it exists) – probably causing major disruptions – • The customer’s executives and managers who
or proceed without such experimentation and analysis – at “own” the problem, the decision-makers,
some potential risk. Simulation allows the identification of • The customer’s engineers, staff, plant and line
problems, bottlenecks and design shortfalls before building managers, and others who are involved in the
or modifying a system. It allows comparison of many al- problem, know key portions of the day-to-day op-
ternative designs and rules of operation. Evaluation and erations and will live with implemented solutions,
comparisons can take place before committing resources • In-house or outside systems designers, who are
and investment to a project. designing a new system or changes to the existing
Simulation allows study of the dynamics of a system, system, and
how it changes over time and how subsystems and com- • The simulation analyst.
ponents interact. A simulation model provides about the
only method to study new, non-existent complex dynamic People who know and understand the actual system
systems for which analytic or static (spreadsheet) models are a key resource for project success. Even if the system
provide at best a low fidelity model with correspondingly itself is new and not yet built or operational, there will be
low accuracy. people who understand the business, the processes and the
On the other hand, often simulations are time- end product or service; their expertise is absolutely needed.
consuming, data is not available or costly to obtain, and the It is quite infrequent that one person alone understands the
time available before decisions must be made is not suffi- whole system in sufficient detail to provide all the informa-
cient for a reliable study. In some companies, an early suc- tion needed; rather, a number of different people are
cess with simulation has evolved into simulation becoming a needed, each providing a bit of expertise for one small part
“checklist” item on every project whether it is justified or of the system.
not for the project at hand. In some situations, the anima- The project team must include all those with questions
tions and other visual displays, combined with the time pres- that they expect the model to address. These questions need
sure present on all projects, may mislead decision makers to be identified and specified up front at project initiation,
into premature conclusions based on insufficient evidence. and not sprung upon the team at a final presentation where
In addition, inexperienced simulation analysts, or those too they will most likely go unanswered leading to misunder-
focused on (and in love with) the simulation software and standing and failure in the minds of some participants.
technology may add too much detail to a model and spend
too much time in model development, resulting in the origi- 5.2 The Simulation Analyst: Skills and Software
nal goals and project timelines being forgotten. This often
leads management to conclude that simulation, while a For a simulation analyst, simulation is both an art and a
promising and interesting technology, is too costly and time- science. As with any art, one learns by training and educa-
consuming for most projects. tion but more importantly by practice and mentoring.
A good simulation model provides not only numerical Good communication skills are a necessity; a willingness
measures of system performance, but provides insight into not to assume anything, not to be afraid to ask “stupid” or
system performance. Insight comes from a tacit under- “obvious” questions, and a willingness to ask the same
standing of system behavior, an understanding that can be question of many team members is a key to understanding
developed by intelligent use of animation and other visual and making accurate assumptions. For the science portion,
aids, and an intelligent set of valid experiments together programming, modeling, and a working knowledge of
with a good statistical analysis. probability and statistics are important skills to attain.
Knowledge of a simulation package, while necessary, by
5 THE SIMULATION TEAM itself is not sufficient to be a good simulation analyst.
Models should be developed in a suitable, commercially
available and supported simulation package. There are
Simulations are conducted by in-house specialists at many
many simulation packages, some more or less general pur-
companies as well as by many engineering, consulting and
pose, and some that specialize in either one or a few applica-
services companies. A few companies specialize in offer-
tion domains (such as manufacturing, material handling, call
ing simulation services. These groups or companies may
centers, medical, transportation, logistics or other limited
supply the simulation expertise and model development
area of applicability). Packages offer differing levels of de-
experience, but the whole team must be broader based.
tail, ease of use, and skill required for effective use, as well

11
 Carson

as differing levels of user customization capability. Some stood, and problem formulation may initially be stated in
have minimal or no programming, emphasizing ease of use terms of observed symptoms (for example, product through-
and quickness of model development for small to medium- put less than desired or expected). During the study, as the
size models with minimal complexity. Other packages offer nature of the problem becomes clearer, problem formulation
total customization but usually at the cost of programming may be restated and clarified with the project team.
skill, time and effort as well as gaining a knowledge of the During this phase, the simulation team should develop
selected package. Large complex models with unique rules a list of specific questions that the model should address,
and algorithms cannot be developed in most (or possibly and develop a list of measures of performance that will be
any) “no-programming” packages. On the other hand, many used to evaluate or compare the alternatives being mod-
simpler simulation models can be developed most efficiently eled. Often, the customer has a goal in mind; for example,
and quickly in the simpler “point & click” environments that the new system under a certain level of resources and
based on flow charting. manning will achieve an expected throughput. This means
The actual choice of software used is beyond the scope that if the study finds that the proposed system design or
of this introduction, but is often heavily influenced by what set of operating rules does not achieve the expected
the simulation analyst has used in the past. This is a valid throughput, then the model is expected to provide informa-
consideration, as learning a new simulation package can be tion and insight into the causes, so that the simulation ana-
time-consuming, and becoming an expert in it takes a lyst and team can develop intelligent alternatives that have
number of projects and an openness for self-education. a better chance of achieving desired goals.
A new simulation analyst with a programming back- At this phase, the simulation analyst (or project leader)
ground may think, on seeing the price range of simulation needs to ask questions of all participants and develop a set
packages, that a model can be developed in a general pur- of working assumptions that will form the basis for model
pose non-simulation programming language, such as C or development. Three important overall considerations are:
C++ or Visual Basic, with less expense and in about the
same time. This judgment is based on inexperience; in the • Model boundary and scope,
author’s experience, using a general purpose language, • Level of detail,
even with a library of simulation routines, generally takes • Project scope.
from 4 to 10 times the amount of analyst time for model
development as using a good simulation package; main- The model boundary or scope determines what is in
taining or extending such a model usually requires the the model, and what is out. The model level of detail
original developer, and can be a challenge (to put it nicely). specifies how in-depth one component or entity is mod-
To be successful with most simulation packages, a eled; it is determined by the questions being asked and data
new analyst needs training from an expert and ongoing availability. Think of model boundary as “width” and
mentoring for a period of time. Self-education is some- level of detail as “depth”. Overall project scope deals with
times possible, but usually results in a “spotty” knowledge the breadth of the questions that the model will be used to
with learning gaps if used as a total solution. address; that is, it deals more broadly with how the model
will be used during the experimentation and analysis
6 STEPS IN A SOUND SIMULATION STUDY phase. As more and more questions can be asked of a
given model (especially a parameterized one), the team
Every simulation project proceeds through a set of phases needs a common understanding of project scope to avoid
and steps whose goal is a successful project. Here are scope creep and a project with no end.
some guidelines.
6.1.2 Overall Project Plan
6.1 Project Initiation
With the information developed during problem formula-
In the first phase, projects begin with a kickoff meeting, tion, the simulation analyst should develop time estimates
problem formulation, objectives setting, determination of and project timelines for model development, verification
measures of performance, and details of modeling assump- and validation, and experimentation and analysis – all the
tions and data requirements, followed by a project plan steps in a simulation.
with time and cost estimates and project timelines. The With these time (and cost) estimates in hand, man-
end results of this phase are the Assumptions Document agement can decide whether to proceed with the simulation
and a project plan. study, or possibly to expand or limit its scope.

6.1.1 Problem Formulation and Setting of Objectives 6.1.3 Conceptual Model and Assumptions Document

All modeling activities should be focused on the objective. The set of agreed-upon assumptions and data is, in essence,
Often, the actual problem may be unknown or little under- the conceptual model. These assumptions and data re-

12
 Carson

quirements should be detailed in an Assumptions Docu- being used, then the customer should review these data as-
ment or Functional Specifications Document. sumptions internally with people knowledgeable in the
The Assumptions Document should be written in the processes involved.
language of the real system and the people who work in Data sources include databases, manual records, auto-
that system. It should not use modeling language or jargon matic data collection systems, sampling studies and time
peculiar to any particular simulation software or language. studies. Unfortunately, it seldom happens that all or even
After all, its purpose is to communicate a set of assump- much of the needed data is readily available, or when avail-
tions and data requirements among all members of the able that it is of the desired quality. In these circumstances,
simulation team, not all of whom will be, or even need to much effort and expense may be required to collect the data
be, simulation experts. With this common document, the or extract it from existing databases.
team can revise the assumptions until all members agree to After collecting it, a further effort may be required to
a common set of working assumptions, or at least to note validate and “cleanse” the data. Even data in customer da-
disagreement until agreement can be reached. tabases, surprisingly to some, may be suspect. Often sim-
In summary, project initiation has these essential ple tests or audits may show that what appears to be data
activities: availability is data garbage. For example, when simulating
a distribution center and using actual customer orders to
• Get all interested parties involved in project kick- drive the model, we found that order files were indeed
off, initial problem formulation and meetings dis- quite accurate (after all, the company is paid by customers
cussing model assumptions. If a person on the who receive what they order!). In contrast, the master SKU
customer or client side will be present at any re- list had many inaccuracies. For each SKU, the master SKU
view meetings or final presentations, that person list was supposed to give pallet weight and pallet height,
must be present at these initial meetings. If a per- but these were inaccurate in up to 50% of the 100,000
son expects the model to address certain ques- SKUs listed. The reason was simple: in the existing manual
tions, that person must put the questions on the system, the forklift drivers did not use this data to decide
table at project initiation. where to store a pallet; no one used it. In the proposed
• Put all assumptions and data requirements into computer-controlled new building which we were simulat-
writing. Include objectives, specific questions to ing, this data was essential as all storages and retrievals of
address, and measures of system performance. A pallets into rack were under software control. A great deal
written Assumptions Documents is essential. A of effort was required to cleanse the data to make it accu-
reviewed, and signed-off, Assumptions Document rate; on the positive side, this effort was required before
is critical. the new system could be put into operation.
When data on an activity is available, and the data ex-
6.2 Project Work
hibits random variability, that is, variability for which no
The project “work” consists of model development and immediate cause is evident, then the activity duration is
data collection. The end result is a working model with usually modeled by a statistical distribution. Sometimes
customer-provided and validated data. The working model the empirical distribution of the data is used; sometimes a
is subjected to verification and validation in the next phase. statistical package is used to fit a distribution to the data.
With some types of data, the analyst may decide to use
6.2.1 Model Development the actual data itself as input to the simulation. This may
be done at customer request, or because it is too difficult to
Model development consists, in a nutshell, of two major ac- represent the data as a statistical distribution. For example,
tivities: (1) development of data structures to represent the we often use customer order files as input to a model of a
data needed by the model, and (2) translation of the model- distribution center or order fulfillment center. Each cus-
ing assumptions in the Assumptions Document into the lan- tomer order may consist of a number of line items, and
guage or representation required by the simulation package. each line item has a quantity of one or more. There is al-
The simulation analyst must design data structures that rep- most always a correlation between number of line items
resent the data and its inter-relationships as well as fit into and quantity, a correlation that would be difficult to
those allowed by the simulation software. For example, al- characterize and represent with a statistical distribution. In
most all packages allow variable arrays, most allow tabular this and similar situations, we have decided to use actual
displays of data (and referencing of that data by model enti- order files to drive models. To get representative variation,
ties and processes), and some allow lists of objects and data. we ask the customer to provide several different samples of
order files (that is, orders from several different days). If
6.2.2 Data Collection, Cleansing, and Analysis there is a need to experiment with greater demand (more
orders), we can combine different order files into a single
The customer or client usually collects the agreed-upon order file. If there is a need for a different order profile
data. If data is not available or (subjective) estimates are
13
 Carson

(perhaps more small orders and fewer large orders), we answer their questions. If a team member will be present
partition the order file appropriately and sample or re- at meetings to present model results, that team member
combine to get a desired profile. should be present at validation review meetings (and, in-
deed, at earlier project kickoff meetings).
6.3 Model Verification and Validation Numerous techniques, similar to those used during
verification, may be used during model validation, includ-
In this phase the simulation analyst verifies the model, and ing (1) use of animations and other visual displays to
working with the customer, validates the model. If prob- communicate model assumptions, (2) output measures of
lems are found, the model or the data, or both, are cor- performance for a model configuration representing an ex-
rected. The end result of the V&V phase is a verified, isting system or an initial design, so that team members
validated model that is judged to be accurate enough for may judge model reasonableness. If sufficient data has
experimentation purposes over the range of system designs been collected on a real-world system that matches one of
contemplated. the model’s possible configurations, more formal tests may
be conducted comparing the real system to the model.
6.3.1 Model Verification For more discussion of V&V, see Carson (2002); in
addition, a subsequent talk on model verification and vali-
In model verification, the simulation analyst checks the dation in the introductory tutorial track will provide more
model, using a number of different techniques, to verify detail on appropriate techniques and issues.
that the running model agrees with the Assumptions
Document. This is more than debugging in the program- 6.4 Experimentation, Analysis, and Reporting
ming sense. All model outputs should make sense and be
reasonable over a range of the input parameters. The purpose of this phase is to meet initial project objec-
Numerous techniques should be applied, including but tives: to evaluate and compare system performance, and to
not limited to: (1) stress testing, or testing with a wide gain insight into the system’s dynamic behavior and, in
range of parameters and different random numbers; (2) a particular, into any problems or bottlenecks identified by
thorough review of all model outputs, not just the primary the analysis.
measures of performance, but numerous secondary meas-
ures; (3) using the software’s debugger, animation and any 6.4.1 Experimental Design
other tools provided; (4) using selective traces, especially
for complex portions of the logic; and (5) review by a more Before conducting simulation experiments, the analyst
senior simulation professional (especially valuable for the must decide a number of issues:
relatively new practitioners).
A valuable attitude to take is the one of a true scientist. 1. The input parameters to be varied, their range and
First, make an hypothesis: the model is “correct”. Second, legitimate combinations,
try as hard as you can to prove the hypothesis is false; that 2. Model runlength (how long to run the simulation),
is, try to prove that the model is “bad” in some way. If 3. For steady-state analyses, the model warm-up
only after great effort, you have only confirmations and no period,
evidence of a faulty model, then conclude (tentatively) that 4. Number of statistical replications.
the model is verified. From a scientific perspective, the
best that can be achieved is a tentative verification; a future Earlier informal experimentation during model devel-
test, or a change in conditions or data, may detect a prob- opment and the V&V phase should assist the analyst in
lem with the model requiring changes. Simply put, there making intelligent decisions regarding these questions:
are virtually an unlimited number of potential tests that For steady-state simulations, what is a reasonable warm-up
could (theoretically) be carried out to test a model’s valid- or transient period? What is a reasonable runlength? How
ity; in practice, we have the time for only a certain number many statistical replications are needed? In earlier phases,
of them. So the best we can achieve is a “failure to reject” the analyst should explore inherent model variability – the
the hypothesis of a “correct” or valid model. range of short-term behavior – which should provide at
least initial insight into appropriate model runlength and
6.3.2 Model Validation number of replications needed for later experiments.
Model runlength may be dictated by the nature of the
Model validation gets the customer involved. After the system or the available data, such as when simulating one
simulation analyst is convinced that the model is accurate day’s operation of a distribution center, one sort tour for a
and verified, the analyst should conduct a thorough model sortation hub for overnight packages, a one-shift ramp-up
review with the customer team. It is important to have all of a manufacturing line, or any other data-driven model
members of the customer team who may have an interest where the data represents a fixed period of time. In one
or “investment” in the model, and who expect the model to project we had a production schedule for a packaging line
14
 Carson

for one week, and hence the model runlength was one In this perplexing situation, the experienced simulation
week based on this customer-supplied data. In contrast, analyst will use the model as a basis for forming hypothe-
inherent and high system variability together with a desire ses regarding the causes of any identified problems. Then
for a certain level of statistical accuracy (width of confi- the analyst may need to add auxiliary measures of per-
dence intervals) combined to require upwards of 100 statis- formance to further pinpoint the cause, and most impor-
tical replications for each point in the experimental design tantly, to confirm that the hypothesis is correct. In any
(each system configuration). Other models with less in- number of simulation projects over many years, the author
herent variability have required only 3 to 5 replications. In has seen the need to use a model to dig into problem causes
other models, model runlength may be under the analyst’s that are not obvious at first sight and to devise new meas-
control, for example, for the future operation of a new port ures of performance to confirm hypotheses regarding the
design or a 24/7 manufacturing system. causes of system failure. The insight gained is invaluable
There is no rule of thumb for runlength or number of when it comes to suggesting changes to improve system
replications. Each is model dependent. The number of performance in order to meet a goal.
replications affects statistical accuracy of performance
measures; specifically, it affects the width of any confi- 6.4.4 Reporting
dence interval estimators. Other talks in the introductory
track address these and other statistical issues. Reporting of the results of experimentation and analyses
usually includes one or more presentations and a written
6.4.2 Experimentation report. It is wise to have both.
Presentations allow question and answer and expan-
The project plan developed during project kickoff and ini- sion of explanations. The response to the presentation
tiation provides the initial guidelines for a set of experi- should be used to finalize the report and to address issues
ments. Usually simulation models are used to compare a and questions that arose during the presentation.
large number of alternatives, and perhaps to evaluate in The final report should include the Assumptions
greater detail a small number (1 or 2) of recommended al- Document, appropriately revised to include any changes
ternatives. The Assumptions Document should include a that arose during the course of the project, as well as, of
description of expected model variations, including the course, the key results and recommendations of the study.
range of each model input parameter, to be simulated to
represent the alternatives of interest. 7 MANAGING A SIMULATION PROJECT
In practice, initial model experiments often raise new
questions and may change the direction of the study after When managing a simulation study through its various
initial experiments are run and analyzed. For example, ini- phases, a good manager should follow these guidelines and
tial experimentation may establish that a proposed new de- watch for a number of potential pitfalls.
sign or set of operating rules leads to major bottlenecks or
other problems, and some major re-thinking of system de- • Have clearly defined and achievable goals. Keep
sign is required. At the very least, initial experimentation the goals in mind during the whole project.
may change the direction of subsequent experiments.
• Allocate adequate resources. Be sure that key
In each phase of the experimentation, actual model
personnel have proper skills.
configurations should be guided by an experimental design
• Get upper management support and buy-in.
that lays out the model parameters being varied, the range
of each parameter, and the parameter combinations that • Have periodic review meetings with all key peo-
make sense. ple present. Keep communications open.
• Don’t be afraid to ask obvious or “stupid” ques-
6.4.3 Analysis tions. It’s always better to confirm than to assume.
• Assume nothing. Confirm everything.
Analysis is based on the agreed-upon measures of system • Develop a common understanding on project
performance. Typically in manufacturing and logistics ap- scope and goals, questions to be addressed, and
plications, there are measures of throughput, resource utili- just as importantly, questions not to be addressed.
zation, queuing and bottlenecks. • Documents assumptions and all changes to as-
It often happens that initial experiments produce out- sumptions.
puts that identify a problem, or symptoms of a problem,
but do not readily provide the causes of the problem or Potential pitfalls and causes of project failure include:
provide enough information to give insight into the nature
of the problem. Such insight is critical if the team is to de- • Scope creep,
velop suggestions for design or operating improvements • Time slippage/project overrun,
that have a good chance of solving the identified problem. • Too much detail,
15
 Carson

• Wrong skill sets, Law, A. M. and W. D. Kelton. 2000. Simulation Model-

• Key people showing up for the first time at the fi- ing and Analysis, 3rd Ed. New York: McGraw-Hill.
nal presentation and asking questions that have Musselman, K. J. 1994. Guidelines for Simulation Project
not been addressed. Success. In Proceedings of the 1994 Winter Simulation
Conference, ed. J. D. Tew, S. Manivannan, D. A.
The pitfalls can be avoided by following the guidelines Sadowski, and A. F. Seila, 88-95. Piscataway, New Jer-
presented here and in more detail in Musselman (1994). sey: Institute of Electrical and Electronics Engineers.

8 FURTHER STUDY AUTHOR BIOGRAPHY

Banks (1998) provides a comprehensive, up-to-date over- JOHN S. CARSON II is the Consulting Technical Man-
view of simulation. Standard textbooks that are not soft- ager for the AutoMod Product Group of Brooks Automa-
ware-specific include Banks et al. (2000) and Law and tion, Software Solutions Group. He has over 29 years ex-
Kelton (2000). perience in simulation in a wide range of application areas,
There are numerous texts and references for software- including manufacturing, material handling, warehousing
specific information and education. See, for example, the and distribution, transportation, ports and shipping, and
software track in this and past year’s proceedings of the medical and health care systems. With the AutoMod
Winter Simulation Conference. And finally, the introduc- Group for 10 years, he was previously President and foun-
tory tutorials track of each WSC provides a continuing der of Carson/Banks & Associates, an independent simula-
source of good information. tion consulting firm. Before that, he taught simulation and
operations research at Georgia Tech and was an independ-
ACKNOWLEDGMENTS ent consultant. He also has taught at the University of
Florida and the University of Wisconsin. He is the co-
The author has benefited from many papers and presenta- author of two university level textbooks including the
tions in previous proceedings of this conference, especially widely used Discrete-Event Systems Simulation (fourth
the papers in the introductory tutorial track. In addition, edition, 2004). He holds a Ph.D. in Industrial Engineering
the author has benefited from many discussions with Jerry and Operations Research from the University of Wiscon-
Banks on these and related topics. This paper is based on a sin-Madison, and is a member of INFORMS. He can be
paper of the same title in the Proceedings of the 2003 Win- reached at <john.carson@brooks.com>.
ter Simulation Conference (Carson 2003), with revisions
and additions.

REFERENCES

Banks, J., ed. 1998. Handbook of Simulation: Principles,

Methodology, Advances, Applications, and Practice.
New York: John Wiley.
Banks, J., J. S. Carson II, B. L. Nelson, and D. M. Nicol.
2000. Discrete-Event System Simulation, 3rd Ed. Up-
per Saddle River, New Jersey: Prentice-Hall.
Carson, J. S. 1993. Modeling and Simulation World Views.
In Proceedings of the 1993 Winter Simulation Con-
ference, ed. G. W. Evans, M. Mollaghasemi, E. C. Rus-
sell, and W. E. Biles, 18-23. Piscataway, New Jersey:
Institute of Electrical and Electronics Engineers.
Carson, J. S. 2002. Model Verification and Validation. In
Proceedings of the 2002 Winter Simulation Confer-
ence, ed. E. Yücesan, C.-H. Chen, J. L. Snowdon, and
J. M. Charnes, 52-58. Piscataway, New Jersey: Insti-
tute of Electrical and Electronics Engineers.
Carson, J. S. 2003. Modeling and Simulation. In Proceed-
ings of the 2003 Winter Simulation Conference, ed. S.
Chick, P. J. Sánchez, D. Ferrin, and D. J. Morrice, 7-
13. Piscataway, New Jersey: Institute of Electrical and
Electronics Engineers.

16