A RT I F I C I A L

INTELLIGENCE
A N D S I M U L AT I O N
IN BUSINESS
Dr. Arash Mahdavi Tyler Wolfe-Adam

W H I T E PA P E R
CONTENTS

About authors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .01

Introduction ...............................................................................................................................................02

01 G ener al P u rp o se S i m u l at i o n To o l s a n d An y L o g i c .............................................0 4

02 How Are S i m u lat i o n an d AI B e i n g U s e d To g e t h e r ? ..........................................08

03 Deep Re i n fo rce m e n t Le a r n i n g i n Ac t i o n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 5

04 Example Model: Traffic Light Timing – AI vs Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 8

05 Case Study: Core Movement Resolution –

an Industrial Problem Resolved by AI and Simulation ...................................................... 21

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Additional resources ............................................................................................................................... 2 5

 ABOUT AUTHORS

ABOUT AUTHORS

Dr. Arash Mahdavi Tyler Wolfe-Adam

Simulation Modeling Expert, Program Support Specialist,
AnyLogic-AI Integration Lead AnyLogic-AI Integration Specialist

Dr. Arash Mahdavi is a simulation modeling Tyler Wolfe-Adam works at The AnyLogic
expert, head of training at The AnyLogic Company in North America as a program
Company in North America, and AnyLog- support and technical specialist. He holds
ic-AI integration lead. He holds a PhD a degree in computer science from De-
degree in civil engineering from Purdue Paul University. He is an active member
University where he applied a system-of-systems approach and agent- and core contributor to the Anylogic-AI
based modeling to profitability analysis of construction companies. Dr. initiative.
Mahdavi recently authored simulation textbook “The Art of Process-Centric
Modeling”. He has trained hundreds of professionals and faculty members
from Fortune 100 companies and elite research universities.

“ This whitepaper is about using simulation and artificial intelligence together. It shows how their
combination benefits practitioners in both fields and how AnyLogic simulation software enables
the joint use of these technologies in business today. From reading this paper and the case stud-
ies it contains, you will develop a deeper understanding of how the technologies work, the three
ways in which they intersect, and why they are powerful when used together.

01
 INTRODUCTION

I N T RO D U C T I O N
Artificial intelligence, machine learning, and deep learning are common S U P E RV I S E D
terms that you can often hear in everyday conversation. Familiar as they
are, before looking at the value of their intersection with simulation, it is In supervised learning, labeled training data is used – inputs are
worth highlighting some important details. provided to an algorithm alongside known solutions. The labeled
training data provides examples for algorithms to learn from. And, using
the labeled pairs of input and output data, an algorithm will learn how to
The Encyclopedia Britannica simply defines artificial intelligence (AI)
map any given input to an output.
as ‘the ability of a computer… to perform tasks commonly associated After training, an algorithm can be used to classify inputs, such as whether
with intelligent beings’. This is a broad definition for a whole email is email, or predict values, such as stock prices.
discipline of terms and methods that aim to make computers behave
intelligently. U N S U P E RV I S E D

Algorithms in unsupervised learning are not given any solutions.

Machine learning (ML) is a subset of artificial intelligence and a Instead, with only unlabeled data for input, they must attempt
field of study directed at enabling computers to develop their own to find hidden structures and patterns. This approach is mainly used for
abilities. A system using ML will have specific algorithms for identifying clustering, such as grouping people based on their purchase history, and
patterns in data, patterns which can subsequently be used for making association, where preferences and tendencies are identified. The results,
predictions and decisions. for example, could be personalized supermarket discounts sent to your
phone, or a tailored homepage for a streaming service.
For ML, there are three basic types of learning paradigms – supervised,
unsupervised, and reinforcement: R E I N FO RC E M E N T
In reinforcement learning, software agents are placed in an
environment and seek to maximize the cumulative reward for
their actions. The significant difference in this approach compared to the
previous two is that no training data is needed, the training data comes
from the cycle of observation, action, and reward. This type of ML can be
applied in areas such as game playing and for control problems.

02
 INTRODUCTION

A limitation of ML paradigms is their need for handcrafted input and AN ARTIFICIAL NEURAL NETWORK
assumptions about what is important. This handcrafted, manual, element
is difficult to scale and is reliant on human intelligence for guidance and
informative feature extraction. There is, however, another way and, as an
advancing subset of ML, deep learning is dealing with these limitations.

Deep learning is used to solve the same types of problems as the other
three ML paradigms but differentiates itself by discovering important
features for itself and learning directly from raw data, without the need for
human intervention.

Deep learning uses artificial neural networks (ANN) – a system loosely

based on how the brain works.

A simple ANN consists of a layer of input nodes which are connected

through multiple hidden layers to a layer of output nodes. By using
multiple layers, each intermediary layer can represent a different layer
of abstraction and contribute to learning by producing a hierarchy of
concepts. In areas where manually coding features is not practical, such
as dictating which pixels are important to an object-identifying program,
deep learning excels.

03
GENERAL
PURPOSE
S I M U L AT I O N
TO O L S A N D
A N Y LO G I C
 GENERAL PURPOSE SIMULATION TOOLS AND ANYLOGIC

GENERAL PURPOSE
S I M U L AT I O N TO O L S A N D A N Y LO G I C
Classical modeling tools such as Microsoft
Excel allow analytical solutions to be
calculated using formulas and scripts.
Going further, more sophisticated analytical
tools are also available, such as linear or
integer programming for optimization.

Where analytical tools fall short

is their low ceiling of complexity
before problems become too difficult
to solve, particularly in dynamic,
nonlinear, and stochastic systems.
For these situations, simulation tools
can be used to capture and define the
behavior of a system.

The logic of a simulation model is a set of rules that tells the simulation Experiments can be performed to explore possibilities by varying the
engine how to get from the current state of the system to the next. As the inputs, running the same model multiple times to see the effect of
model progresses in virtual time, various trajectories of the system can be randomness on the outputs, or a combination of both. This dynamic
observed, and output statistics collected. approach ultimately allows a more detailed analysis of a system, and for
problems to be solved, in ways that other tools cannot provide.

05
 GENERAL PURPOSE SIMULATION TOOLS AND ANYLOGIC

There are three primary simulation methodologies for modeling a real-world system – system dynamics, discrete event, or agent-based:

SYSTEM DYNAMICS DISCRETE EVENT AGENT-BASED

System dynamics operates at a high abstraction Discrete event simulation models a system at a Agent-based modeling can work at any
level (less detail), in a typically deterministic medium-low abstraction level and focuses on abstraction level and is made up of subsystems
environment, and with a continuous modeling systems in which a sequence of operations or (local models) that may work independently or
of time. These models use stock-and-flow tasks need to be performed. This type of mod- collaboratively. Behaviors are typically described
diagrams to capture causal relationships, elling is built using generic blocks that control through states and through communication
feedback loops, and non-linear behaviors of the flow of entities in different contexts (e.g., between agents. Both discrete event and agent-
complex systems over time. patients in a hospital, cars in a manufacturing based models tend to be stochastic but can also
facility, packages in a warehouse). be deterministic.

06
 GENERAL PURPOSE SIMULATION TOOLS AND ANYLOGIC

How does all this relate to AnyLogic simulation software?

While most simulation tools specialize in one simulation methodology,

AnyLogic supports them all.

AnyLogic supports using GIS maps, provides database connectivity,

and has specific vertical libraries for industry areas such as material
handling, fluid, pedestrian, road traffic, and rail – all these features give
users a flexible environment for elegantly and efficiently modeling the
structure, behavior, and rules of a real system.

Models are built using a drag-and-drop interface, greatly simplifying the

creation process. The visual interface is superior to coding the system
from scratch, helping to better structure models. Coding specifics are
not lost, however. AnyLogic is written in Java and has a rich API, so the
flexibility of programming is not abstracted away from the user. Any further
customization to a model can be made by writing custom code or importing
and making use of external Java libraries. No matter how complicated
the system being modeled is, an elegant, flexible model can be built
with no workarounds.

07
H OW A R E
S I M U L AT I O N
AND AI
BEING USED
TO G E T H E R ?
 HOW ARE SIMULATION AND AI BEING USED TOGETHER?

H OW A R E S I M U L AT I O N A N D A I B E I N G
U S E D TO G E T H E R ?
There exists immediate potential for AI and simulation to further develop their mutually beneficial relationship, specifically in three key areas:

Synthetic data – simulation Testbed – for examining AI Learning environment – for RL

generated data for training AI behavior in new situations, not when the real world is too costly,
just on historical data dangerous or unavailable

Before examining these areas, it’s important to first understand how labeled data that is representative of a real system’s outputs; second,
companies and organizations currently benefit from simulation. that a simulation model can be used as a virtual environment to test the
implications of incorporating AI into existing systems.
In the process of building a mature simulation model to represent a real-life
system, it is necessary to perform verification and validation. Verification is The third area comes from how useful information is extracted from
the iterative process of running a model and checking for any errors where a simulation model. A business or organization that has developed a
the programmed logic does not match assumptions and specifications, then simulation model will use it to run experiments and learn from the virtual
going back to fix those issues. Validation is the subsequent process that representation of their real system in a risk-free way. These experiments –
checks the model representativeness of the real or proposed system. Once a such as parameter variation, Monte Carlo, and optimization – facilitate the
simulation model has been sufficiently verified and validated, the outputs of decision-making process and help analysis and strategy development. One
simulation and the real world parallel each other. shortcoming of these types of experiments is that they have limitations
when extracting useful strategies from systems with high stochasticity
There are two implications to this that line up with the first two key areas or with many decision variables. These experiments cannot automatically
of where AI and simulation intersect: the first is that simulation models can suggest adaptive dynamic policies and rely on human experts to produce
be used as a source to generate an unlimited amount of synthetic, clean, potential policies, algorithms, or heuristics for testing in simulation.

09
 HOW ARE SIMULATION AND AI BEING USED TOGETHER?

As simulation models are dynamic tools, they may reasonably be expected

to propose dynamic solutions – this is the final key area where AI and
simulation intersect: a simulation model can be used as an environment
for a neural network based learning agent to train in. A trained policy is like
a human-curated algorithm or heuristics that adaptively suggest optimal
decisions for different situations over time. To test the learned policies before
putting them into live production, they can be placed back into a simulation
and rigorously examined.

1. SYNTHETIC DATA

Lack of data might be the biggest Achilles heel of machine learning –

almost all machine learning algorithms heavily rely on input data being
provided in both quantity and quality. Data collection is a significant
task that needs many expertly designed, and often costly, processes to
be put into place. Even when the process can be streamlined, there is no
guarantee that sufficient data can be gathered due to the hard constraints the behavior of a system or the influence of interacting parts; they only
of monitoring a real-world system (e.g., one day of data takes one day to generate new data by inference from historical data. Unlike statistical or
produce). Moreover, machine learning algorithms, especially in supervised mathematical methods, a sufficiently verified and validated simulation
learning, need labeled data. The labeling process usually needs to be model can be used to generate unlimited amounts of relevant, clean,
done by humans, adding to the time, cost, and difficulty of generating structured, and labeled synthetic input data.
the necessary data.
Compared to mathematically generated data, simulation-based synthetic
To combat this, there are methods that can be used to create synthetic data is not entirely constrained to historical data. Simulation models
data from historical data, such as statistical algorithms. However, the main rely on rules and inherently capture the causal relationships in a system.
issue with this approach and similar methods, is that they do not capture Since these models uncover the essentials of how the system works, their

10
 HOW ARE SIMULATION AND AI BEING USED TOGETHER?

outputs may also uncover counterfactuals and scenarios that have never
been experienced. This ability gives simulation unique capabilities for
generating synthetic data in terms of comprehensiveness, accuracy, and
representativeness.

The data generated from a simulation model is almost entirely free of

measurement bias. This is true because every aspect of a virtualized
system is controlled, and all the recorded outputs are direct readings of
values, not estimates or approximated measurements. This feature can
be effectively utilized for testing and fine-tuning learning algorithms
because there is absolute confidence that the input data, and its labels,
are not influenced or diluted by irrelevant external factors or measurement
errors. Moreover, unlike the real-world, simulations give you control over
model conditions from start to finish. Modelers also have control over the
random components of a system because they can set the seed of the
random number generator (RNG). In this way, it is possible to tweak input
parameters and analyze output data knowing that any differences resulted
from an input change and not randomness.

Simulation models can generate an unlimited amount of clean, synthetic

data for all parts of a system.

The model can track all levels of outputs and statistics, ranging from broad
information – such as the yearly profit of a company – down to the most
minute detail – such as a breakdown of the minutes worked by a specific
worker. This level of precision allows the data output by a model to have
a similar level of precision applied to the labels. Since each data point
originates from a specific, known part of a model, both the individual data
points and the results are accurately labeled.

11
 HOW ARE SIMULATION AND AI BEING USED TOGETHER?

2. TESTBED

As artificial intelligence (AI) and machine learning (ML) solutions become

increasingly mainstream, more of these solutions are deployed and
integrated with existing systems. In many cases, data is first gathered
from real world systems and then fed into ML algorithms for training.
Datasets are partitioned, leaving one part for training on and the other
for testing, allowing the performance of the algorithm to be evaluated. If
the performance of a trained model is satisfactory, it can be deployed and
incorporated into a live system.

The objective of adding AI components to a system is to improve the

system’s overall performance and not just that of the components
being substituted by AI. Consequently, testing a trained model on its
own does not verify that the performance of the modified system is
sufficiently improved.

One of the main advantages of simulation is the ability to test changes

to a system in a virtual, risk-free environment; this advantage can also
be applied when testing the effectiveness of an AI component on the
overall performance of a complex system.
implementing the new pre-qualification ML model will benefit overall
For example, imagine a mortgage approval process in a bank. It involves closing rates. This kind of uncertainty means simulation is an ideal way
several consecutive steps such as pre-qualification, application, appraisal, to find the answer. In this specific case, a simulated system incorporating
credit report, underwriting and closing. To streamline the process, the bank the ML model might uncover a bottleneck which has now moved from
has trained and tested a ML model that automates some part of the pre- the pre-qualification stage to the appraisal stage. In addition, when in
qualification process by automatically detecting whether an applicant is the designing phase of any new system that will have AI components,
considered a high-risk applicant based on their personal information. The alternative designs can be simulated to select the option with the best
policy performs better than the current system, but it is uncertain whether overall performance.

12
 HOW ARE SIMULATION AND AI BEING USED TOGETHER?

Simulation can also be used to test polices that were trained in a

virtualized learning environment, as the result of reinforcement learning
schemes – a concept further described in the next section. For these cases,
the policies learned in a simulated environment can be put back to into a
simulation for performance testing.

3. LEARNING ENVIRONMENT

Reinforcement learning is a machine learning paradigm that has shown

great potential in solving problems that were previously deemed
unsolvable for machines.

RL borrows many ideas from how intelligent beings, like humans, learn
from their environment, specifically how experience is gained through
trial and error. The only difference is that a computer program, or learning
agent, does the learning.

The interesting part about the RL process is that no direct transfer of

knowledge to the learning agent takes place; it is only through a reward
system that the agent learns to understand what actions in any given
situation are most beneficial and lead toward a learning objective. a playground. Although the learning agent itself is a computer program,
According to Richard S. Sutton1 (2018), “Reinforcement learning is learning the environment can either be physical and in the real world, or a virtual
what to do — how to map situations to actions — so as to maximize a representation of it. For example, to learn how to fly a drone, a learning
numerical reward signal. The learner is not told which actions to take, but agent can take control of a physical drone and fly it, or it could learn
instead must discover which actions yield the most reward by trying them”. in a computer simulation. Although more realistic and comprehensive,
the real-world environment has several problems. Many drones will
For a learning agent to learn, it needs to interact with an environment or be needed, as the first few batches will likely be destroyed, and the

1 “Reinforcement Learning: An Introduction” Richard S. Sutton, Andrew G. Barto, 2nd Edition, ch. 1, Nov 13, 2018

13
implementation costs will be high.

Additional costs from learning in the real world may be incurred through
injury to people or damage to property. Legal restrictions can also limit
testing and impose further constraints.

In contrast, simulated environments are controllable, reproduceable, and

the direct results of slight changes to situations are visible.

Simulations can also run faster than real-time and in parallel, allowing
quick and concurrent accumulation of knowledge.

Many current simulated environments used for RL algorithms are either

research-oriented toy environments or hyper-specific custom environments
made for a single use case. They are often developed from scratch for
substantial cost and effort.

The creation of these custom environments is mainly due to an assumption

that general-purpose simulation tools are not flexible or comprehensive
enough. And this is not far from the truth, since almost all available
general-purpose simulation tools are intentionally simplified applications
and and restricted as a result. In contrast, AnyLogic is developed with
connecting Python, Java, or any other language that can interface via
scalability, interoperability, connectivity, and flexibility in mind.
REST. This feature specifically is important for the integration of simulation
models into machine learning or deep learning frameworks and their
AnyLogic is a multi-scale, multi-method simulation platform. It can model
development environments.
the entire business lifecycle and is the natural choice as the learning
environment for real-world use-cases. Furthermore, AnyLogic Cloud
provides scalable computational power and the ability to abstract away
from a model’s implementation language. It also exposes a rich API for

14
DEEP
R E I N FO RC E M E N T
LEARNING IN
AC T I O N
 DEEP REINFORCEMENT LEARNING IN ACTION

D E E P R E I N FO RC E M E N T L E A R N I N G
I N AC T I O N
There are many examples in the past half-decade that showcase
the potential of deep reinforcement learning. A significant moment
in the field was the 2016 defeat of Go world champion, Lee Sedol,
by DeepMind’s AlphaGo.

Although the rules are simple, the game complexity of Go make it

formidably difficult and it was seen as the biggest challenge in classical
games for artificial intelligence to master. It is estimated that there are
more valid ways to play the game to conclusion than there are atoms
in the observable universe. Of the five games played, AlphaGo won all
but the fourth game.

“ AlphaGo showed anomalies and moves from

a broader perspective which professional Go
players described as looking like mistakes at the
first sight but an intentional strategy in hindsight

Not content with its board game success, Google’s self-driving research led
to Waymo starting a self-driving taxi service in Phoenix, Arizona, in April
2017. As the first commercial autonomous vehicle hire service, Waymo One
is currently in use by 400 members of the public who pay to be driven to
their schools and workplaces within a 100 square mile area.

16
 DEEP REINFORCEMENT LEARNING IN ACTION

Another popular example features Dota 2 – a multiplayer online battle

arena video game, where two teams of five players both defend their base
and seek to win by destroying a large structure in the opposing team’s
base. It is a highly strategic game that involves significant cooperation
between players in order to win.

In April of 2019, OpenAI Five – a machine learning project composed of five

neural networks - won a best-of-three series against the 2018 international
champions. In the same month, the bots made their public debut, and
anyone could sign up to play against them – from a total of 42,729 games,
the bots won all but 4,075, a win rate of 99.4%.

The training for the AI was entirely based on self-play. The system
started with no experience and was only provided with an incentive
to win. It then played games against itself, experiencing 45,000 years
of play over a period of 10 real months – averaging 250 years of
simulated gameplay per day.

“ We use Dota as a testbed for general-purpose

AI systems which start to capture the messiness
and continuous nature of the real world, such
as teamwork, long time horizons, and hidden
information. Our Dota training system showed
that current AI algorithms can learn long-term
planning with large but achievable scale

17
EXAMPLE MODEL:
TRAFFIC LIGHT
TIMING – AI VS
O P T I M I Z AT I O N
 EXAMPLE MODEL: TRAFFIC LIGHT TIMING – AI VS OPTIMIZATION

EXAMPLE MODEL: TRAFFIC LIGHT TIMING –

AI VS OPTIMIZATION
To showcase the capabilities of a powerful general-purpose simulation
tool as the training environment, AnyLogic, in partnership with Skymind,
developed a simple but illustrative example model based on the simulation
of a traffic light-controlled intersection. The objective of this was to train
an artificial neural network to be able to optimally control a traffic light. It
did this by learning a policy which best controlled the traffic light based on
the current status of the traffic.

The model consisted of a very straight forward intersection: cars start at

beginning of one of four roads and have their destination set to the end
of the opposing road. The single traffic light in the model controls the
right of way in each direction. The time that each car spends in the model
is affected by the traffic level and how efficiently the traffic light’s signal
is controlled. The logic of the traffic and intersection was modeled using
standard flowchart blocks within the AnyLogic Road Traffic Library.

Two Schedule elements were used to determine the arrival rates for
each axis of traffic (one for the north/south, one for the east/west). The
simulation starts at 8:00 AM and finishes 8 hours later at 4:00 PM. The cars
traveling east/west had a consistently high rate throughout the simulation
(500 cars per hour). However, the north/south direction experienced light
traffic in the morning (50 cars per hour) and heavier traffic that peaked
between 2:00 to 3:00 PM (900 cars per hour).

19
 EXAMPLE MODEL: TRAFFIC LIGHT TIMING – AI VS OPTIMIZATION

For this model, the key performance indicator (KPI) was defined as the The model was then advanced another ten seconds. The reward given to
average time a car spends in the system; this was tracked through a the learning agent is a function that was adopted from the work by Arel
Dataset object which was added to whenever cars left the system. As et al.1 (2010) and is a numeric value between -1 to 1. For the observations,
this model made use of reinforcement learning, the observation and the sum is taken for the cars in the forward lanes (moving towards the
actions taken by the learning agent, and the reward structure also center) and for the cars in the intersection. Initially, the reward is set as the
needed to be defined. difference between these two values. It is then normalized by dividing the
maximum of the values.
Every ten seconds, an observation was sent to the learning agent that
consisted of ten values: the first eight described the total amount of time The learning agent was trained on 100 episodes using the cyclical process
spent in the model for all cars within 150 meters of the intersection for each described above and then added back into the model for testing. To give
of the eight lanes; the ninth element was also the total time spent in the a comparative idea of performance, an Optimization Experiment was also
model, but for the cars in the intersection; the tenth element was the index run on the model and output what phase length it determined was best for
of the current traffic light phase. From that information, the learning agent each axis. When using the optimized values, the mean time in system value
then took one of two actions: do nothing and keep the current phase of the was 1.54 minutes. However, using the AI policy in the model, achieved a
traffic light or move to the next phase. mean time in system of 1.03 minutes – a 33% improvement!

The power of replacing static values with a dynamic ANN allows dynamic
adaptation to any given situation. Even for situations never experienced
before, with knowledge from similar situations, a policy can outperform
values determined by black-box optimization experiments.

The model is simple for illustrative purposes and, as a result, the policy it
learned can be outperformed by a human. However,

the beauty of the policy is that no human assistant was involved in the
learning process; the AI learned a meaningful policy on its own, based on
its interaction with the simulation model.

If a more realistic case were setup and trained on, the learning agent would
start to show its advantages over human curated algorithms.

1 I. Arel, C. Liu, T. Urbanik and A. G. Kohls, “Reinforcement learning-based multi-agent system for network traffic signal control,” in IET Intelligent Transport Systems, vol. 4, no. 2, pp. 128-135, June 2010.

20
C A S E ST U DY:
C O R E M OV E M E N T
RESOLUTION – AN
I N D U ST R I A L P RO B L E M
R E S O LV E D BY A I A N D
S I M U L AT I O N
 CASE STUDY: CORE MOVEMENT RESOLUTION – AN INDUSTRIAL PROBLEM RESOLVED BY AI AND SIMULATION

CASE STUDY: CORE MOVEMENT

RESOLUTION – AN INDUSTRIAL PROBLEM
RESOLVED BY AI AND SIMULATION
Engineering Ingegneria Informatica (EII) developed advance. Despite all efforts, bottlenecks and line
A FERROMAGNETIC CORE AT THE
an automated decision system using reinforcement blockages occur and result in costly delays.
END OF THE MANUFACTURING
learning and AnyLogic for Lagor S.R.L. The decision
PROCESS
system was applied to resolve the problem of One possible solution for this problem is to use
movement planning on a factory shop floor. mathematical optimization. However, the scope of the
problem is too large for daily execution of a search
Lagor’s factory produces ferromagnetic cores used for algorithm. Indeed, this is a sequential decision-making
electrical transformers. The production line consists problem for which reinforcement learning is the
of several automated workstations, each connected best approach. All the required heavy computations
by heavy-duty conveyor systems. Each core is highly only need performing during the training stage and
customized to meet the specific requirements of then solutions can be obtained repetitively with
each transformer, resulting in production cycles and relatively no time-cost.
movements within the production line that are neither
unique nor able to be automated. Using a simulation solution that enabled the client
to evaluate their production plan, EII developed
Since at any given time several cores can be under a reinforcement learning-based agent that could
process, a line manager needs to consider each automatically generate the best movement sequence.
individual production cycle and plan movements in

Being highly customizable, AnyLogic allowed the utilization of Java-based machine learning packages and
provided a robust simulation environment in which to train the learning agent.

22
 CASE STUDY: CORE MOVEMENT RESOLUTION – AN INDUSTRIAL PROBLEM RESOLVED BY AI AND SIMULATION

The adopted learning algorithm was a double deep Q network (DDQN), AN EXAMPLE OF A BOTTLENECK AND BLOCKAGE DURING THE
implemented by Skymind in their DL4J package. Since the problem MANUFACTURING OF CORES WITH DIFFERENT PRODUCTION CYCLES
consisted of different tasks, each one was addressed separately, and the
combination of these solutions was used to address the main problem. The
result was a policy that replaced the complicated heuristics, which were
difficult to maintain and debug.

It was observed that the robustness and flexibility of the AnyLogic

simulation platform was critical for the implementation. The learned policy
successfully proved that the application of advanced machine learning
techniques with simulation modelling helps solve complicated tasks when
other solutions are not readily available.

For more information about this project, please see

EII’s 2019 AnyLogic Conference presentation

23
CONCLUSION
We are now at a point where AI is expanding beyond research environments and into the real world. To facilitate this, the power of
simulation is increasingly being used to leverage the promises of AI. Companies working with either technology stand to benefit
from their combination.

With the origins of machine learning starting in the 1950’s, it is not a new spanning thousands of real projects that are deployed across almost all
topic. However, it wasn’t until recently, and the availability of the necessary industries. This will allow an expansion beyond the focus on toy and game
computing power, that the theorized concepts could start to be applied. simulations and into real-world systems.

Simulation practitioners will soon notice a dramatic increase in the demand Regardless of background, both industries can easily collaborate. However,
for their expertise. With real-world systems being too complicated for while both simulation and AI are clearly compatible with one another, there
the constant redevelopment of hand-coded environments, there will be a is still a separation of expertise. Overall, the compatibility of the two fields
significant need for general-purpose simulation modelers. This will provide enables easy collaboration and practitioners can benefit without the need
simulation practitioners with many opportunities in the upcoming boom. for retraining. With AnyLogic’s interoperability and flexibility, it provides a
natural fit for these two industries to come together.
Concurrently, the benefit to the AI practitioners is that they can use
simulation models today; simulation has a rich and mature ecosystem

24
ADDITIONAL White Papers

Case Studies
R E S O U RC E S
Videos

For up-to-date AnyLogic AI resources Books

visit our dedicated page.
Download AnyLogic Simulation Software

Seminars and Training

25
C O N TAC T S
T HE ANY LOG I C COM PAN Y

[email protected]
Contact your local office >>