www.pwc.

com

Integrating Artificial Intelligence

and Simulation Modeling
PwC Artificial Intelligence Accelerator

Fall 2017

PwC | Artificial Intelligence Accelerator | 12/4/2017

PwC Artificial Intelligence Accelerator is working with AnyLogic simulation and
other AI technologies to create a new generation of simulation models

• The pervasive adoption of AI technology in business and society means that our
simulation models must reflect and adopt AI as well

• In the foreseeable future all but the simplest simulation models will incorporate
AI tech either in the model itself or in the development of the model

2
PwC | Artificial Intelligence Accelerator | 12/4/2017
Agenda

• Impact of AI and key AI technologies

• Survey of AI and simulation integrated applications
• Example Problem: Autonomous Vehicle behavior using Deep Reinforcement Learning
• Making it real: Integrating AI directly into AnyLogic simulation

3
PwC | Artificial Intelligence Accelerator | 12/4/2017
Application of AI ranges from automating existing processes to disrupting markets
and business models

AI Game Changing Impact

Disrupting your Core Business Innovating with New Services Redesigning your Business Model
Automate your business processes & augment your decision Innovate with new products and services for your Fundamentally redesign your business model or disrupt
making before other disruptors do it to you customers based on big data, analytics and AI adjacent markets based on your core capabilities
Example: Legal and accounting firms are using robotic and Example: Electronic retailers are using their extensive Example: Auto manufacturers are fundamentally
cognitive process automation, and blockchain to disrupt and transactional and behavioral data of their customers to offer rethinking their business model as ‘Personal Mobility’ service
re-engineer their business processes (e.g., Ross as a them new ways of trying, experiencing and purchasing their providers instead of manufacturers of vehicles to exploit
disruptor, PwC) products (e.g., Amazon with Alexa, Baidu with Chatbots) autonomous cars and car share/ride share trends (e.g., GM
with MAVEN, BMW with DriveNow)

Automating business processes with AI will soon become table-stakes, leaving the
differentiation to those using AI to change the way they interact with customers
4
PwC | Artificial Intelligence Accelerator | 12/4/2017
Artificial Intelligence is not a single technology. Here are five key areas that PwC is focusing
on. We consider simulation to be an AI technology.

Application Areas Success Stories

• Workflow automation • Consumer fraud behavior anomalous activity
Machine • Decision support detection using unsupervised methods
Learning • Anomaly detection • Deals asset reclassification automation support

• Unstructured data analysis for • Audio analysis used to identify sales call features and
Deep audio, image, and text assess traits of effective conversations
Learning • Automation and learning systems • Deep reinforcement learning used to learn optimal
operating policies for a rideshare business
• Information extraction, workflow • Tax workflow automation that could save the client
Natural automation, and text generation ~$65M annually
Language • Insights from unstructured data • Customer contact center complaint analytics platform
Understanding • Chat bots and IPA • Comprehensive chat bot study
• Industrial IoT • Predictive maintenance applying ML on oil/gas pump
• Streaming and real time data sensor streaming data in a scalable manner
Data at Scale • Large scale data processing
architecture
• Operations analysis • Designed a go to market strategy for a new personal
• Strategy support mobility service
Simulation • Consumer behavior modelling • Patent for autonomous vehicle resource management
• New market entry analysis in a rideshare environment

PwC | Artificial Intelligence Accelerator | 12/4/2017 5

Survey of AI and
simulation integrated
applications
Digital supply chain, smart factory, and other industrial processes all incorporate AI technology.
Simulation models of these systems will necessarily include AI components as well.

AI and Simulation Integration

AI Application Objective
Approach

AI components from the real

Near real-time digital twins,
system are embedded in the
• Material routing “what-if” system analysis
simulation model
• Loading plans
• Real-time decision making
based on sensor data
• Real-time “control tower” AI components for the real
system are developed using System design
simulation model

7
PwC | Artificial Intelligence Accelerator | 12/4/2017
Deep learning components can replace rules based models of human behavior and decision
making in new service and strategic simulation models

AI and Simulation Integration

AI Application Objective
Approach

AI components trained in other

Testing of alternative business
• Neural nets that describe environments are embedded in
models
consumer choice the simulation model
• Competitive reaction
• Modelling other human
behaviors
Self generation of data to train Strategy identification and
neural nets development

8
PwC | Artificial Intelligence Accelerator | 12/4/2017
AI techniques can address crippling optimization/calibration problems in large scale agent
based models

• Large-scale ABM simulation models have

– Lots of parameters to calibrate
– Long run times

• Traditional optimization approaches often fail because they take much too long

• Machine learning and intelligent sampling approaches can be used to create meta models
that dramatically speed up calibration
– Meta models run much faster than the full simulation model
– Machine learning makes it possible to capture the non-linear behavior and
discontinuities in agent based models

9
PwC | Artificial Intelligence Accelerator | 12/4/2017
Example Problem: Autonomous
Vehicle behavior using Deep
Reinforcement Learning
A common application for simulation is develop “optimal” decision rules for agents in a
complex system. This is a problem that can be tackled with AI technologies combined with
simulation.

• Many business problems consist of multiple decisions makers either collaborating or

competing towards a particular goal

• PwC is working with a team at a large car company that is looking to roll out autonomous
vehicles for the purpose of transporting customers

• Using a very complex AnyLogic AB/DE model simulated vehicles that follow hand coded
rules to make their decisions

• As an experiment we used “Deep Reinforcement Learning” to train AV to maximize fleet

efficiency while satisfying customer trip demand

11
PwC | Artificial Intelligence Accelerator | 12/4/2017
Reinforcement Learning - an area of machine learning focused on teaching a computer an
optimal decision policy over time using rewards and penalties as signals

Key Takeaways
Simple Physics RL Examples
Cart Pole • The agent’s goal is to select
actions that maximize the
future reward

Pendulum Balance
• Representative model of how
humans learn from experiences

• Lies in between supervised and

unsupervised learning

• We implement Deep
Reinforcement Learning for
more complex problems using
Deep Neural Networks

Mountain Car Hill

12
PwC | Artificial Intelligence Accelerator | 12/4/2017
RL Agent Vs Environment – RL is composed of two entities, the agent making decisions
and the world it is acting in. An Agent (Car) will communicate actions to the world and as a
result it will recieve a reward and a new observation.

RL Agent (Car) Environment (City Map)

Action (e.g., Left, Right)

State/Observation

Reward

13
PwC | Artificial Intelligence Accelerator | 12/4/2017
Deep Q Network – The CNN acts as a value function for a particular state and outputs a Q-
Value for each action. Depending on the action policy, the action that yields the highest value
is chosen by the agent and then performed in the environment.

Environment

New State DQN Attributes

New • Bellman’s Updates: iteratively used to

State train the neural network to represent the
Choose Q function
Action With
Best Q Value
• Prioritized Experience Replay: used to
Q- Value 3 store all experiences in a replay memory
(Action Left)
CNN in order of importance and it s sampled in
minibatches during training

• Exploration–Exploitation: є-greedy
Q- Value 2
(Action Up)
exploration is used where with
State
probability є choose a random action,
otherwise go with “greedy action

State • Target Networks: Training is stabilized

Q- Value 1
(Action Stay) with a second network that is only
updated with the weights every X training
period

14
PwC | Artificial Intelligence Accelerator | 12/4/2017
Making it real: Integrating AI directly
into AnyLogic simulation
Specialized AI toolsets are fine for developing specific components but to model
entire systems we need to integrate AI directly into AnyLogic

Python is well known as the prominent language for data science toolkits. However, we needed a java-based library to achieve
our goal of integrating AI into an AnyLogic simulation.

We looked for a library that met these requirements:

• Java based to allow for integration with AnyLogic
• Well supported by an engaged community to keep up with AI trends
• Designed for scale as our applications are likely to be for big businesses
• Interfaces with a variety of data stores as is the likely in a commercial scenario

With these requirements, we landed on Deeplearning4j (dl4j). Deeplearning4j is a commercial grade library designed for data
scientists in business environments. Whilst it is always recommended to research libraries for each use case, here are some dl4j
highlights:
• JVM-based distributed deep learning framework
• Dl4j leverages ND4J for data management which is also distributed
• Integrates with Hadoop, Spark and Kafka
• Strong documentation and community

16
PwC | Artificial Intelligence Accelerator | 12/4/2017
The deep learning model is constructed externally and may be trained before
integrating into AnyLogic

Build AnyLogic Simulation

1a 2 3 4 5 6
Import NN to Run AnyLogic
Develop neural network Train neural Export NN Train NN inside simulation
AnyLogic as a simulation with agent
(NN) in Eclipse with dl4j network in dl4j (.jar file) model
dependency behavior driven by NN

1b
Build simulation model in
AnyLogic Simulation data
Training data

External source of data

Optional Steps
17
PwC | Artificial Intelligence Accelerator | 12/4/2017
To explore this process we modified the “Schelling Segregation model” to train and
utilize a simple neural net that replaces the rules based agent decision

Schelling Segregation model

Agents have a minimum threshold for ‘like-minded’ neighbors and are placed in a random location.

Rule-based simulation Neural-network driven simulation

Agents evaluate whether there are The neural network is trained on ‘historical
enough like-minded neighbors each movement data’ i.e. data generated from the
time they move or a neighbor changes. rule-based simulation
Agents can also move randomly for no
reason

If there are enough like- If there are not enough Given the number of like-minded individuals
minded agents, they stay in like-minded agents, they and threshold, the neural network makes a
their current location move to a random location decision to move or not move

Both models converge but in the neural-network driven simulation, we arrive at the
result without prior knowledge of the relationship between threshold and like-
minded neighbor count

18
PwC | Artificial Intelligence Accelerator | 12/4/2017
AI can also be embedded into AnyLogic and used in replacement of manually
encoded behaviors or processes

Objective: Demonstrate direct integration Example model: Classifying priority customers

of AI and AnyLogic
AnyLogic Model Dependencies
Direct integration means we are able to:
• Instantiate and configure a neural network in
the AnyLogic environment
• Train the neural network with data during Customer Neural Network
runtime agent has library
• Replace manually coded rules that govern agent properties that
behavior with the neural network
change over
In our example model: time
• A neural network is trained on historical
Historical
customer data that contains some Classifies customer
characteristics of the customers and whether or customer based dataset
not they are a priority customer on historical (synthetic)

• In replacement of business rules to approximate data

priority status, the neural network detects
trends in the data and classifies each customer
19
PwC | Artificial Intelligence Accelerator | 12/4/2017
Concluding Thoughts

• Pervasive AI in the real world means that we will have pervasive AI in our simulated world
as well

• AI will not only appear in our models but will help us build them as well

• AnyLogic’s architecture makes it practical to integrate AI components directly into

simulations

20
PwC | Artificial Intelligence Accelerator | 12/4/2017
Thank you

Lyle Wallis Mark Paich

Analytics Director, PwC Analytics Director, PwC
Tel: +1-720-931-7404 Tel: +1-719-332-2591
Email: [email protected] Email: [email protected]

21
PwC | Artificial Intelligence Accelerator | 12/4/2017