Organized Complexity Talking to reality Calibrating the model Estimating the model

Calibrating and Estimating Agent-based Models

Vipin P. Veetil†

† Assistant Professor, Indian Institute of Management Kozhikode

LinkedIn ABM Network

Group 9588199

www.vipinveetil.com

1 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Simple Systems

◮ Simple systems involve stable relations between two or three variables

◮ Boyle's Law: Pressure is inversely proportional to volume
◮ Gravitational Constant
◮ Maxwell's equations of electromagnetism

◮ Much of 19th century science was about analysing simple systems

◮ Understanding of simple systems along with engineering created many marvels

like the internal combustion engine

2 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Systems with Disorganized Complexity

◮ Simple Systems involve one or two variables in stable relationships

◮ Galileo's experiment of a ball falling to earth from a tower

◮ Disorganized Complexity involves large number of variables interacting in

complicated ways
◮ A dozen balls colliding on a billiard's table

◮ Examples of Disorganized Complexity

◮ Calls into a telephone exchange
◮ Life insurance company
◮ Motion of atoms
◮ Law of hereditery

3 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Studying Disorganized Complexity

◮ In the 20th century developed statistical techniques to caricature Systems of

Disorganized Complexity

◮ Statistics allow us to:

◮ Say something about average rate of number of collisions between balls on a billiard
table without tracing the path of each ball
◮ Say something about the probability of death and thereby price insurance instruments
without knowing the life histories, stories, and the contributory causes of death of each
individual
◮ Say something about collisions between atoms and galaxies without knowing the each
trajectory of each atom or each galaxies

4 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Systems of Organized Complexity

◮ Large number of variables that are "organized" in someway

◮ System-level outcomes emerge from "structured" interactions, not helter-skelter

collisions

◮ Economic and Social System

◮ Why are there recessions?
◮ Why do some firms succeed, others fail?
◮ What causes wars between nations?

◮ Biological System
◮ What causes schizophrenia?
◮ How genotypes generate phenotypes?

5 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Studying Organized Complexity

◮ We cannot study them by tracing the path of each individual variable/agent

◮ Cannot use techniques of classical physics

◮ Nor can be assume that variables/agents interact without "structure"

◮ Cannot use techniques of statistical physics

◮ Agent-based modeling is one of the useful techniques

◮ Define model at the level of individual agents
◮ Define their interactions
◮ Study the dynamics of the resulting system by running it forward in time

6 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Calibrating models of Organized Complexity

◮ Most problems of calibrating agent-based models arise from the fact that they are
models of Organized Complexity

◮ Many output variables, so no single measure of "goodness of fit"

◮ Many input variables related to each other in complicated ways, so models can be
sensitive to changes in parts (LLN does not work)

◮ Large number of parameters unlike Simple System and Disorganized Complexity

◮ In Simple System, few parameters because few variables at work
◮ In Disorganized Complexity, few parameters though large number of variables because
each variable is just some "original/primordial" variable plus noise

◮ System of agents following simple rules can exhibit complex behavior

7 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Four Levels of Calibration

◮ Level 0: Model is caricature of reality

◮ Level 1: Model is in qualitative agreement with empirical macro structures

◮ Level 2: Model is in quantitative agreement with empirical macro structures

◮ Level 3: Model is in quantitative agreement with empirical macro and micro

structures

8 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Two models

◮ Foraging by ants
◮ There are multiple sources of food
◮ Each individual ant creates pheromone trail upon finding food
◮ Parameters include release rate and evaporation rate of pheromones
◮ Can we reproduce the empirically observed behavior of ants' foraging?

◮ Axtell's model of firm dynamics

◮ Individual workers form firms
◮ Each worker is paid: individual's productivity + average productivity
◮ As firm grows bigger difficult to observe "individual's productivity", incentive to shrik
◮ More productive workers leave to form new firms
◮ Can we reproduce empirically observed dynamics of firm sizes?

9 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Level 0: Model is caricature of reality

◮ Ant's foraging
◮ Visualization - pheromone trails are created, ants move about exploiting sources of food

◮ Axtell's firms
◮ Creation of firms, death of firms, with some firms living longer and growing larger than
others

10 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Level 1: Model is in qualitative agreement with empirical macro structures

◮ Ant's foraging
◮ Pheromone trails are created, not all food sources are exploited simultaneously, i.e. ants
violate the optimality principle of equating margins

◮ Axtell's firms
◮ Creation of some very large and some very small firms: Wallmart has 1 million workers,
1 million firms with 1 worker

11 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Level 2: Model is in quantitative agreement with empirical macro structures

◮ Ant's foraging
◮ Rate of moving from one food source to another matches the empirical observed rate

◮ Axtell's firms
◮ Firm size distribution follows a powerlaw

12 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Level 3: Model is in quantitative agreement with empirical macro and

micro structures

◮ Ant's foraging
◮ Rate of moving from one food source to another and the rate of excreting pheromones
are both the empirically observed rates

◮ Axtell's firms
◮ Firm size distribution follows a powerlaw and rate of difficulty in observing workers'
efforts is the empirically observed rate

13 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Four types of parameters of the model

◮ Type 0 Parameters: direct correspondence with micro attributes

◮ Rate of release of pheromones by ants

◮ Type 1 Parameters: near-direct correspondence with micro attributes

◮ Buyer-seller network between firms (i.e. whether a link exists or not)

◮ Type 2 Parameters: loose and presumed correspondence with micro attributes

◮ Exponents of Cobb-Douglas production function of firms
◮ Weight of old price in forming new price (price stickiness)

◮ Type 3 Parameters: no correspondence with micro attributes

◮ Parameters that are used to make the model "work"
◮ Sequence in which events unfold in the model
◮ Which agents move first
◮ How much wealth do agents begin the model with?

14 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Our approach

◮ Estimate as many of the parameters as possible from real world micro data

◮ For the remaining parameters:

◮ Randomize to test for sensitivity
◮ Using macro outcomes to estimate "workable" ranges

15 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Type 0 Parameters: direct correspondence with micro attributes

◮ Measure the variable in the real world

◮ Set the measured quantity with the model, along with some noise reflecting the
measurement error

◮ In case the variable follows a distribution, set the moments of the distribution

◮ Model must be run many times to see the establish the variation in macro
outcomes that emerge from noise in deterministic variables and distributional
variables

16 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Type 1 Parameters: near-direct correspondence with micro attributes

◮ Establish why "near" direct and not direct

◮ Measure the variable in the real world

◮ Bridge the gap between variable in the model and in the real world
◮ Firm buyer-seller connection: cutoff for size of transaction, cutoff for frequency of
transaction

◮ Understand the difficulties involved in using real world measure in the model

17 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Type 2 Parameters: loose and presumed correspondence with micro

attributes

◮ Note that the parameter may be the outcome of a "model" of micro behavior

◮ Estimate the parameter using the model of micro behavior using traditional
methods like GMM

◮ Use the estimated parameter within the model with an understand that
robustness needs to be tested by:
◮ Introducing noise in the parameter
◮ Estimating other parameters for other models of micro behavior

18 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Type 3 Parameters: no correspondence with micro attributes

◮ There is not much we can do here

◮ We can test for robustness by varying the parameter

19 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Calibration versus estimation

◮ Calibration involves getting model parameters from real world micro events

◮ Estimation involves computing model parameters from the distance between

model macro and real world macro

20 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Estimation

◮ Suppose the model has one Type 2 parameter

◮ Real world micro events do not tell us much about the parameter

◮ Explore the parameter space to see which regions generate macro outcomes that
correspond to reality

21 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Technically

◮ Typically not possible to explore full parameter space

◮ Intelligent honning in:

◮ Define distance between model output and observed macro
◮ Move parameter a little, if the distance decline keep moving in same direction

◮ The are many sophisticated way of intelligent search

22 / 23
Organized Complexity Talking to reality Calibrating the model Estimating the model

Problems of Estimation

◮ Overfitting
◮ Model does very well in reproducing the past at the cost of how well it predicts the future

◮ Underfitting
◮ Model can do better at fitting the past and predicting the future

23 / 23