Advances in Agent-Based Modeling

In: Agent-Based Models

By: Nigel Gilbert

Pub. Date: 2011
Access Date: February 24, 2019
Publishing Company: SAGE Publications, Inc.
City: Thousand Oaks
Print ISBN: 9781412949644
Online ISBN: 9781412983259
DOI: http://dx.doi.org/10.4135/9781412983259
Print pages: 69-74
© 2008 SAGE Publications, Inc. All Rights Reserved.
This PDF has been generated from SAGE Research
Methods. Please note that the pagination of the online version
will vary from the pagination of the print book.
Advances in Agent-Based Modeling

Agent-based modeling is a rapidly advancing field, and new

approaches are being introduced all the time. In this chapter, we
shall briefly describe some of the directions of current research.

5.1 Geographical Information Systems

Most agent-based models involve agents moving over a

landscape, but usually this terrain is a rectangular plane or a
toroid. In place of these abstract surfaces, newer simulation
environments are offering the possibility of creating complex
artificial surfaces, or incorporating terrains mapped from real
landscapes. They do this by integrating a geographical
information system (GIS) into the model. A GIS is a software
system for managing, storing, and displaying spatial data such
as maps (Chang, 2004; Heywood, Cornelius, & Carver, 2002).
GIS study is a research specialty in its own right, and this is
one of the reasons why using a GIS for agent-based modeling
has been slow to develop: GISs have their own software
technologies and techniques that have been somewhat difficult
to merge with the tools of agent-based modeling.

GISs store spatial data in specially built or adapted “spatially

aware” database systems. These are designed to be efficient
at answering the kind of queries that one needs for managing
geographical data, such as “tell me about all the other objects
that are no more than 10 units away from this object.” GIS data
are often arranged in layers, containing data on one or a few
variables. When displaying or manipulating a map, one can turn
some layers on or off, to see just the variables in the visible
layers. For example, a map might include roads and lakes in
separate layers. If one wanted to see the road network, but was
not interested in the lakes, the lake layer could be turned off.
There are two kinds of GIS variables: raster and vector. Rasters
are regular grids of cells, each cell storing a single value. Raster
variables are used for data that vary continuously over the
surface, such as height and hours of sunshine. Vector variables
are used for data that are in the form of points, lines, or areas
(known in GISs as polygons). Spatial data may be displayed
using a projection, which is a way of showing the surface of a
three-dimensional body such as the earth on a two-dimensional
map. There are a wide variety of projections available, although
the question of which projection to use is more likely to trouble
a geographer wanting to map the whole earth than a modeler
who wants to map just a town or region, small enough that the
differences between alternative projections become irrelevant.
With a GIS, it is possible to build an agent-based model in which
the agents traverse a more realistic landscape, for example,
moving only along city streets. This is vital for projects such as
modeling traffic flows (Raney, Cetin, Vollmy, & Nagel, 2002) and
the spread of epidemics (Dibble & Feldman, 2004; Dunham,
2005), although it may be a distraction if the intention is to
model an abstract process not specifically located in any
particular place. One decision that needs to be made early is
whether it is sufficient for the GIS landscape to be unchanging
throughout the simulation run, or whether the landscape needs
to change dynamically. For example, a static representation of
a city is likely to be sufficient for a traffic model because in
the time period represented by the simulation (a few hours or
days), the arrangement of city streets will not change materially.
On the other hand, a model of the effects of a hurricane on a
city population may need to have its topography updated during
the simulation to account for floods, closed streets, and storm
damage. However, managing time-varying data is at the state of
the art for GISs and a challenge if one wants to integrate it with
agent-based modeling.

A second issue that needs to be considered when designing

spatially aware agents is how they will detect the features of the
terrain they are traversing. For example, if one wants to ensure
that pedestrians walk down streets, and not through buildings,
there needs to be a way for agents to determine that the way
ahead is a walkway. This is best done by sending a query to the
GIS to determine the locations of the objects ahead of the agent
and then decomposing (“parsing”) the returned answer to check
that forward movement is not obstructed. In principle, this is not
complicated, but in practice, it can take a lot of processing by
both the underlying GIS and the agents and may slow down the
model. These are problems with which the current generation
of model builders is struggling, but over the next few years,
they will be solved, and spatially aware agent-based modeling
frameworks will become more common and easier to use.

5.2 Learning

The agents in most of the models mentioned so far are not

capable of learning from experience. Simple agents driven by
production rule systems have a memory in which their current
and past state is recorded, so that they “learn” about the state
of the environment as they proceed through it. But usually we
mean something much more than this by learning. One kind
of learning is learning more effective rules, and then changing
behavior as a result of the learning. In an earlier section, we
touched briefly on one technique that allows this kind of
learning; although neural networks (see Section 2.1.3) are not
based on explicit rules, the learning that they perform can be
akin to rule learning. There are two other common techniques
for modeling learning in agents: reinforcement learning and
evolutionary computation. Both have been transferred from
artificial intelligence, where the development of learning
methods is an important topic under the heading of machine
learning.

5.2.1 Reinforcement Learning

Reinforcementlearning(RL)is akin to what we do when exploring

a new city to find a tourist attraction by trial and error: You try
one street, and if it looks promising you carry on, or if not, you
double back to try another. Agents that engage in RL have a
state (e.g., their current location) and a small set of possible
actions (e.g., move one cell north, east, south, or west). The
environment is set up so that it will eventually provide a reward
(gratification in reaching the attraction). The agent has to find
a policy that maximizes the reward it receives (Sutton & Barto,
1998). Usually, each additional step along the way to the goal
reduces the eventual reward and so the goal is not just to find
the reward, but also to do so in the most efficient way. The agent
can try many times to find the best policy, exploring different
routes, and so one of the issues in designing RL algorithms is
the balance between exploration (trying new routes, which may
be less efficient and waste time) and exploitation (keeping to
the old routes, which may not be the best way). RL has been
used to model firms that learn about the preferences of their
customers (Sallans, Pfister, Karatzoglou, & Dorffner, 2003) and
decision making in organizations (Sun & Naveh, 2004), among
other phenomena. Several authors have used simple models
of reinforcement learning to explain and predict behavior in
the context of experimental game theory with human subjects
(Chen & Tang, 1998; Erev & Roth, 1998; Flache & Macy, 2002;
Macy & Flache, 2002; Mookherjee & Sopher, 1994, 1997).

RL is well suited to problems in which the environment and the

reward structure remain constant during the simulation run. On
the other hand, it does badly if the environment is dynamic (in
particular, if all the agents are learning and the actions of one
agent affect the state or the rewards of other agents, RL is not
likely to be a good technique to use).

5.2.2 Evolutionary Computation

A very different approach to learning is that of evolutionary
computation, a family of techniques of which the simplest and
best known is the genetic algorithm (and others include
evolutionary programming, evolution strategy, genetic
programming, and learning classifier systems). Evolutionary
computation (Eiben & Smith, 2003; Engelbrecht, 2002) is
loosely based on natural selection and involves two basic
processes: selection and reproduction. Fundamental to
evolutionary computation is a population whose members
reproduce to form successive generations of individuals who
inherit the characteristics of their parents. The probability of
successful reproduction is determined by an individual's fitness:
fit individuals are more likely to breed and pass on their
characteristics to their offspring than unfit ones.

The individuals involved in a genetic algorithm (Holland, 1975)

may, but need not, be the agents in the simulation. For example,
in a model of an industrial sector, the agents could be firms,
with the sector as a whole learning how to be productive in
the industrial landscape through successive generations of firm
bankruptcies and start-ups (Marengo, 1992). On the other
hand, each agent may learn using a genetic algorithm that
acts on the agent's rules, each rule being an “individual” as
far as the genetic algorithm is concerned (this is the way that
learning classifier systems work) (Bull, 2004). Simulations of
stock market trading have been built with such agents
(Johnson, 2002; LeBaron, Arthur, & Palmer, 1999).

For a genetic algorithm, each individual must have its features

encoded in a chromosome. The encoding may be in the form
of a binary string, zero representing the absence of a feature,
and one, its presence, or as a more complicated structure. It is
also important that it is possible to assess the fitness of every
individual—for example, the fitness may be measured by the
accumulated capital stock of a firm agent, or the effectiveness
of a cooperation strategy in a model of altruism. Individuals
are selected for reproduction by the algorithm in proportion
to their fitness, so that very fit individuals are very likely to
breed and very unfit ones very unlikely. The reproduction takes
place by combining the parents’ chromosomes using a process
called crossover: Slices are taken from each chromosome and
combined to form a new one made of some sections from
one parent's and some sections from the other's. In addition,
to ensure that there continues to be some variation in the
population even after a great deal of interbreeding, some bits
from the offspring's chromosomes are randomly changed, or
mutated.

The offspring's fitness is evaluated to determine its likelihood

to reproduce to yield grandchildren. Eventually, the individuals
who are relatively unfit die out and are replaced by individuals
who have inherited from fit parents and are likely to be fit
themselves. Although no individual does any learning as a
result of the genetic algorithm, the population as a whole can be
considered to be learning or optimizing as the individuals within
it become fitter.

The genetic algorithm is usually a very effective optimization

device. Whether it is a good model of any social phenomenon is
more debatable (Chattoe, 1998). One difficulty is that it is often
hard to see what would be an appropriate way of measuring
“fitness,” or even that the concept has any clear meaning when
applied to social phenomena. Sometimes, this does not matter.
For example, in a model of a simple society, one does not
need a carefully elaborated definition of fitness: Agents can
be designed to breed if they have the opportunity and have
not previously died from lack of resources. Another common
problem with the basic genetic algorithm is the difficulty of
coding all the salient features of an agent into a binary string
for the chromosome. However, other evolutionary computation
techniques allow a wider range of structures. Genetic
programming (Banzhaf, 1998; Koza, 1992, 1994), for example,
substitutes program code for the binary string: Individuals
evolve a program that can be used to guide their actions, and
learning classifier systems (Bull, 2004; Meyer & Hufschlag,
2006) use condition-action rules as the equivalent of
chromosomes.

5.3 Simulating Language

One of the defining features of multiagent models is that the

agents have the potential to interact. It is this that separates
multiagent models from, for example, microsimulation or
equation-based modeling (see Section 1.4). The interaction
may represent a simple perception of the presence of other
agents, to avoid them or to imitate them, or it may involve
more sophisticated communication of knowledge, opinions, or
beliefs, depending on the requirements of the domain being
simulated. However, even in the most sophisticated models, it
is almost always the case that agent interaction occurs through
unmediated and direct agent-to-agent message transfer. This
is quite different from human communication, in which our
thoughts (and intentions) have to be conveyed through an
external natural language that is inevitably ambiguous and
whose meaning has to be learned, rather than being innate.

As well as being unrealistic, the modeling of information

transmission directly from agent to agent can give a misleading
impression of a wide range of social phenomena, from
cooperation to leadership, all of which change their character
when modeled using error-free and unambiguous messages
that go directly from one agent rule system to another. Partly
to respond to this issue, and partly because how language
evolved is an important topic in linguistics, research on the
“evolution of language” has been increasing in volume and
sophistication over the past decade (Perfors, 2002). Much of
this work is specialized and complex; here we give only a rather
simplified introduction to the topic and some hints about how
language can evolve.

Suppose we have two agents, Adam and Betty, and that they
are standing near to each other in a virtual landscape. Both
can see a small red ball and a large blue cube, as well as a
number of other objects nearby. Adam wants to direct Betty's
attention to the red ball by making an utterance (implemented
by transmitting a string of characters from one agent to the
other). Initially, the agents do not share a language with which
to speak about what they can see. Although Adam could utter
some sequence of syllables, Betty would have no idea what
this sequence meant and no way of knowing that Adam was
mentioning the ball rather than the cube or any of the other
objects they could both see. The problem is even more
profound than this, because initially Adam has no concept of
“ball” or “cube;” all that the agents start with is the ability to
detect features such as color, shape, size, and location.
Everything else has to be learned. The goal is to evolve a
shared lexicon that is, in the words of Hutchins and Hazlehurst
(1995), a “consensus on a set of distinctions” (p. 151).

The following is one algorithm that Adam and Betty could use
(based on the work of Vogt, 2005, and Vogt & Coumans, 2003).
Adam begins by looking at the ball and sees that it is round, red,
and small. These features all distinguish the ball from the cube,
which is square, blue, and large. Adam is able to categorize
it as a “round, red, small” object, and Adam can henceforth
discriminate such balls from square, blue, large objects. If there
were two round, red, small objects in the scene, Adam would
need to find another feature that discriminates the ball from
the other object, for example, the objects’ distance away from
Adam. For Adam, “red” might be a range of shades: It would
not be helpful to have a different category and thus a different
name for every shade of red, so the color space is crudely cut
into segments by the agent's perceptual system, and the same
applies to the perception of other features such as shape and
distance.

In this way, Adam is able to distinguish objects in his view,

assigning each a bundle of categories chosen to differentiate
the object from others previously encountered or currently
present in the scene. The bundle of categories is called a
concept. All of this takes place within the “head” of the agent.
Betty will also have viewed the scene and will also have derived
a concept for the ball, one that is likely to be similar to Adam's
(but because they are standing in different places and so have
different perspectives on the ball, Betty's concept may not be
identical to Adam's). Adam now has to signal “ball” to Betty
using an utterance. Assuming that this is the first time that
Adam has spoken about balls, he invents a word for his
concept—any string of characters will do, provided that it differs
from any other word he knows. Adam “utters” the word and it
is communicated to Betty. However, Betty has no idea what the
word means at this stage.

In order to learn a common vocabulary, Adam and Betty engage

in a language game (Steels, 2004). This involves repeated
dialogues in which a score measuring the associations between
all of Betty's concepts and the word in question are updated
according to whether she judges that she has successfully
understood the utterance. This judgment can be based on some
nonverbal reinforcement (e.g., Adam pointing at the object,
Betty succeeding in an action that depends on understanding
the word, or Adam rewarding Betty) or the similar use of the
word in other contexts, although the latter requires more
inference and is more prone to error. When Betty has selected
a concept to associate with the word (or more precisely, has
associated each concept that she knows with a probability that
that concept is expressed by this word), she can take the role of
speaker. If she wants to communicate “round, red, small” ball,
she will choose the word that is most closely associated with
that concept, that is, the word that she has now learned from
Adam.

Techniques such as the one we have sketched will allow agents

to develop a shared lexicon to describe a scene that they can
both see. But this is only the beginning. To allow effective
communication to proceed, the agents need a shared grammar
as well as syntax. They also need to be able to use language
not only as a method of transmitting descriptions but also as
a way of carrying out actions such as commanding, asking,
persuading, and so on. We are only just beginning to discover
how to build systems in which agents can evolve such
capabilities (Cangelosi & Parisi, 2002), but it would be
surprising if great progress were not made in the next decade.

http://dx.doi.org/10.4135/9781412983259.d65