Event Study Approach for Validating Agent-based Trading Simulations

Shih-Fen Cheng
School of Information Systems
Singapore Management University
Republic of Singapore
[email protected]

Abstract—In this paper, we introduce how one can validate longer) of simulation time. Secondly, since the simulation
an event-centric trading simulation platform that is built with tracks real markets, we have little control over the kind
multi-agent technology. The issue of validation is extremely of scenario to be constructed. Finally, financial markets are
important for agent-based simulations, but unfortunately, so
far there is no one universal method that would work in known to demonstrate fat-tailed behavior and experiences
all domains. The primary contribution of this paper is a for these cases are extremely valuable; however, since these
novel combination of event-centric simulation design and event events are rare, it is highly unlikely that we will be able to
study approach for market dynamics generation and validation. guarantee having such events in our simulation.
In our event-centric design, the simulation is progressed by To address these concerns, during the past few years our
announcing news events that affect market prices. Upon
receiving these events, event-aware software agents would research group has developed a simulation platform that
adjust their views on the market and act accordingly. Their allows a wide variety of trading simulation scenarios to be
actions would be based on their roles and also their private created (the platform is generic, however, we mainly use it
information, and collectively the market dynamics will be in simulating commodity futures; for details, see [2], [3]).
shaped. The generated market dynamics can then be validated The major difference between our system and most existing
by a variant of the event study approach. We demonstrate how
the methodology works with several numerical experiments trading simulation platforms is that we allow the creation
and conclude by highlighting the practical significance of such of arbitrary market scenarios that would be more intensive
simulation platform. and dramatic than the paces of real markets. For example,
Keywords-Trading Simulation; Agent-based Computational instead of having to wait for days or even weeks for market-
Economy; Simulation Validation changing events, a scenario could be easily designed to
contain a year worth of important events within an hour
I. I NTRODUCTION of simulation time. Extreme market conditions that are rare
Simulations and serious games have been widely used in but important could also be easily designed to develop
various domains to enable more effective knowledge trans- participant’s crisis management ability or test emergency
fer. Of all the disciplines, trading is probably the one that market rescue measures [8]. As demonstrated by the case
has benefited most from the rapid development of serious studies presented in [3] and [8], this platform enables us to
games and simulations. This is so because trading games are design highly effective training programs for novice traders
becoming more and more like real trading platforms thanks and effective simulations for policy analysis.
to the aggressive push for markets to become fully electronic We achieved this flexibility with an event-centric de-
in recent years. As rich web technologies become ubiquitous, sign powered by the multi-agent framework. Event-centric
a large number of sites are being set up to teach people design allows us to design arbitrary market scenarios by
how to trade via highly interactive web games. For exam- introducing sequences of events. The multi-agent framework
ple, UMOO (http://www.umoo.com/) is a popular fantasy allows us to build the simulated market from the bottom-
stock trading site that allows people to practice day trading up. By designing individual building blocks carefully and
on selected real stocks. In commodity trading, FACTSim introducing appropriate market mechanisms, close-to-reality
(http://www.factsim.org/) developed at University of Florida market dynamics can be generated.
allows participants to trade in a simulated environment that One major challenge we face is the validation of the
tracks real commodity prices over several weeks or even simulation outcome. Unlike other modeling techniques, real-
months. world data necessary for the validation usually is not avail-
All the above mentioned systems track real market prices. able for the scenarios designed within our platform. This is
Although tracking real markets has the appeal of realism, it because introduced events could be completely fictitious, and
is not always the most preferred approach in constructing the sequence in which they appear could also be arbitrary.
training scenarios or simulations, for the following three The major contribution of this paper is the use of the event
reasons. Firstly, real markets move slowly most of time, study approach as a way to validate event-centric agent-
and a meaningful scenario might require several weeks (or based market simulations.
 II. BACKGROUND human traders. Both “impact” and “action time” are only
For the interest of readers, we provide a brief overview available to the market agents. Of course, besides qualitative
on the system design. A more comprehensive description of information, latest market information is also available as
the system design can be found in [3]. price quotes.
By putting events with different impact values in a se-
A. Software Architecture quence, the scenario designer can then realize the kind of
Three important components are included in our commod- market movement she wants to have in the simulation.
ity trading simulation: a) human trader terminals, b) market
server (servicing market mechanism and dispatching events), III. D ESIGNING M ARKET AGENTS
and c) market agents (to be described later). Using agents in modeling complex economic or financial
The software architecture is distributed in nature and all systems is not new, in fact, a large number of literature has
components are network-connected. Unlike most classical been devoted to the subject of “Agent-based Computational
market simulations that usually run under strict theoretic Economics” (ACE) (e.g., see [7], [12]). The ACE models is
assumptions on price dynamics, the price dynamics in our probably best explained in Tesfatsion’s own words [12]: “the
simulation is entirely determined by agent interactions in defining characteristic of ACE models is their constructive
ways that are similar to real exchanges. All agents are grounding in the interactions of agents, ... Starting from
allowed to buy and sell at any time and transactions are an initially specified system state, the motion of the state
matched continuously; these requirements can be easily met through time is determined by endogenously generated agent
by a standard continuous double auction (CDA). All human interactions.” Our model follows similar constructive prin-
traders are also subject to a hard limit on their standing ciple, creating various agent roles that interact to generate
positions, and they are expected to clear all their positions market dynamics. Hedgers and speculators are two primary
before simulation ends. To keep the simulation compact and agent roles implemented in our trading simulations. In this
focused, most non-critical exchange features, such as daily section we provide a very concise overview on major market
settlements and margin calls are not explicitly modeled. agent types that are included in our simulation.
B. Event-Centric Design A. Hedger Model
As mentioned earlier, one critical design that enables Hedgers are the original users of the futures market. They
artificial scenarios to be created is the introduction of events are usually producers or consumers of the commodity who
in the simulation. Events don’t alter price stream directly, would like to lock in at some specific prices and quantities
however, market agents that are part of the simulation will well before the time of production (for producers) or usages
read the news events and react to them following a rational (for consumers). These hedgers represent fundamentalists in
model. We do want to highlight that the agents we design do the market.
not actually read the news events in textual format; instead, To properly incorporate producers and consumers in our
we request that when an event is being introduced, the model, we assume that they exhibit stationary behaviors,
designer should quantify the impact of the event in numerical i.e., the rate of their production and consumption will be
form. These impact values are only accessible to market stationary. Since we assume that only one futures market
agents and are invisible to human traders. exists for this commodity, this assumption implies that all
Formally speaking, an event is defined by the following producers and consumers have to constantly establish new
four parameters: hedges in this market, and their collective actions will create
• Title and content: This qualitative information is the market dynamics accordingly. We further assume that all
meant for human participants only. producers and consumers will employ a simple hedge-and-
• Impact: This integer number describes how strong an forget strategy, meaning that they will establish new hedges
event is: 1 being the weakest and 5 being the strongest. based on their own needs (new produces or usages), the
A positive (negative) value corresponds to a bullish current market condition, and their expectation; once the
(bearish) event. hedges are established, they will hold them to the end (in
• Arrival time: This is the time when an event is sent to other words, no dynamic hedge will be considered).
both human and market agents.
• Effective time window: This is the interval in which B. Speculator
the event is effective. Note that consecutive time win- While “mean-reverting” hedgers constitute the “funda-
dows could overlap with each other to emulate a chain mental” part of the simulated market, most of the market
of events. volatility, on the other hand, is generated by the specula-
For all agents, an event only comes into existence when its tor agents. In our simulation, we adopt the classical zero
designated arrival time has passed. Once an event arrives, intelligence (ZI) strategy [6] in constructing our speculator
only the “title and content” will be made available to the agent. To prevent ZI agents from destroying the market trend
generated by producer and consumer agents, we limit the 1) First, the events of interest are identified, and for each
price range to fall in the bid-ask spread. identified event, a time window surrounding that event
After the price is randomly decided, the ZI agent will is defined so that security price information could be
decide to long or short equally likely. Since each ZI agent is collected. For event j, let the beginning and the ending
granted the same trading limit as human traders, it will also of the time window be τj1 and τj2 respectively.
randomly decide how much remaining position allowance 2) Second, determine the firms to be included as data
it would devote to the new trade. Again, just like human samples. The selection criteria usually involve data
traders, ZI agents are required to exit all positions at the end, availability and the characteristics of the firm such as
and they are programmed to gradually exit their positions market capitalization and industry, so that an unbiased
when the end draws near. set of samples could be constructed.
In our case, since the target we are studying is the price
IV. VALIDATING S IMULATION WITH E VENT S TUDY of the commodity derivative itself, the concept of firms
A PPROACH does not apply here. Alternatively, we will construct
A. Introduction the sample set by executing multiple simulation in-
The modern event study approach, introduced by Fama stances for the same event series (this is conceptually
et al. [4], is widely applied in economics and finance in identical to the reactions of multiple securities to the
measuring the effects of an event on the value of firms using same event series).
financial data. As MacKinlay [9] puts it: “the usefulness 3) To understand the impact of the event, we need a
of such a study comes from the fact that, given rationality measure on the event-induced abnormal return (AR),
in the marketplace, the effects of an event will be reflected which is simply the actual return minus the normal
immediately in security prices.” This feature is important return over the event window. The normal return is
since the impact of an event could be measured in a relatively the expected return when no event is introduced. In our
shorter time periods of several weeks or days, as opposed to study, the normal return is computed by finding the
several months or even years if we use other more lagging mean price of the commodity before the occurrence
indicators (e.g., production levels, revenues). This descrip- of any event. To accommodate events with multiple
tion also explains our choice in picking the event study periods, we define the cumulative abnormal return
as the tool in validating the event-based trading simulation (CAR) for event j as:
we built, since our simulation progresses by letting market jτ2
agents react to events. CAR(τj1 , τj2 ) =
X
ARτ , (1)
The event study approach has already been applied in
τ =τj1
detecting a wide-variety of events, e.g., mergers and acquisi-
tions, earning announcements (both examples are discussed where ARτ represents the abnormal return in time τ .
in [9]), or even macroeconomic news [11]. In most appli-
cations, common equity price of the studied firm is used; With the computed CAR, various statistical tests could
however, the event study approach could also be applied to be administrated for different purposes. In our study, we are
other type of securities with little modifications [5]. interested in testing the occurrence of an event and asserting
Roughly speaking, event studies try to statistically test that proper response strengths are generated. To detect the
for abnormal returns from the security prices within a occurrence of event j, we define the null hypothesis to be
predetermined time window. Due to practical limitations no event occurrence and compute the test statistics as:
resulting from data (could be related to both security prices CAR(τj1 , τj2 )
and events), many event study variants have been sug- θ= q , (2)
gested. Binder [1] reviewed a wide variety of event study var(CAR(τj1 , τj2 ))
methodologies, and discussed some frequently encountered
where CAR(τj1 , τj2 ) represents the average CAR from all ex-
empirical issues.
periment instances1 , and var(·) represents the variance of all
B. Event Study Procedure results. θ computed in (2) should follow the standard normal
These past researches on event study approaches provide distribution of N (0, 1). To test the occurrence of a bullish
a sound analytical framework for us to analyze whether or bearish event, we should define a one-tailed positive or
artificial events generate consistent market dynamics in our negative alternative hypothesis (i.e., H1 : CAR(τj1 , τj2 ) > 0
simulation. Despite differences in the statistical techniques for bullish events, H1 : CAR(τj1 , τj2 ) < 0 for bearish events).
applied, most event study methodologies have the following In either case, the rejection of the null hypothesis could lead
general procedures. To stay focused, we only include steps 1 A scenario can be repeatedly simulated, generating a number of price
that are relevant to our analysis (complete coverage on the streams. A separate CAR(·) is computed for every price stream, and
methodology can be found in [9]): CAR(·) is then obtained by taking the average.
 120
us to the conclusion that a bullish or a bearish event has
occurred. 115
To assert that impact levels from 1 to 5 indeed generate
110
appropriate price dynamics in the market, we would like to
establish that the events with higher impact levels indeed 105

Price ($)
produce larger CAR. To establish this result statistically, we
100
compare mean CAR for consecutive levels in pairs using
t-tests, i.e., comparing levels 1 and 2, 2 and 3, and so on. 95
The null hypothesis will be no difference in mean CAR. The
90
alternative hypothesis is similarly defined to be one-tailed,
stating that the mean CAR from the stronger event is greater 85 Normal
Bullish
than that of the weaker event, i.e., H1 : CAR(τj1 , τj2 ) > Bearish
CAR(τk1 , τk2 ), assuming that j is stronger than k. 80
0 50 100 150 200 250 300
Day

C. Validating Occurrences of Events (a)

To simplify the simulation and avoid clustering effects 10

from overlapping events, we create a special scenario with Normal
Bullish
8
only one event. For the market agents, we include 12 Bearish

producers, 13 consumers, and 2 ZI agents (to emulate 6

Cumulative Abnormal Return

human trader’s actions). Both producers and consumers are 4
constructed following the hedger model. The length of a
2
simulation day is defined to be 1 second, and the length
of the simulation is just over 370 days. The event occurs 0
in day 160 (which is known to all agents), and the event
−2
window is defined to be 20 days before the event occurrence
and 20 days after the event occurrence2 . In other words, −4

τ 1 = 140 and τ 2 = 180. For bullish, bearish, and no −6

event scenarios, the impact levels are set to +5, -5, and 0
−8
respectively. Sample price evolutions of these three scenarios −20 −15 −10 −5 0 5 10 15 20
Event Day
are shown in Figure 1(a).
(b)
To collect enough sample data points, the same scenario
is executed 15 times. Following the event study procedures Figure 1: (a) Sample price dynamics of the one-event scenar-
described in Section IV-B, we test the null hypothesis for ios. The event is announced in day 160. (b) The cumulative
bullish, bearish, and no event cases. One sample CAR from abnormal returns (CAR) for bullish, bearish, and normal (no
each case is plotted in Figure 1(b). Note that in Figure 1(b), event) scenarios.
we use -20 and 20 to represent 20 days before and after the
event occurrence respectively. Table I: Summary statistics of CAR and paired comparisons
For both the bullish and the bearish cases, the p-values for all impact levels.
∼ 0, implying that positive/negative abnormal returns are
statistically significant. For the normal (no-event) case, p- Lv Mean Std. Sample p-value against
value ∼ 0.065, indicating that no significant abnormal return dev. size previous level
is detected during the event window. 1 6.042 3.067 20 –
2 10.562 2.897 20 0
D. Validating Relative Strength of Events 3 14.206 4.661 60 0.0008
The events in our simulation are labeled from 1 to 5, 4 16.661 5.962 60 0.0067
indicating their respective strengths. It would be impractical 5 21.138 7.895 40 0.0009
to try to validate the absolute response for each impact
level, since many other factors (e.g., number of agents,
percentage of human traders, size of price tick, just to name a few) also affect the absolute return. Alternatively, we
would focus on comparing the relative magnitudes of events
2 As discussed earlier, the 20-day period before the event occurrence is with difference strengths. Regardless of the absolute return
used for estimating normal return. On the other hand, the 20-day period levels, events with higher strength level should consistently
after the event is required since in the simulation and the real-world market
alike, the impact of an event usually take a number of days to be fully generate greater market return magnitudes than events with
realized. lower strength level.
 To validate relative event strengths we construct a similar to assure the simulation users (human traders and policy
one-event scenario as specified in the previous subsection makers) that the platform generates reliable dynamics.
and vary its impact level from 1 to 5. The setup of the
VI. ACKNOWLEDGEMENTS
experiment is almost identical except that we now have five
scenarios, each with different impact level. This work was supported in part by the International
For each impact level, 20 to 60 samples are generated Trading Institute and Wharton-SMU Research Centre (under
depending on how variable the samples are. For impact grant C220/MSS7C008) at Singapore Management Univer-
level 3, 4, and 5, the results are particularly noisy, thus we sity. The author would like to acknowledge the dedicated
have executed more simulations. The experiment results are efforts by Chao-Chi Liu and Sharom Lim in running sim-
summarized in Table I. From Table I, we can see that the ulation experiments. The author would also like to thank
impact level of an event indeed dictates the strength of re- Samuel Owens and Anil Shamdasani for lending their ex-
sponse from market agents (all comparisons are statistically pertise in commodity trading.
significant). R EFERENCES
V. C ONCLUSIONS [1] J. J. Binder. The event study methodology since 1969.
Review of Quantitative Finance and Accounting, 11(2):111–
This paper attempts to address the validation issue in con- 137, 1998.
structing agent-based trading simulations. Without proper
validation mechanisms, it would be difficult to claim the the [2] S.-F. Cheng. Designing the market game for a commodity
trading simulation. In IEEE/WIC/ACM International Confer-
trustworthiness of the simulation, and hence void any pre- ence on Intelligent Agent Technology, pages 445–449, San
diction or analysis derived from the simulation. As recently Jose, CA, USA, 2007.
argued by Marks [10], the lack of formal validation protocol
is probably one of the major reasons why agent-based [3] S.-F. Cheng and Y. P. Lim. An agent-based commodity trading
simulation. In Twenty-First Annual Conference on Innovative
simulation is still not widely accepted by the mainstream Applications of Artificial Intelligence, pages 72–78, Pasadena,
economic, finance, and social science research. CA, USA, 2009.
The primary contribution of this paper is the introduction
and the application of the event study approach as one [4] E. F. Fama, L. Fisher, M. C. Jensen, and R. Roll. The
adjustment of stock prices to new information. International
candidate in performing micro-level validation for agent- Economic Review, 10(1):1–21, 1969.
based trading simulations. The event study methodology
is ideal and natural for this application since the trading [5] T. R. Fortenbery and D. A. Sumner. Tthe effects of USDA
simulation is built with the event-centric principle, and reports in futures and options markets. The Journal of Futures
Market, 13:157–173, 1993.
the fundamental market movement is generated by event-
aware agents. The novelty of our approach thus lies in the [6] D. K. Gode and S. Sunder. Allocative efficiency of markets
combination of the event-centric design in generating market with zero-intelligence traders: Market as a partial substitute
dynamics and the use of event study approach in validating for individual rationality. The Journal of Political Economy,
generated market dynamics. 101(1):119–137, 1993.
We do not attempt to claim that our methodology would [7] B. LeBaron. A builder’s guide to agent-based financial
be universally applicable (for other types of agent-based markets. Quantitative Finance, 1:254–261, 2001.
simulations where there are no event, our methodology
[8] B. Lee, S.-F. Cheng, and A. Koh. An analysis of extreme
will not be applicable), however, for agent-based trading price shocks and illiquidity among systematic trend followers.
simulation that relies on events in scenario generation, the Review of Futures Market, 18(4):385–419, 2010.
use of event study approach is indeed very effective.
With this validated simulation platform, a wide spectrum [9] A. C. MacKinlay. Event studies in economics and finance.
Journal of Economic Literature, 35(1):13–39, March 1997.
of applications are granted a sound foundation. For example,
in [3], the platform allows human traders to participate, [10] R. Marks. Validating simulation models: A general frame-
and the simulation platform is basically a training ground: work and four applied examples. Computational Economics,
exposing human traders to controlled and realistic market 30(3):265–290, Oct. 2007.
scenario so that they can practice their trading skills. A [11] G. McQueen and V. V. Roley. Stock prices, news, and
very different application is discussed in [8], where the same business conditions. Review of Financial Studies, 6(3):683–
platform is used for policy analysis, allowing policy maker 707, 1993.
to evaluate the effectiveness of different regulation changes
[12] L. Tesfatsion. Agent-based computational economics: A con-
under different market composite. In this application, there structive approach to economic theory. In L. Tesfatsion and
is no human trader and the market only includes autonomous K. L. Judd, editors, Handbook of Computational Economics,
software agents with different strategies. For both cases, the Volume 2: Agent-based Computational Economics. Elsevier,
validation procedure is extremely important since we need 2006.