Using Event Data to Monitor Contemporary Conflict

in the Israel-Palestine Dyad

Philip A. Schrodt, Deborah J. Gerner, and mr Yilmaz

Department of Political Science

University of Kansas
1541 Lilac Lane
Lawrence, KS 66044-3177
USA

March 2004

[email protected], [email protected], [email protected]

phone: +1.785.864.3523
fax: +1.785.864.5700
KEDS project web site: http://www.ku.edu/~keds
FAST project web site: http://www.swisspeace.org/fast/

Paper prepared for delivery at the Annual Meeting of the International Studies Association,
Montreal, Quebec, Canada, March 2004.
This research was supported in part by the swisspeace FAST Project. We would like to thank
Heinz Krummenacher and Vera Troeger of swisspeace for their continued support for, and
assistance with, this project. Bradley Lewis and Lauren Prather at the University of Kansas
assisted in the updating of the event data sets. The opinions expressed in this paper are those of
the authors and do not necessarily reflect those of the swisspeace or the FAST Project
The data sets, and software discussed in this paper, as well as a pdf version of the paper, can be
downloaded from the KEDS project web site: http://www.ku.edu/~keds.
2004 by the authors

Abstract
For the past eighteen months, the Kansas Event Data System (KEDS) project has been using
event data and other web-based sources to produce quarterly reports on the Israel-Palestine
conflict for the swisspeace (Swiss Peace Foundation) FAST Project, which is sponsored by
Swiss Agency for Development and Cooperation and a number of non-governmental
organizations. This paper describes the indicators that we are monitoring, the process we have
developed to generate the reports, and the supplemental sources we are using. We address the
issue of the differences between newspaper and news wire reports with respect to media
fatigue effects and also analyze some of the strengths and weaknesses of this approach to
conflict monitoring.

Schrodt, Gerner and, Yilmaz

Introduction
The broad phenomena of globalization has had a wide variety of impacts on the
international system. While much of the popular political focus on globalization has been on the
negative impacts, there have been some positive elements as well. This paper will describe how
contemporary web-based communication technology and advances in inexpensive computer
processing have been used to seamlessly implement a conflict monitoring system using resources
in Bern, Switzerland, Lawrence, Kansas, and non-governmental organization (NGO) groups in
Israel and Palestine.
For the past fifteen years, the Kansas Event Data System (KEDS) project (Gerner et al.
1994; Schrodt and Gerner 1994) has developed technologies for the automated coding of
international event data (see Merritt, Muncaster and Zinnes 1993; Schrodt 1994; Schrodt, Gerner
and Simpson 2001) and the application of those data to the monitoring and forecasting of
political conflict. Most of this work has been funded by the U.S. National Science Foundation,
and much of it has been academic in orientation. We were also interested, however, in trying to
adapt the system to do near-real-time forecasting similar to the work on the Kosovo conflict
using KEDS-based data by Pevehouse and Goldstein (1999).
In 2002, we were contacted by the Bern-based swisspeace (Swiss Peace Foundation)
FAST project concerning the possibility of incorporating our data-generation capabilities, along
with the extensive area expertise and field experience that one of us had in the region, for the
monitoring of the conflict in Israel-Palestine. Because the Levant has always been the highest
priority in data collection for our projectin fact the entire initial motivation for the KEDS
effort was to get contemporary data on the Levantthis was a natural match.
Swisspeace describes itself as:
an action-oriented peace-research institute [working] in the area of conflict analysis and
peacebuilding. We research the causes of wars and violent conflicts, develop tools for early
recognition of tensions, and formulate conflict mitigation and peacebuilding strategies.
swisspeace contributes to information exchange and networking on current issues of peace
and security policy through its analyses and reports as well as meetings and conferences.

Schrodt, Gerner and, Yilmaz

FAST is a project sponsored by the Swiss Federal Department of Foreign Affairs (DFA)
and numerous non-governmental organizations (NGOs) that support peacebuilding
activities by Swiss authorities, aid agencies, and other NGOs and promote their
cooperation at the conceptual and operational levels.
swisspeace was founded in 1988 as the "Swiss Peace Foundation" with the goal of
promoting independent peace research in Switzerland. During recent years it developed
into an internationally renowned institution in peace and conflict research. Today
swisspeace engages about 25 staff members. Its most important clients include the Swiss
Federal Department of Foreign Affairs (DFA) and the Swiss National Science Foundation.
http://www.swisspeace.org/about/default.htm (accessed 26 Februrary 2004)
The swisspeace FAST project is specifically focused on conflict monitoring and early warning:
FAST's objective is the early recognition of impending or potential crisis situations for the
purpose of early action and prevention of violent conflict. FAST aims at enhancing the
political decision makers' ability to identify critical developments in a timely manner in
order to formulate coherent political strategies to either a) prevent or limit destructive
effects of violent conflict, or b) recognize windows of opportunities for peace building.
http://www.swisspeace.org/fast/default.htm (accessed 26 Februrary 2004)
While there are hundreds if not thousands of NGOs with similar concerns about conflict,
the FAST project is virtually unique in combining systematic quantitative analysis with
traditional qualitative analysis and field reports; most NGOs use only qualitative methods.
Swisspeace in general, and FAST in particular, is also actively engaged with the policy
community at both the national and intergovernmental organization (IGO) levels: This is
facilitated by Switzerlands traditional position of neutrality and the proximity of Bern to
Geneva, the headquarters for a large number of humanitarian IGOs and NGOs.
The director of swisspeace, Heinz Krummenacher, makes an interesting distinction
between the early warning function of swisspeace and related NGOs, and the traditional
intelligence function of national governments. This is summarized in Table 1.

Schrodt, Gerner and, Yilmaz

Table 1: Early Warning vs. Intelligence

Early Warning by NGOs

Intelligence Analysis by Governments

Transparent sources and methods

Secret sources and methods

Decentralized

Centralized

International

National

Information sharing

Information hoarding

Human security (welfare of others)

National security (welfare of the state)

Source: Heinz Krummenacher, FAST project, swisspeace Conference on Global Conflict Monitoring, Bern, 6 March
2003

The KEDS/FAST collaboration is illustrative of all of these characteristics. The KEDS

project uses open-source informationcurrently news sources readily available at most academic
institutionsand all of its specialized software is available as open-source computer code that
can be freely downloaded and modified by other researchers. Our coded event data are posted to
our web site more or less concurrently with the publication of the FAST reports or any other
analysis by the project and these data are never embargoed to prevent other researchers from
analyzing them in a timely manner.1 In fact, due to our quarterly updating of the data sets, the
KEDS Middle East data sets are probably the most up-to-date data regularly available in the field
of quantitative international relations. An analysis shared between KEDS in Lawrence, Kansas,
USA2 and FAST in Bern, Switzerland,3 is clearly both decentralized and international. Finally,

Unfortunately, due to licensing and other intellectual property constraints, we are not able to post or share the
source texts used to code the data. This is not a major problem with North American universities, since the
NEXIS service is almost universally accessible and one could readilyif tediouslyreconstruct the set of source
texts locally by simply duplicating our procedures. It is a greater problem for European institutions, where
NEXIS is not as easily available.

view from project office windows: tallgrass prairie and in the distance, the decommissioned De Soto Army
Ammunition depot, a 1,200 hectare toxic-waste site.

view from project office windows: medieval Bern, and in the distance, the Alps.

Schrodt, Gerner and, Yilmaz

the overall focus of our analysis is the general security situation in Israel-Palestine, not the
interests of any particular state or proto-state.
The FAST project is currently monitoring 20 conflict regions: eight in Africa, nine in Central
Asia, Albania and the former Yugoslavia, and Israel-Palestine. The choice of areas is determined
by a combination of donor interests and the ability of FAST to set up monitoring networks.
FAST reports are brief, typically four pages of charts and analysis, in bullet-point format, plus
additional text describing the data. FASTs general methodological approach is to use:
...both quantitative and qualitative methods for its analysis. The centerpiece in the
quantitative analysis is based on event data analysis and the respective tools developed in
the framework of the Program on Nonviolent Sanctions and Cultural Survival (PONSACS)
at Harvard University. The logic of event data analysis is fairly simple: all events
considered relevant to conflict escalation and de-escalation are assigned certain numeric
values according to a distinct conflict scale. These values can then be added up for specific
time interval and graphically displayed in a curve over time. The quality and quantity of
data is crucial for the success of such a method. In order to gather the quality and quantity
of data required to suit early warning purposes, FAST sets up its own local information
networks (LINs) and thus overcomes shortcomings of existing information sources (e.g.
international news wires). A unique set of data is collected for each country completely
independently from Western news-media coverage. For qualitative data analysis, FAST
contracts with internationally renowned country experts.
(swisspeace 2003:8)
Because of the unique characteristics of the Israel-Palestine conflict, and the fact that FAST
contracted with the KEDS project to extend an existing data collection project, the KEDS
analysis differs from this approach in several ways. First, because the Israel-Palestine conflict is
intensely and consistently covered by the international mediainternational wire services
typically provide about 50 to 100 reports per day on an area covering only about 26,000 square
kilometers with a population of less than 10-million peoplewe can rely on news wires as our
primary event data source, although we also supplement this information with specialized
publications and NGO reports from Israel and Palestine. Second, we are using the Goldsteinscaled data (Goldstein 1992) developed by the KEDS project rather than the PONSACS system,
which is somewhat different. FAST has not set up an LIN in the region, but the KEDS projects
Middle East expert (Gerner) maintains extensive telephone and email correspondence with her
own sources in the region, as well as making periodic field research trips.

Schrodt, Gerner and, Yilmaz

Transmission of information between the Bern and Lawrence components of the project
uses the now-routine medium of email attachments. The monthly data, graphs, and text are
prepared in Kansas, then sent to Bern for editing and reformatting, then Bern produces the final
reports in PDF format, sending these immediately to subscribers and also making them available
on the Web (http://www.swisspeace.org/fast/). While this process appears unexceptional
in 2004, it is worth noting that this exchange of information that is done in near real time at zero
marginal cost would, as recently as ten years ago, have required either expensive international
courier services or slow international mail. More tellingly, we use the same approach to exchange
documents with Bern that we use to exchange documents within our project: it is irrelevant
whether one is moving information between offices separated by 76 meters or 7,600 kilometers.

Conflict Monitoring and the Problem of Media Fatigue

The first question that one might ask about this exercise is whether it is necessary at all.
The Israel-Palestine issue seems to be a very frequent topic in newspaper articlestoo frequent,
in the assessment of many readersso what can be added by yet another analysis, particularly
one that relies on the same news wire articles upon which most newspapers depend?
The answer lies in the phenomenon of media fatigue, which we systematically analyzed
in Gerner and Schrodt (1998). Media fatigue refers to the tendency of news sources to provide
a high level of coverage when a conflict first breaks out and then to reduce and simplify that level
of coverage as the conflict continues. As the erudite political analyst Davr Barry has observed:
My employer, the Miami Herald, has had to cut costs. One way were doing this is by
recycling stories. Rather than go to the expense of writing a whole new story, were
rerunning earlier ones. For example, every day weve run the same story on fighting in the
Middle East (Middle East Fighting Again).
To further cut costs, many newspapers around the country are now using fewer words.
Not need adjectives, adverbs. Nouns, verbs can communicate story gist. (Barry 2003)4
More seriously, an article in The New York Times in August 2003 titled Suffering News
Burnout? Rest of America Is, Too discussed the precipitous decline in U.S. audiences watching

For the benefit of non-North American readers: Barry is a nationally-syndicated humor columnist; this story is a
joke. That said, Middle East Fighting is a close approximation to how TABARIour automated coding
programreads a news story.

Schrodt, Gerner and, Yilmaz

news stories compared to the number watching during the 2003 Iraq war (CNN went from 2.5
million viewers during the war to 413,000 during the summer, Fox from 3.2-million to 753,000,
and MSNBC from 1.3-million to 197,000)
People have been through two years of very heavy-duty, stressful news, from Sept. 11
through the war with Iraq, said Jim Murphy, executive producer of the CBS Evening
News with Dan Rather. I think theres probably just a little bit of a break-taking going on
across the spectrum.
Steve Sternberg, senior vice president for audience research at Magna Global USA in New
York, an advertising buying agency, takes a similar view. Considering how much news
there was with the Iraq war, he said, people are probably just taking a breath and saying,
O.K., thats enough news for a while. (Rutenberg 2003)
As we demonstrated in our 1998 analysis, one of the best ways to detect media fatigue is
to compare the newspaper coverage of a distinct, newsworthy eventfor instance, the
occurrence of deaths due to violencewith an independently created time series of the same
event. Since we have access to independently-compiled Israeli and Palestinian death totals from
the Israeli human rights organization BTselem and the well-respected Palestine Red Crescent
Society (PRCS), we can compare these with the coverage in U.S. newspapers, and also examine
the differences between newspaper and news wire service coverage. 5
The differences over time have been quite dramatic. During the most intense period of the
second Palestinian intifada (uprising)the spring of 2002, which saw both an upsurge in
Palestinian suicide bombings and Israels military reoccupation of the West Bankthere was
very substantial U.S. media coverage of the conflict. This continued, although with less detail,
through the remainder of 2002, but was largely displaced beginning in 2003 by coverage of the
U.S. invasion and subsequent occupation of Iraq. There was a brief resurgence of interest in the
region in the summer of 2003 during the short-lived Bush administration road map peace
initiative, but this ended quickly. At the present time, the only acts of violence that are
consistently covered in The New York Times and Washington Post are Palestinian suicide
bombings and, less consistently, Israeli military operations that result in large number of

PRCS compiles its death totals from hospital and ambulance reports, which it has access to because of its
position as a provider of medical services. BTselem uses a variety of sources, including some media reports, but
those reports of independent of those provided by the U.S.-based press.

Schrodt, Gerner and, Yilmaz

Palestinian casualties. Routine day-to-day violencewhich has continued at the almost constant
level of a couple of Palestinian deaths per day since May 2002receives virtually no coverage: it
has faded into the background.
Fortunately, this media fatigue does not affector at least does not affect as severelythe
news wire reports from sources such as Agence France Presse (AFP). For example, when
preparing our most recent report for FAST covering November and December 2003, Schrodt was
surprised to see PRCS figures showing a total of 90 Palestinian deaths in the period, while the file
he was keeping of stories from The New York Times had virtually no reports of Palestinian
deaths. 6 Nonetheless, cross-checking the AFP reports showed that the deaths were in fact
reported in the wire services, typically occurring in small numbers, at checkpoints, near
settlements, in small-scale raids, and so forth.
Table 2 and Figure 1 show the dramatic difference between the level of coverage of
newspaper and news-wire sources. Various sources in the NEXIS electronic data service were
searched using the phrase Palestinian killed and the headline, lead paragraph or terms option
for the two month period 1 November 2003 to 31 December 2003. 7 We then eliminated multiple
stories on a single day and stories on multiple days that clearly referred to the same incident. For
the newspapers this had little effect but it substantially changed the wire service figures. For
example, the AFP search initially returns 64 stories, but this is reduced to 18 stories after
duplicates are removed.

The Israeli human rights group BTselem reports 79 deaths for the same period: as noted below, BTselems
figures for Palestinian deaths are usually about 10% below those of the PRCS, possibly because PRCS has direct
access to hospitals and is more likely to identify deaths that occur in the days following an initial wound.

Caveats: The NEXIS search engine automatically includes the plural Palestinians in the search. Based on
extensive work on related projects searching for stories dealing with civilian casualties, we have found that the
simple search phrase killed is very reliable for locating incidents of violent death, although it is possible that
the addition of synonymous phrases such as shot dead might alter the results slightly. New York Times, full
text used the full text search option in NEXIS. We experimented with Hamas and militants as alternatives
to Palestinian in The New York Times headline search and this made no difference. We searched the BBC using
the Factiva data service; for reasons that are unclear the NEXIS search on BBC returned no stories. We also
attempted to assess coverage in the English-language version of the Israeli newspaper Haaretz but it is currently
unavailable on NEXIS due to a contract dispute and its own archival search engine did not appear to be retrieving
basic news reports, only commentaries. The AP search covers all AP files in NEXIS. We attempted to get
Reuters totals in Factiva but a search returned only one story, which indicates either problems in the Factiva
search engine, problems in how we were using Factiva, or that Reuters is in even bigger trouble than we thought.
Searches were done 6 Mar 2004.

Schrodt, Gerner and, Yilmaz

As shown in Table 2, the lowest level of coverage (Los Angeles Times) and the highest
(AFP) differ by a factor of six. While this can be improved somewhat by going to a full-text
search, even that strategy generates only eight stories in The New York Times and in most
instances the report of Palestinian deaths was buried near the end of a story that dealt with other
news about the region (for example analyses of U.S. policy, internal Israeli politics, or the
building of the Separation Wall).
Table 2. Number of stories found with Palestinian killed NEXIS search string
Newspaper

Wire Service

Los Angeles Times

Xinhau

New York Times

BBC (Factiva)

10

Washington Post

Associated Press

11

Jerusalem Post

Agence France Presse

18

New York Times, full text

Figure 1 juxtaposes information on the dates of the stories for each of the sources with the
daily death totals reported on the PRCS web site, which are shown in an inverted scale below the
X-axis. Two things are evident from this figure. First, the wire services usuallybut not
alwaysreport on situations where there are more than two Palestinian deaths. The exception
to this occurs in the period 8 to 12 November, where there is no follow-up on the 11 deaths that
occurred after the seven deaths on 6-7 November. Situations where there are only one or two
deaths, in contrast, generate stories in only about 50% of the cases. Second, the news wire
stories are generallybut not perfectlyaligned with each other; this is less true of the
newspaper stories (for example, only 50% of The New York Times and Washington Post stories
occur on the same days.8 There are no cases where the newspaper reports provide stories that
do not already exist in the wire service reports.

Due to time zone and publication time differences, all of these story dates should be considered accurate to only
within a day or two. Also note that the PRCS data records the date of deathin the case of fatal wounds this can
occur well after the date of an incident that might generate a news report.

Schrodt, Gerner and, Yilmaz

Figure 1: Comparison of newspaper and wire service coverage of Palestinian

deaths
NYT full
NYT lead
Wash Post
Jerus Post
Xinhau
AP
BBC
AFP

10

Event Data Sets

For the FAST Israel-Palestine monitoring, we are using English-language reports from the
Agence France Presse (AFP) wire service. AFP is the worlds second largest wire service after
Reuters and, unlike Reuters, is available on NEXIS data service. The University of Kansas, like
most North American research universities, has a general subscription to the NEXIS service
through its library; consequently these reports are available to us at no cost.
Because we are maintaining a general Levant data set, we locate stories using the search
string for the headline and first paragraph:

Schrodt, Gerner and, Yilmaz

10

Israel or Palestine or Jordan or Syria or Egypt or

Lebanon or PLO
This search will typically yield about 1,500 AFP stories per month, though a number of these
will be duplicates from developing stories. The number of stories can be substantially greater
during periods of high conflict; it would presumably decrease in periods of low conflict (for
example the mid-1980s after the invasion of Lebanon and before the first Palestinian intifada) but
we havent seen such periods since beginning the monitoring for FAST. Stories are downloaded
and reformatted for coding using a specialized computer program, nexispider.pl, written in the
perl text processing computer language (see Appendix).
The downloaded stories are coded by using the open-source automated event data coding
program TABARI that has been developed by the KEDS project (Schrodt 2000, 2001; also see
http://www.ukans.edu/~keds/tabari.html) and the WEIS (World Events Interaction
Survey; McClelland 1976) event coding scheme.9. Following standard KEDS project procedures,
we are only coding the lead (first) sentence of the story. For AFP news storiesas distinct from
feature stories dealing with historical background material or human interest storiesthe leads are
generally relatively straightforward to code. We are using dictionaries developed over the past 15
years at the KEDS project; the dictionaries for the Levant coding now contain about 5,500 verb
phrases and 1,800 phrases identifying political actors.
The final step in the event data acquisition is one-a-day filtering that allows only one
instance of each dyad-event per day. This step is necessitated by the fact that AFP provides
near-real-time coverage of breaking stories. Thus, an incident such as a missile attack or a suicide
bombing may generate a dozen or more individual stories as the circumstances and casualty figures
are updated while the story develops. One-a-day filtering has the undesirable side-effect of
eliminating situations where there are, in fact, separate incidents (for example missile attacks in

In the near future, we anticipate switching from WEIS to the new CAMEO coding scheme that we have
developed over the past three years under U.S. National Science Foundation funding (see Gerner et al. 2002).
Although specifically designed for the study of international mediation, CAMEO is a general-purpose event
coding scheme that has been optimized for automated coding by eliminating distinctions between categories in
WEIS that cannot be easily distinguished using automated methods. In many cases, those distinctions could not
consistently be made by human coders either, so the result should be more reliable event coding. CAMEO also
provides far more detail on violent events and provides a systematic method of coding sub-state actors. See
http://www.ukans.edu/~keds/cameo.html for more detail.

Schrodt, Gerner and, Yilmaz

11

both Gaza and Nablus, or near simultaneous suicide attacks in Tel Aviv and Jerusalem), but these
are relatively rare compared to the problem of multiple stories. At some future point, we may be
able to deal with this problem by extracting greater levels of geographical information from the
news report, but the existing system is still working reasonably well, particularly since the events
are aggregated over an extended period of time.

Goldstein Totals and Averages

The first two pages of the swisspeace FAST update use two measures of activity between
Israel and the Palestinians, aggregated by month. We employ Goldsteins widely-used scale for
converting WEIS events to numerical values. In the Goldstein scale, the most cooperative events
are assigned +10, the most conflictual events (notably the WEIS 223 code for acts of violence) are
assigned -10, neutral comments are assigned 0, and the remaining WEIS categories have
intermediate values. The Goldstein scale generates data on a cooperation-conflict continuum,
with higher negative values reflecting greater levels of conflict. Two series are reported, one for
Israel actions towards Palestinians, the other for Palestinian actions toward Israel. While it is
possible to distinguish sub-state actors using event data, we have aggregated to the national level
so, for example, violent acts by Israel include those of Israeli settlers as well as the Israeli
military; similarly criticisms of Israel by Palestinians include those by Palestinian militant
groups such as Hamas and Islamic Jihad as well as those of the Palestine Authority and the PLO.
While most of the work we have done on the KEDS project uses monthly Goldstein totals
(e.g., Schrodt and Gerner 1997, 2000), for the FAST work we are also using Goldstein averages.
The advantage of the Goldstein total is that it reflects the overall level of activity in the region;
the disadvantage is that it is sensitive to the number of reported events. The Goldstein average
does not have this sensitivity to event frequency and therefore shows whether the level of
conflictual interaction is increasing or decreasing in intensity. The disadvantage of the average is
that two events per week could generate the same values as two hundred events per week, and
the second situation clearly involves a greater level of activity.
Two general criticisms of event data scales are that the conflict-cooperation continuum is
inappropriate in situations where increased levels of interaction can lead to a rise in both
cooperation and conflict (think newlyweds) and that equal numbers of conflict and cooperation

Schrodt, Gerner and, Yilmaz

12

events could cancel each other out and give the appearance that nothing was happening. The first
criticism is not really relevant to the Israel-Palestine conflict, since almost all issues are defined in
terms of security and most cooperation is oriented towards reducing conflict. The second
criticism is usually not relevant because the level of conflict is so much greater than the level of
cooperation, although we have found one case where this problem occurred.
We are currently experimenting with alternative measures in our statistical worktypically
event counts of broad categories of events such as verbal cooperation or physical conflictand at
some point we may incorporate these into the FAST updates. However, the levels produced by
the Goldstein scaling is generally consistent with our qualitative analysis of the conflict and has
the advantage of simplicity as well as greater comparability with reports produced by FAST for
other regions.
Figure 2. Goldstein Totals
0
-100
-200
-300
-400
-500
-600
-700
-800

ISR>PAL

PAL>ISR

Schrodt, Gerner and, Yilmaz

13

Figures 2 and 3 show the Goldstein totals and averages for January 2002 through December
2003. The two most conspicuous events of the past two years show up clearly: The downward
spike in the spring of 2002 corresponds to the upsurge in Palestinian suicide bombings and
subsequent Israeli military reoccupation of West Bank cities in that period. The positive hump
that peaks in July 2003 corresponds to the short-lived U.S.-sponsored road map peace initiative.
Figure 3. Goldstein averages

0.00
-1.00
-2.00
-3.00
-4.00
-5.00
-6.00

ISR>PAL

PAL>ISR

The difference between the two charts illustrates the advantage of considering both the
total and average scores. The Goldstein totals show an overall upward trend, indicating a decline
in conflict. However, a comparison with the Goldstein average chart (and with the qualitative
analysis) shows that this decline in conflict (that is, increasing slope of the line) is due to a
decrease in the number of incidents, not the type of incidents. As shown by the Goldstein
average, over the period from May 2002 to April 2003, there was a steady decline in the average,
corresponding to the combination of the effects of the re-occupation and the virtual end of
cooperative interactions as the Israeli government of Ariel Sharon tried to physically and

Schrodt, Gerner and, Yilmaz

14

diplomatically isolate Yasir Arafat. The U.S.-mediated peace process in the summer of 2003
dramatically reversed this decline in cooperation for a short period of time, after which average
interactions returned to a relatively low level (but not quite as low as before May 2003).
November 2003 shows an interesting anomaly where the score for Palestinian actions
directed to Israel score is almost zero. This is, in fact, a situation where conflict and cooperation
events have in fact cancelled out, one of the first times we have observed this problem in
aggregated event data. During this month, the Goldstein total of Palestinian actions towards
Israelis came very close to the neutral point where reports of conflictual acts are cancelled out by
reports of cooperative acts. This was due both to the low level of Palestinian violence towards
Israelis and extensive citizen negotiations between Palestinians and Israeli elites. These
negotiations culminated in the announcement of the Geneva Accord on 1 December 2003.
Mediation
From the perspective of general event data research strategies, the mediation measure is the
most interesting of the four we are using because it involves the use of a composite pattern of
events to generate a secondary measure that, nonetheless, appears to be quite accurate. In other
words, even though the WEIS system does not explicitly have a category for mediation, we can
derive a measure of mediation using patterns.
Specifically, following the approach we used in Schrodt and Gerner (forthcoming), a
mediation event occurs when there are cooperative WEIS events (WEIS codes <=100 except 02x
comments) involving the mediator (see below) and each of the parties in the conflict (Israel and
Palestinians) within a 7-day period. While this pattern is not a sufficient condition for mediation
a representative of a state might visit multiple parties to a conflict without trying to
mediateit is probably a necessary condition in the sense that any mediation will involve such
cooperation, at least within the limitation of the news reports. This definition has been
implemented as a relatively simple 100-line computer program written in the C programming
language that operates on the original stream of event data, detects occurrences of the pattern, and
then produces a tab-delimited file that can be copied in a Microsoft Excel spreadsheet.
We are monitoring the activities of three mediators: the USA, UN, and the Europeans, with
the latter defined as the EU, UK, France, Germany, Italy and Netherlands. Figure 4 shows these

Schrodt, Gerner and, Yilmaz

15

statistics for the period 1 Jan 2002 to 31 December 2003. There are two conspicuous spikes of
mediation activity. The first coincides with the outbreak of violence in spring 2002 and deals
with the diplomatic efforts responding to Israels re-occupation of areas that it had evacuated
earlier under the Oslo Accords, and particularly over Israels siege of Yasir Arafats Ramallah
headquarters and the Church of the Nativity in Bethlehem. These events caused an increase in
activity in all three of the mediating groups.
The second spike corresponds with the U.S.-led road map negotiations during the
summer of 2003. There is a slight increase in European involvement at this point but the level is
nowhere near that seen a year earlier, and there is essentially no UN involvement. The mediation
efforts show an almost perfectly symmetric rise-and-fall pattern consistent with the very limited
effort that the U.S. put into this initiative; the rise-and-fall is also mirrored in the increase in the
Goldstein averages (Figure 2) and, over a shorter period, the decline in fatalities (Figure 5).
Figure 4. Mediation efforts

180
160
140
120
100
80
60
40
20
0

USA

UN

European

Schrodt, Gerner and, Yilmaz

16

Deaths
Unlike many of the conflict areas being monitored by FAST, which are currently quiescent,
there has been almost non-stop violence in Israel-Palestine since the outbreak of the second
intifada in September 2000. While these casualties have been quite low compared to areas where
there has been a total breakdown of control such as the former Yugoslavia, Rwanda, Liberia, or
the former Zaire, death totals in this region are of considerable interest to Western governments.
In principle, we could extract approximate death totals from the AFP news reports.
However, it would be nearly impossible to do this accurately using fully automated methods,
AFPs coverage is not completely comprehensive, and based on our prior experience (Gerner and
Schrodt) working in the field with the Palestine Human Rights Information Center during the first
intifada, we knew that getting reliable casualty totals required full-time specialized efforts.
Consequently we have used two NGO sources for these numbers. For Israeli deaths, we use the
statistics compiled by B'Tselem posted at
http://www.btselem.org/English/Statistics/Al_Aqsa_Fatalities_Tables.asp

For Palestinian deaths, we use the figures compiled by the Palestine Red Crescent Society posted
at
http://www.palestinercs.org/crisistables/table_of_figures.htm

Neither of these two data sources include deaths of foreign nationals; B'Tselem has totals
for these but does not report them by month. The standard PRCS web site provides daily data
for the current month, then totals for past months and is usually is updated daily. The PRCS
web site also contains links to a database that can generate more detailed reports by date or
location. BTselem provides only monthly data and tends to update these figures once a month,
usually two or three weeks after the end of a month. BTselems historical data on the web are
less detailed than those of PRCS, although they helpfully provided us with an Excel file giving
monthly figures for some data which were only available in yearly aggregations on the web sites.
BTselem reports figures for Palestinian deaths as well as Israeli deaths. BTselems
numbers are generally about 10% lower than the PRCS figures. BTselem makes it clear that
they are not able to investigate all cases in depth (see http://www.btselem.org/ Main >
Statistics > Clarification on the Data on Persons

accessed 8 March 2004), whereas

Schrodt, Gerner and, Yilmaz

17

PRCSs data is, to the best of our knowledge, derived from their operational databases.10 The
10% discrepancy is consistent with PRCS having more complete information on individuals who
were wounded in incidents and subsequently died, and to incidents that never were reported in
sources available to BTselem. The patterns of deaths in the two series are quite similar so we do
not regard these differences as significant.
Figure 5. Deaths

300
250
200
150
100
50
0

Israeli

Palestinian

Supplementary Information and Qualitative Analysis

In the FAST updates, the graphs of the various quantitative indicators are accompanied by
qualitative interpretations. Schrodt usually drafts the qualitative analysis based on the event data
and newspaper reports, then Gerner modifies itinvariably in the direction of making it more

10

We made a site visit to the main PRCS office in Ramallah in May 2002; Gerner has visited BTselem on several
occasions.

Schrodt, Gerner and, Yilmaz

18

pessimistic, and, unfortunately, making the analysis more accurateon the basis of the
specialized sources she is monitoring.
When preparing these qualitative assessments, we use a variety of different sources of
information to supplement the statistical indicators. First, despite the problem of media fatigue,
we keep an on-going chronology of newspaper stories about major events in the region. Our
primary U.S.-based sources are from The New York Times (http://www.nytimes.com),
Washington Post (http://www.washingtonpost.com) and Christian Science Monitor
(http://www.csmonitor.com); we also monitor the English-language web site of Haaretz
(http://www.haaretzdaily.com). We supplement these sources with a variety of IGO and
NGO sources; 11 these are particularly useful in providing information on overall living
conditions. As part of her on-going work on the Middle East, Gerner subscribes to a number of
specialized publications covering the region, both in print form12 and electronic form.13 There
are also a number of individuals living in the region with whom Gerner is in regular contact by
phone or email.
Because the FAST updates are targeted are policy-makers who are looking for a condensed
summary of the most important trends in a conflict, these qualitative assessments are relatively
briefFAST aims for only four or five bullet points per page. In this respect the updates differ
substantially from most of the source material we are using, and the challenge is reducing a great
deal of information to a small number of salient observations. One issue that has become very
apparent with this processalbeit it is something that weve always known to be a problem

11

Examples include the ReliefWeb site http://www.reliefweb.int/hic-opt/, the NGO health

monitoring site http://www.healthinforum.org/ and the Israeli branch of Physicians for Human
Rights http://www.phr.org.il/phr/Pages/PhrHomepage.asp

12

Print sources include The Journal of Palestine Studies, The Jerusalem Report, al-Majdal, Washington Report on
Middle East Affairs, and Middle East Report.

Schrodt, Gerner and, Yilmaz

19

with event data analysisis the cumulative effect of slow, on-going actions that either do not, in
themselves, constitute events, or else are continuous and therefore are not reported as discrete
events.
Two cases can illustrate this. The problem of cumulative change is illustrated in the effects
of closures and other economic dislocations on the region. This is most dramatic for the
Palestinians: a recent report (The Economist, 31 January 2004) indicates that Inside the
occupied territories, life has grown increasingly desperate. Food consumption has fallen by onethird since the start of the intifada in late 2000. Nearly a quarter of young children are acutely
malnourished. Two-thirds of Palestinians have no job. Because this situation has continued for
so long, any savings of the typical Palestinian family have been long depleted. The economic
problems are also affecting Israel, which has experienced unusually high levels of unemployment
and limited economic growth since the outbreak of the second intifada. Because event data looks
for actions in the form of who-does-what-to-whom (subject-verb-object), an evolution in
economic circumstances is not, by itself, an event. To a large extent, these economic changes
are clearly caused by discrete eventsnotably closures, checkpoints, house demolitions, and
other disruptions of normal lifebut due to media fatigue this information is not systematically
reported, even in the news wires.
A more dramatic example of an on-going event that is not reflected well in the event data is
Israels recent construction of the massive Separation Wall mentioned earlier. This presents
several problems. First, because the construction of massive concrete walls is a relatively rare
international event (and in the case of a wall that is not on an internationally-recognized border or
cease-fire line, virtually unprecedented), it is not at all clear how this event would be reflected in
WEIS codes. Second, due to media fatigue effects (and, more generally, the narrative norms of

13

Electronic sources include AIPAC - American Israel Public Affairs Committee (http://www.aipac.org),
Alternative Information Center (http://www.alternativenews.org), American Friends Service
Committee - Middle East (http://www.afsc.org/middleeast/default.htm), B'Tselem: The Israeli
Information Center for Human Rights in the Occupied Territories (http://www.btselem.org), The
Electronic Intifada (http://electronicintifada.net/new.shtml), Israel/Palestine Center for
Research and Information (http://www.ipcri.org/index1.html), Middle East Research and
Information Project (http://www.merip.org), Miftah: The Palestinian Initiative for the Promotion of
Global Dialogue & Democracy (http://www.miftah.org/Index.cfm). For a more complete list of
Web-based sources of information of the Israel-Palestine conflict, see Gerners home page
(http://people.ku.edu/~gerner) or the appendix in International Quaker Working Party (2004).

Schrodt, Gerner and, Yilmaz

20

journalism) event data have always been weak in dealing with events that occur over a long period
of time: news reports do not say Construction of the Separation Wall continued today.
Finally, the political, economic, and psychological impact of the wall depends not only on the
fact of its construction, but on where it is being constructed. The earliest parts of the Wall were
erected in high-population areas in the northern West Bank. The Wall was then extended through
areas of relatively low population density, but it is currently (March 2004) again being built in
areas of high population densityand high political significancein the immediate vicinity of
Jerusalem. Because our event data is aggregated to just Israel and Palestine, we do not pick
up any of these geographical distinctions. Reports of the construction of the Wall are subject to
media fatiguemost reports occur in the context of new portions being constructed (but only
rarely that) or protests about the constructionand the effects, whether in terms of economics
or security, will occur only gradually.
These weaknesses certainly do not suggest that event data indicators are useless, but they
do indicate that such data alone do not present a complete picture. As we have argued elsewhere,
event data need to be supplemented with additional sources of information such as economic and
demographic indicators that do not involve events and the monitoring of qualitative information
(ideally supplemented with first-hand reports from the region) that will shed light on the overall
impact of the publicly-reported events.

Conclusion
Based on our experience over the past eighteen months, the combination of event data and
supplemental quantitative indicators, the systematic monitoring of web-based general and
specialized media, NGO sources and personal contacts in the region have proven effective in
allowing us to construct succinct, policy-relevant summaries that monitor the Palestinian-Israeli
conflict despite the fact that we are operating from Kansas. As we noted initially, we are able to
do this in part because the area is extensively covered by both the international media and by a
dense network of NGOs, and is usually directly accessible to telephone and email
communication.
Since we are not part of the policy communities to whom these are updates are directed, it
is difficult for us to assess the extent to which the event data are providing added value to our

Schrodt, Gerner and, Yilmaz

21

qualitative analysis, but we can speculate on this. First, based on the scores of presentations
weve given to various audiences over the years, the first part of the answer is it depends on the
recipient. Some individuals prefer to hear narratives, others prefer to see information presented
visually. In particular, we have found that policy analysts who are accustomed to working with
economic analysis tend to be more receptive to event data than those dealing with more
traditional diplomatic issues, which is unsurprising given the prevalence of quantitative
information (and the visual presentation thereof) in economic affairs.
Because event data are virtually the only current mode for aggregating and presenting
political interactions as a time series, it is likely to be an attractive tool for individuals who are
accustomed to absorbing data from charts quickly. We have also noted that, with very few
exceptions, the event data support the narrative and vice versa. On only one or two occasions,
usually involving the Goldstein averages, have we had to explain why we thought that the actual
situation was different than what the statistical series suggested. In several instancesnotably
the rise and fall of the road map mediation efforts and the continued high level of Palestinian
casualties despite the failure of newspapers to report thisthe statistical series provide insights
that would be much more difficult to establish through narrative materials.
Finally, as academic political scientists, it has been a valuable exercise for us to continually
track the accuracy of our analysis. Two cases have been particularly interesting. Our greatest
success was correctly (and unambiguously) predicting the failure of the road map process.
This was done largely on the basis of our detailed assessment of the situation drawn from the
specialized secondary literature, whereas our initial assessment prior to completing the
analysisbased largely on newspaper sourceshad been neutral to slightly optimistic. While
we did not use the event data to predict this collapse, the event data patterns track the shortlived nature of this process very clearly, and provide ample evidence of the accuracy of the
qualitative analysis.
The most conspicuous place where weand, frankly, pretty much everyone in the
IGO/NGO communityhave been consistently incorrect is assessing the impact of the high level
of economic deprivation that has accompanied the second intifada, particularly in Gaza. Our
assessment is that this would not be sustainable and could result in a major humanitarian crisis

Schrodt, Gerner and, Yilmaz

22

due to starvation or a serious outbreak of epidemic disease. This has not, so far, been the case,
although the predictions of imminent economic collapse continue. This may be the result of a
self-negating forecast: IGO and NGO warnings of impending crisis may have motivated
additional aidsuch as the 1-million-euro contribution made by European Union in early March
2004 to secure alternative accommodation for 13,000 Palestinians in the Gaza whose homes
have been demolished in recent months by Israeli security forces. (AP 2004) that has
prevented the crisis from worsening dramatically in the past year, although the situation has not
improved. We may simply have incorrectly gauged the magnitude of the problem, although
based on Gerners most recent trip to the region, we think this unlikely. Or the catastrophe may
be just around the corner.

Schrodt, Gerner and, Yilmaz

23

References
Note: Copies of the conference papers produced by KEDS project are available at the project
web site, http://www.ku.edu/~keds/papers.html.
Associated Press (AP). 2004. EU grants [euro]1 million in humanitarian aid to Palestinians. 5
March 2004. (Accessed 8 March 2004 on NEXIS http://web.lexisnexis.com/universe/)

Azar, Edward E. 1982. The Codebook of the Conflict and Peace Data Bank (COPDAB) .
College Park, MD: Center for International Development, University of Maryland.
Barry, Dave. 2003. The Dave Barry 2003 Desk Calendar. Kansas City, MO: Andrew McMeel
Publishing.
Burgess, Philip M. and Raymond W. Lawton. 1972. Indicators of International Behavior: An
Assessment of Events Data Research. Beverly Hills: Sage Publications.
Gerner, Deborah J., Philip A. Schrodt, Ronald A. Francisco, and Judith L. Weddle. 1994. The
Machine Coding of Events from Regional and International Sources. International
Studies Quarterly 38, 1: 91-119.
Gerner, Deborah J., and Philip A. Schrodt. 1998. The Effects of Media Coverage on Crisis
Assessment and Early Warning in the Middle East. In Early Warning and Early
Response, ed. Susanne Schmeidl and Howard Adelman. New York: Columbia University
Press-Columbia International Affairs Online.
Gerner, Deborah J., Rajaa Abu-Jabr, Philip A. Schrodt, and mr Yilmaz. 2002. Conflict and
Mediation Event Observations (CAMEO): A New Event Data Framework for the
Analysis of Foreign Policy Interactions. Paper presented at the International Studies
Association, New Orleans, March 2002.
Goldstein, Joshua S. 1992. A Conflict-Cooperation Scale for WEIS Events Data. Journal of
Conflict Resolution 36, 3: 369-385.

Schrodt, Gerner and, Yilmaz

24

Goldstein, Joshua S., and Jon C. Pevehouse. 1997. Reciprocity, Bullying and International
Cooperation: A Time-Series Analysis of the Bosnia Conflict. American Political Science
Review 91, 3: 515-530.
International Quaker Working Party. 2004. When the Rain Returns: Toward Justice and
Reconciliation in Palestine and Israel. Philadelphia: American Friends Service Committee.
McClelland, Charles A. 1976. World Event/Interaction Survey Codebook. (ICPSR 5211). Ann
Arbor: Inter-University Consortium for Political and Social Research.
Merritt, Richard L., Robert G. Muncaster and Dina A. Zinnes, eds. 1993. International Event
Data Developments: DDIR Phase II. Ann Arbor: University of Michigan Press.
Pevehouse, Jon C., and Joshua S. Goldstein. 1999. Serbian Compliance or Defiance in Kosovo?
Statistical Analysis and Real-Time Predictions. Journal of Conflict Resolution 43, 4:
538-546.
Rutenberg, Jim. 2003. Suffering News Burnout? Rest of America Is, Too. The New York
Times, 11 August 2003, Section C, page 1.
Schrodt, Philip A. 1994. "Event Data in Foreign Policy Analysis" in Laura Neack, Jeanne A.K.
Hey, and Patrick J. Haney. Foreign Policy Analysis: Continuity and Change. New York:
Prentice-Hall, pp. 145-166.
Schrodt, Philip A., and Deborah J. Gerner. 1997. Empirical Indicators of Crisis Phase in the
Middle East, 1982-1995. Journal of Conflict Resolution 41, 4: 529-552.
Schrodt, Philip A., and Deborah J. Gerner. 2000. Cluster-Based Early Warning Indicators for
Political Change in the Contemporary Levant. American Political Science Review 94, 4:
803-818.
Schrodt, Philip A. 2000. Automated Coding of International Event Data Using Sparse Parsing
Techniques. Fifth International Conference on Social Science Methodology, Cologne,
Germany, October.
Schrodt, Philip A. 2001. TABARI: Text Analysis by Augmented Replacement Instructions.
Paper presented at the International Studies Association meetings, Chicago, February.

Schrodt, Gerner and, Yilmaz

25

Schrodt, Philip A., and Deborah J. Gerner. 1994. Validity assessment of a machine-coded
event data set for the Middle East, 1982-1992. American Journal of Political Science,
38, 3: 825-854.
Schrodt, Philip A., and Deborah J. Gerner. 2001. Analyzing the dynamics of international
mediation processes. Presented at the International Studies Association joint conference,
Hong Kong, Peoples Republic of China, July.
Schrodt, Philip A., and Deborah J. Gerner. forthcoming. An Event Data Analysis of ThirdParty Mediation in the Middle East and Balkans. Journal of Conflict Resolution.
Schrodt, Philip A., Shannon G. Davis, and Judith L. Weddle. 1994. Political Science:
KEDSA Program for the Machine Coding of Event Data. Social Science Computer
Review 12, 3: 561-588.
Schrodt, Philip A., Deborah J. Gerner, and Erin M. Simpson. 2001. Monitoring conflict using
automated coding of newswire sources: a comparison of five geographical regions. Paper
presented at the PRIO/Uppsala University/DECRG High-Level Scientific Conference on
Identifying Wars: Systematic Conflict Research and Its Utility in Conflict Resolution and
Prevention, Uppsala. June 2001.
Schrodt, Philip A., Deborah J. Gerner, Rajaa Abu-Jabr, mr Yilmaz, and Erin M. Simpson.
2001. Analyzing the dynamics of international mediation processes in the Middle East
and Balkans. Presented at the annual meeting of the American Political Science
Association, San Francisco, 1 September.
swisspeace. 2003. FAST Update: Palestine Quarterly Risk Assessment November to December
2003. PDF file Palestine_1_04.pdf; downloaded from
http://www.swisspeace.org/fast/; accessed 26 February 2004.

Schrodt, Yilmaz and Gerner

26

Appendix: Instructions for updating FAST data sets

This appendix shows the current process we go through to update the data sets on a
quarterly basis. A number of the steps are idiosyncratic to our specific technical environment
and this should be seen as illustrative only. Over time we have gradually simplified the process
by automating various tasksfor example the calculation of the mediation scores used to involve
three separate runs of a program originally developed for the research reported in Schrodt et al
2001, but that program was subsequently modified to allow the statistics to be computed in a
single run. In the description, computer program names are in Helvetica font; file names
and Unix commands are in Courier font; elements of Web pages and other internal file
information such as Excel worksheet titles are in Times italic. All of the non-commercial
computer programs used here are open source; most are available on the KEDS project web site
and any others can be provided by the authors on request.

Acquiring AFP text

1. In a browser, go to LEXIS-NEXIS page on KU library electronic data sources page:
http://web.lexis-nexis.com/universe
2. Click "Sources" in the blue bar that goes across the top of the page. This will take you to a
page titled "Source List"
3. In the Source List "Find Title" box, enter "Agence France Press" and click "Find Title".
This will produce a list of several AFP sources.
4. Click the "Search this title" link of the first option (English) in the list. This will take you to a
standard NEXIS search page, except that it has now been set to search only AFP.
5. Enter the search strings below.
israel or palestin! or syria or jordan or lebanon or egypt or
plo
This string (and other search strings used in the KEDS project) can be found in the
www.input file that will be used below.
6. Change the "in" selection to "Full text"

Schrodt, Yilmaz and Gerner

27

7. Enter a series of dates that will give you fewer than 1000 storiesthis is the limit for the
stories returned by a NEXIS search on the Academic-Universe site. Sometimes this takes a
bit of experimenting, but a search period of two weeks generally works. Click "Search"
8. Click the first story in the resulting list. Copy the address, which will look something like
http://web.lexis-nexis.com/universe/document?
_m=69565bab69eb293ee7c03173007d31bd&_docnum=1&wchp=dGLbVzb-lSlzV
&_md5=7178503bdb21eac6c35eb90a4d666814

so be sure you get the whole thing. (Netscape messed this up in the past; MS-Explorer,
Safari and Mozilla seem to work fine.)
9. Use BBEdit to open the file "www.input" in the "download.dir" directory on the KU
Unix server raven. Paste the address you just copied into the first line of this file (there can
be more stuff in the file; the program just reads the first line) [BBEdit is a text editing
program for the Macintosh]
10. Log onto the raven account. Enter
cd download.dir

[puts you into the download subdirectory]

perl nexispider.pl

[runs the nexispider program]

11. The nexispider.pl program will ask for a file prefix; enter AFPLE and hit <return>
12. You should see the stories start scrolling bythis will go pretty quickly and periodically
pause when there is a delay retrieving the story.
[You'all young whippersnappers should know that in the old days, we had to do this over
a phone line with a 300 baud modem. Before we left to walk to school 5 miles barefoot in
the snow uphill both ways in July]
If the program stops immediately you probably gave it the URL of the NEXIS index, rather
than the URL of the first story; return to step [8].
13. Tricks if you are downloading a bunch of stories:
a. You can log into raven multiple times and have several copies of nexispider.pl going
simultaneously (this is why we use Unix systems rather than MS-Windows...).
nexispider automatically assigns non-conflicting file names to the resulting files based on
the prefix you entered in step [11] and the beginning and end of the download (it gets this
information from the file; you dont need to enter it).
b. As soon as nexispider.pl starts in any window, you can change the www.input file,
putting a new URL at the top of the file.
c. Click the "Edit search" option in the top bar of the NEXIS window to change the dates on
a search.
14. Now use the Unix "mv" command to move the file(s) to the appropriate subdirectory, e.g.
mv AFPLE.020601-020615 text_files/levant/yr2003/

Schrodt, Yilmaz and Gerner

28

where "text_files/levant/yr2003/" gets one into the year 2003 subdirectory inside
the levant subdirectory inside text_files, which is how we've got things set up at the
moment.
Hint: You can move a bunch of files at once using a wild-card
mv AFPLE.* text_files/levant/yr2003/
15. Move into that directory using the "cd" command
cd text_files/levant/yr2003/
16. Copy the contents of the subdirectory into a file named "filelist" using the command
ls > filelist
17. Open the file "filelist" in BBEdit using BBEdits Open from FTP server... menu
option (you'll need to go through the intermediate directories to get there) and delete any file
names that are not AFP downloads (this will be "filelist", "nexisreverse.pl" and
possibly some other left-over junk). The remaining files will be in chronological order
assuming they have the names given them by nexispider.
18. Run
perl nexisreverse.pl
which combines all of the files, puts the entries into correct chronological order (they are in
reverse order in the NEXIS output), and delete everything except the lead (first) sentence.
The output will be in a file with a ".rev" suffix; the original files are left unchanged.
19. Change the name of the output to something informative. This file is now ready to be run
through TABARI.

Coding and Aggregating

1. Go to Schrodt's G4 and go into the Data/FAST Data folder. This has a set of files that look
like
FAST YYYY-MM
Each folder contains CAMEO and WEIS dictionaries for doing the update, and a file
containing the data up until the end of the previous update (this will be data from the
beginning of the current year). There should also be a copy of the TABARI automated coding
program in there somewhere.
Make a new copy of this folder and re-name it with the last month (and year, if doing a new
year) of the data.
2. Open the new reversed data and the previous updated data, and append the new data to the
old. Rename this data set
AFPLE.YYYY-MM.L (e.g. AFPLE.2003-07.L)

Schrodt, Yilmaz and Gerner

29

where MM is the last month of the data. (The .L suffix indicates that these are leads only)
3. Move the copy of TABARI and the new text file into the CAMEO folder for the relevant
region. It will probably replace an existing folder, unless you've switched to a new year.
4. Update the project file
LVNT.PROJ.FASTYYMM (e.g. LVNT.PROJ.FAST0307)
by modifying the name of the text file (adjusting the month) and then also change the name of
that file (updating the month)
5. Run TABARI with the project file that is in the folder; A)utocode the new file. Rename the
output file
LVNT.WEIS.YYYY-MM (e.g. LVNT.WEIS.2003-07)
Delete the earlier version of the file (there is still a copy in the file you copied the directory
from)
6. Run this file through the program One-A-Day Filter; output file will have a .filt suffix.
7. Run the program Trim_Events on the filtered file; delete the .filt infix so that .trim
remains as the suffix.
8. Copy the trimmed file into K_Count folder in FAST Data. Edit the file
Goldstein.FAST.TAB
and edit the "Prefix" and "Files" fields to correspond to the new yy-mm. combination.
9. Run the program K-Count. Make a new folder FASTyy-mm and put all of the output files
(which also contain Israel-Lebanon information) plus the .trim file in that folder. Copy the
PAL>ISR and ISR>PAL files back to the FAST YYYY-MM folder
10. Make a copy of the file
FAST.YYMM.xls (e.g. FAST.0307.xls)
and update the final year-month infix. Open this file in MS-Excel.
11. Open the PAL>ISR and ISR>PAL files in BBEdit and transfer the results into the
Goldstein total and Goldstein average cells, adding months as needed. (Calculate the
averages using a calculator based on the Goldstein score and the number of events per month;
both are reported in the K-Count output files.). Select (i.e. click on) the various lines on the
embedded graphs and update the final cell of the series; note that this involves updating both
the $A entry to get the labels as well as the $B/$C entry to get the numbers.
12. Run the FAST_Mediation program with the *.trim file as input and YY-MM as the
suffix. The file
Mediate.out.YY-MM (e.g. Mediate.out.03-07)

Schrodt, Yilmaz and Gerner

30

contains the counts of mediation events for the UN, USA, and Europe. (Note: just enter the
suffix, not the entire file name, since the latter will cause a buffer overflow and freeze the
computer...)
13. Enter these figures into the Mediation worksheet of the FAST.YYMM.xls file; update the
final cells of the series.
14. Update the Documentation worksheet in the spreadsheet.

Updating the casualty data

1. In a browser, go to
http://www.btselem.org/
click the Statistics link, then on that page, the Monthly Totals link under Fatalities.
(Note that the data may or may not have been compiled to the end of the month. If it hasnt,
wait a few days until it has been completed)
2. Get the total of the following figures:
Table: Israelis Killed in the Occupied Territories (Including East Jerusalem)
Israeli civilians killed by Palestinians
Israeli security forces personnel killed by Palestinians
Table:Persons Killed within the Green Line
Israeli civilians killed by Palestinians
Israeli security forces personnel killed by Palestinians
Total these, and enter in the Deaths worksheet of the FAST.YYMM.xls file.
3. Go to
http://www.palestinercs.org/crisistables/table_of_figures.htm
Again, note that the data may or may not have been compiled to the end of the most recent
month, although usually PRCS does daily updates.
4. Copy the appropriate monthly totals from the Total Deaths column. Enter this total in the
Deaths worksheet of the FAST.YYMM.xls file; update the final cells of the series.

Updating the WEIS data

1. Make a copy of the Data/Levant.WEIS.yy.mm folder, rename with the new final date.

Schrodt, Yilmaz and Gerner

31

2. Folder for updating the .summary file is currently Vinland G3/Programming/K_Count.03.

Move the LVNT.WEIS.yyyy-mm.trim into thiscurrently this is moving across volumes
on the G3 so it will automatically make a copy.
3. Edit the Goldstein.TAB file, changing the lines
PREFIX=LVyy-mm. [near top]
LVNT.WEIS.yyyy-mm.trim

[near bottom]

4. Run the program KEDS_Count, which is the product of an ancient Pascal compiler, takes
forever to run, and produces a very large number of individual dyadic files.
5. Edit the file merge.files to correspond to the new prefix. Run the program KEDS.Merge.pl,
which produces the file merge.output. This contains the new summary information.
6. Open Levant.79-yy.mm.summary and merge.output in BBEdit. Cut and paste the
new data into the .summary file, and change the name. Wouldnt hurt to read this into Excel
and plot a few of the series to make sure nothing weird has happened.
7. Open LVNT.WEIS.yyyy-mm.filt and add new records from this to LEVANT.WEIS.
8. Edit Levant.WEIS.ReadMe to reflect the new datesthese occur in three or four different
places and this probably needs some additional re-writing. Also update the total event count.
9. Stuff and zip the files using the StuffIt utility; when doing the .zip version, change the
preferences so that the text files will be converted to MS-DOS line endings.
10. Upload to the KEDS web site, and change the HTML text to reflect the new dates and event
totals. Test the downloads to make sure they work correctly.