See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/2802361

Predictable Parallel Performance: The BSP Model

Article · September 2000

DOI: 10.1007/978-1-4757-3609-0_4 · Source: CiteSeer

CITATIONS READS
5 87

1 author:

David Skillicorn
Queen's University
254 PUBLICATIONS   3,839 CITATIONS   

SEE PROFILE

All content following this page was uploaded by David Skillicorn on 09 January 2015.

The user has requested enhancement of the downloaded file.

Chapter 1

PREDICTABLE PARALLEL
PERFORMANCE: THE BSP MODEL
D.B. Skillicorn

Department of Computing and Information Science

Queen's University
Kingston Canada
[email protected]

Abstract There are three big challenges for mainstream parallel computing: build-
ing useful hardware platforms, designing programming models that are
eective, and designing a software construction process that builds cor-
rectness into software. The rst has largely been solved, at least for
current technology. The second has been an active area of research for
perhaps fteen years, while work on the third has barely begun. In
this chapter, we describe the Bulk Synchronous Parallel (BSP) model
which, at present, represents the best compromise among programming
models for simplicity, predictability, and performance. We describe the
model from the a software developer's perspective and show how its
high-level structure is used to build ecient implementations. Almost
alone among programming models, BSP has an associated cost model
so that the performance of programs can be predicted on any target
without laborious benchmarking. Some progress towards software con-
struction has also been made in the context of BSP.

1. APPLYING PARALLEL COMPUTING

Parallel computing requires at least three distinct components to be in
place before it becomes eective. The rst is the parallel hardware that
will execute parallel applications; the second is the abstract machine,
or programming model, in which parallel applications will be written;
and the third is the software design process that allows applications to
be built from specications [39]. All three are essential if parallelism is
to be a mainstream technology for computation; as it will have to be

1
2

to meet the demands of high-end commercial and scientic applications

such as data mining and simulation.
The past fty years have seen the development of a wide variety of
parallel architectures. In the past decade, this variety has gradually
reduced to two main alternatives: shared-memory MIMD systems of
modest size (perhaps 30 processors) using cache coherence, sometimes
gathered into larger ensembles that have come to be called clumps [15];
and distributed-memory MIMD systems based on commodity processor
boards and o-the-shelf interconnects, called clusters [1]. This picture
may change with the widespread deployment of optical computing de-
vices or other new technology, but at present almost all deployed systems
are of these two kinds [24].
The second requirement for eective parallel computing is an abstract
machine or parallel programming model, forming the bridge between the
hardware and the level at which software is written. All computers con-
sume instructions much faster than they can be written. An abstract
machine is only useful if some of its instructions can generate much
larger sets of instructions for the hardware in a more or less automatic
way. In sequential computing, loops play this role. In parallel comput-
ing, we need to include abstract machine instructions that can generate
activity in many processors at the same time. A parallel computer can
consume trillions of instructions in a short period of time, and also move
data between processors; we clearly need abstractions that can insulate
software developers from expressing this amount of detail directly.
Finding such abstractions has proved to be enormously dicult, be-
cause the following properties are essential [35]:
Portability { parallel computers, even of the same general kind,
dier in their internal arrangements; it is important to be able to
move programs from one to another without having to redesign
them.
Eciency { parallel programs exist for performance, so they must
exploit a signicant amount of the performance available from the
platforms on which they execute.
Simplicity { the speed with which software can be constructed
seems relatively invariant in terms of semantic units; the larger
these can be, the faster it is to build eective code.
Predictable performance { there are many ways to implement any
given specication and it is not possible to try all of them; therefore
software developers must be able to judge the (true) complexity of
their code as it is being developed.
 The BSP Model 3

The third requirement for eective parallel computing is the existence

of a process for moving from specication to program. We have managed
without such a process in sequential computing, at considerable cost. In
parallel computing, the wide range of possible platforms means that test-
ing cannot more than skim the surface of a program's behaviour. Hence,
it is much more important that programs be correct by construction.
So far, three major advances towards a complete parallel computing
environment have taken place. The rst was the development, in the
1980s, of higher-order functional programming implemented using graph
reduction. This was the rst parallel programming model that provided
a signicant abstraction from the underlying hardware, and which also
provided a way to build software (equational transformation) [27, 22,
21]. This model met the requirements for portability and simplicity.
Unfortunately, it could not achieve either eciency or predictability,
although progress towards the former continues to be made.
The second advance was the development of a model that was both
portable and ecient across a range of architectures, the Bird-Meertens
theory of lists [33]. This model met the requirements of portability, sim-
plicity, eciency, and predictability, and also possessed a software con-
struction methodology, again based on equational transformation and
initiality [35]. However, it was limited because it assumed that inter-
connection topology was important, and also because it applied only to
problems whose data could be represented using some particular, though
fairly general, structures.
The third advance was the development of Bulk Synchronous Paral-
lelism (BSP), [40, 38, 14] which also meets all four requirements, but does
not have the same restrictions on the patterns of data or communication.
It can reasonably claim to be the rst model that is expressive, while still
achieving portability, eciency, and predictable performance. BSP is by
no means the nal answer to the problem of parallel programming, but
it is a signicant step forward. Its major weaknesses are: it is relatively
low-level in the sense that programs have to explicitly partition the data
and the computations; and it is cannot naturally expressed dynamic and
irregular computations. The open problem in parallel programming is
how to take the next step to a model that retains the good properties of
BSP while simplifying further the programming framework.
2. INTRODUCTION TO BSP
BSP is a programming model whose abstract machine consists of a
set of processors and their local memories, connected by a network. In
other words, it is a distributed-memory abstraction. This means that
4

Figure 1.1 A superstep

Virtual Processors

Local
Computation

Global
Comms
Barrier

almost any real parallel computer can readily simulate the BSP abstract
machine with little loss of performance. It means that some features of
shared-memory computers cannot be directly exploited [38].
The essence of the BSP model is the notion of a superstep which can
be considered the basic parallel instruction of the abstract machine. A
superstep is a global operation of the entire parallel computer, and is
decomposed into three phases:
Local computation within each processor on local data.
Non-blocking communication between processors.
A barrier synchronisation of all processors, after which all commu-
nication is guaranteed to have completed, and the moved data
becomes visible to the destination processors.
Communication and synchronisation are completely decoupled. This
avoids one of the big problems with e.g. message passing, where each
communication action overloads both data movement and synchronisa-
tion. A single process is easy to understand; but the global state that
results from the parallel composition of more than one process is ex-
tremely hard to understand. When a send is written in a process, the
entire global state must be understood in order to know whether the op-
eration is safe or will cause deadlock. Thus the send-receive abstraction
of message passing is easy to understand in the abstract but hard to use
in practice.
 The BSP Model 5

Barrier synchronisation collapses the global state at the end of every

superstep. Every process that exits the barrier knows that every other
process has reached it. This control of the state makes software easier
to write and understand. It can also be exploited by the runtime system
to make optimisations that are not possible in other systems.
The rst question that is always asked about supersteps is this: isn't it
ridiculously inecient to wait until the end of the communication phase
to start moving data? Overlapping computation and communication
seems like an obvious optimisation. In fact, however, the most that can
be saved by a perfect overlap is a factor of two, and it is often much less
in practice. Postponing communication allows many optimisations that
produce much larger improvements in performance.
The second question that is always asked about supersteps is: aren't
barrier synchronisations ridiculously expensive? The answer is that they
are if they are done poorly, as they are in many implementations, even
by manufacturer's who ought to know better. BSP works hard to make
barriers extremely fast. (In fact, barriers are one of the easiest things to
arrange physically { the Papers project [9] at Purdue implements barriers
in 10s of nanoseconds. There has not been enough demand from users
for manufacturers to include barrier hardware in their systems.)
3. IMPLEMENTATION OF BSP
Although BSP is an MIMD programming model, its most popular im-
plementation is an SPMD one { the BSPLib library, sometimes called
the Oxford BSP Toolset [19]. This library was designed by an interna-
tional group and implemented at Oxford. It is freely available and runs
on a wide variety of platforms. See www.bsp-worldwide.org for more
details.
Two styles of communication are available in the library:
Direct Remote Memory Access (DRMA) in which processes
can copy data from or to a remote process's memory, without the
active participation of the remote process.
Bulk Synchronous Message Passing (BSMP) in which pro-
cesses send a packet to a queue in a remote process, which can be
accessed by the remote process in the following superstep.
The BSPLib library is small compared to message passing libraries.
It contains 20 operations, compared to 129 in MPI-1, 202 in MPI-2, and
90 in PVM. The BSPLib operations are shown in Figure 1.1
The standard way of communicating between processors is to use the
DRMA operations of the library. These allow data copies to be initiated
6

Table 1.1 Basic BSPLib primitives

Class Operation Meaning
Initialisation bsp begin Start of SPMD code
bsp end End of SPMD code
bsp init Simulate dynamic processes
Halt bsp abort One process halts all
Enquiry bsp pid Find my process id
bsp nprocs Number of processes
bsp time Local time
Superstep bsp sync Barrier synchronisation
DRMA bsp push reg Make region globally visible
bsp pop reg Remove global visibility
bsp put Copy to remote memory
bsp get Copy from remote memory
BSMP bsp set tag size Choose tag size
bsp send Send to remote queue
bsp qsize Number of messages in queue
bsp get tag Match tag with message
bsp move Move from queue
High Performance bsp hpput Unbuered versions
bsp hpget of communication
bsp hpmove primitives

by one processor to (put) or from (get) the memory of another processor

without interrupting the remote processor.
To simplify the writing of computational code, puts and gets may
appear textually anywhere within a superstep. In normal use, data are
copied from program data structures at the occurrence of a put so that
data structures may be reused during the superstep (transfer of the data
between processors does not take place until the end of the superstep).
High-performance variants exist which do not buer data, and which
may begin communication before the end of the superstep. These are
harder to use, since program data structures are o-limits to the program
for the remainder of the superstep, once they have been mentioned in a
data transfer operation. A simple BSP program is shown in Figure 1.2.
An alternative set of data transfer operations, bulk synchronous mes-
sage passing, exist for the situation where data distribution is irregular
 The BSP Model 7

Figure 1.2 The simplest BSP program

Example: \Hello World"
void main(void) {
int i;
bsp_begin(bsp_nprocs());
for(i=0; i<bsp_nprocs(); i++) {
if (bsp_pid()==i)
printf("Hello from process %d of %d\n",
i,bsp_nprocs());
fflush(stdout);
bsp_sync();
}
bsp_end();
}

and only the data recipient knows where incoming data may be safely
stored.
The message-passing style typical of MPI and PVM has the following
drawbacks:
It is possible to write programs that deadlock, and it is dicult to
be sure that a given program will not deadlock;
The ability to use a send or receive anywhere leads to spaghetti
communication logic { message passing is the communication ana-
logue of goto control ow;
There is no global consistent state which makes debugging and
reasoning about programs extremely dicult;
There is no simple cost model for performance prediction, and
those that do exist are too complex to use on programs of even
moderate size;
On modern architectures with powerful global communication, they
cannot use the full power of the architecture.
BSP avoids all of these drawbacks. Provided BSP's apparent inecien-
cies can be avoided, it is clearly a more attractive programming model,
at least for algorithms that are fairly regular.
It is possible to program in a BSP style using libraries such as MPI
which provide an explicit barrier operation. However, many global op-
timisations cannot be applied in this setting, and the library operations
8

themselves are not optimised appropriately. For example, the MPI bar-
rier synchronisation on the SGI Power Challenge is 32 times slower than
BSPlib's barrier. Replacing MPI calls in a superstep structured program
with BSPLib calls typically produces a 5-20% speedup.
BSPLib is available for the following machines: Silicon Graphics Power
Challenge (Irix 6.x), Origin 2000; IBM SP; Cray T3E, T3D, C90; Parsytec
Explorer; Convex SPP; Digital 8400 and Digital Alpha farm; Hitachi
SR2001; plus generic versions on top of either UDP/IP, TCP/IP, sys-
tem V shared memory, MPI, and certain clusters.
4. BSP ALGORITHM DESIGN
BSP has a simple but accurate cost model that can be used for per-
formance prediction at a high level, before any code is written. Hence, it
provides the basis for parallel algorithm design, since the costs of dier-
ent potential implementations can be compared without building them
all.
There are three components to the cost of a BSP superstep: the com-
putation that occurs locally at each processor, the communication that
takes place between them, and the cost of the barrier synchronisation.
Costing the local computation is uses the conventional approach of
counting instructions executed. This approach, however, is becoming
increasingly fragile, not only for BSP but for other parallel and, for that
matter, sequential models. The reason is that modern processors are
not simple linear consumers of instructions with predictable execution
times. Instead, the time a given instruction takes depends strongly on
its context: the instructions that execute around it and their use of
functional units, and whether or not its arguments are present in the
cache. We have been exploring better models for instruction execution,
but this is an issue bigger than parallel programming.
Costing communication is an area in which BSP has made a major
contribution to our understanding of parallel computation. Most other
cost models for communication are bottom up in the sense that they try
to compute the cost of a single message, and then sum these costs to get
an overall communication cost. Typically, the cost of a single message
is modelled as a startup term followed by a per-byte transmission time
term. The problem with this approach is that the dominant term in the
per-message cost often comes from congestion, which is highly nonlin-
ear and prevents the costs of individual messages being summed in a
meaningful way.
Such communication cost models implicitly assume that the bottle-
neck in a network is somewhere in the centre, where messages collide.
 The BSP Model 9

Figure 1.3 Costing a superstep

wi

hi

This may be a fair assumption for a lightly loaded network, but it does
not hold when a network is busy. In typical parallel computer networks,
the bottlenecks are at the edges of the network, getting trac into and
out of the communication substrate itself [13, 16]. This continues to be
true even for clusters where many of the protocol and copy overheads of
older networks (e.g. TCP/IP) have been avoided [11].
If it is the network boundary that is the bottleneck, then the cost of
communication will not be dominated by the distance between sender
and receiver (i.e. the transit time), nor by congestion in transit, nor by
the communication topology of the network. Instead, the communica-
tion that will take the longest is the one whose processor has the most
other trac coming from or to it. This insight is at the heart of the
BSP cost model, and has led to its adoption as a de facto cost model for
other parallel programming models too.
BSP has a further advantage because of its superstep structure { it is
possible to tell which communication actions will aect each other: those
from the same superstep. The communications actions of a superstep
consist of a set of transfers between processors. If the maximum volume
of data entering and leaving any processor is h then this communication
pattern is called an h-relation. The cost of the communication part of
the superstep is given by hg, where g captures the permeability of the
network to high-load trac. It is expressed in units of instruction times
per byte, so that hg is in units of instruction times. Hence a small value
for g is good.
The cost of the barrier synchronisation is similarly expressed in units
of instruction times.
10

The overall cost of a superstep is given by:

| {z w}i + MAX
superstep cost = MAX | i  g + }l
h{z
Computation Communication
In the BSP cost model, a communication pattern in which one proces-
sor sends a value to each of the others, and one in which all processors
send a value to all other processors are treated as if they cost the same
{ they are both p-relations. However, the total amount of data being
moved is p in the rst case and p2 in the second. Can these two pat-
terns really have the same cost? In practice, they don't quite; but the
time taken versus total amount of trac is linear and quite at, not the
quadratic that might have been the more intuitive prediction. The BSP
cost model doesn't quite hold, but it's much more accurate than other
cost models.
Because each superstep ends with a global synchronisation event, the
cost of a series of supersteps is just the sum of the costs of each separate
superstep.
Some representative performance gures for some common parallel
computers are shown in Figures 1.4 and 1.5.
A BSP DESIGN STRATEGY
The BSP cost model makes it clear that the following strategies should
be used to write ecient BSP programs:
balance the computation in each superstep between processes, since
the computation time for a superstep is a maximum over the com-
putation times in each process, and the barrier synchronisation
must wait for the slowest process;
balance the communication between processes, since h is a maxi-
mum over sent and received data; and
minimise the number of supersteps, since this determines the num-
ber of times l appears in the nal cost.
As an example of the eectiveness of balance, we consider a basic
building block: broadcasting the same value from one processor to all of
the others. Suppose that the message to be sent is of size N and that
there are p processors.
The simplest implementation (Figure 1.6) is just to send the message
out p times (so that the communication is a tree of depth 1 and width
p).
 The BSP Model 11

Figure 1.4 Performance for manufactured systems

Machine s p l l=s g g=s
ops s ops/wd s/wd
O2000 101 2 804 7.9 8.3 0.08
4 1789 17.9 10.2 0.10
8 3914 39.8 15.1 0.15
16 15961 158.5 44.9 0.45
32 39057 386.6 66.7 0.66
62 138836 1374.2 121.2 1.20
PowerChallenge 74 2 1132 15.3 10.2 0.14
4 1902 25.7 9.3 0.13
Cray T3E 47 2 269 5.7 2.14 0.05
4 357 7.6 1.77 0.04
8 506 10.8 1.64 0.03
16 751 16.0 1.66 0.04
SP2 switch 26 2 1903 73.2 7.8 0.30
4 3583 137.8 8.0 0.31
8 5412 208.2 11.4 0.43
Cray T3D 12 2 5.6 0.3 0.02
8 175 14.4 0.8 0.07
64 148 12.3 1.7 0.14
256 387 32.1 2.4 0.20
12

Figure 1.5 Performance for clusters

Machine s p l l=s g g=s
ops s ops/wd s/wd
SP2 26 2 1903 73.2 7.8 0.30
4 3583 137.8 8.0 0.31
8 5412 208.2 11.4 0.43
Alpha farm 10 2 17202 1703.1 81.1 8.0
3 34356 3401.6 83.0 8.2
4 47109 4664.3 81.3 8.1
Pentium NOW 61 2 52745 870.4 484.5 8.00
266MHz, 64Mb 4 139981 2309.9 1128.5 18.62
10Mbit shrd enet 8 826054 13631.3 2436.3 32.91
Pentium II NOW 88 2 13683 155.5 37.8 0.43
400MHz, 128Mb 4 27583 313.4 39.6 0.45
100Mbit switch enet 8 38788 440.8 38.7 0.44
Pentium II Krnl-NOW 88 2 5654 64.2 33.5 0.38
400MHz, 128Mb 4 11759 133.6 31.5 0.36
100Mbit switch enet 8 18347 208.5 30.9 0.35

Figure 1.6 Naive broadcast

Source

Destination
 The BSP Model 13

Figure 1.7 Tree broadcast

Superstep 1

Superstep 2

Superstep 3

Figure 1.8 Two stage broadcast

Source

Destination

Superstep
One

Superstep
Two

The cost of this implementation is:

Naive broadcast cost = pNg + l (1.1)
A second possible implementation (Figure 1.7) is to have the source
of the broadcast send it to two other processors. These in turn send it
on to two others, and so on (so the communication forms a tree of depth
log p).
The cost of this implementation is:
Tree broadcast cost = (log p)Ng + (log p)l (1.2)
The communication cost has been reduced, but at the expense of more
barrier synchronisations.
There is a third way to implement broadcast (Figure 1.8). In the rst
step, the source processor sends 1=pth of the data to each of the other
processors. In the second step, all of the processors send their piece of
the data to all of the other processors [2].
14

Table 1.2 Costs for broadcasting

Algorithm Communication Sync

One-stage pNg l
Tree (log p)Ng (log p)l
Two-stage 2Ng 2l

The cost of this implementation is:

N  N 
Two stage broadcast cost = pg + l + pg + l (1.3)
p p
= 2Ng + 2l (1.4)
It is easy to see that this implementation is much better than either of
the two previous ones. The communication term is smaller than direct
broadcast, while the number of supersteps is a small constant. This
technique is an extremely good way to handle broadcasts, but it seems
not to have been known before BSP, and it is still not used as often as it
should be because conventional cost metrics make it look too expensive.
It requires the insight to see that the total exchange in the second step
is not ridiculously expensive. Table 1.2 illustrates the dierent costs of
these three implementations.
The only time when it is worth using the one-stage broadcast is when
N is small, i.e. N < l=(pg ? 2g), and it is never worth using the tree
broadcast.
Does the cost model re ect reality for this technique? Figure 1.9 shows
the measured cost of the three dierent implementations on a powerful
parallel computer (the Cray T3E).
5. BSP OPTIMISATION
BSPLib works quite hard to optimise performance, but it's an un-
usual kind of optimisation. Instead of optimising programs, it optimises
the BSP abstract machine (Figure 1.10). To put it another way, all
of the optimisations result in improved values for g and l, rather than
improvements to particular parts of particular programs [10].
Using global knowledge of the communication patterns in BSPLib
programs, improves the global communication bandwidth g. Some of
the standard techniques used by BSPLib are shown in Table 1.3. They
are discussed in the next few subsections.
 The BSP Model 15

Figure 1.9 Broadcasting 100,000 doubles on a p = 20 Cray T3E

0.06
BSPlib: One stage broadcast
CRI/EPCC MPI
BSPlib: Two stage broadcast

0.05

0.04
Time in seconds

0.03

0.02

0.01

0
2 4 6 8 10 12 14 16 18 20
Number of processors on a Cray T3E

Figure 1.10 Optimising the model rather than programs

Program Abstract Executing Abstract
Machine Program Machine
Optimisation Optimisation
Executing Real Real
Program Machine Machine

Conventional BSP

Table 1.3 BSPLib optimisations

Optimisation Kind Improvement

Message packing local small packet bandwidth
Destination scheduling global congestion
Pacing pairwise congestion
Barrier implementation global l
16

MESSAGE PACKING
On most computers, there is a signicant startup cost to initiating
a transfer of data. Even if the network itself introduces only minor
overheads, it's not unusual to require a system call with its overheads
of saving and restoring state. The BSP cost model claims that sending
1000 1-byte messages from a processor should cost the same as sending
a single 1000-byte message, and that seems unlikely.
However, BSP has one advantage over conventional message passing
systems { it is not required to begin transmitting data at the point where
the program indicates that it should. Instead, a BSP implementation
can wait until the end of each superstep before initiating data transfers.
This makes it possible to pack all of the trac between each pair of
processors into a single message so that the overhead is paid only once
per processor pair rather than once per message. In other words, no
matter how the programmer expresses the messaging requirements, BSP
will treat them the same: 1000 1-byte messages become a single 1000-
byte message before leaving the sending processor. The communication
interface sees a single transfer regardless of the actual sequence of puts
in the program text.
DESTINATION SCHEDULING
Each processor knows that, when it is about to start transmitting
data, other processors probably are too.
If all processors choose their destinations in the same way, say Pro-
cessor 0 rst, then Processor 1, and so on, then Processor 0 is going
to be bombarded with messages. It will only be able to upload one of
them, and the rest will block back through the network. This situation
is particularly likely in SPMD programs, where puts to the same desti-
nation will tend to be executed simultaneously because each processor
is executing the same code.
Because BSPLib does not transmit messages as they are requested, it
is able to rearrange the order in which they are actually despatched; in
particular each processor may use a dierent despatch order even though
the same order was implied by the puts executed by each processor.
Choosing a schedule that makes collisions at the receivers unlikely has a
large eect on performance. It can be done by choosing destinations ran-
domly, or by using a latin square as a delivery schedule. Table 1.4 shows
this eect on a variety of computers. Notice that in almost every case,
the factor of two lost by not overlapping computation and communica-
tion is earned back; in the more complex case of the NIC cluster, where
 The BSP Model 17

Table 1.4 8-processor total exchange in s. small=16k words per processor pair,
big=32k words.
latin square dst order factor
SP2 big 102547 107073 1.0
T3D big 14658 29017 2.0
PowerChallenge big 37820 47910 1.3
Cluster(TCP) small 61336 119055 1.9
big 134839 248724 1.8
Cluster(NIC) small 39468 76597 1.9
big 78683 14043686 178

the huge increase in time happens because the central switch saturates,
using a careful schedule prevents a performance disaster.
PACING
Each processor knows that, when it is about to start transmitting
data, other processors probably are too.
Networks all contain bottlenecks { as the applied load increases, the
throughput eventually drops. It is always useful to control (throttle) in-
put rates to maintain the applied load at (just below) the region of max-
imum throughput. If the applied load is maintained slightly below the
point of maximum throughput, then uctuations in load create slightly
larger throughputs which tend to restore equilibrium. On the other
hand, if the applied load goes above the point of maximum throughput,
then positive feedback tends to drive the network into congestion and
collapse. The ability to control applied load makes it possible to keep
the network running at its maximum eective capacity.
This can be done in two ways:
High-level. The target's g value gives its permeability to data under
conditions of continuous trac { and this can easily be converted
into units of secs/Mbit, say. There is no point in trying to insert
data into the network faster than this value, because g is the best
estimate of what the network's ideal throughput value is.
Low-level. When the communication medium is shared (e.g. Ether-
net), using a slotting system with a carefully-chosen slot size re-
duces collisions (a distributed approximation to the latin square
scheme). For TCP/IP over Ethernet, this improves performance
by a factor of 1.5 over mpich.
18

Table 1.5 Shared-memory barriers (times in s, 4 processors.)

SGI SUN DEC

central semaphore 320.8 129.3 97.9
manufacturer 33.5 | |
tree of semaphores 335.6 167.6 89.7
cache coherence 6.6 2.2 3.8

Table 1.6 Distributed-memory barriers (times in s, 4 processors).

Powerchallenge SP2
SGI MPI Arg MPI IBM mpl BSPLib
provided 73.5 217.7 197.9 |
message tree 74.1 165.9 95.6 |
central message 92.0 234.0 156.2 |
total exchange 105.7 224.9 124.3 137.8

BARRIER IMPLEMENTATION
Most standard barrier implementations are quite poor. Because bar-
riers are so important in BSP, BSPLib implements them very carefully
[17].
On shared-memory architectures, a technique that exploits cache co-
herency hardware is used to make barriers very fast. Table 1.5 shows
some performance data for dierent barrier implementations on shared-
memory systems.
On distributed-memory manufactured architectures, access to the com-
munication substrate is usually possible only through a manufacturer's
API that is already far removed from the hardware. This makes it
dicult to build fast interfaces. Some performance data are shown in
Table 1.6. These times are close to the sum of the times required to send
individual messages to p ? 1 destinations.
Total exchange is an attractive way to implement barriers in the con-
text of global exchange of data, although once again this is counterin-
tuitive. Rather than exchanging single bits at the barrier, BSPLib ex-
changes the sizes of messages about to be communicated between each
pair of processors. This costs almost nothing, but makes new optimi-
sations possible because the receiver knows how much data to expect.
Distributed-memory implementations of BSPLib execute the barrier im-
 The BSP Model 19

Table 1.7 Optimisations for clusters

Optimisation Kind Improvement

Hacking protocol stack local low-latency messaging
Hole lling error recovery pairwise # retransmissions
Resource level acking pairwise explicit acks
Barrier acking global explicit acks
Shared buers local buer avail

mediately after the local computation phase. Since each processor knows
how much data to expect, it is simple to preserve the semantics of the
program barrier.
The opportunities for optimisation are constrained because BSPLib
is a library and does not have access to e.g. the kernel, where further
optimisation opportunities exist.
6. OPTIMISATION IN CLUSTERS
It is now possible to build a distributed-memory parallel computer
from o-the-shelf components (a cluster). Such systems are charac-
terised by: high performance processor boards, a simple interconnection
network such as switched Ethernet or Myrinet, and Network Interface
Cards (NICs) to handle the interface between processors and network.
Protocols such as TCP/IP that were originally designed for long-haul
networks are not appropriate for such an environment. Instead, new
protocols have been designed to provide low latencies (end-to-end laten-
cies in the 10s of microseconds are typical) and as few data copies as
possible [4, 41, 28, 6]. Changes to the kernel to make it possible to access
the network without a system call are also common.
When the target is a cluster, with full access to the kernel, commu-
nication hardware, and protocols, further optimisations to BSPLib are
possible [12, 11]. Some of them are listed in Table 1.7.
HOLE FILLING ERROR RECOVERY
TCP/IP is a reliable protocol, but it was designed for an error model
that is not relevant to clusters. Clusters do lose packets, but almost in-
variably because of buer problems at the receiving end. High-performance
protocols need to provide reliability in a dierent way.
20

TCP error recovery keeps track of the last packet acknowledged. Once
a packet is lost, everything after it is also resent. This is a poor strategy
if errors are not in fact bursty.
In the BSPLib cluster protocol, packets contain an extra eld with
the sequence number after the last one in sequence (i.e. acknowledging
both a hole, and the rst packet past the end of a hole). Only the actual
missing packets need to be resent.
REDUCING ACKNOWLEDGEMENT TRAFFIC
Acknowledgements are piggy-backed onto normal trac, and therefore
use no extra bandwidth, provided that trac is symmetric. However,
often it isn't, and so the general strategy is to avoid acknowledgements
for as long as possible, preferably for ever. Also, because each processor
knows what it is expecting, recovery is driven by the receiver, not by
the sender. Because of this, a sender need not know for sure whether
a packet has been received for a very long time, unless it needs to free
some send buers.
JUST IN TIME ACKNOWLEDGEMENTS
Acknowledgements are forced by a sender prediction that it will run
out of buers (because it must keep copies of unacked data). It uses
knowledge of the roundtrip message time to send ack requests just in
time.
MANAGING BUFFERS MORE EFFECTIVELY
Because the architecture is dedicated, and communication patterns
will often be skewed, it makes sense to use a single buer pool at each
processor (rather than separate buer pools for each pair of processors).
The combination of these optimisation makes the BSPLib cluster pro-
tocol one of the fastest reliable protocols. Its performance is illustrated
in Figures 1.11, 1.12, and 1.13.
These optimisations have a direct eect on application level perfor-
mance. This is shown in Tables 1.8 and 1.9, using some of the NAS
benchmarks.
The combined eect of these communication optimisations is that the
communication performance of BSPLib is about a factor of four better
than comparable libraries.
 The BSP Model 21

Figure 1.11 Link performance { round-trip times in s

PII-BSPlib-NIC-100mbit-wire
PII-BSPlib-NIC-100mbit-2916XL
500 PII-BSPlib-UDP-100mbit-2916XL
PII-BSPlib-TCP-100mbit-2916XL
PII-MPI-ch_p4-100mbit-2916XL
SP2-MPI-IBM-320mbit-vulcan
SP2-MPL-IBM-320mbit-vulcan
O2K-MPI-SGI-700mbit-ccnuma
400
Round-trip time (usecs)

300

200

100

0
4 8 16 32 64 128 256 512 1024
Message size (bytes, log2 scale)

Figure 1.12 Link performance { bandwidth in Mbps

120 PII-BSPlib-NIC-100mbit-wire
PII-BSPlib-NIC-100mbit-2916XL
PII-BSPlib-UDP-100mbit-2916XL
PII-BSPlib-TCP-100mbit-2916XL
PII-MPI-ch_p4-100mbit-2916XL
100 SP2-MPI-IBM-320mbit-vulcan
SP2-MPL-IBM-320mbit-vulcan
O2K-MPI-SGI-700mbit-ccnuma

80
Bandwidth (Mbps)

60

40

20

0
0 200 400 600 800 1000 1200 1400
Message size (bytes)
22

Figure 1.13 Link performance { latency per packet in s

512

256
Latency per packet (usecs, log2 scale)

128

64

32

PII-BSPlib-NIC-100mbit-2916XL
16 PII-BSPlib-UDP-100mbit-2916XL
PII-BSPlib-TCP-100mbit-2916XL
PII-MPI-ch_p4-100mbit-2916XL
SP2-MPI-IBM-320mbit-vulcan
SP2-MPL-IBM-320mbit-vulcan
O2K-MPI-SGI-700mbit-ccnuma
8
1 4 16 64 256 1024
Number of packets (log2 scale)

Table 1.8 NAS benchmark performance (M ops/s per process).

p SP2 Cluster Cluster Origin 2000

(MPI) (MPI) (BSPlib) (MPI)
BT 4 31.20 51.57 56.18 51.10
SP 4 24.89 33.15 42.36 56.22
MG 8 36.33 21.02 39.62 36.16
LU 8 38.18 46.34 63.09 87.34

Table 1.9 NAS benchmarks communication times

p computation communication time
time BSPlib mpich

BT 4 724:6s 24:2s 91:1s 3.8

SP 4 458:9s 42:8s 182:2s 4.3
MG 8 11:1s 1:1s 4:6s 4.2
LU 8 205:9s 30:5s 115:9s 3.8
 The BSP Model 23

7. SOFTWARE DEVELOPMENT
The third requirement for successful parallel programming is the exis-
tence of a process for developing programs from specications. This is, of
course, also important for sequential computing; but parallel computing
adds an extra spatial dimension to all of the complexities of sequential
programming and a software construction process is therefore even more
critical.
Formal methods for software development in sequential settings often
have large overheads to generate a single program statement. Partly
for this reason, they have made little inroad into practical software con-
struction. A big dierence in a parallel setting is that the application
of a single formal construction can often build the entire parallel struc-
ture of a program. Thus the same overhead of using a formal technique
results in a much larger gain in program construction.
BSP is particularly attractive from the perspective of a methodology
for program construction because:
Each BSP superstep has a well-dened interface to those that pre-
cede and follow it. BSP supersteps resemble a general form of
skeleton whose details can be instantiated from a few parameters.
The extensive skeleton literature has documented the advantages
this brings to software development [8, 34, 33].
BSP programs can be given a simple semantics in which each pro-
cessor can act arbitrarily on any program variables, with the nal
eect of a superstep being dened by a merge operator which de-
nes what happens if multiple assignments to the same variable
are made by dierent processors. In fact, it is clear that BSP is
just one of a family of models dened by dierent merge functions.
This idea has been pursued by He and Hoare [20], and in slightly
dierent way by Lecomber [23].
Too much should not be made of the ability to reason formally about
parallel programs. Nevertheless, BSP is noticeable as the only known
programming model that is both ecient and semantically clean enough
to be reasoned about [36].
8. APPLICATIONS OF BSP
BSP has been used extensively across the whole range of standard
scientic and numeric applications. Some examples are:
computational uid dynamics [18, 7],
computational electromagnetics [25],
24

molecular dynamics [26],

computational geometry [31],
standard algorithmic building blocks (sorting, queues),
discrete event simulation [5],
However, BSP's limits with respect to irregular programs handicap it
slightly compared to MPI, and so it is far from being the most common
parallel programming environment in these applications.
However, the application areas for parallel and high-performance com-
puting are changing { away from these numerical areas to applications
that involve more symbolic processing in support of Internet applica-
tions. In these newer application areas, there is less pressure on per-
formance as the only variable of interest, and more emphasis on pre-
dictability, portability, and ease of maintenance (which is correlated with
properties such as lack of deadlock and clean semantics). In these ar-
eas, BSP is attracting greater interest. To illustrate the avour of such
applications, we consider data mining in the next section.
PARALLEL DATA MINING
Data mining is concerned with extracting useful enterprise knowledge
from data that many organisations collect more or less incidentally be-
cause of the use of computers as point-of-sale units. Retail businesses
are an obvious example; but increasingly business-to-business interac-
tions are computer-mediated and the same opportunities to gather data
exist. Interactions at web sites are another opportunity to gather huge
amounts of data.
Once this data exists, most organisations feel pressure to use it to
improve their interactions with customers (broadly understood). This
requires building models to explain the variation that is observed in
interactions { the fundamental question is almost always some variant
of \why did this potential customer buy, while this one did not". The
algorithms required to build such models are often conceptually rather
simple, but they face two challenges:
The datasets involved are huge, typically terabytes, so that the
time required simply to move the data around is substantial; and
Even with these enormous datasets, many algorithms are compute
bound, so that many processor cycles are required.
In a sense, almost all of the work goes into controlling the size of the con-
stants, rather than attacking the asymptotic complexity of algorithms.
 The BSP Model 25

Hence, the ability to get a performance improvement of a factor of p is

interesting even if p is in the range 4{30. Data mining is a `killer appli-
cation' for parallel computing, especially as those who want the results
of parallel programs derive a direct commercial advantage from them,
and are thus willing to pay the costs involved [43]. Many data mining
algorithms can partition the data evenly, and are thus a good t with
BSP-structured programs [32].
Data mining techniques can be roughly divided into two kinds. The
rst kind is given data about `customers' for whom the outcome is known
and computes predictors that can be used to predict the outcome for
future, new customers. The second kind produce clusters showing how
`customers' are related to each other. Learning these similarities can be
used, for example, to predict something about a new `customer' based
on those customers who are `similar'.
The most obvious way to parallelize a data mining technique is to
parallelize the algorithm itself. It turns out that this is usually not the
best approach. Most data mining algorithms have the property that,
given some of the available data, they can compute an approximation of
their usual output. In other words, if the algorithm is given a sample of
the data, it will compute an approximation to the predictor, or a partial
set of clusters. As the size of the sample grows, the approximation gets
more and more similar to the result on the entire dataset.
If there is a way to combine these approximations to produce a good
global solution, then a much better generic parallelization strategy is:
partition the data across processors, execute the approximating versions
of the algorithm sequentially, and then use the combining to produce a
global answer. This strategy is remarkably eective, partly because the
amount of communication it requires is small [37].
BAGGING
There is a generic way in which the results of dierent algorithms
can be combined: if the output is categorical, combine the results by
voting, if the output is numeric, combine the results by regression (i.e.
averaging). Thus there is always a way to apply the idea described
above, an approach that is known as bagging [3]. (The deeper question is
whether the results of the dierent algorithms can be combined in a more
integrated way, since the cost of voting grows linearly with the number
of voters. Bagging is therefore always useful as a training technique, but
its results may not be attractive as a deployed system.)
Standard bagging algorithms select uniformly random subsets of the
training dataset (\bags") and then execute a copy of the base data min-
26

ing algorithm on each one. Clearly, since these copies are executed
independently on their own data, bagging can be trivially parallelized.
Bagging illustrates an important feature of data mining algorithms. It
is often possible to achieve much better results by combining the outputs
trained on several small bags than to use the output trained on the entire
dataset. It's as if the most cost-eective training occurs from the rst
few examples seen. Hence a bagging algorithm often produces better
results from much less input data, and does so in parallel as well.
It is easy to see how to use BSP to implement bagging. It requires
only a single BSP superstep. If the bags are chosen to have the same size,
then the computations at each processor usually takes similar amounts
of time. If the predictors are left at the processors that computed them,
then evaluation can also be done in parallel by broadcasting each new
data point to the processors and then combining their votes using a
gather.
EXCHANGING INFORMATION EARLIER
Bagging provides a generic parallelization technique. However, the ob-
servation that much of the gain occurs after having seen only a few data
points suggest that performance might be improved still further by ex-
changing information among processors before the end of the algorithm.
That way, each processor can learn from its own data, and simultane-
ously from the data that other processors are using. This turns out to
be very productive.
The basic structure of the parallel algorithm is:
Either partition the data among the processors, or give each pro-
cessor a bag;
Execute the approximating algorithm on each processor for a while;
Exchange the results obtained by each processor with all of the
others (a total exchange);
Update the model being built by each processor to incorporate the
information received from other processors;
Repeat as appropriate.
Of course, this requires the possibility of being able to combine informa-
tion learned by other processors into the model under construction, so
this approach will not work for all data mining algorithms. However, it
is eective for several, including neural networks [30, 29], and inductive
logic programming [42].
 The BSP Model 27

The information that is exchanged is quite small compared to the

size of typical datasets, so the communication overhead of this approach
is low. Interestingly, superlinear speedup occurs because there are two
improvements over executing the plain sequential algorithm. The rst
is that p processors are being used so we expect roughly a factor of
p speedup, less the costs of communication. The second is that each
processor nishes its own model faster by integrating the information
learned from the other processors, so the overall execution time is even
better than expected.
Since this is a true superlinear speedup, not simply some artifact of
cache behaviour or whatever, it implies that the sequential algorithm
can be improved, and that is indeed the case. Whenever such a su-
perlinear speedup parallelization exists, the sequential algorithm should
not attempt to build a model monolithically, but should instead use the
same approach of building partial models on small subsets of the data
and combining them.
Both of these parallelization strategies are easily implemented in BSP;
and BSP's attention to communication patterns such as total exchange
means that they execute eciently.
9. SUMMARY
BSP has made a number of contributions to both the design of parallel
programming models, and implementation strategies. We summarise by
listing some of these contributions here:
BSP has shown that many intuitions about networks and their
performance are not valid. In particular, BSP has shown that, even
for NIC-based networks with low-overhead protocols, crossing the
boundary into and out of a network is the performance bottleneck.
Congestion inside the network is negligible by comparison.
This implies that it takes as long to move data between adjacent
processors in the network topology as it does to move data across
the network diameter. As a result locality is of limited importance
(although co-locality is still critical). This explains the `death of
topology' { almost all modern distributed-memory computers use
a hierarchy of crossbar switches (in contrast to the thousands of
papers on network topologies that have appeared over the past
three decades).
This property of networks also invalidates the intuition that a
dense communication pattern, such as a total exchange, ought to
be much more expensive than a broadcast by a single processor;
and that a total exchange ought to be quadratically more expensive
28

than a shift. While there is an decrease in network performance

for denser communication patterns, it is small.
BSP has shown that it isn't always best to do things as early as
possible. In particular, the ability to wait to carry out data trans-
fers long past the point in the program where they are requested
buys considerable opportunities for arranging the pattern and tim-
ing of those data transfers to use the network to best eect. We
have seen that this can produce a performance improvement of a
factor of four over systems such as MPI. Note that this depends on
BSP semantics; other messaging systems may create deadlock by
postponing communication, and are certain to do so if they reorder
it.
More generally, BSP shows that structured parallel programming
is not only a performance win, but it's also a program construction
win. Much of the discussion about structured versus unstructured
parallel programming resembles the discussion of structured versus
unstructured programming in the Sixties. At rst glance, it seems
as if structure carries a performance cost; the truth is actually the
opposite { structure improves performance overall.
BSP's structure also allows a powerful set of optimisations that rely
on global knowledge to be applied. In essence, each processor infers
a lot about the global state from its own local state. We have seen
how this can be exploited to: arrange the order of communication,
control the rate at which data is inserted into the network, and
avoid the need for some acknowledgements.
Furthermore, the BSP approach means that optimisation is a prop-
erty of the implementation of the model rather than of individual
programs. Software developers do not get performance surprises
because the compiler has done something they weren't expecting.
Instead, they can tell what to expect by examining the values of g
and l for their implementation system.
BSP takes a global approach to performance modelling, measuring
macroscopic properties of architectures rather than trying to build
them up from measurements of microscopic properties. As a result,
BSP's architectural parameters are both stable and simple.
BSP also uses a set of program parameters that are easily accessible
to software developers. As a result, the BSP cost model is tractable
in the sense that it can be used to guide software construction well
before code is actually written, but also reasonably accurate. In
 The BSP Model 29

fact, the biggest weakness in the BSP cost model is not related
to parallelism at all, but to the diculty of modelling instruction
execution within a single processor.
BSP's cost model makes parallel software design possible, and pro-
vides predictable performance without benchmarking.
The benets of the BSP model are not theoretical, nor do they
apply only at low levels. The benets of the model play through to
the application level, allowing high-performance real applications
to be built and maintained.
The BSP model shows that there are signicant benets, both
in simplicity and performance, in using DRMA for data transfer
rather than message passing. The biggest weakness of message
passing is that it takes two to play { forcing programmers to do
a complex matching when they write programs, and processors to
synchronise when they wish to transfer data.
Finding eective ways to program parallel computers is dicult be-
cause the requirements are mutually exclusive: simplicity and abstrac-
tion, but also performance (and preferably predictable performance).
Very few models are known that score reasonably well in both dimen-
sions. BSP is arguably the best positioned of known models. Yet it is
clear that BSP is still too low level and restricted to become the model
for parallel programming. However, it is an important step on the way
to such a model.
Acknowledgements.. A large number of people have been involved in
the design and implementation of BSP. In particular, the implementation
of BSPLib and much of the performance analysis was done by Jonathan
M.D. Hill and Stephen Donaldson.
References
[1] D.J. Becker, T. Sterling, D. Savarese, J. E.Dorbandi, U.A. Ranawak,
and C.V. Packer. Beowulf: A parallel workstation for scientic com-
putation. In Proceedings of the International Conference on Parallel
Processing (ICPP), pages 11{14, 1995.
[2] Rob H. Bisseling. Basic techniques for numerical linear algebra
on bulk synchronous parallel computers. In Lubin Vulkov, Jerzy
Wasniewski, and Plamen Yalamov, editors, Workshop Numerical
Analysis and its Applications 1996, volume 1196 of Lecture Notes
in Computer Science, pages 46{57. Springer-Verlag, Berlin, 1997.
30

[3] L. Breiman. Bagging predictors. Machine Learning, 24:123{140,

1996.
[4] Philip Buonadonna, Andrew Geweke, and David E. Culler. Imple-
mentation and analysis of the Virtual Interface Architecture. In
SuperComputing'98, 1998.
[5] Radu Calinescu. Conservative discrete-event simulations on bulk
synchronous parallel architectures. Technical Report TR-16-95, Ox-
ford University Computing Laboratory, 1995.
[6] Giuseppe Ciacco. Optimal communication performance on Fast
Ethernet with GAMMA. In Parallel and Distributed Processing,
volume 1388 of Lecture Notes in Computer Science, pages 534{548.
Springer, 1998.
[7] P.I. Crumpton and M.B. Giles. Multigrid aircraft computations
using the OPlus parallel library. In Parallel Computational Fluid
Dynamics: Implementation and Results using Parallel Computers.
Proceedings Parallel CFD '95, pages 339{346, Pasadena, CA, USA,
June 1995. Elsevier/North-Holland.
[8] M. Danelutto, F. Pasqualetti, and S. Pelagatti. Skeletons for data
parallelism in p3l. In C. Lengauer, M. Griebl, and S. Gorlatch,
editors, Proc. of EURO-PAR '97, Passau, Germany, volume 1300
of LNCS, pages 619{628. Springer-Verlag, August 1997.
[9] H.G. Dietz, T. Muhammad, J.B. Sponaugle, and T. Mattox. PA-
PERS: Purdue's adapter for parallel execution and rapid synchro-
nization. Technical Report TR-EE-94-11, Purdue School of Electri-
cal Engineering, March 1994.
[10] S.R. Donaldson, J.M.D. Hill, and D.B. Skillicorn. Exploiting
global structure for performance in clusters. In Proceedings of
IPPS/SPDP'99, pages 176{182. IEEE Computer Society Press,
1999.
[11] S.R. Donaldson, J.M.D. Hill, and D.B. Skillicorn. Performance re-
sults for a reliable low-latency cluster communication protocol. In
PCNOW'99, Workshop at IPPS/SPDP'99, number 1586 in Lec-
ture Notes in Computer Science, pages 1097{1114. Springer-Verlag,
April 1999.
[12] S.R. Donaldson, J.M.D. Hill, and D.B. Skillicorn. BSP clusters:
High-performance, reliable, and very low cost. Parallel Computing,
26(2{3):199{242, February 2000.
[13] Stephen R. Donaldson, Jonathan M.D. Hill, and David B. Skillicorn.
Predictable communication on unpredictable networks: Implement-
 The BSP Model 31

ing BSP over TCP/IP. In Europar'98, number 1470 in Springer Lec-

ture Notes in Computer Science, pages 970{980, September 1998.
[14] M. Goudreau, K. Lang, S. Rao, T. Suel, and T. Tsantilas. Towards
eciency and portability: Programming the BSP model. In Pro-
ceedings of the 8th Annual Symposium on Parallel Algorithms and
Architectures, pages 1{12, June 1996.
[15] W. W. Gropp and E. L. Lusk. A taxonomy of programming models
for symmetric multiprocessors and SMP clusters. In Programming
Models for Massively Parallel Computers, pages 2{7, October 1995.
[16] J.M.D. Hill and D.B. Skillicorn. Lessons learned from implementing
BSP. Future Generation Computer Systems, 13(4{5):327{335, April
1998.
[17] J.M.D. Hill and D.B. Skillicorn. Practical barrier synchronisation.
In 6th Euromicro Workshop on Parallel and Distributed Processing
(PDP'98), pages 438{444, Barcelona, Spain, January 1998. IEEE
Computer Society Press.
[18] Jonathan M D Hill, Paul I Crumpton, and David A Burgess. The
theory, practice, and a tool for BSP performance prediction applied
to a CFD application. Technical Report TR-4-96, Programming
Research Group, Oxford University Computing Laboratory, Wolf-
son Building, Parks Road, Oxford, England. OX1 3QD, February
1996.
[19] Jonathan M. D. Hill, Bill McColl, Dan C. Stefanescu, Mark W.
Goudreau, Kevin Lang, Satish B. Rao, Torsten Suel, Thanasis Tsan-
tilas, and Rob H. Bisseling. BSPlib: The BSP programming library.
Parallel Computing, 24(14):1947{1980, December 1998.
[20] C.A.R. Hoare and J. He. Unied Theories of Programming.
Prentice-Hall International, 1998.
[21] P. Hudak. The conception, evaluation, and application of functional
programming. ACM Computing Surveys, 21(3):359{411, 1989.
[22] P. Kelly. Functional Programming for Loosely-Coupled Multiproces-
sors. Pitman, 1989.
[23] D. Lecomber. Methods of BSP Programming. PhD thesis, Oxford
University Computing Laboratory, 1998.
[24] W. F. McColl. General purpose parallel computing. In A. M. Gib-
bons and P. Spirakis, editors, Lectures on Parallel Computation,
Cambridge International Series on Parallel Computation, pages
337{391. Cambridge University Press, 1993.
32

[25] P.B. Monk, A.K. Parrott, and P.J. Wesson. A parallel nite element
method for electromagnetic scattering. COMPEL, 13, Supp.A:237{
242, 1994.
[26] M. Nibhanupudi, C. Norton, and B. Szymanski. Plasma simulation
on networks of workstations using the bulk synchronous parallel
model. In Proceedings of the International Conference on Parallel
and Distributed Processing Techniques and Applications, Athens,
GA, November 1995.
[27] S.L. Peyton-Jones and David Lester. Implementing Functional Pro-
gramming Languages. Prentice-Hall International Series in Com-
puter Science, 1992.
[28] Loic Prylli and Bernard Tourancheau. A new protocol designed
for high performance networking on Myrinet. In Parallel and Dis-
tributed Processing, volume 1388 of Lecture Notes in Computer Sci-
ence, pages 472{485. Springer, 1998.
[29] R.O. Rogers and D.B. Skillicorn. Using the BSP cost model for opti-
mal parallel neural network training. Future Generation Computer
Systems, 14:409{424, 1998.
[30] R.O. Rogers and D.B. Skillicorn. Using the BSP cost model to opti-
mize parallel neural network training. Future Generation Computer
Systems, 14:409{424, 1998.
[31] Constantinos Siniolakis. Bulk-synchronous parallel algorithms in
computational geometry. Technical Report PRG-TR-10-96, Oxford
University, Computing Laboratory, May 1996.
[32] D. Skillicorn. Strategies for parallel data mining. IEEE Concur-
rency, 7(4):26{35, October{December 1999.
[33] D.B. Skillicorn. Architecture-independent parallel computation.
IEEE Computer, 23(12):38{51, December 1990.
[34] D.B. Skillicorn. Structuring data parallelism using categorical data
types. In Programming Models for Massively Parallel Computers,
pages 110{115, Berlin, September 1993. Computer Society Press.
[35] D.B. Skillicorn. Foundations of Parallel Programming. Number 6 in
Cambridge Series in Parallel Computation. Cambridge University
Press, 1994.
[36] D.B. Skillicorn. Building BSP programs using the Renement
Calculus. In Third International Workshop on Formal Methods
for Parallel Programming: Theory and Applications (FMPPTA'98),
Springer Lecture Notes in Computer Science 1388, pages 790{795,
March/April 1998.
 The BSP Model 33

[37] D.B. Skillicorn. Parallel predictor generation. In Proceedings of a

Workshop on Large-Scale Parallel KDD Systems, KDD'99, num-
ber 1759 in Lecture Notes in Articial Intelligence, pages 190{196.
Springer-Verlag, 2000.
[38] D.B. Skillicorn, J.M.D. Hill, and W.F. McColl. Questions and an-
swers about BSP. Scientic Programming, 6(3):249{274, 1997.
[39] D.B. Skillicorn and D. Talia. Models and programming languages
for parallel computation. Computing Surveys, 30(2):123{169, June
1998.
[40] L.G. Valiant. A bridging model for parallel computation. Commu-
nications of the ACM, 33(8):103{111, August 1990.
[41] Thorsten von Eicken, David E. Culler, Seth Copen Goldstein, and
Klaus Erik Schauser. Active Messages: A mechanism for integrated
communication and computation. In The 19th Annual Interna-
tional Symposium on Computer Architecture, volume 20(2) of ACM
SIGARCH Computer Architecture News. ACM Press, May 1992.
[42] Y. Wang and D.B. Skillicorn. Parallel inductive logic for data min-
ing. In Workshop on Distributed and Parallel Knowledge Discovery,
KDD2000, Boston, to appear. ACM Press.
[43] M. Zaki. Parallel and distributed data mining: A survey. IEEE
Concurrency, 7(4):14{25, October{December 1999.

View publication stats