10 R. A. Kendall et al.

run on a single-processor machine. Parallel programs are commonly developed on

single-processor laptops, even with multiple processes. If there are more than a few
processes per processor, however, the program may run very slowly because of con-
tention among the processes for the resources of the processor.

Point-to-Point Communication

The program in Figure 1.4 is a very simple parallel program. The individual processes
neither exchange data nor coordinate with each other. Point-to-point communication
allows two processes to send data from one to another. Data is sent by using rou-
tines such as MPI Send and is received by using routines such as MPI Recv (we
mention later several specialized forms for both sending and receiving).
We illustrate this type of communication in Figure 1.5 with a simple program that
sums contributions from each process. In this program, each process first determines
its rank and initializes the value that it will contribute to the sum. (In this case, the
sum itself is easily computed analytically; this program is used for illustration only.)
After receiving the contribution from the process with rank one higher, it adds the
received value into its contribution and sends the new value to the process with rank
one lower. The process with rank zero only receives data, and the process with the
largest rank (equal to size−1) only sends data.
The program in Figure 1.5 introduces a number of new points. The most obvi-
ous are the two new MPI routines MPI Send and MPI Recv. These have similar
arguments. Each routine uses the first three arguments to specify the data to be sent
or received. The fourth argument specifies the destination (for MPI Send) or source
(for MPI Recv) process, by rank. The fifth argument, called a tag, provides a way to
include a single integer with the data; in this case the value is not needed, and a zero
is used (the value used by the sender must match the value given by the receiver).
The sixth argument specifies the collection of processes to which the value of rank
is relative; we use MPI COMM WORLD, which is the collection of all processes in the
parallel program (determined by the startup mechanism, such as mpiexec in the
“Hello World” example). There is one additional argument to MPI Recv: status.
This value contains some information about the message that some applications may
need. In this example, we do not need the value, but we must still provide the argu-
ment.
The three arguments describing the data to be sent or received are, in order, the
address of the data, the number of items, and the type of the data. Each basic datatype
in the language has a corresponding MPI datatype, as shown in Table 1.1.
MPI allows the user to define new datatypes that can represent noncontiguous
memory, such as rows of a Fortran array or elements indexed by an integer array
(also called scatter-gathers). Details are beyond the scope of this chapter, however.
This program also illustrates an important feature of message-passing programs:
because these are separate, communicating processes, all variables, such as rank
or valOut, are private to each process and may (and often will) contain different
values. That is, each process has its own memory space, and all variables are private
 1 Parallel Programming Models 11
#include "mpi.h"
#include <stdio.h>
int main( int argc, char *argv[] )
{
int size, rank, valIn, valOut;
MPI_Status status;

MPI_Init( &argc, &argv );

MPI_Comm_size( MPI_COMM_WORLD, &size );

MPI_Comm_rank( MPI_COMM_WORLD, &rank );

/* Pick a simple value to add */

valIn = rank;

/* receive the partial sum from the right processes

(this is the sum from i=rank+1 to size-1) */
if (rank < size - 1) {
MPI_Recv( &valOut, 1, MPI_INT, rank + 1, 0,
MPI_COMM_WORLD, &status );
valIn += valOut;
}
/* Send the partial sum to the left (rank-1) process */
if (rank > 0) {
MPI_Send( &valIn, 1, MPI_INT, rank - 1, 0,
MPI_COMM_WORLD );
}
else {
printf( "The sum is %d\n", valOut );
}

MPI_Finalize( );
return 0;
}

Fig. 1.5. A simple program to add values from each process.

Table 1.1. Some major predefined MPI datatypes.

C Fortran
int MPI INT INTEGER MPI INTEGER
float MPI FLOAT REAL MPI REAL
double MPI DOUBLE DOUBLE PRECISION MPI DOUBLE PRECISION
char MPI CHAR CHARACTER MPI CHARACTER
short MPI SHORT
12 R. A. Kendall et al.

to that process. The only way for one process to change or access data in another
process is with the explicit use of MPI routines such as MPI Send and MPI Recv.
MPI provides a number of other ways in which to send and receive messages, in-
cluding nonblocking (sometimes incorrectly called asynchronous) and synchronous
routines. Other routines, such as MPI Iprobe, can be used to determine whether a
message is available for receipt. The nonblocking routines can be important in ap-
plications that have complex communication patterns and that send large messages.
See [30, Chapter 4] for more details and examples.

Collective Communication and Computation

Any parallel algorithm can be expressed by using point-to-point communication.

This flexibility comes at a cost, however. Unless carefully structured and docu-
mented, programs using point-to-point communication can be challenging to under-
stand because the relationship between the part of the program that sends data and
the part that receives the data may not be clear (note that well-written programs using
point-to-point message passing strive to keep this relationship as plain and obvious
as possible).
An alternative approach is to use communication that involves all processes (or
all in a well-defined subset). MPI provides a wide variety of collective communica-
tion functions for this purpose. As an added benefit, these routines can be optimized
for their particular operations (note, however, that these optimizations are often quite
complex). As an example Figure 1.6 shows a program that performs the same com-
putation as the program in Figure 1.5 but uses a single MPI routine. This routine,
MPI Reduce, performs a sum reduction (specified with MPI SUM), leaving the re-
sult on the process with rank zero (the sixth argument).
Note that this program contains only a single branch (if) statement that is used
to ensure that only one process writes the result. The program is easier to read than
its predecessor. In addition, it is effectively parallel; most MPI implementations will
perform a sum reduction in time that is proportional to the log of the number of
processes. The program in Figure 1.5, despite being a parallel program, will take
time that is proportional to the number of processes because each process must wait
for its neighbor to finish before it receives the data it needs to form the partial sum.7
Not all programs can be conveniently and efficiently written by using only col-
lective communications. For example, for most MPI implementations, operations on
PDE meshes are best done by using point-to-point communication, because the data
exchanges are between pairs of processes and this closely matches the point-to-point
programming model.

7
One might object that the program in Figure 1.6 doesn’t do exactly what the program in
Figure 1.5 does because, in the latter, all of the intermediate results are computed and available
to those processes. We offer two responses. First, only the value on the rank-zero process
is printed; the others don’t matter. Second, MPI offers the collective routine MPI Scan to
provide the partial sum results if that is required.
 1 Parallel Programming Models 13
#include "mpi.h"
#include <stdio.h>
int main( int argc, char *argv[] )
{
int rank, valIn, valOut;
MPI_Status status;

MPI_Init( &argc, &argv );

MPI_Comm_rank( MPI_COMM_WORLD, &rank );

/* Pick a simple value to add */

valIn = rank;

/* Reduce to process zero by summing the values */

MPI_Reduce( &valIn, &valOut, 1, MPI_INT, MPI_SUM, 0,
MPI_COMM_WORLD );
if (rank == 0) {
printf( "The sum is %d\n", valOut );
}

MPI_Finalize( );
return 0;
}

Fig. 1.6. Using collective communication and computation in MPI.

Other Features

MPI contains over 120 functions. In addition to nonblocking versions of point-

to-point communication, there are routines for defining groups of processes, user-
defined data representations, and testing for the availability of messages. These are
described in any comprehensive reference on MPI [73, 30].
An important part of the MPI design is its support for programming in the large.
Many parallel libraries have been written that make use of MPI; in fact, many appli-
cations can be written that have no explicit MPI calls and instead use libraries that
themselves use MPI to express parallelism. Before writing any MPI program (or any
program, for that matter), one should check to see whether someone has already done
the hard work. See [31, Chapter 12] for a summary of some numerical libraries for
Beowulf clusters.

1.2.3 The MPI-2 Extensions

The success of MPI created a desire to tackle some of the features not in the original
MPI (henceforth called MPI-1). The major features include parallel I/O, the creation
of new processes in the parallel program, and one-sided (as opposed to point-to-
point) communication. Other important features include bindings for Fortran 90 and