Parallel Computing Platforms
Chieh-Sen (Jason) Huang
Department of Applied Mathematics
National Sun Yat-sen University
Thank Ananth Grama, Anshul Gupta, George Karypis, and Vipin
Kumar for providing slides.
Topic Overview
• Dichotomy of Parallel Computing Platforms
• Communication Costs in Parallel Machines
• The first example of MPI: the Message Passing Interface
Explicitly Parallel Platforms
Dichotomy of Parallel Computing Platforms
• An explicitly parallel program must specify concurrency and
interaction between concurrent subtasks.
• The former is sometimes also referred to as the control structure
and the latter as the communication model.
Control Structure of Parallel Programs
• Parallelism can be expressed at various levels of granularity –
from instruction level to processes.
• Between these extremes exist a range of models, along with
corresponding architectural support.
Control Structure of Parallel Programs
• Processing units in parallel computers either operate under
the centralized control of a single control unit or work
independently.
• If there is a single control unit that dispatches the same
instruction to various processors (that work on different data),
the model is referred to as single instruction stream, multiple
data stream (SIMD).
• If each processor has its own control control unit, each
processor can execute different instructions on different data
items. This model is called multiple instruction stream, multiple
data stream (MIMD).
SIMD and MIMD Processors
PE: Processing Element
PE
+
INTERCONNECTION NETWORK
INTERCONNECTION NETWORK
PE control unit
PE PE
+
control unit
PE
Global
control
unit
PE
+
PE control unit
PE PE
+
control unit
(a) (b)
A typical SIMD architecture (a) and a typical MIMD architecture
(b).
SIMD Processors
• Some of the earliest parallel computers such as the Illiac IV,
MPP, DAP, CM-2, and MasPar MP-1 belonged to this class of
machines.
• SIMD relies on the regular structure of computations (such as
those in image processing).
• CUDA programs compile to the PTX instruction set. That
instruction set does not contain SIMD instructions. So,
CUDA programs cannot make explicit use of SIMD. Individual
threads are part of groups called warps, within which every
thread executes exactly the same sequence of instructions.
Nvidia call it Single Instruction, Multiple Thread (SIMT), but it’s
essentially SIMD.
Parallel Thread Execution (PTX, or NVPTX) is a pseudo-assembly language used
in Nvidia’s CUDA programming environment. The nvcc compiler translates
code written in CUDA, a C++-like language, into PTX, and the graphics driver
contains a compiler which translates the PTX into a binary code which can be
run on the processing cores.
MIMD Processors
• In contrast to SIMD processors, MIMD processors can execute
different programs on different processors.
• A variant of this, called single program multiple data streams
(SPMD) executes the same program on different processors.
• It is easy to see that SPMD and MIMD are closely related in
terms of programming flexibility and underlying architectural
support.
• Examples of such platforms include current generation Sun
Ultra Servers, SGI Origin Servers, multiprocessor PCs, workstation
clusters, and the IBM SP.
SIMD-MIMD Comparison
• SIMD computers require less hardware than MIMD computers
(single control unit).
• However, since SIMD processors are specially designed, they
tend to be expensive and have long design cycles.
• Not all applications are naturally suited to SIMD processors.
• In contrast, platforms supporting the SPMD paradigm can be
built from inexpensive off-the-shelf components with relatively
little effort in a short amount of time.
• MPI is primarily for SPMD/MIMD. HPF is an example of a SIMD
interface.
Communication Model of Parallel Platforms
• There are two primary forms of data exchange between
parallel tasks – accessing a shared data space and
exchanging messages.
• Platforms that provide a shared data space are called shared-
address-space machines or multiprocessors.
1. Consistency problems
2. Global view of memory
• Platforms that support messaging are also called message
passing platforms or multicomputers.
1. No consistency problems
2. Lack of global view of memory
3. Communication is costly
Shared-Address-Space Platforms
• Part (or all) of the memory is accessible to all processors.
• Processors interact by modifying data objects stored in this
shared-address-space.
• If the time taken by a processor to access any memory
word in the system global or local is identical, the platform
is classified as a uniform memory access (UMA), else, a non-
uniform memory access (NUMA) machine.
NUMA and UMA Shared-Address-Space Platforms
P P
P
M M
Interconnection Network
Interconnection Network
Interconnection Network
C C M
P P P
M M
C C M
P P
M M
P
C C M
(a) (b) (c)
Typical shared-address-space architectures: (a)
Uniform-memory-access shared-address-space computer; (b)
Uniform-memory-access shared-address-space computer with
caches and memories; (c) Non-uniform-memory-access
shared-address-space computer with local memory only.
NUMA and UMA Shared-Address-Space Platforms
• The distinction between NUMA and UMA platforms is important
from the point of view of algorithm design. NUMA machines
require locality from underlying algorithms for performance.
• Programming these platforms is easier since reads and writes
are implicitly visible to other processors.
• However, read-write data to shared data must be coordinated
(this will be discussed in greater detail when we talk about
threads programming).
• Caches in such machines require coordinated access to
multiple copies. This leads to the cache coherence problem.
• A weaker model of these machines provides an address map,
but not coordinated access. These models are called non
cache coherent shared address space machines.
Shared-Address-Space vs. Shared Memory Machines
• It is important to note the difference between the terms shared
address space and shared memory.
• We refer to the former as a programming abstraction and to
the latter as a physical machine attribute.
• It is possible to provide a shared address space using a
physically distributed memory.
Message-Passing Platforms
• These platforms comprise of a set of processors and their own
(exclusive) memory.
• Instances of such a view come naturally from clustered
workstations and non-shared-address-space multicomputers.
• These platforms are programmed using (variants of) send and
receive primitives.
• Libraries such as MPI and PVM provide such primitives.
Message Passing vs. Shared Address Space Platforms
• Message passing requires little hardware support, other than a
network.
• Shared address space platforms can easily emulate message
passing. The reverse is more difficult to do (in an efficient
manner).
Communication Costs in Parallel Machines
• Along with idling and contention, communication is a major
overhead in parallel programs.
• The cost of communication is dependent on a variety of
features including the programming model semantics, the
network topology, data handling and routing, and associated
software protocols.
Message Passing Costs in Parallel Computers
The total time to transfer a message over a network comprises
of the following:
• Startup time (ts): Time spent at sending and receiving nodes
(executing the routing algorithm, programming routers, etc.).
• Per-hop time (th): This time is a function of number of hops and
includes factors such as switch latencies, network delays, etc.
• Per-word transfer time (tw ): This time includes all overheads that
are determined by the length of the message. This includes
bandwidth of links, error checking and correction, etc.
Store-and-Forward Routing
• A message traversing multiple hops is completely received at
an intermediate hop before being forwarded to the next hop.
• The total communication cost for a message of size m words to
traverse l communication links is
tcomm = ts + (mtw + th)l.
• In most platforms, th is small and assume l = 1 then the above
expression can be approximated by
tcomm = ts + mtw .
• See homework.
MPI: the Message Passing Interface
• MPI defines a standard library for message-passing that can
be used to develop portable message-passing programs using
either C or Fortran.
• The MPI standard defines both the syntax as well as the
semantics of a core set of library routines.
• Vendor implementations of MPI are available on almost all
commercial parallel computers.
• It is possible to write fully-functional message-passing programs
by using only the six routines.
• Syntax is the grammar. It describes the way to construct a correct sentence.
For example, this water is triangular is syntactically correct.
• Semantics relates to the meaning. this water is triangular does not mean
anything, though the grammar is ok.
MPI: the Message Passing Interface
The minimal set of MPI routines.
MPI_Init Initializes MPI.
MPI_Finalize Terminates MPI.
MPI_Comm_size Determines the number of processes.
MPI_Comm_rank Determines the label of the calling process.
MPI_Send Sends a message.
MPI_Recv Receives a message.
Starting and Terminating the MPI Library
• MPI_Init is called prior to any calls to other MPI routines. Its
purpose is to initialize the MPI environment.
• MPI_Finalize is called at the end of the computation,
and it performs various clean-up tasks to terminate the MPI
environment.
• The prototypes of these two functions are:
int MPI_Init(int *argc, char ***argv)
int MPI_Finalize()
• MPI_Init also strips off any MPI related command-line
arguments.
• All MPI routines, data-types, and constants are prefixed
by “MPI_”. The return code for successful completion is
MPI_SUCCESS.
Communicators
• A communicator defines a communication domain – a set of
processes that are allowed to communicate with each other.
• Information about communication domains is stored in
variables of type MPI_Comm.
• Communicators are used as arguments to all message transfer
MPI routines.
• A process can belong to many different (possibly overlapping)
communication domains.
• MPI defines a default communicator called MPI_COMM_WORLD
which includes all the processes.
Querying Information
• The MPI_Comm_size and MPI_Comm_rank functions are used to
determine the number of processes and the label of the calling
process, respectively.
• The calling sequences of these routines are as follows:
int MPI_Comm_size(MPI_Comm comm, int *size)
int MPI_Comm_rank(MPI_Comm comm, int *rank)
• The rank of a process is an integer that ranges from zero up to
the size of the communicator minus one.
Our First MPI Program
#include <mpi.h>
main(int argc, char *argv[])
{
int npes, myrank;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &npes);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
printf("From process %d out of %d, Hello World!\n",
myrank, npes);
MPI_Finalize();
}
Sending and Receiving Messages
• The basic functions for sending and receiving messages in MPI
are the MPI_Send and MPI_Recv, respectively.
• The calling sequences of these routines are as follows:
int MPI_Send(void *buf, int count, MPI_Datatype datatype,
int dest, int tag, MPI_Comm comm)
int MPI_Recv(void *buf, int count, MPI_Datatype datatype,
int source, int tag, MPI_Comm comm, MPI_Status *status)
• MPI provides equivalent datatypes for all C datatypes. This is
done for portability reasons.
• The datatype MPI_BYTE corresponds to a byte (8 bits) and
MPI_PACKED corresponds to a collection of data items that has
been created by packing non-contiguous data.
• The message-tag can take values ranging from zero up to the
MPI defined constant MPI_TAG_UB.
A sending-receiving MPI Program
#include <mpi.h>
main(int argc, char *argv[])
{
int npes, myrank, a[10], b[10];
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &npes);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
if (myrank==0){
a[0]=0; b[0]=1;
MPI_Send(a,1,MPI_INT,1,1,MPI_COMM_WORLD);
}
else{
a[0]=2; b[0]=3;
MPI_Recv(a,1,MPI_INT,0,1,MPI_COMM_WORLD,&status);
}
printf("From processor: %d out of %d a=%d b=%d\n"
,myrank,npes,a[0],b[0]);
MPI_Finalize();
}
• Homework: Compute the start up time (ts) and transmission
time per word (tm), note that this is so called uncongested
time.
