UNIT IV DISTRIBUTED MEMORY PROGRAMMING WITH MPI

MPI program execution – MPI constructs – libraries – MPI send and receive – Point-to-point and
Collective communication – MPI derived datatypes – Performance evaluation

Textbook

• Peter S. Pacheco, “An Introduction to Parallel Programming”, Morgan-Kauffman/Elsevier, 2011.

1. Getting Started using MPI

Distributed Memory Systems - MPI programming

• In message-passing programs, a program running on one core-memory pair is usually called a

process.

• Two processes can communicate by calling functions:

– one process calls a send function

– and the other calls a receive function.

• The implementation of message-passing can be done using Message-Passing Interface(MPI).

• MPI defines a library of functions that can be called from C, C++, and Fortran programs.

• In Parallel Programming common practice to identify processes by nonnegative integer ranks.

• p processes are numbered 0, 1, 2, .. p-1

• One process to do the output, and the other processes will send it messages, which it will print

Example: Hello World in MPI

Compile:

mpicc -g -Wall -o mpi_hello mpi_hello.c

Execution:
mpiexec -n <number of processes> <executable>

mpiexec -n 4 ./mpi_hello

• Written in C.

– Has main.

– Uses stdio.h, string.h, etc.

• Need to add mpi.h header file.

• Identifiers defined by MPI start with “MPI_”.

• First letter following underscore is uppercase.

– For function names and MPI-defined types.

– Helps to avoid confusion.

2. MPI Components (MPI_Init)

2.1 MPI_Init

– Tells MPI to do all the necessary setup.

– It might allocate storage for message buffers,

– It might decide which process gets which rank.

– no other MPI functions should be called before the program calls MPI Init.

2.2 MPI_Finalize

– Tells MPI that any resources allocated for MPI can be freed.

– no MPI functions should be called after the call to MPI Finalize.

– It’s also not necessary that the calls to MPI Init and MPI Finalize be in main function.
2.3 Communicators MPI_Comm_size & MPI_Comm_rank

• In MPI a communicator is a collection of processes that can send messages to each other.

• MPI_Init defines a communicator that consists of all the processes created when the program is
started.

• It is called MPI_COMM_WORLD.

2.4 Communication - MPI_Send()

• msg_buf_p - is a pointer to the block of memory containing the contents of the message.

– In the program, this is just the string containing the message, greeting.

• The second and third arguments, msg_size and msg_ type, determine the amount of data to be
sent.

• msg_size - is the number of characters in the message plus one character for the ‘\0’ character
that terminates C strings.

• msg_type - is MPI CHAR. These two arguments together tell the system that the message
contains strlen(greeting)+1 chars.
• dest - specifies the rank of the process that should receive the message.

• tag - is a nonnegative int.

• It can be used to distinguish messages that are otherwise identical. For example, suppose
process 1 is sending floats to process 0. Some of the floats should be printed, while others
should be used in a computation.

• Then the first four arguments to MPI Send provide no information regarding which floats should
be printed and which should be used in a computation.

• So process 1 can use, say, a tag of 0 for the messages that should be printed and a tag of 1 for
the messages that should be used in a computation.

MPI Data types:

 • The final argument to MPI Send is a communicator. All MPI functions that involve
communication have a communicator argument.

• One of the most important purposes of communicators is to specify communication universes.

• a communicator is a collection of processes that can send messages to each other.

• Conversely, a message sent by a process using one communicator cannot be received by a

process that’s using a different communicator.

• Since MPI provides functions for creating new communicators, this feature can be used in
complex programs to insure that messages aren’t “accidentally received” in the wrong place.

2.5 Communication – MPI_Recv()

• First three arguments specify the memory available for receiving the message:

– msg_buf_p points to the block of memory,

– buf_size determines the number of objects that can be stored in the block,

– buf_type indicates the type of the objects.

• The next three arguments identify the message.

– source - specifies the process from which the message should be received.

– tag - should match the tag argument of the message being sent,

– communicator - must match the communicator used by the sending process.

2.6 Semantics of MPI_Send and MPI_Recv

when we send a message from one process to another

• The sending process will assemble the message. For example, it will add the “envelope”
information to the actual data being transmitted.
 • The destination process rank, the sending process rank, the tag, the communicator, and some
information on the size of the message.

• Once the message has been assembled, there are essentially two possibilities: the sending
process can buffer the message or it can block.

• If it buffers the message, the MPI system will place the message (data and envelope) into its
own internal storage, and the call to MPI Send will return.

• If the system blocks, it will wait until it can begin transmitting the message, and the call to
MPI_Send may not return immediately.

• If we use MPI_Send, when the function returns, we don’t actually know whether the message
has been transmitted. We only know that the storage we used for the message.

• If the size of a message is less than the cutoff, it will be buffered. If the size of the message is
greater than the cutoff, MPI Send will block.

• Unlike MPI Send, MPI_Recv always blocks until a matching message has been received. Thus,
when a call to MPI_Recv returns, we know that there is a message stored in the receive buffer
(unless there’s been an error).

• There is an alternate method for receiving a message, in which the system checks whether a
matching message is available and returns, regardless of whether there is one.

• MPI requires that messages be nonovertaking.

• This means that if process q sends two messages to process r, then the first message sent by q
must be available to r before the second message.

• However, there is no restriction on the arrival of messages sent from different processes.

• That is, if q and t both send messages to r, then even if q sends its message before t sends its
message, there is no requirement that q’s message become available to r before t’s message.

2.7 Parallelizing the Trapezoidal Rule

• Two types of task

– one type is finding the area of a single trapezoid.

– the other is computing the sum of these areas.

• Then the communication channels will join each of the tasks of the first type to the single task of
the second type.

• split the interval [a,b] up into comm sz subintervals.

• If comm sz evenly divides n, the number of trapezoids, we can simply apply the trapezoidal rule
with n=comm sz trapezoids to each of the comm sz subintervals.

• To finish, we can have one of the processes, say process 0, add the estimates.
3. Collective communication

• Each process with rank greater than 0 is, in effect, saying “add this number into the
total(trapezoidal rule).”

• Process 0 is doing nearly all the work in computing the global sum, while the other processes are
doing almost nothing
communication functions that involve all the processes in a communicator are called collective
communications

3.1 A tree-structured Communication global sum

Tree-structured communication

1. In the first phase:

(a) Process 1 sends to 0, 3 sends to 2, 5 sends to 4, and 7 sends to 6.
(b) Processes 0, 2, 4, and 6 add in the received values.
(c) Processes 2 and 6 send their new values to processes 0 and 4, respectively.
(d) Processes 0 and 4 add the received values into their new values.

2. (a) Process 4 sends its newest value to process 0.

(b) Process 0 adds the received value to its newest value.

3.2 MPI_Reduce

Example
Predefined reduction operators in MPI

3.3 MPI_Allreduce

• Useful in a situation in which all of the processes need the result of a global sum in order to
complete some larger computation.
3.4 Butterfly Structured

3.5 Broadcast

Get_input that uses MPI_Bcast

3.6 Scatter

• MPI_Scatter can be used in a function that reads in an entire vector on process 0 but only sends
the needed components to each of the other processes.
3.7 Gather

• Collect all of the components of the vector onto process 0, and then process 0 can process all of
the components.
3.8 Allgather

• Concatenates the contents of each process’ send_buf_p and stores this in each process’
recv_buf_p.

• As usual, recv_count is the amount of data being received from each process.

4. Collective vs. Point-to-Point Communications

Point-to-point communications Collective communications

• Point-to-point communications are • Collective communications don’t use tags.
matched on the basis of tags and
• They’re matched solely on the basis of the
communicators.
communicator and the order in which
they’re called.

Each process communicates using send and • All the processes in the communicator
receive function must call the same collective function.

For example, a program that attempts to match a

call to MPI_Reduce on one process with a call to
MPI_Recv on another process is erroneous, and, in
all likelihood, the program will hang or crash
The corresponding destination alone is used • The output_data_p argument is only used
on dest_process.

• However, all of the processes still need to

pass in an actual argument corresponding
to output_data_p, even if it’s just NULL.

The arguments passed by each process to an MPI • The arguments passed by each process to
communication may vary with each process an MPI collective communication must be
“compatible.”

• For example, if one process passes in 0 as

the dest_process and another passes in 1,
then the outcome of a call to MPI_Reduce
is erroneous, and, once again, the program
is likely to hang or crash.

5.Library Functions

Library functions listed below can be explained with an example.

1. MPI_Init()
2. MPI_Comm_size()
3. MPI_Comm_rank()
4. MPI_Finalize()
5. MPI_Send()
6. MPI_Receive()
7. MPI_Reduce()
8. MPI_Allreduce()
9. MPI_Bcast()
10. MPI_Scatter()
11. MPI_Gather()
12. MPI_Allgather()

13.MPI_Address()

#include <mpi.h>

int MPI_Address(void *location, MPI_Aint *address)

Description
The address of a location in memory can be found by invoking this function. Returns the (byte) address
of location.

14.MPI_Barrier()

#include <mpi.h>

int MPI_Barrier(MPI_Comm comm)

MPI_Get_version

Description

An MPI barrier completes after all group members have entered the barrier.

#include <mpi.h>

int MPI_Get_version(int *version, int *subversion)

Description

Since Open MPI is MPI 3.1 compliant, this function will return a version value of 3 and a subversion value
of 1 for this release.

15.MPI_Wait

#include <mpi.h>

int MPI_Wait(MPI_Request *request, MPI_Status *status)

Input Parameter

Request - Request (handle).

Output Parameters

Status - Status object (status).

Description

A call to MPI_Wait returns when the operation identified by request is complete.

PART B – 16 Marks

1.MPI Send and Receive

2.1 to 2.7

2.Point to Point Communication

2.1 to 2.7
3.Collective Communication

3. 1 to 3.8

4.Library Functions

Chapter 5

PART A

1.Difference between point to point and collective communication.

2,Syntax

3.MPI Derived data types

4.MPI reduction operator