MPI for Cluster

Computing
 Module Objectives

After studying this module, you should be able to:

 Distinguish the existence of message passing

paradigms for cluster computing.
 Identify some collective operations of MPI.
 Understand the concept of collective
communication and the need for unique identifiers
for each computational process.

2
 Cluster
Computing
Cluster computing is a collection of tightly or loosely connected computers
that work together so that they act as a single entity.

3
 Cluster
Computing

"How do you think

computers in a cluster
coordinate their tasks?"
4
 Cluster
Computing
Message Passing Paradigms for Cluster Computing

 Message passing is a fundamental paradigm in

cluster computing where processes communicate
by exchanging messages.

 This approach is particularly well-suited for clusters

due to their distributed nature and the ability to
scale horizontally.

5
 Message Passing
Interface

Message Passing Interface (MPI)

 Is a standardized interface for exchanging

messages between multiple computers running a
parallel program across distributed memory.

 Defines the syntax and semantics of library

routines helpful in writing portable message-
passing programs in C, C++, or Fortran.

6
 Message Passing
Interface
Message Passing Interface (MPI)
 Supports both point-to-point and collective
communication between processes, as well as derived
data types, one-sided communication, dynamic process
management, and I/O.

 There are several open-source and vendor-specific

implementations of MPI, such as Open MPI, MPICH,
Microsoft MPI, Intel MPI, etc.

7
 Message Passing
Interface

Message Passing Interface (MPI)

 Widely used for developing and running parallel

applications on the Windows platform, as well as on
Linux and other operating systems.

8
Message Passing
Interface

9
 Message Passing
Interface
Message Passing Paradigm:

10
 Message Passing
Interface
Peer to Peer Paradigm:

Direct communication between processes: no client or server

11
 Message Passing
Interface
Client Server Paradigm

the server acts as a service provider, the client issues the request and
waits for the response from the server.
12
 Message Passing
Synchronous Message Passing
 the sender and the receiver have
to "meet" at their respective
send/receive operations so that
data can be transferred.
 This is also called 'rendezvous' or
'handshaking’.
 This form of transfer is simple but
might be inefficient because the
sender may have to wait even if it
has done its duty and prepared the
data to be sent.
'rendezvous' or 'handshaking

13
 Message Passing

Asynchronous Message Passing

 the sender does not wait for the receiver to

reach its receive operation, rather it gets rid of
the prepared data and continues its execution.

 This form of transfer does not force the sender

to wait, but creates another problem: there may
be messages that have already been sent but
not yet received, and they have to be stored
somewhere.

14
 Message Passing
Interface
Message Passing Vs Message systems
This is where message queues come into play, they are the buffers for in-
transit messages.

15
 Collective
Communication

COLLECTIVE COMMUNICATION

 where data is aggregated or disseminated from/to

multiple processes.

16
 Collective
Communication
Collective communication operations are made of the following types:

• Barrier Synchronization – Blocks until all processes have reached a

synchronization point

• Data Movement (or Global Communication) – Broadcast, Scatters,

Gather, All to All transmission of data across the communicator.

• Collective Operations (or Global Reduction) – One process from

which the communicator collects data from each process and
operates on that data to compute a result.

17
 Collective
Communication
Operations

Barrier Synchronization
Meaning: MPI_Barrier blocks until all processes have reached this routine.

18
 Collective Communication
Operations

Data Movement (or Global Communication)

Meaning: MPI_Bcast broadcasts a message from the process with rank "root" to
all other processes of the group.

MPI_Bcast( void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm );

19
 Collective Communication
Operations

Data Movement (or Global Communication)

Meaning: MPI_Scatter sends data from one task to all other tasks in a group

MPI_Scatter( void *sendbuf, int sendcnt, MPI_Datatype

sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype,
int root, MPI_Comm comm );
20
 Collective Communication
Operations

Data Movement (or Global Communication)

Meaning: MPI_Gather gathers together values from a group of processes.

MPI_Gather( void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype
recvtype, int root, MPI_Comm comm );
21
 Collective Communication
Operations

Data Movement (or Global Communication)

Meaning: MPI_Allgather gathers data from all tasks and distribute it to all.

MPI_Allgather( void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int
recvcount, MPI_Datatype recvtype, MPI_Comm comm );

22
 Collective Communication
Operations

Collective Operations (or Global Reduction)

MPI_Reduce( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root,
MPI_Comm comm );

23
 Unique Identifiers

Meaning: A unique identifier (UID) is a numeric or alphanumeric string that is

associated with a single entity within a given system

Some agencies give people a 'unique identifier' instead of using their name. Examples are a
driver's licence number, a passport number, a student ID number, or an IRD number.

24
 Unique Identifiers

Unique Identifiers
Lists several process identifiers commonly used in operating systems:

• PID (Process ID): A unique numerical identifier assigned to each process

running on a system. It serves as a way to uniquely identify and track
processes.

• PPID (Parent Process ID): The PID of the process that created the current
process. This is useful for understanding the process hierarchy and for
debugging purposes.

• SID (Session ID): A unique identifier for a group of processes that are
associated with a particular user session. This is often used for security and
resource allocation purposes.

25
 Unique Identifiers

Unique Identifiers
Lists several process identifiers commonly used in operating systems:

• PGID (Process Group ID): A unique identifier for a group of processes

that are associated with a particular terminal or session. This is often used
for job control and signal handling.

• UID (User ID): A numerical identifier that represents the user associated
with the process. This is used for security and access control purposes.

Task Manager. Task Manager can be opened in a number of ways, but the simplest is to select
Ctrl+Alt+Delete, and then select Task Manager. In Windows, first click More details to expand the
information displayed. From the Processes tab, select Details to see the process ID listed in the PID
column

26
 Unique Identifiers

Unique Identifiers
 are essential for collective communication
because they allow processes to:

27
 Unique Identifiers

Why do we need a unique identifier?

1. To determine the correct path for data to follow between processes,

ensuring efficient and accurate communication.
2. To establish synchronization points between processes, ensuring
that they proceed at the same pace and avoid race conditions.
3. To identify the affected processes and initiate recovery procedures.
This helps to maintain the integrity and reliability of collective
communication.

28
 Unique Identifiers

Conclusion:

Unique identifiers are indispensable for effective

collective communication in parallel and distributed
systems. They provide a mechanism for processes to
identify themselves, address messages, and coordinate
their actions, enabling scalable, reliable, and efficient
communication patterns.

29
THANKS!

30