PARALLEL & DISTRIBUTED COMPUTING
LECTURE NO: 06
PARALLEL PROGRAMMING MODELS
Lecturer: Sardar Un Nisa
[email protected] Department of Computer Science
NUML, Rawalpindi
Parallel Computers
• Programming mode types
– Shared Memory
– Distributed Memory
– Hybrid Model
PARALLEL PROGRAMING
MODELS
Parallel programming models exist as an abstraction above hardware and
memory architectures
These models are NOT specific to a particular type of machine or memory
architecture
These models can (theoretically) be implemented on any underlying hardware
Examples from past
SHARED memory model on a DISTRIBUTED memory machine. Kendall Square
Research (KSR) ALLCACHE approach, “virtual shared memory"
DISTRIBUTED memory model on a SHARED memory machine. Message Passing
Interface (MPI) on SGI Origin 2000, employed the CC-NUMA type of shared memory
architecture, however, MPI commonly done over a network of distributed memory
machines
Which model to use?
Combination of what is available and personal choice
Shared Memory
• Architecture
Processors have direct access to global memory and
I/O through bus or fast switching network
• Cache Coherency Protocol guarantees
consistency of memory and I/O accesses
• Each processor also has its own memory (cache)
• Data structures are shared in global address space
• Concurrent access to shared memory must be coordinated
• Programming Models
– Multithreading (Thread Libraries)
– OpenMP P
P0 P1 ... Pn
0
Cach Cach Cach
e e e
Shared
Bus
Global Shared
Memory
THREADS MODEL
Threads implementations commonly comprise:
A library of subroutines that are called from within parallel
source code
A set of compiler directives imbedded in either serial or parallel
source code
Historically, hardware vendors have implemented their
own proprietary versions of threads, making it difficult for
programmers to develop portable threaded applications
Standardization efforts: POSIX Threads (IEEE POSIX
1003.1c) and OpenMP (Industry standard)
POSIX Part of Unix/Linux, Library based
OpenMP Compiler directive based, Portable / multi-platform
Mircosoft threads, Java, Python threads, CUDA threads for GPUs
OpenMP
• OpenMP: portable shared memory parallelism
• Higher-level API for writing portable
multithreaded applications
• Provides a set of compiler directives and library routines
for parallel application programmers
• API bindings for Fortran, C, and C++
Distributed Memory
Architecture
• Each Processor has direct access only to its local memory
• Processors are connected via high-speed interconnect
• Data structures must be distributed
• Data exchange is done via explicit processor-to-
processor communication: send/receive messages
• Programming Models
– Widely used standard: MPI
– Others: PVM, Express, P4, Chameleon, PARMACS, ...
Memory Memory Memory
P0 P1 ... Pn
Communicati
on
Interconne4 ct
Message Passing Interface
MPI provides:
• Point-to-point communication
• Collective operations
– Barrier synchronization
– gather/scatter operations
– Broadcast, reductions
• Different communication modes
– Synchronous/asynchronous
– Blocking/non-blocking
– Buffered/unbuffered
• Predefined and derived datatypes
• Virtual topologies
• Parallel I/O (MPI 2)
• C/C++ and Fortran bindings
HYBRID MODEL
A hybrid model combines more
than one of the previously
described programming models.
A common example of a hybrid
model is the combination of the
message passing model (MPI) with
the threads model (OpenMP).
Threads perform computationally
intensive kernels using local, on-
node data
Communications between
processes on different nodes
occurs over the network using MPI
HYBRID MODEL
Another similar and increasingly
popular example of a hybrid
model is using MPI with CPU-GPU
(Graphics Processing Unit)
programming.
MPI tasks run on CPUs using local
memory and communicating with
each other over a network.
Computationally intensive kernels
are off-loaded to GPUs onnode.
Data exchange between node-
local memory and GPUs uses
CUDA (or something equivalent)
HIGH LEVEL
PROGRAMMING MODEL
Single Program Multiple Data (SPMD)
Multiple Program Multiple Data (MPMD)
SPMD
Built upon any combination of the previously mentioned
parallel programming models
SINGLE PROGRAM: All tasks execute their copy of the
same program simultaneously. This program can be
threads, message passing, data parallel or hybrid.
MULTIPLE DATA: All tasks may use different data
-tasks do not necessarily have to
execute the entire program
- perhaps only a portion of it
MPMD
built upon any combination of the previously mentioned
parallel programming models
MULTIPLE PROGRAM: Tasks may execute different
programs simultaneously.
The programs can be threads, message passing, data
parallel or hybrid.
MULTIPLE DATA: All tasks may use different data
MPMD applications are not as
common as SPMD applications
THAT’S ALL FOR TODAY!!
