Recap

• Issues in Parallel Computing

• Design of Parallel Computing
• Design of Efficient Algorithms
• Method of Evaluating Parallel Algorithms
• Parallel Computing Language
• Parallel Progarming Tool
• Portable Parallel Programs
• Types of Parallel Computing
• SISD
• SIMD
• MISD
• MIMD
 Communication Model of Parallel Platforms

• There are two primary forms of data exchange between parallel tasks –
• accessing a shared data space (Shared Memory Architecture)
• exchanging messages. (Distributed Memory Architecture)
• Platforms that provide a shared data space are called shared-address-space machines or
multiprocessors machines.
• Platforms that support messaging are also called message passing platforms or
multicomputers machines.

Distributed Memory Architecture

Shared Memory Architecture
 Shared-Address-Space Platforms
• Part (or all) of the memory is accessible to all
processors.
• Processors interact by modifying data objects stored in
this shared-address-space.
• Consistency problems Shared Memory Address
• Global view of Memory
• Memory in shared-address-space platforms can be local
or global (common to all processors).

• If the time taken by a processor to access any memory

UMA NUMA
word in the system (global or local) is identical, the
platform is classified as a uniform memory access
(UMA)

• On the other hand if the time taken to access certain

memory words is longer than others, the platform is
called a non-uniform memory access (NUMA).
 Uniform Memory Access - UMA
• Identical Processor, Symmetric
Multiprocessor (SMP).
• Equal access and access times to memory. CPU
• Sometimes called CC-UMA: Cache
Coherent UMA.
• Cache coherent mean if one processor
update a location in shared memory, all the CPU Memory CPU

other processor know about the update.

• Cache coherency is accomplished at
hardware level CPU
 Non-Uniform Memory Access - NUMA
• NUMA memory access is a computer
memory designed used in multiprocessing,
where the memory access time depends on
memory location relative to the processors.
• The identical processors are connected to a
network and each processor has a portion
of memory attached directly to it.
• The processors can have mapping to
memory connected to other processors
NUMA and UMA Shared-Address-Space Platforms

Typical shared-address-space architectures: (a) Uniform-memory access

shared-address-space computer; (b) Uniform-memory-access shared-
address-space computer with caches and memories; (c) Non-uniform-
memory-access shared-address-space computer with local memory
only.
NUMA and UMA Shared-Address-Space Platforms

• The distinction between NUMA and UMA platforms is important from the point of
view of algorithm design. NUMA machines require locality from underlying algorithms
for performance.
• Programming these platforms is easier since reads and writes are implicitly visible to
other processors.
• However, read-write data to shared data must be coordinated (this will be discussed in
greater detail when we talk about threads programming).
• Caches in such machines require coordinated access to multiple copies. This leads to
the cache coherence problem.
• A weaker model of these machines provides an address map, but not coordinated
access. These models are called non cache coherent shared address space machines.
Message-Passing Platforms

• The architecture is used to communicate data among a set of processors without the
need for global memory.
• These platforms comprise of a set of processors and their own (exclusive) memory.
• In the message passing model, an application runs as a collection of autonomous
processes, each with its own local memory.
• In this model, processes communicate with other processes by sending and receiving
messages
• Massage passing model is used widely on parallel computer with distributed memory
and on clusters of servers.
• Libraries such as MPI (Message Passing Interface) and PVM (Parallel Virtual
Machines) provide such primitives.
Message Passing vs. Shared Address Space Platforms

How Communication is achieved between tasks

Message Passing Model:
 Explicit communication via messages
 Loose coupling of program components

Shared Memory programming Model

 Implicit communication via memory operation (load/store)
 Tight coupling of program components
Message Passing vs. Shared Address Space Platforms

How tasks communication is supported in Hardware

 Shared Memory Model
 All processors see a global shared address space
o Ability to access all memory from each processor
 A write to a location is visible to the reads of other processors.
Message Passing Model
 No global shared address space
 Send and receive variants are the only method of communication between
processors.
Limitations of Memory System Performance

• Memory system, and not processor speed, is often the bottleneck for many applications.
• Memory system performance is largely captured by two parameters:
• Latency and
• Bandwidth.
• Latency is the time from the issue of a memory request to the time the data is available
at the processor.
• Bandwidth is the rate at which data can be pumped to the processor by the memory
system.
• .
Memory System Performance: Bandwidth and Latency
• It is very important to understand the difference between
latency and bandwidth.
• Consider the example of a fire-hose. If the water comes out
of the hose two seconds after the hydrant is turned on, the
latency of the system is two seconds.
• Once the water starts flowing, if the hydrant delivers water
at the rate of 5 gallons/second, the bandwidth of the system
is 5 gallons/second.
• If you want immediate response from the hydrant, it is
important to reduce latency.
• If you want to fight big fires, it is important to high
bandwidth.
• .
Memory Latency: An Example
• Consider a processor operating at 1 GHz (1 ns clock) connected to a DRAM with a
latency of 100 ns (no caches). Assume that the processor has two multiply-add units
and is capable of executing four instructions in each cycle of 1 ns. The following
observations follow:
• The peak processor rating is 4 GFLOPS.
• Since the memory latency is equal to 100 cycles and block size is one word, every
time a memory request is made, the processor must wait 100 cycles before it can
process the data
Memory Latency: An Example
• On the above architecture, consider the problem of computing a dot-product of two
vectors.
• A dot-product computation performs one multiply-add on a single pair of vector
elements, i.e., each floating point operation requires one data fetch.
• It follows that the peak speed of this computation is limited to one floating point
operation every 100 ns, or a speed of 10 MFLOPS, a very small fraction of the
peak processor rating!
Improving Effective Memory Latency Using Caches
• Caches are small and fast memory elements between the processor and DRAM.
• This memory acts as a low-latency high-bandwidth storage.
• If a piece of data is repeatedly used, the effective latency of this memory system can be
reduced by the cache.
• The fraction of data references satisfied by the cache is called the cache hit ratio of the
computation on the system.
• Cache hit ratio achieved by a code on a memory system often determines its
performance.
Impact of Caches: Example

• Consider the architecture from the previous example.

• In this case, we introduce a cache of size 32 KB with a latency of 1 ns or one cycle.
• We use this setup to multiply two matrices A and B of dimensions 32 × 32.
• We have carefully chosen these numbers so that the cache is large enough to store
matrices A and B, as well as the result matrix C
Impact of Caches: Example (continued)

• The following observations can be made about the problem:

• Fetching the two matrices into the cache corresponds to fetching 2K words, which
takes approximately 200 µs.
• Multiplying two n × n matrices takes 2n3 operations. For our problem, this
corresponds to 64K operations, which can be performed in 16K cycles (or 16 µs)
at four instructions per cycle.
• The total time for the computation is therefore approximately the sum of time for
load/store operations and the time for the computation itself, i.e., 200 + 16 µs.
• This corresponds to a peak computation rate of 64K/216 or 303 MFLOPS.
Impact of Memory Bandwidth

• Memory bandwidth is determined by the bandwidth of the memory bus as well as the
memory units.
• Memory bandwidth can be improved by increasing the size of memory blocks.
• The underlying system takes L time units (where L is the latency of the system) to
deliver B units of data (where B is the block size).
Impact of Memory Bandwidth: Example

• Consider the same setup as before, except in this case, the block size is 4 words instead
of 1 word. We repeat the dot-product computation in this scenario:
• Assuming that the vectors are laid out linearly in memory, eight FLOPs (four
multiply-adds) can be performed in 200 cycles.
• This is because a single memory access fetches four consecutive words in the
vector.
• Therefore, two accesses can fetch four elements of each of the vectors. This
corresponds to a FLOP every 25 ns, for a peak speed of 40 MFLOPS.
Impact of Memory Bandwidth

• It is important to note that increasing block size does not change latency of the system.
• Physically, the scenario illustrated here can be viewed as a wide data bus (4 words or
128 bits) connected to multiple memory banks.
• In practice, such wide buses are expensive to construct.
• In a more practical system, consecutive words are sent on the memory bus on
subsequent bus cycles after the first word is retrieved.
Impact of Memory Bandwidth

• The above examples clearly illustrate how increased bandwidth results in higher peak
computation rates.
• The data layouts were assumed to be such that consecutive data words in memory were
used by successive instructions (spatial locality of reference).
• If we take a data-layout centric view, computations must be reordered to enhance
spatial locality of reference.
Impact of Memory Bandwidth: Example

Consider the following code fragment:

for (i = 0; i < 1000; i++)
column_sum[i] = 0.0;
for (j = 0; j < 1000; j++)
column_sum[i] += b[j][i];

The code fragment sums columns of the matrix

b into a vector column_sum.
Impact of Memory Bandwidth: Example
• The vector column_sum is small and easily fits into the cache
• The matrix b is accessed in a column order.
• The strided access results in very poor performance.

Multiplying a matrix with a vector: (a) multiplying column-by-column,

keeping a running sum; (b) computing each element of the result as a
dot product of a row of the matrix with the vector.
Impact of Memory Bandwidth: Example

We can fix the above code as follows:

for (i = 0; i < 1000; i++)
column_sum[i] = 0.0;
for (j = 0; j < 1000; j++)
for (i = 0; i < 1000; i++)
column_sum[i] += b[j][i];

In this case, the matrix is traversed in a row-order and

performance can be expected to be significantly better.
Memory System Performance: Summary

• The series of examples presented in this

section illustrate the following concepts:
– Exploiting spatial and temporal locality in
applications is critical for amortizing memory
latency and increasing effective memory
bandwidth.
– The ratio of the number of operations to number
of memory accesses is a good indicator of
anticipated tolerance to memory bandwidth.
– Memory layouts and organizing computation
appropriately can make a significant impact on the
spatial and temporal locality.