Scribd
Upload
20 views17 pages

Week 01 Lec 2 - 05!03!2024 (Types of Parallelism)

Uploaded by

Kashif Shahid
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
20 views17 pages

Week 01 Lec 2 - 05!03!2024 (Types of Parallelism)

Uploaded by

Kashif Shahid
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 17

CS416 – Parallel

and Distributed
Computing
Lecture # 02
Spring 2021
Agenda
2

 Amdahl’s Law of Parallel Speedup

 Karp-Flatt Metric

 Types of Parallelism
 Data-parallelism
 Functional-parallelism
 Pipelining

 Multi-processor vs Multi-computer

 Cluster vs Network of workstations


CS443
Amdahl’s Law
3
 Amdahl’s was formulized in 1967
 It shows an upper-bound on the maximum speedup
that can be achieved
 Suppose you are going to design a parallel algorithm
for a problem
 Further suppose that fraction of total time that the
algorithm must consume in serial executions is ‘F’
 This implies fraction of parallel potion is (1- F)
 Now, Amdahl’s law states that

 Here ‘p’ is total number of available processing


nodes.
CS443
Amdahl’s Law
4

Derivation
 Let’s suppose you have a sequential code for a
problem that can be executed in total T(s)time.
 T(p) be the parallel time for the same algorithm
over p processors.
Then speedup can be calculated using:-

 T(p) can be calculated as:

CS443
Amdahl’s Law
5

Derivation
 Again

 What if you have infinite number of processors?


 What you have to do for further speedup?

CS443
Amdahl’s Law
6

 Example 1: Suppose 70% of a sequential


algorithm is parallelizable portion. The remaining
part must be calculated sequentially. Calculate
maximum theoretical speedup for parallel variant
of this algorithm using i). 4 processors and ii).
infinite processors.

 F and 1-F use Amdahl’s law to calculate


theoretical speedups.

CS443
Amdahl’s Law
7

 Example 2: Suppose 25% of a sequential


algorithm is parallelizable portion. The remaining
part must be calculated sequentially. Calculate
maximum theoretical speedup for parallel variant
of this algorithm using 5 processors and infinite
processors.
 ???

 Little challenge: Determine, according to


Amdahl's law, how many processors are needed to
achieve maximum theoretical speedup while
sequential portion remains the same?
 The answer may be surprising?
CS443
8
Karp-Flatt Metric

CS443
Karp-Flatt Metric
9

 The metric is used to calculate serial fraction for a


given parallel configuration.
 i.e., if a parallel program is exhibiting a speedup S while
using P processing units then serial fraction e is given by
:-

Example task: Suppose in a parallel program,


for 5 processors, you gained a speedup of 1.25x,
determine sequential fraction of your program.

CS443
10
Types of Parallelism

CS443
Types of Parallelism
11

1. Data-parallelism
 When there are independent sub-tasks applying
the same operation to different elements of a
data set
 When just distributing the data provides sufficient
parallelism
 Example code
for i=0 to 99 do
a[ i ] = b[ i ] + c [ i ]
Endfor
 Here same operation (i.e., addition) is being performed on
first 100 elements of ‘b’ and ‘c’
 All 100 iterations of the loop could be executed
CS443
simultaneously.
Types of Parallelism
12

2. Functional-parallelism
 When there are independent tasks applying
different operations to different data elements
 Example code
1) a=2
2) b=3
3) m= (a+b)/2
4) s= ( + )/2
5) v= s -
 Here (1, 2) and (3, 4) statements could be
performed concurrently.

CS443
Types of Parallelism
13

3. Pipelining
 Usually used for the problems where single
instance of the problem can not be parallelized
 The output of one stage is input of the other stage
 Divide whole computation of each instance into
multiple stages (if there are multiple instances of
the problem)
 An effective method of attaining parallelism
(concurrency) on the uniprocessor architectures
 Also depends on pipelining abilities of the
processor

CS443
Types of
Parallelism

3. Pipelining
 Example:
Assembly line
analogy

Sequential Execution

14 CS443
Types of
Parallelism

3. Pipelining
 Example:
Assembly line
analogy

Pipelining

15 CS443
Types of
Parallelism

3. Pipelining
 Example:
Overlap
instructions in
a single
instruction
cycle to
achieve
parallelism 4-stage Pipelining

16 CS443
Questions
2
6

CS443

You might also like