Lecture 4: Principles of Parallel

Algorithm Design

1
 Constructing a Parallel Algorithm

• identify portions of work that can be performed

concurrently
• map concurrent portions of work onto multiple
processes running in parallel
• distribute a program’s input, output, and
intermediate data
• manage accesses to shared data: avoid conflicts
• synchronize the processes at stages of the
parallel program execution

2
 Task Decomposition and Dependency Graphs

Decomposition: divide a computation into smaller

parts, which can be executed concurrently
Task: programmer-defined units of computation.

Task-dependency graph:
Node represent s task.
Directed edge represents
control dependence.

3
 Example 1: Dense Matrix-Vector Multiplication

• Computing y[i] only use ith row of A and b – treat

computing y[i] as a task.
• Remark:
– Task size is uniform
– No dependence between tasks
– All tasks need b
4
 Example 2: Database Query Processing
• Executing the query:
Model =“civic” AND Year = “2001” AND (Color = “green” OR
Color = “white”)
on the following database:

5
• Task: create sets of elements that satisfy a (or several)
criteria.
• Edge: output of one task serves as input to the next

6
• An alternate task-dependency graph for query

• Different task decomposition leads to different

parallelism 7
 Granularity of Task Decomposition
• Fine-grained decomposition: large number of
small tasks
• Coarse-grained decomposition: small number of
large tasks
Matrix-vector multiplication example
-- coarse-grain: each task computes 3 elements of y[]

8
 Degree of Concurrency

• Degree of Concurrency: # of tasks that can

execute in parallel
-- maximum degree of concurrency: largest # of
concurrent tasks at any point of the execution
-- average degree of concurrency: average # of tasks
that can be executed concurrently
• Degree of Concurrency vs. Task Granularity
– Inverse relation

9
 Critical Path of Task Graph
• Critical path: The longest directed path between
any pair of start node (node with no incoming
edge) and finish node (node with on outgoing
edges).
• Critical path length: The sum of weights of nodes
along critical path.
– The weights of a node is the size or the amount of
work associated with the corresponding task
• Average degree of concurrency = total amount of
work / critical path length

10
 Example: Critical Path Length

Task-dependency graphs of query processing operation

Left graph:
Critical path length = 27
Average degree of concurrency = 63/27 = 2.33
Right graph:
Critical path length = 34
Average degree of concurrency = 64/34 = 1.88
11
 Limits on Parallelization
• Facts bounds on parallel execution
– Maximum task granularity is finite
• Matrix-vector multiplication O(n2)
– Interactions between tasks
• Tasks often share input, output, or intermediate data, which may
lead to interactions not shown in task-dependency graph.

Ex. For the matrix-vector multiplication problem, all tasks are

independent, and all need access to the entire input vector b.

12
• Speedup = sequential execution time/parallel
execution time
• Parallel efficiency = sequential execution
time/(parallel execution time × processors used)

13
 Task Interaction Graphs
• Tasks generally share input, output or intermediate data
– Ex. Matrix-vector multiplication: originally there is only one
copy of b, tasks will have to communicate b.
• Task-interaction graph
– To capture interactions among tasks
– Node = task
– Edge(undirected/directed) = interaction or data exchange
• Task-dependency graph vs. task-interaction graph
– Task-dependency graph represents control dependency
– Task-interaction graph represents data dependency
– The edge-set of a task-interaction graph is usually a superset
of the edge-set of the task-dependency graph

14
 Example: Task-Interaction Graph
Sparse matrix-vector multiplication
• Tasks: each task computes an entry of y[]
• Assign ith row of A to Task i. Also assign b[i] to
Task i.

15
 Processes and Mapping

• Mapping: the mechanism by which tasks are

assigned to processes for execution.
• Process: a logic computing agent that performs
tasks, which is an abstract entity that uses the
code and data corresponding to a task to produce
the output of that task.
• Why use processes rather than processors?
– We rely on OS to map processes to physical
processors.
– We can aggregate tasks into a process

16
 Criteria of Mapping
1. Maximize the use of concurrency by mapping independent
tasks onto different processes
2. Minimize the total completion time by making sure that
processes are available to execute the tasks on critical path
as soon as such tasks become executable
3. Minimize interaction among processes by mapping tasks
with a high degree of mutual interaction onto the same
process.

Basis for Choosing Mapping

Task-dependency graph
Makes sure the max. concurrency
Task-interaction graph
Minimum communication.
17
 Example: Mapping Database Query to Processes

P3 P2 P1 P0 P3 P2 P1 P0
P0

P0
P2 P0
P0
P0

• 4 processes can be used in total since the max. concurrency is 4.

• Assign all tasks within a level to different processes.

18
 Decomposition Techniques

How to decompose a computation into a set of

tasks?
 Recursive decomposition
Data decomposition
• Exploratory decomposition
• Speculative decomposition

19
 Recursive Decomposition

• Ideal for problems to be solved by divide-and-

conquer method.
• Steps
1. Decompose a problem into a set of independent
sub-problems
2. Recursively decompose each sub-problem
3. Stop decomposition when minimum desired
granularity is achieved or (partial) result is
obtained
20
 Quicksort Example
Sort a sequence A of n elements in the increasing order.

• Select a pivot
• Partition the sequence around the pivot
• Recursively sort each sub-sequence

Task: the work of partitioning a given sub-sequence

21
 Recursive Decomposition for Finding Min
Find the minimum in an array of numbers A of length n

procedure Serial_Min(A,n) procedure Recursive_MIN(A,n)

begin begin
min = A[0] if (n == 1) then
for i:= 1 to n-1 do min := A[0];
if(A[i] < min) min := A[i] else
endfor; lmin := Recursive_MIN(A,n/2);
return min; rmin := Recursive_MIN(&[A/2],n-n/2);
end Serial_Min if( lmin < rmin) then
min := lmin;
else
min := rmin;
endelse;
endelse;
return min;
end Recursive_MIN

22
 Data Decomposition
• Ideal for problems that operate on large data
structures
• Steps
1. The data on which the computations are
performed are partitioned
2. Data partition is used to induce a partitioning of
the computations into tasks.
• Data Partitioning
– Partition output data
– Partition input data
– Partition input + output data
– Partition intermediate data
23
 Data Decomposition Based on Partitioning Output Data

• If each element of the output can be computed

independently of others as a function of the input.
• Partitioning computations into tasks is natural. Each
task is assigned with the work of computing a
portion of the output.
• Example. Dense matrix-vector multiplication.

24
 Example: Output Data Decomposition
Matrix-matrix multiplication: 𝐶𝐶 = 𝐴𝐴 × 𝐵𝐵
• Partition matrix C into 2× 2 submatrices
• Computation of C then can be partitioned into four tasks.

Remark: data-decomposition is different from task decomposition.

Same data decomposition can have different task decompositions.
25
26
 Data Decomposition Based on Partitioning Input Data

• Ideal if output is a single unknown value or

the individual elements of the output can not
be efficiently determined in isolation.
– Example. Finding the minimum, maximum, or sum
of a set of numbers.
– Example. Sorting a set.
• Partitioning the input data and associating a
task with each partition of the input data.

27
Data Decomposition Based on Partitioning Intermediate Data

• Applicable for problems which can be solved

by multi-stage computations such that the
output of one stage is the input to the
subsequent stage.
• Partitioning can be based on input or output
of an intermediate stage.

28
 Example: Intermediate Data Decomposition

Dense matrix-matrix multiplication

• Original output data decomposition yields a
maximum degree of concurrency of 4.

29
Stage 1: 𝐷𝐷𝑘𝑘,𝑖𝑖,𝑗𝑗 = 𝐴𝐴𝑖𝑖,𝑘𝑘 𝐵𝐵𝑘𝑘,𝑗𝑗

A1,1 B1,1 B1,2 D1,1,1 D1,1,2

A2,1 D1,2,1 D1,2,2

A1,2 D2,1,1 D2,1,2

A2,2 B2,1 B2,2 D2,2,1 D2,2,2

Stage 2: 𝐶𝐶𝑖𝑖,𝑗𝑗 = 𝐷𝐷1,𝑖𝑖,𝑗𝑗 + 𝐷𝐷2,𝑖𝑖,𝑗𝑗

D1,1,1 D1,1,2 D2,1,1 D2,1,2 C1,1 C1,2

+
D1,2,1 D1,2,2 D2,2,1 D2,2,2 C2,1 C2,2

30
 Let 𝑫𝑫𝒌𝒌,𝒊𝒊,𝒋𝒋 = 𝑨𝑨𝒊𝒊,𝒌𝒌 � 𝑩𝑩𝒌𝒌,𝒋𝒋

Task-dependency graph

31
 Owner-Computes Rule

• Decomposition based on partitioning

input/output data is referred to as the owner-
computes rule.
– Each partition performs all the computations involving
data that it owns.
• Input data decomposition
– A task performs all the computations that can be done
using these input data.
• Output data decomposition
– A task computes all the results in the partition
assigned to it.

32
 Characteristics of Tasks
Key characteristics of tasks influencing choice of mapping and
performance of parallel algorithm:
1. Task generation
• Static or dynamic generation
– Static: all tasks are known before the algorithm starts execution. Data or
recursive decomposition often leads to static task generation.
Ex. Matrix-multiplication. Recursive decomposition in finding min. of a set of
numbers.
– Dynamic: the actual tasks and the task-dependency graph are not explicitly
available a priori. Recursive, exploratory decomposition can generate tasks
dynamically.
Ex. Recursive decomposition in Quicksort, in which tasks are generated
dynamically.
2. Task sizes
• Amount of time required to compute it: uniform, non-uniform
3. Knowledge of task sizes
4. Size of data associated with tasks
• Data associated with the task must be available to the process
performing the task. The size and location of data may determine the
data-movement overheads.
33
 Characteristics of Task Interactions

1) Static versus dynamic

– Static: interactions are known prior to execution.
2) Regular versus irregular
– Regular: interaction pattern can be exploited for
efficient implementation.
3) Read-only versus read-write
4) One-way versus two-way

34
 Static vs. Dynamic Interactions

• Static interaction
– Tasks and associated interactions are predetermined:
task-interaction graph and times that interactions
occur are known: matrix multiplication
– Easy to program
• Dynamic interaction
– Timing of interaction or sets of tasks to interact with
can not be determined prior to the execution.
– Difficult to program using massage-passing; Shared-
memory space programming may be simple
35
 Regular vs. Irregular Interactions

• Regular interactions
– Interaction has a spatial structure that can be
exploited for efficient implementation: ring, mesh
Example: Explicit finite difference for solving PDEs.
• Irregular Interactions
– Interactions has no well-defined structure
Example: Sparse matrix-vector multiplication

36
37
 Mapping Technique for Load Balancing
Minimize execution time → Reduce overheads of execution
• Sources of overheads:
– Inter-process interaction
– Idling
– Both interaction and idling are often a function of mapping
• Goals to achieve:
– To reduce interaction time
– To reduce total amount of time some processes being idle
(goal of load balancing)
– Remark: these two goals often conflict
• Classes of mapping:
– Static
– Dynamic
38
Remark:
1. Loading balancing is only a necessary but not sufficient condition for reducing
idling.
• Task-dependency graph determines which tasks can execute in parallel and
which must wait for some others to finish at a given stage.
2. Good mapping must ensure that computations and interactions among processes
at each stage of execution are well balanced.

Two mappings of 12-task decomposition in which the last 4 tasks can be started only
after the first 8 are finished due to task-dependency.
39
 Schemes for Static Mapping

Static Mapping: It distributes the tasks among

processes prior to the execution of the algorithm.

• Mapping Based on Data Partitioning

• Task Graph Partitioning
• Hybrid Strategies

40
 Mapping Based on Data Partitioning

• By owner-computes rule, mapping the relevant

data onto processes is equivalent to mapping
tasks onto processes
• Array or Matrices
– Block distributions
– Cyclic and block cyclic distributions
• Irregular Data
– Example: data associated with unstructured mesh
– Graph partitioning

41
 1D Block Distribution
Example. Distribute rows or columns of matrix to different
processes

42
 Multi-D Block Distribution
Example. Distribute blocks of matrix to different processes

43
 Load-Balance for Block Distribution

Example. 𝑛𝑛 × 𝑛𝑛 dense matrix multiplication 𝐶𝐶 = 𝐴𝐴 × 𝐵𝐵

using 𝑝𝑝 processes
– Decomposition based on output data.
– Each entry of 𝐶𝐶 use the same amount of computation.
– Either 1D or 2D block distribution can be used:
𝑛𝑛
• 1D distribution: rows are assigned to a process
𝑝𝑝
• 2D distribution: 𝑛𝑛/ 𝑝𝑝 × 𝑛𝑛/ 𝑝𝑝 size block is assigned to a process
– Multi-D distribution allows higher degree of concurrency.
– Multi-D distribution can also help to reduce interactions

44
Suppose the size of matrix is 𝑛𝑛 × 𝑛𝑛, and 𝑝𝑝 processes are used.
𝑛𝑛2
(a): A process need to access + 𝑛𝑛2 amount of data
𝑝𝑝
(b): A process need to access 𝑂𝑂(𝑛𝑛2 / 𝑝𝑝) amount of data 45
 Cyclic and Block Cyclic Distributions

• If the amount of work differs for different

entries of a matrix, a block distribution can
lead to load imbalances.
• Example. Doolittle’s method of LU factorization
of dense matrix
– The amount of computation increases from the top
left to the bottom right of the matrix.

46
 Doolittle’s method of LU factorization

𝑎𝑎11 𝑎𝑎12 … 𝑎𝑎1𝑛𝑛 1 0 … 0 𝑢𝑢11 𝑢𝑢12 … 𝑢𝑢1𝑛𝑛

𝑎𝑎21 𝑎𝑎22 … 𝑎𝑎2𝑛𝑛 𝑙𝑙21 1 … 0 0 𝑢𝑢22 … 𝑢𝑢2𝑛𝑛
𝐴𝐴 = ⋮ ⋮ ⋱ ⋮ = 𝐿𝐿𝐿𝐿 = ⋮ ⋮ ⋱ ⋮
⋮ ⋮ ⋱ ⋮
𝑎𝑎𝑛𝑛𝑛 𝑎𝑎𝑛𝑛𝑛 … 𝑎𝑎𝑛𝑛𝑛𝑛 𝑙𝑙𝑛𝑛𝑛 𝑙𝑙𝑛𝑛𝑛 … 1 0 0 … 𝑢𝑢𝑛𝑛𝑛𝑛

By matrix-matrix multiplication

𝑢𝑢1𝑗𝑗 = 𝑎𝑎1𝑗𝑗 , 𝑗𝑗 = 1,2, … , 𝑛𝑛 (1𝑠𝑠𝑠𝑠 row of 𝑈𝑈)

𝑙𝑙𝑗𝑗𝑗 = 𝑎𝑎𝑗𝑗𝑗 /𝑢𝑢11 , 𝑗𝑗 = 1,2, … , 𝑛𝑛 (1𝑠𝑠𝑠𝑠 column of 𝐿𝐿)
For 𝑖𝑖 = 2,3, … , 𝑛𝑛 − 1 do
𝑢𝑢𝑖𝑖𝑖𝑖 = 𝑎𝑎𝑖𝑖𝑖𝑖 − ∑𝑖𝑖−1
𝑡𝑡=1 𝑙𝑙𝑖𝑖𝑖𝑖 𝑢𝑢𝑡𝑡𝑖𝑖

𝑢𝑢𝑖𝑖𝑖𝑖 = 𝑎𝑎𝑖𝑖𝑖𝑖 − ∑𝑖𝑖−1

𝑡𝑡=1 𝑙𝑙𝑖𝑖𝑖𝑖 𝑢𝑢𝑡𝑡𝑡𝑡 for 𝑗𝑗 = 𝑖𝑖 + 1, … , 𝑛𝑛 (𝑖𝑖𝑖𝑖𝑖 row of 𝑈𝑈)
𝑎𝑎𝑗𝑗𝑗𝑗 −∑𝑖𝑖−1
𝑡𝑡=1 𝑙𝑙𝑗𝑗𝑗𝑗 𝑢𝑢𝑡𝑡𝑡𝑡
𝑙𝑙𝑗𝑗𝑗𝑗 = for 𝑗𝑗 = 𝑖𝑖 + 1, … , 𝑛𝑛 (𝑖𝑖𝑖𝑖𝑖 column of 𝐿𝐿)
𝑢𝑢𝑖𝑖𝑖𝑖

End
𝑢𝑢𝑛𝑛𝑛𝑛 = 𝑎𝑎𝑛𝑛𝑛𝑛 − ∑𝑛𝑛−1
𝑡𝑡=1 𝑙𝑙𝑛𝑛𝑛𝑛 𝑢𝑢𝑡𝑡𝑡𝑡

47
 Serial Column-Based LU

• Remark: Matrices L and U share space with A

48
Work used to compute Entries of L and U

49
• Block distribution of LU factorization tasks
leads to load imbalance.

50
 Block-Cyclic Distribution

• A variation of block distribution that can be

used to alleviate the load-imbalance.

• Steps
1. Partition an array into many more blocks than
the number of available processes
2. Assign blocks to processes in a round-robin
manner so that each process gets several non-
adjacent blocks.

51
(a) The rows of the array are grouped into blocks each consisting of two rows,
resulting in eight blocks of rows. These blocks are distributed to four processes
in a wrap-around fashion.
(b) The matrix is blocked into 16 blocks each of size 4×4, and it is mapped onto a
2×2 grid of processes in a wraparound fashion.
• Cyclic distribution: when the block size =1
52
Randomized Block Distribution

53
 Graph Partitioning
Sparse-matrix vector multiplication

Work: nodes
Interaction/communication: edges

Partition the graph:

Assign roughly same number of nodes to each process
Minimize edge count of graph partition
54
Finite element simulation of water contaminant in a lake.
• Goal of partitioning: balance work & minimize communication

Random Partitioning Partitioning for Minimizing Edge-Count

• Assign equal number of nodes (or cells) to each process

– Random partitioning may lead to high interaction overhead due to data
sharing
• Minimize edge count of the graph partition
– Each process should get roughly the same number of elements and the
number of edges that cross partition boundaries should be minimized as 55well.
 Mappings Based on Task Partitioning

• Mapping based on task partitioning can be used

when computation is naturally expressed in the
form of a static task-dependency graph with
known sizes.
• Finding optimal mapping minimizing idle time and
minimizing interaction time is NP-complete
• Heuristic solutions exist for many structured
graphs

56
 Mapping a Sparse Graph
Example. Sparse matrix-vector multiplication using 3
processes
• Arrow distribution

57
• Partitioning task-interaction graph to reduce
interaction overhead

58
 Techniques to Minimize Interaction Overheads

• Maximize data locality

– Maximize the reuse of recently accessed data
– Minimize volume of data-exchange
• Use high dimensional distribution. Example: 2D block
distribution for matrix multiplication
– Minimize frequency of interactions
• Reconstruct algorithm such that shared data are accessed
and used in large pieces.
• Combine messages between the same source-destination
pair

59
• Minimize contention and hot spots
– Competition occur when multi-tasks try to access the same
resources concurrently: multiple processes sending
message to the same process; multiple simultaneous
accesses to the same memory block

𝑝𝑝−1
• Using 𝐶𝐶𝑖𝑖,𝑗𝑗 = ∑𝑘𝑘=0 𝐴𝐴𝑖𝑖,𝑘𝑘 𝐵𝐵𝑘𝑘,𝑗𝑗 causes contention. For example, 𝐶𝐶0,0 ,
𝐶𝐶0,1 , 𝐶𝐶0, 𝑝𝑝−1 attempt to read 𝐴𝐴0,0 , at the same time.
• A contention-free manner is to use:
𝑝𝑝−1
𝐶𝐶𝑖𝑖,𝑗𝑗 = ∑𝑘𝑘=0 𝐴𝐴𝑖𝑖, 𝑖𝑖+𝑗𝑗+𝑘𝑘 % 𝑝𝑝 𝐵𝐵 𝑖𝑖+𝑗𝑗+𝑘𝑘 % 𝑝𝑝,𝑗𝑗
All tasks 𝑃𝑃∗,𝑗𝑗 that work on the same row of C access block
𝐴𝐴𝑖𝑖, 𝑖𝑖+𝑗𝑗+𝑘𝑘 % 𝑝𝑝 , which is different for each task. 60
• Overlap computations with interactions
– Use non-blocking communication
• Replicate data or computations
– Some parallel algorithm may have read-only access to
shared data structure. If local memory is available,
replicate a copy of shared data on each process if
possible, so that there is only initial interaction during
replication.
• Use collective interaction operations
• Overlap interactions with other interactions

61
 Parallel Algorithm Models
• Data parallel
– Each task performs similar operations on different data
– Typically statically map tasks to processes
• Task graph
– Use task dependency graph to promote locality or reduce
interactions
• Master-slave
– One or more master processes generating tasks
– Allocate tasks to slave processes
– Allocation may be static or dynamic
• Pipeline/producer-consumer
– Pass a stream of data through a sequence of processes
– Each performs some operation on it
• Hybrid
– Apply multiple models hierarchically, or apply multiple models
in sequence to different phases
62
• Reference
– A. Grama, et al. Introduction to Parallel
Computing. Chapter 3.

63