Principles of Parallel Algorithm Design

Ananth Grama, Anshul Gupta, George Karypis, and Vipin Kumar

To accompany the text “Introduction to Parallel Computing”,

Addison Wesley, 2003.
 Unit 2: Parallel Algorithm Design

Syllabus:
Principles of Parallel Algorithm Design:
Preliminaries, Decomposition Techniques, Characteristics
of Tasks and Interactions, Mapping Techniques for Load
Balancing, Methods for Containing Interaction overheads

Parallel Algorithm Models: Data, Task, Work Pool and

Master Slave Model

Complexities: Sequential and Parallel Computational

Complexity, Anomalies in Parallel Algorithms.
Course Object- Outcome Mapping

Course Objectives:
To analyze the performance and modeling of parallel
programs

Course Outcomes:
CO2: Design and Develop an efficient parallel
algorithm to solve given problem
 Parallel Algorithm

Recipe to solve a problem using multiple processors

Typical steps for constructing a parallel algorithm
— identify what pieces of work can be performed concurrently
— partition and map work onto independent processors
— distribute a program’s input, output, and intermediate data
— coordinate accesses to shared data: avoid conflicts
— ensure proper order of work using synchronization
Why “typical”? Some of the steps may be omitted.
— if data is in shared memory, distributing it may be unnecessary
— if using message passing, there may not be shared data
— the mapping of work to processors can be done statically by the
programmer or dynamically by the runtime

4
 Chapter Overview: Algorithms and Concurrency

• Introduction to Parallel Algorithms

– Tasks and Decomposition
– Processes and M apping
– Processes Versus Processors

• Decomposition Techniques
– Recursive Decomposition
– Recursive Decomposition
– Exploratory Decomposition
– Hybrid Decomposition

• Characteristics of Tasks and Interactions

– Task Generation, Granularity, and Context
– Characteristics of Task Interactions.
 Chapter Overview: Concurrency and Mapping

• Mapping Techniques for Load Balancing

– Static and Dynamic M apping

• Methods for Minimizing Interaction Overheads

– Maximizing Data Locality
– Minimizing Contention and Hot-Spots
– Overlapping Communication and Computations
– Replication vs. Communication
– Group Communications vs. Point-to-Point Communication

• Parallel Algorithm Design Models

– Data-Parallel, Work-Pool, Task Graph, Master-Slave, Pipeline, and Hybrid
Models
 Preliminaries: Decomposition, Tasks, and
Dependency Graphs

• The first step in developing a parallel algorithm is to decompose

the problem into tasks that can be executed concurrently

• A given problem may be docomposed into tasks in many

different ways.

• Tasks may be of same, different, or even interminate sizes.

• A decomposition can be illustrated in the form of a directed

graph with nodes corresponding to tasks and edges indicating
that the result of one task is required for processing the next.
Such a gra ph is called a task d e p e n d e n cy gra ph.
 Example: Multiplying a Dense Matrix with a Vector
A b y
0 1 n
Task 1
2

n-1
Task n

Computation of each element of output vector y is independent of other

elements. Based on this, a dense matrix-vector product can be decomposed
into n tasks. The figure highlights the portion of the matrix and vector
accessed by Task 1.

Observations: While tasks share data (namely, the vector b), they
do not have any control dependencies – i.e., no task needs to
wait for the (partial) completion of any other. All tasks are of the
same size in terms of number of operations. Is this the maximum
number of tasks we could d e c o m p o s e this problem into?
 Example: Database Query Processing

Consider the execution of the query:

MODEL = ‘‘CIVIC’’ AND YEAR = 2001 AND
(COLOR = ‘‘GREEN’’ OR COLOR = ‘‘WHITE)

on the following database:

ID# Model Year Color Dealer Price

4523 Civic 2002 Blue MN $18,000
3476 Corolla 1999 White IL $15,000
7623 Camry 2001 Green NY $21,000
9834 Prius 2001 Green CA $18,000
6734 Civic 2001 White OR $17,000
5342 Altima 2001 Green FL $19,000
3845 Maxima 2001 Blue NY $22,000
8354 Accord 2000 Green VT $18,000
4395 Civic 2001 Red CA $17,000
7352 Civic 2002 Red WA $18,000
 Example: Database Query Processing

The execution of the query can be divided into subtasks in

various ways. Each task can be thought of as generating an
intermediate table of entries that satisfy a particular clause.
ID# Year
ID# Model ID# Color
7623 2001
4523 Civic 6734 2001 ID# Color 7623 Green
6734 Civic 5342 2001 9834 Green
4395 Civic 3845 2001 3476 White 5342 Green
7352 Civic 4395 2001 6734 White 8354 Green

Civic 2001 White Green

ID# Color

ID# Model Year 3476 White

7623 Green
6734 Civic 2001 Civic AND 2001 White OR Green 9834 Green
4395 Civic 2001 6734 White
5342 Green
8354 Green

Civic AND 2001 AND (White OR Green)

ID# Model Year Color

6734 Civic 2001 White

Decomposing the given query into a number of tasks.

Edges in this graph denote that the output of one task
is needed to accomplish the next.
 Example: Database Query Processing

Note that the same problem can be decomposed into subtasks

in other ways as well.
ID# Year
ID# Model ID# Color
7623 2001
4523 Civic 6734 2001 7623 Green
ID# Color
6734 Civic 5342 2001 9834 Green
4395 Civic 3845 2001 3476 White 5342 Green
7352 Civic 4395 2001 6734 White 8354 Green

Civic 2001 White Green

ID# Color
3476 White
White OR Green
7623 Green
9834 Green
6734 White
5342 Green
8354 Green

2001 AND (White or Green) ID# Color Year

7623 Green 2001
6734 White 2001
5342 Green 2001

Civic AND 2001 AND (White OR Green)

ID# Model Year Color

6734 Civic 2001 White

An alternate decomposition of the given problem into

subtasks, along with their data dependencies.
Different task decompositions may lead to significant differences
with respect to their eventual parallel performance.
 Granularity of Task Decompositions

• The number of tasks into which a problem is decomposed

determines its granularity.

• Decomposition into a large number of tasks results in fine-

grained decomposition and that into a small number of tasks
results in a coarse grained decomposition.
A b y
0 1 ... n

Task 1

Task 2

Task 3

Task 4

A coarse grained counterpart to the dense matrix-vector product

example. Each task in this example corresponds to the computation of three
elements of the result vector.
 Degree of Concurrency

• The number of tasks that can be executed in parallel is the

degree of concurrency of a decomposition.

• Since the number of tasks that can be executed in parallel

may change over program execution, the maximum degree
of concurrency is the maximum number of such tasks at any
point during execution. What is the maximum degree of
concurrency of the da ta base query examples?

• The average degree of concurrency is the average number

of tasks that can be processed in parallel over the execution
of the program. Assuming that e a c h tasks in the database
example takes identical processing time, what is the average
degree of concurrency in e a c h decomposition?

• The degree of concurrency increases as the decomposition

becomes finer in granularity and vice versa.
 Critical Path Length

• A directed path in the task dependency graph represents a

sequence of tasks that must be processed one after the other.

• The longest such path determines the shortest time in which the
program can be executed in parallel.

• The length of the longest path in a task dependency graph is

called the critical path length.
 Critical Path Length

Consider the task dependency graphs of the two database

query decompositions:

Task 4 Task 3 Task 2 Task 1 Task 4 Task 3 Task 2 Task 1

10 10 10 10 10 10 10 10

6 Task 5
9 Task 6 6 Task 5
11 Task 6

8 Task 7 7 Task 7

(a) (b)

What are the critical path lengths for the two task dependency graphs?
If each task takes 10 time units, what is the shortest parallel execution time
for each decomposition? How many processors are needed in each case to
achieve this minimum parallel execution time? What is the maximum degree
of concurrency?
 Limits on Parallel Performance

• It would appear that the parallel time can be made arbitrarily

small by making the decomposition finer in granularity.

• There is an inherent bound on how fine the granularity of a

computation can be. For example, in the c a s e of multiplying
a dense matrix with a vector, there c a n b e no more than (n 2 )
concurrent tasks.

• Concurrent tasks may also have to exchange data with other

tasks. This results in communication overhead. The tradeoff
between the granularity of a decomposition and associated
overheads often determines performance bounds.
 Amdahl’s Law

A hard limit on the speedup that can

be obtained using multiple CPUs
Two expressions of Amdahl’s law
—execution time on N CPUs

—speedup on N processors

https://en.wikipedia.org/wiki/Amdahl's_law

17
 Amdahl’s Law

https://en.wikipedia.org/wiki/Amdahl's_law
18
 Task Interaction Graphs

• Subtasks generally exchange data with others in a decomposition.

For example, even in the trivial decomposition of the dense
matrix-vector product, if the vector is not replicated across all
tasks, they will have to communicate elements of the vector.

• The gra ph of tasks (nodes) and their interactions/data

exchange (edges) is referred to as a task interaction graph.

• Note that task interaction graphs represent data dependencies,

whereas task d e p e n d e n c y graphs represent control dependencies.
 Task Interaction Graph Example

Sparse matrix-vector multiplication

• Computation of each result element = independent task
• Only non-zero elements of sparse matrix A participate
• If, b is partitioned among tasks …
structure of the task interaction graph = graph of the matrix A
(i.e. the graph for which A represents the adjacency structure)

20
Interaction Graphs, Granularity, & Communication
• Finer task granularity increases communication overhead
• Example: sparse matrix-vector product interaction graph

• Assumptions:
— each node takes unit time to process
— each interaction (edge) causes an overhead of a unit time
• If node 0 is a task: communication = 3; computation = 4
• If nodes 0, 4, and 5 are a task: communication = 5; computation
= 15
— coarser-grain decomposition  smaller communication/computation21
 Processes and Mapping

• In general, the number of tasks in a decomposition exceeds

the number of processing elements available.

• For this reason, a parallel algorithm must also provide a

ma pping of tasks to processes.

Note: We refer to the mapping as being from tasks to processes, as opposed

to processors. This is because typical programming APIs, as we shall see, do
not allow easy binding of tasks to physical processors. Rather, we aggregate
tasks into processes and rely on the system to map these processes to physical
processors. We use processes, not in the UNIX sense of a process, rather, simply
as a collection of tasks and associated data.
 Processes and Mapping

• Appropriate mapping of tasks to processes is critical to the

parallel performance of an algorithm.

• Mappings are determined by both the task dependency and

task interaction gra phs.

• Task dependency graphs can be used to ensure that work

is equally spread across all processes at any point (minimum
idling and optimal load balance).

• Task interaction graphs can be used to make sure that

processes need minimum interaction with other processes
(minimum communication).
 Processes and Mapping

An appropriate mapping must minimize parallel execution time

by:

• Mapping independent tasks to different processes.

• Assigning tasks on critical path to processes as soon as they

become available.

• Minimizing interaction between processes by mapping tasks

with dense interactions to the same process.

Note: These criteria often conflict eith each other. For example,
a decomposition into one task (or no decomposition at all)
minimizes interaction but does not result in a speedup at all! Can
you think of other such conflicting cases?
 Processes and Mapping: Example

Task 4 Task 3 Task 2 Task 1 Task 4 Task 3 Task 2 Task 1

10 10 10 10 10 10 10 10
P3 P2 P1 P0 P3 P2 P1 P0

P0 6 Task 5
P2 9 Task 6 P0 6 Task 5
P0 11 Task 6

P0 8 Task 7 P0 7 Task 7

(a) (b)

Mapping tasks in the database query decomposition to

processes. These mappings were arrived at by viewing the
dependency graph in terms of levels (no two nodes in a level
have dependencies). Tasks within a single level are then assigned
to different processes.
 Decomposition Techniques

So how does one decompose a task into various subtasks?

While there is no single recipe that works for all problems, we
present a set of commonly used techniques that apply to broad
classes of problems. These include:

• recursive decomposition

• data decomposition

• exploratory decomposition

• speculative decomposition
 Recursive Decomposition

• Generally suited to problems that are solved using the divide-

and-conquer strategy.

• A given problem is first decomposed into a set of sub-problems.

• These sub-problems are recursively decomposed further until a

desired granularity is reached.
 Recursive Decomposition: Example

A classic example of a divide-and-conquer algorithm on which

we can apply recursive decomposition is Quicksort.
5 12 11 1 10 6 8 3 7 4 9 2

1 3 4 2 5 12 11 10 6 8 7 9

1 2 3 4 5 6 8 7 9 12 11 10

1 2 3 4 5 6 7 8 9 10 12 11

5 6 7 8 10 11 12

11 12

In this example, once the list has been partitioned around the pivot,
each sublist can be processed concurrently (i.e., each sublist represents an
independent subtask). This can be repeated recursively.
 Recursive Decomposition: Example

The problem of finding the minimum number in a given list

(or indeed any other associative operation such as sum, AND,
etc.) can be fashioned as a divide-and-conquer algorithm. The
following algorithm illustrates this.

We first start with a simple serial loop for computing the

minimum entry in a given list:

1. procedure SERIAL MIN (A , n)

2. begin
3. min = A [0];
4. for i := 1 to n −1 do
5. if (A [i ] < min) min := A [i ];
6. endfor;
7. return min;
8. end SERIAL MIN
 Recursive Decomposition: Example

We can rewrite the loop as follows:

1. procedure RECURSIVE MIN (A , n)

2. begin
3. if (n = 1) then
4. min := A [0];
5. else
6. lmin := RECURSIVE MIN (A , n/2);
7. rmin := RECURSIVE MIN (&(A[n/2]), n −n/2);
8. if (lmin < rmin) then
9. min := lmin;
10. else
11. min := rmin;
12. endelse;
13. endelse;
14. return min;
15. end RECURSIVE MIN
 Recursive Decomposition: Example

The code in the previous foil can be decomposed naturally using

a recursive decomposition strategy. We illustrate this with the
following example of finding the minimum number in the set {4,
9, 1, 7, 8, 11, 2, 12}. The task dependency graph associated with
this computation is as follows:

min(1,2)

min(4,1) min(8,2)

min(4,9) min(1,7) min(8,11) min(2,12)

 Data Decomposition

• Identify the data on which computations are performed.

• Partition this data across various tasks.

• This partitioning induces a decomposition of the problem.

• Data can be partitioned in various ways – this critically impacts

performance of a parallel algorithm.
 Data Decomposition: Output Data Decomposition

• Often, each element of the output can be computed

independently of others (but simply as a function of the input).

• A partition of the output across tasks decomposes the problem

naturally.
 Output Data Decomposition: Example

Consider the problem of multiplying two n × n matrices A and B

to yield matrix C. The output matrix C can be partitioned into four
tasks as follows:

A B C
. →
A 2,1 B2,1 C 2,1

Task 1: C 1, 1 = A 1, 1B 1, 1 + A 1, 2B 2, 1
Task 2: C 1, 2 = A 1, 1B 1, 2 + A 1, 2B 2, 2

Task 3: C 2, 1 = A 2, 1B 1, 1 + A 2, 2B 2, 1

Task 4: C 2, 2 = A 2, 1B 1, 2 + A 2, 2B 2, 2
 Output Data Decomposition: Example

A partitioning of output data does not result in a unique

decomposition into tasks. For example, for the same problem
as in previus foil, with identical output data distribution, we can
derive the following two (other) decompositions:

Decomposition I Decomposition II

Task 1: C1,1 = A1,1B1,1 Task 1: C1,1 = A1,1B1,1

Task 2: C1,1 = C1,1 + A1,2B2,1 Task 2: C1,1 = C1,1 + A1,2B2,1
Task 3: C1,2 = A1,1B1,2 Task 3: C1,2 = A1,2B2,2
Task 4: C1,2 = C1,2 + A1,2B2,2 Task 4: C1,2 = C1,2 + A1,1B1,2
Task 5: C2,1 = A2,1B1,1 Task 5: C2,1 = A2,2B2,1
Task 6: C2,1 = C2,1 + A2,2B2,1 Task 6: C2,1 = C2,1 + A2,1B1,1
Task 7: C2,2 = A2,1B1,2 Task 7: C2,2 = A2,1B1,2
Task 8: C2,2 = C2,2 + A2,2B2,2 Task 8: C2,2 = C2,2 + A2,2B2,2
 Output Data Decomposition: Example

Consider the problem of counting the instances of given itemsets in a

database of transactions. In this case, the output (itemset frequencies) can
be partitioned across tasks.
(a) Transactions (input), itemsets (input), and frequencies (output)

A, B, C, E, G, H A, B, C 1
B, D, E, F, K, L D, E 3

Itemset Frequency
Database Transactions

A, B, F, H, L C, F, G 0

Itemsets
D, E, F, H A, E 2
F, G, H, K, C, D 1
A, E, F, K, L D, K 2
B, C, D, G, H, L B, C, F 0
G, H, L C, D, K 0
D, E, F, K, L
F, G, H, L

(b) Partitioning the frequencies (and itemsets) among the tasks

1 1
Itemset Frequency

Itemset Frequency
A, B, C, E, G, H A, B, C A, B, C, E, G, H C, D
Itemsets

Itemsets
B, D, E, F, K, L D, E 3 B, D, E, F, K, L D, K 2
Database Transactions

Database Transactions

A, B, F, H, L C, F, G 0 A, B, F, H, L B, C, F 0

D, E, F, H A, E 2 D, E, F, H C, D, K 0
F, G, H, K, F, G, H, K,
A, E, F, K, L A, E, F, K, L
B, C, D, G, H, L B, C, D, G, H, L
G, H, L G, H, L
D, E, F, K, L D, E, F, K, L
F, G, H, L F, G, H, L

task 1 task 2
 Output Data Decomposition: Example

From the previous example, the following observations can be

made:

• If the database of transactions is replicated across the

processes, each task can be independently accomplished
with no communication.

• If the database is partitioned across processes as well (for

reasons of memory utilization), each task first computes partial
counts. These counts are then aggregated at the a ppropriate
task.
 Input Data Partitioning

• Generally applicable if each output can be naturally

computed as a function of the input.

• In many cases, this is the only natural decomposition because

the output is not clearly known a-priori (e.g., the problem of
finding the minimum in a list, sorting a given list, etc.).

• A task is associated with each input data partition. The task

performs as much of the computation with its part of the data.
Subsequent processing combines these partial results.
 Input Data Partitioning: Example

In the database counting example, the input (i.e., the transaction set) can be
partitioned. This induces a task decomposition in which each task generates
partial counts for all itemsets. These are combined subsequently for aggregate
counts.

Partitioning the transactions among the tasks

Database Transactions

Database Transactions
A, B, C, E, G, H A, B, C 1 A, B, C 0
B, D, E, F, K, L

Itemset Frequency
Itemset Frequency
D, E 2 D, E 1
A, B, F, H, L C, F, G 0 C, F, G 0

Itemsets
Itemsets

D, E, F, H A, E 1 A, E, F, K, L A, E 1
F, G, H, K, C, D 0 B, C, D, G, H, L C, D 1
D, K 1 G, H, L D, K 1
B, C, F 0 D, E, F, K, L B, C, F 0
C, D, K 0 F, G, H, L C, D, K 0

task 1 task 2
 Partitioning Input and Output Data

Often input and output data decomposition can be combined for a higher
degree of concurrency. For the itemset counting example, the transaction set
(input) and itemset counts (output) can both be decomposed as follows:

Partitioning both transactions and frequencies among the tasks

Database Transactions

Database Transactions
A, B, C, E, G, H A, B, C 1 A, B, C, E, G, H
B, D, E, F, K, L D, E B, D, E, F, K, L

Itemset Frequency

Itemset Frequency
2
A, B, F, H, L C, F, G 0 A, B, F, H, L
Itemsets

Itemsets
D, E, F, H A, E 1 D, E, F, H
F, G, H, K, F, G, H, K, C, D 0
D, K 1
B, C, F 0
C, D, K 0

task 1 task 2
Database Transactions
Database Transactions

A, B, C 0

Itemset Frequency
D, E
Itemset Frequency

1
C, F, G 0

Itemsets
Itemsets

A, E, F, K, L A, E 1 A, E, F, K, L
B, C, D, G, H, L B, C, D, G, H, L C, D 1
G, H, L G, H, L D, K 1
D, E, F, K, L D, E, F, K, L B, C, F 0
F, G, H, L F, G, H, L C, D, K 0

task 3 task 4
 Intermediate Data Partitioning

• Computation can often be viewed as a sequence of

transformation from the input to the output data.

• In these cases, it is often beneficial to use one of the

intermediate stages as a basis for decomposition.
 Intermediate Data Partitioning: Example

Let us revisit the example of dense matrix multiplication. We first show how we
can visualize this computation in terms of intermediate matrices D .

A 1, 1 B 1,1 B 1,2
. D1,1,1 D1,1,2

A 2, 1
D1,2,1 D1,2,2

+
A 1, 2
. D2,1,1 D2,1,2

A 2, 2 B 2,1 B 2,2 D2,2,1 D2,2,2

C 1,1 C 1,2

C 2,1 C 2,2
 Intermediate Data Partitioning: Example

A decomposition of intermediate data structure D leads to the following

decomposition into 8 + 4 tasks:

Stage I
0 „ « 1
„ « „ « D 1, 1, 1 D1,1,2
D 1, 2, 2
→ B
A 1, 1 A 1, 2 B 1, 1 B 1, 2 „ D1,2,2 « C
A 2, 1 A 2, 2
.
B 2, 1 B 2, 2
@ D2,1,1 D2,1,2 A
D2,2,2 D2,2,2

Stage II «
„ « „ « „
D 1, 1, 1 D1,1,2 D 2, 1, 1 D2,1,2 C 1, 1 C
+ → 1, 2
D1,2,2 D1,2,2 D2,2,2 D2,2,2 C 2C, 12, 2

Task 01: D1,1,1 = A1,1B1,1 Task 02: D2,1,1 = A1,2B2,1

Task 03: D1,1,2 = A1,1B1,2 Task 04: D2,1,2 = A1,2B2,2
Task 05: D1,2,1 = A2,1B1,1 Task 06: D2,2,1 = A2,2B2,1
Task 07: D1,2,2 = A2,1B1,2 Task 08: D2,2,2 = A2,2B2,2
Task 09: C1,1 = D1,1,1 + D2,1,1 Task 10: C1,2 = D1,1,2 + D2,1,2
Task 11: C2,1 = D1,2,1 + D2,2,1 Task 12: C2,2 = D1,2,2 + D2,2,2
 Intermediate Data Partitioning: Example

The task dependency graph for the decomposition (shown in

previous foil) into 12 tasks is as follows:

1 2 3 4 5 6 7 8

9 10 11 12
 The Owner Computes Rule

• The Owner Computes Rule generally states that the process

assined a particular data item is responsible for all computation
associated with it.

• In the case of input data decomposition, the owner computes

rule imples that all computations that use the input data are
performed by the process.

• In the case of output data decomposition, the owner

computes rule implies that the output is computed by the
process to which the output data is assigned.
 Exploratory Decomposition

• In many cases, the decomposition of the problem goes hand-

in-hand with its execution.

• These problems typically involve the exploration (search) of a

state space of solutions.

• Problems in this class include a variety of discrete optimization

problems (0/1 integer programming, QAP, etc.), theorem
proving, game playing, etc.
 Exploratory Decomposition: Example

A simple application of exploratory decomposition is in the

solution to a 15 puzzle (a tile puzzle). We show a sequence of
three moves that transform a given initial state (a) to desired final
state (d).

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
5 6 8 5 6 7 8 5 6 7 8 5 6 7 8
9 10 7 11 9 10 11 9 10 11 9 10 11 12
13 14 15 12 13 14 15 12 13 14 15 12 13 14 15

(a) (b) (c) (d)

Of-course, the problem of computing the solution, in general,

is much more difficult than in this simple example.
 states of the current state and to view them as independent tasks.
The state space can be explored by generating various successor

1 2 3 4
5 6 7 8
9 10 11
13 14 15 12
1 2 3 4 1 2 3 4
task 4

5 6 7 8 5 6 7
9 10 11 9 10 11 8
13 14 15 12 13 14 15 12
1 2 3 4
Exploratory Decomposition: Example

5 6 7 8
9 10 11 12
13 14 15
1 2 3 4
5 6 7 8
9 10 11
13 14 15 12
1 2 3 4
5 6 7 8
1 2 3 4 9 10 11
5 6 7 8 13 14 15 12
9 10 11 1 2 3 4
13 14 15 12 5 7 8
task 3

9 6 10 11
13 14 15 12
1 2 3 4
5 6 7 8
1 2 3 4 9 14 10 11
5 6 7 8 13 15 12
9 10 11
13 14 15 12 1 2 3 4
5 6 8
9 10 7 11
13 14 15 12
1 2 3 4
5 6 8
1 2 3 4 9 10 7 11
5 6 8 13 14 15 12

task 2
9 10 7 11 1 2 4
13 14 15 12 5 6 3 8
9 10 7 11
13 14 15 12
1 2 3 4
5 6 7 8
9 10 11
13 14 15 12
1 2 3 4
5 6 7 8
9 10 15 11
13 14 12
1 2 3 4 1 2 3 4
5 6 7 8 5 6 7 8

task 1
9 10 15 11 9 10 15 11
13 14 12 13 14 12
1 2 3 4
5 6 7 8
9 10 11
13 14 15 12
 Exploratory Decomposition: Anomalous
Computations

• In many instances of exploratory decomposition, the decomposition

technique may change the amount of work done by the
parallel formulation.

• This change results in super- or sub-linear speedups.

m m m m m m m m

Solution
Total serial work: 2m+1 Total serial work: m
Total parallel work: 1 Total parallel work: 4m

(a) (b)
 Speculative Decomposition

• In some applications, dependencies between tasks are not

known a-priori.

• For such applications, it is impossible to identify independent

tasks.

• There are generally two approaches to dealing with

such applications: conservative approaches, which identify
independent tasks only when they are guaranteed to not have
dependencies, and, optimistic approaches, which schedule
tasks even when they may potentially be erroneous.

• Conservative approaches may yield little concurrency and

optimistic approaches may require roll-back mechanism in the
case of an error.
 Speculative Decomposition: Example

A classic example of speculative decomposition is in discrete

event simulation.

• The central data structure in a discrete event simulation is a

time-ordered event list.

• Events are extracted precisely in time order, processed, and if

required, resulting events are inserted back into the event list.

• Consider your day today as a discrete event system – you get

up, get ready, drive to work, work, eat lunch, work some more,
drive back, eat dinner, and sleep.

• Each of these events may be processed independently,

however, in driving to work, you might meet with an
unfortunate accident and not get to work at all.

• Therefore, an optimistic scheduling of other events will have to

be rolled back.
 Speculative Decomposition: Example

Another example is the simulation of a network of nodes (for

instance, an assembly line or a computer network through which
packets pass). The task is to simulate the behavior of this network
for various inputs and node delay parameters (note that networks
may become unstable for certain values of service rates, queue
sizes, etc.).

C
System Inputs

A D

System Output
E G I
B
F H

System Components
 Hybrid Decompositions

Often, a mix of decomposition techniques is necessary for decomposing a

problem. Consider the following examples:

• In quicksort, recursive decomposition alone limits concurrency (Why?). A

mix of data and recursive decompositions is more desirable.
• In discrete event simulation, there might be concurrency in task processing.
A mix of speculative decomposition and data decomposition may work
well.
• Even for simple problems like finding a minimum of a list of numbers, a mix
of data and recursive decomposition works well.

Data
3 7 2 9 11 4 5 8 7 10 6 13 1 19 3 9
decomposition

2 1 Recursive
decomposition

1
 Characteristics of Tasks

Once a problem has been decomposed into independent tasks,

the characteristics of these tasks critically impact choice and
performance of parallel algorithms. Relevant task characteristics
include:

• Task generation.

• Task sizes.

• Size of data associated with tasks.

 Task Generation

• Static task generation: Concurrent tasks can be identified a-

priori. Typical matrix operations, graph algorithms, image
processing applications, and other regularly structured
problems fall in this class. These can typically be decomposed
using data or recursive decomposition techniques.

• Dynamic task generation: Tasks are generated as we perform

computation. A classic example of this is in game playing
– each 15 puzzle board is generated from the previous
one. These a pplications are typically decomposed using
exploratory or speculative decompositions.
 Task Sizes

• Task sizes may be uniform (i.e., all tasks are the same size) or
non-uniform.

• Non-uniform task sizes may be such that they can be

determined (or estimated) a-priori or not.

• Examples in this class include discrete optimization problems, in

which it is difficult to estimate the effective size of a state space.
 Size of Data Associated with Tasks

• The size of data associated with a task may be small or large

when viewed in the context of the size of the task.

• A small context of a task implies that an algorithm can easily

communicate this task to other processes dynamically (e.g.,
the 15 puzzle).

• A large context ties the task to a process, or alternately, an

algorithm may attempt to reconstruct the context at another
processes as opposed to communicating the context of the
task (e.g., 0/1 integer programming).
 Characteristics of Task Interactions

Tasks may communicate with each other in various ways. The

associated dichotomy is:

• Static interactions: The tasks and their interactions are known

a-priori. These are relatively simpler to code into programs.

• Dynamic interactions: The timing or interacting tasks cannot

be determined a-priori. These interactions are harder to code,
especitally, as we shall see, using message passing APIs.
 Characteristics of Task Interactions

• Regular interactions: There is a definite pattern (in the graph

sense) to the interactions. These patterns can be exploited for
efficient implementation.

• Irregular interactions: Interactions lack well-defined topologies.

 Characteristics of Task Interactions: Example

A simple example of a regular static interaction pattern is

in image dithering. The underlying communication pattern is a
structured (2-D mesh) one as shown here:

Tasks

Pixels
 Characteristics of Task Interactions: Example

The multiplication of a sparse matrix with a vector is a good

example of a static irregular interaction pattern. Here is an
example of a sparse matrix and its associated interaction pattern.

A b 0
2 3
0 1 2 3 4 5 6 7 8 9 1011 1
Task 0

5
4
4 6

7
8

9
Task 11 8 10 11

(a) (b)
 Characteristics of Task Interactions

• Interactions may be read-only or read-write.

• In read-only interactions, tasks just read data items associated

with other tasks.

• In read-write interactions tasks read, as well as modily data

items associated with other tasks.

• In general, read-write interactions are harder to code, since

they require additional synchronization primitives.
 Characteristics of Task Interactions

• Interactions may be one-way or two-way.

• A one-way interaction can be initiated and accomplished by

one of the two interacting tasks.

• A two-way interaction requires participation from both tasks

involved in an interaction.

• One way interactions are somewhat harder to code in

message passing APIs.
 Mapping Techniques

• Once a problem has been decomposed into concurrent tasks,

these must be ma pped to processes (that can be executed on
a parallel platform).

• Mappings must minimize overheads.

• Primary overheads are communication and idling.

• Minimizing these overheads often represents contradicting

objectives.

• Assigning all work to one processor trivially minimizes

communication at the expense of significant idling.
 Mapping Techniques for Minimum Idling

Mapping must simultaneously minimize idling and load

balance. Merely balancing load does not minimize idling.
start synchronization finish start synchronization finish

P1 1 5 9 P1 1 2 3

P2 2 6 10 P2 4 5 6

P3 3 7 11 P3 7 8 9

P4 4 8 12 P4 10 11 12

t=0 t=2 t=3 t=0 t=3 t=6

(a) (b)
 Mapping Techniques for Minimum Idling

Mapping techniques can be static or dynamic.

• Static Mapping: Tasks are mapped to processes a-priori. For

this to work, we must have a good estimate of the size of each
task. Even in these cases, the problem may be NP complete.

• Dynamic Mapping: Tasks are mapped to processes at runtime.

This may be because the tasks are generated at runtime, or
that their sizes are not known.

Other factors that determine the choice of techniques include

the size of data associated with a task and the nature of
underlying domain.
 Schemes for Static Mapping

• Mappings based on data partitioning.

• Mappings based on task gra ph partitioning.

• Hybrid ma ppings.
 Mappings Based on Data Partitioning

We can combine data partitioning with the “owner-

computes” rule to partition the computation into subtasks. The
simplest data decomposition schemes for dense matrices are 1-D
block distribution schemes.
row-wise distribution column-wise distribution

P0
P1
P2
P3
P0 P1 P2 P3 P4 P5 P6 P7
P4
P5
P6
P7
 Block Array Distribution Schemes

Block distribution schemes can be generalized to higher

dimensions as well.

P0 P1 P2 P3
P0 P1 P2 P3 P4 P5 P6 P7
P4 P5 P6 P7

P8 P9 P 10 P 11
P 8 P 9 P10 P 11 P12 P13 P14 P15
P 12 P 13 P 14 P 15

(a) (b)
 Block Array Distribution Schemes: Examples

• For multiplying two dense matrices A and B, we can partition

the output matrix C using a block decomposition.

• For load balance, we give each task the same number of

elements of C. (Note that each element of C corresponds to a
single dot product.)

• The choice of precise decomposition (1-D or 2-D) is determined

by the associated communication overhead.

• In general, higher dimension decomposition allows the use of

larger number of processes.
Data Sharing in Dense Matrix Multiplication
A B C
P0
P1
P2
P3
P4
P5
P6
X =
P7
P8
P9
P10
P 11
P1 2
P 13
P14
P 15

(a)

A B C

P0 P1 P2 P3

P4 P5 P6 P7
X =
P8 P9 P10 P11

P12 P13 P14 P15

(b)
 Cyclic and Block Cyclic Distributions

• If the amount of computation associated with data items

varies, a block decomposition may lead to significant load
imbalances.

• A simple example of this is in LU decomposition (or Gaussian

Elimination) of dense matrices.
 LU Factorization of a Dense Matrix

A decomposition of LU factorization into 14 tasks – notice the

significant load imbalance.

A 1,1 A 1,2 A 1, 3 L 1, 1 0 0 U 1,1 U 1,2 U 1,3

AA2, 12, 2 A 2, 3 → L 2, 1 L2,2 0 . 0 U 2,2 U 2,3
AA3, 13, 2 A 3, 3 L 3, 1 L3,2 L3,3 0 0 U 3,3

1: A 1, 1 → L 1, 1U 1, 1 6: A 2, 2 = A 2, 2 −L 2, 1U 1, 2 11: L 3, 2 = A 3, 2U 2,−12
2: L 2, 1 = A 2, 1U 1,−11 7: A 3, 2 = A 3, 2 −L 3, 1U 1, 2 −1
12: U 2, 3 = L 2, 2 A 2, 3
3: L 3, 1 = A 3, 1U 1,−11 8: A 2, 3 = A 2, 3 −L 2, 1U 1, 3 13: A 3, 3 = A 3, 3 −L 3, 2U 2, 3
−1
4: U 1, 2 = L 1, 1 A 1, 2 9: A 3, 3 = A 3, 3 −L 3, 1U 1, 3 14: A 3, 3 → L 3, 3U 3, 3
−1
5: U 1, 3 = L 1, 1 A 1, 3 10: A 2, 2 → L 2, 2U 2, 2
 Block Cyclic Distributions

• Variation of the block distribution scheme that can be used to

alleviate the load-imbalance and idling problems.

• Partition an array into many more blocks than the number of

available processes.

• Blocks are assigned to processes in a round-robin manner so

that each process gets several non-adjacent blocks.
 Block-Cyc lic Distribution for Gaussian Elimination

The active part of the matrix in Gaussian Elimination changes.

By assigning blocks in a block-cyclic fashion, each processor
receives blocks from different parts of the matrix.

Column k

Column j
Inactive part

Row k (k,k) (k,j) A[k,j] := A[k,j]/A[k,k]

Active part

Row i (i,k) (i,j) A[i,j] := A[i,j] - A[i,k] x A[k,j]

 Block-Cyclic Distribution: Examples

One- and two-dimensional block-cyclic distributions among 4

processes.

P0 P3 P6
T1 T4 T5

P1 P4 P7
T2 T 6 T10 T 8 T12

P2 P5 P8
T3 T 7 T11 T 9 T13T14
 Block-Cyclic Distribution

• A cyclic distribution is a special case in which block size is one.

• A block distribution is a special case in which block size is n/p,

where n is the dimension of the matrix and p is the number of
processes.

P0
P0 P1 P0 P1
P1
P2
P2 P3 P2 P3
P3
P0
P0 P1 P0 P1
P1
P2
P2 P3 P2 P3
P3

(a) (b)
 Graph Partitioning Dased Data Decomposition

• In case of sparse matrices, block decompositions are more

complex.

• Consider the problem of multiplying a sparse matrix with a

vector.

• The gra ph of the matrix is a useful indicator of the work (number

of nodes) and communication (the degree of each node).

• In this case, we would like to partition the graph so as to assign

equal number of nodes to each process, while minimizing
edge count of the graph partition.
Partitioning the Graph of Lake Superior

Random Partitioning

Partitioning for minimum edge-cut.

 Mappings Based on Task Paritioning

• Partitioning a given task-dependency graph across processes.

• Determining an optimal ma pping for a general task-

dependency gra ph is an NP-complete problem.

• Excellent heuristics exist for structured graphs.

Task Paritioning: Mapping a Binary Tree Dependency
Graph

Example illustrates the dependency graph of one view of

quick-sort and how it can be assigned to processes in a
hypercube.
0

0 4

0 2 4 6

0 1 2 3 4 5 6 7
 Task Paritioning: Mapping a Sparse Graph

Sparse graph for computing a sparse matrix-vector product

and its ma pping.
A b
0 1 2 3 4 5 6 7 8 9 1011

Process 0 C0 = (4,5,6,7,8)

Process 1 C1 = (0,1,2,3,8,9,10,11)

Process 2 C2 = (0,4,5,6)

C1 = (0,5,6) Process 1
0
2 3
1

Process 0
4 5
C0 = (1,2,6,9) 6

8 9 10 11
Process 2 C2 = (1,2,4,5,7,8)
 Hierarchical Mappings

• Sometimes a single ma pping technique is inadequate.

• For example, the task mapping of the binary tree (quicksort)

cannot use a large number of processors.

• For this reason, task ma pping can be used at the top level and
data partitioning within each level.
 Hierarchical Mapping: Example

An example of task partitioning at top level with data

partitioning at the lower level.
P0 P1 P4 P5
P2 P3 P6 P7

P0 P1 P4 P5
P2 P3 P6 P7

P0 P1 P2 P3 P4 P5 P6 P7

P0 P1 P2 P3 P4 P5 P6 P7
 Schemes for Dynamic Mapping

• Dynamic mapping is sometimes also referred to as dynamic

load balancing, since load balancing is the primary motivation
for dynamic ma pping.

• Dynamic mapping schemes can be centralized or distributed.

 Centralized Dynamic Mapping

• Processes are designated as masters or slaves.

• When a process runs out of work, it requests the master for more
work.

• When the number of processes increases, the master may

become the bottleneck.

• To alleviate this, a process may pick up a number of tasks (a

chunk) at one time. This is called Chunk scheduling.

• Selecting large chunk sizes may lead to significant load

imbalances as well.

• A number of schemes have been used to gradually decrease

chunk size as the computation progresses.
 Distributed Dynamic Mapping

• Each process can send or receive work from other processes.

• This alleviates the bottleneck in centralized schemes.

• There are four critical questions: how are sensing and receiving
processes paired together, who initiates work transfer, how
much work is transferred, and when is a transfer triggered?

• Answers to these questions are generally application specific.

We will look at some of these techniques later in this class.
 Minimizing Interaction Overheads

• Maximize data locality: Where possible, reuse intermediate

data. Restructure computation so that data can be reused
in smaller time windows.

• Minimize volume of data exchange: There is a cost associated

with each word that is communicated. For this reason, we must
minimize the volume of data communicated.

• Minimize frequency of interactions: There is a startup cost

associated with each interaction. Therefore, try to merge
multiple interactions to one, where possible.

• Minimize contention and hot-spots: Use decentralized

techniques, replicate data where necessary.
 Minimizing Interaction Overheads (continued)

• Overlapping computations with interactions: Use non-blocking

communications, multithreading, and prefetching to hide
latencies.

• Replicating data or computations.

• Using group communications instead of point-to-point primitives.

• Overlap interactions with other interactions.

 Parallel Algorithm Models

An algorithm model is a way of structuring a parallel algorithm

by selecting a decomposition and mapping technique and
applying the appropriate strategy to minimize interactions.

• Data Parallel Model: Tasks are statically (or semi-statically)

mapped to processes and each task performs similar
operations on different data.

• Task Graph Model: Starting from a task dependency graph,

the interrelationships among the tasks are utilized to promote
locality or to reduce interaction costs.
 Parallel Algorithm Models (continued)

• Master-Slave Model: One or more processes generate work

and allocate it to worker processes. This allocation may be
static or dynamic.

• Pipeline / Producer-Comsumer Model: A stream of data is

passed through a succession of processes, each of which
perform some task on it.

• Hybrid Models: A hybrid model may be composed either

of multiple models applied hierarchically or multiple models
applied sequentially to different phases of a parallel algorithm.
 References

Adapted from slides “Principles of Parallel Algorithm Design”

by Ananth Grama
Based on Chapter 3 of “Introduction to Parallel Computing”
by Ananth Grama, Anshul Gupta, George Karypis, and Vipin
Kumar. Addison Wesley, 2003

92
Complexities:
5.4 Complexities ......................................................233
5.4.1 Sequential Computation Complexity ....................234
5.4.2 Parallel Computation Complexity .........................234
5.5 Anomalies in Parallel Algorithms ............................237

Seyed H. Roosta, “Parallel Processing and Parallel Algorithms

Theory and Computation‖”, Springer-Verlag 2000 ,ISBN 978-1-
4612-7048-5 ISBN 978-1-4612-1220-1
 Approaches for parallel algorithms design

1. Parallelize the existing sequential algorithms or

modify an existing sequential algorithm exploiting
those parts of the algorithm that are naturally
parallelizable.
2. Design a completely new parallel algorithm that
might be adapted to parallel architectures.
3. Design a new parallel algorithm from the
existing parallel algorithm.
 Relation between parallel architectures and
parallel algorithms

Parallel Algorithm Parallel Architecture

• Granularity Complexity Concurrency Operation Mode

• Data Structure Memory Structure

• Communication Environment Interconnection Network

• Algorithm Size Number of Processing Elements

• Programming Approach Architecture Category

 Anomalies

Sometimes the unbounded parallelism reflects an

undesirable situation. For example, if we assume the
algorithm can use as many processors as it wants and
there are no communication and memory access
restrictions, the computational time cannot be reduced
below a certain limit. This is due to the fact that some
intermediate results must be known before other parts
of the computation proceed.

The unbounded parallel time complexity of a problem

reflects this characteristic of a problem and is referred
to as anomalies.
 Time Complexities

The time complexity for an algorithm expresses

its time requirements by giving, for each
possible input length, the largest amount of
time needed by the algorithm to solve a
problem of that size
 Types of anomalies
Increasing p will lower the time for the first phase and raise the time for
the second phase, which can increase the total execution time. In
summary, for some executions the parallel version provides a solution
after examining more alternatives, resulting in sublinear speedups. This
is referred to as deceleration anomalies. Execution yielding speedups
greater than p by using p processors (superlinear speedup) is referred
to as acceleration anomalies.

As a consequence, there is little advantage to expand more than k

vertices
THANK YOU