Lecture 4: Principles of Parallel

Algorithm Design (part 4)

1
 Mapping Technique for Load Balancing

• Sources of overheads:
– Inter-process interaction
– Idling
• Goals to achieve:
– To reduce interaction time
– To reduce total amount of time some processes being
idle
– Remark: these two goals often conflict
• Classes of mapping:
– Static
– Dynamic
2
 Schemes for Static Mapping

• Mapping Based on Data Partitioning

• Task Graph Partitioning
• Hybrid Strategies

3
 Mapping Based on Data Partitioning

• By owner-computes rule, mapping the relevant

data onto processes is equivalent to mapping
tasks onto processes
• Array or Matrices
– Block distributions
– Cyclic and block cyclic distributions
• Irregular Data
– Example: data associated with unstructured mesh
– Graph partitioning

4
 1D Block Distribution
Example. Distribute rows or columns of matrix to different
processes

5
 Multi-D Block Distribution
Example. Distribute blocks of matrix to different processes

6
 Load-Balance for Block Distribution

Example. 𝑛 × 𝑛 dense matrix multiplication 𝐶 = 𝐴 × 𝐵

using 𝑝 processes
– Decomposition based on output data.
– Each entry of 𝐶 use the same amount of computation.
– Either 1D or 2D block distribution can be used:
𝑛
• 1D distribution: rows are assigned to a process
𝑝
• 2D distribution: 𝑛/ 𝑝 × 𝑛/ 𝑝 size block is assigned to a process
– Multi-D distribution allows higher degree of concurrency.
– Multi-D distribution can also help to reduce interactions

7
8
 Cyclic and Block Cyclic Distributions

• If the amount of work differs for different

entries of a matrix, a block distribution can
lead to load imbalances.
• Example. Doolittle’s method of LU factorization
of dense matrix

9
 Doolittle’s method of LU factorization

𝑎11 𝑎12 … 𝑎1𝑛 1 0 … 0 𝑢11 𝑢12 … 𝑢1𝑛

𝑎21 𝑎22 … 𝑎2𝑛 𝑙21 1 … 0 0 𝑢22 … 𝑢2𝑛
𝐴= ⋮ ⋮ ⋱ ⋮ = 𝐿𝑈 = ⋮ ⋮ ⋱ ⋮
⋮ ⋮ ⋱ ⋮
𝑎𝑛1 𝑎𝑛2 … 𝑎𝑛𝑛 𝑙𝑛1 𝑙𝑛2 … 1 0 0 … 𝑢𝑛𝑛

By matrix-matrix multiplication

𝑢1𝑗 = 𝑎1𝑗 , 𝑗 = 1,2, … , 𝑛 (1𝑠𝑡 row of 𝑈)

𝑙𝑗1 = 𝑎𝑗1 /𝑢11 , 𝑗 = 1,2, … , 𝑛 (1𝑠𝑡 column of 𝐿)
For 𝑖 = 2,3, … , 𝑛 − 1 do
𝑢𝑖𝑖 = 𝑎𝑖𝑖 − 𝑖−1𝑡=1 𝑙𝑖𝑡 𝑢𝑡𝑗

𝑖−1
𝑢𝑖𝑗 = 𝑎𝑖𝑗 − 𝑡=1 𝑙𝑖𝑡 𝑢𝑡𝑗 for 𝑗 = 𝑖 + 1, … , 𝑛 (𝑖𝑡ℎ row of 𝑈)
𝑎𝑗𝑖 − 𝑖−1
𝑡=1 𝑙𝑗𝑡 𝑢𝑡𝑖
𝑙𝑗𝑖 = for 𝑗 = 𝑖 + 1, … , 𝑛 (𝑖𝑡ℎ column of 𝐿)
𝑢𝑖𝑖

End
𝑛−1
𝑢𝑛𝑛 = 𝑎𝑛𝑛 − 𝑡=1 𝑙𝑛𝑡 𝑢𝑡𝑛

10
 Serial Column-Based LU

• Remark: Matrices L and U share space with A

11
Work used to compute Entries of L and U

12
• Block distribution of LU factorization tasks
leads to load imbalance.

13
 Block-Cyclic Distribution

• A variation of block distribution that can be

used to alleviate the load-imbalance.

• Steps
1. Partition an array into many more blocks than
the number of available processes
2. Assign blocks to processes in a round-robin
manner so that each process gets several non-
adjacent blocks.

14
(a) The rows of the array are grouped into blocks each consisting of two rows,
resulting in eight blocks of rows. These blocks are distributed to four processes
in a wraparound fashion.
(b) The matrix is blocked into 16 blocks each of size 4×4, and it is mapped onto a
2×2 grid of processes in a wraparound fashion.
• Cyclic distribution: when the block size =1
15
 Graph Partitioning
• Assign equal number of nodes (or cells) to each process
• Minimize edge count of the graph partition

Random Partitioning Partitioning for Minimizing Edge-Count

16
 Mappings Based on Task Partitioning

• Mapping based on task partitioning can be used

when computation is naturally expressed in the
form of a static task-dependency graph with
known sizes.
• Finding optimal mapping minimizing idle time and
minimizing interaction time is NP-complete
• Heuristic solutions exist for many structured
graphs

17
Mapping a Binary Tree Task-Dependency Graph

• Finding min.

• Mapping the tree graph onto 8 processes

• Mapping minimizes the interaction overhead by mapping independent
tasks onto the same process (i.e., process 0) and others on processes
only one communication link away from each other
• Idling exists. This is inherent in the graph 18
 Mapping a Sparse Graph
Example. Sparse matrix-vector multiplication using 3
processes
• Arrow distribution

19
• Partitioning task interaction graph to reduce
interaction overhead

20
 Schemes for Dynamic Mapping

• When static mapping results in highly imbalanced

distribution of work among processes or when
task-dependency graph is dynamic, use dynamic
mapping
• Primary goal is to balance load – dynamic load
balancing
– Example: Dynamic load balancing for AMR
• Types
– Centralized
– Distributed

21
 Centralized Dynamic Mapping

• Processes
– Master: mange a group of available tasks
– Slave: depend on master to obtain work
• Idea
– When a slave process has no work, it takes a portion of available
work from master
– When a new task is generated, it is added to the pool of tasks in
the master process
• Potential problem
– When many processes are used, mast process may become
bottleneck
• Solution
– Chunk scheduling: every time a process runs out of work it gets
a group of tasks.

22
 Distributed Dynamic Mapping

• All processes are peers. Tasks are distributed

among processes which exchange tasks at run
time to balance work
• Each process can send or receive work from other
processes
– How are sending and receiving processes paired
together
– Is the work transfer initiated by the sender or the
receiver?
– How much work is transferred?
– When is the work transfer performed?

23
 Techniques to Minimize Interaction Overheads

• Maximize data locality

– Maximize the reuse of recently accessed data
– Minimize volume of data-exchange
• Use high dimensional distribution. Example: 2D block
distribution for matrix multiplication
– Minimize frequency of interactions
• Reconstruct algorithm such that shared data are accessed
and used in large pieces.
• Combine messages between the same source-destination
pair

24
 Techniques to Minimize Interaction Overheads
• Minimize contention and hot spots
– Contention occur when multi-tasks try to access the same resources
concurrently: multiple processes sending message to the same
process; multiple simultaneous accesses to the same memory block

𝑝−1
• Using 𝐶𝑖,𝑗 = 𝑘=0 𝐴𝑖,𝑘 𝐵𝑘,𝑗 causes contention. For example, 𝐶0,0 ,
𝐶0,1 , 𝐶0, 𝑝−1 attempt to read 𝐴0,0 , at once.
• A contention-free manner is to use:
𝑝−1
𝐶𝑖,𝑗 = 𝑘=0 𝐴𝑖, 𝑖+𝑗+𝑘 % 𝑝 𝐵 𝑖+𝑗+𝑘 % 𝑝,𝑗
All tasks 𝑃∗,𝑗 that work on the same row of C access block
𝐴𝑖, 𝑖+𝑗+𝑘 % 𝑝 , which is different for each task. 25
 Techniques to Minimize Interaction Overheads
• Overlap computations with interactions
– Use non-blocking communication
• Replicate data or computations
– Replicate a copy of shared data on each process if
possible, so that there is only initial interaction during
replication.
• Use collective interaction operations
• Overlap interactions with other interactions

26
 Parallel Algorithm Models
• Data parallel
– Each task performs similar operations on different data
– Typically statically map tasks to processes
• Task graph
– Use task dependency graph to promote locality or reduce
interactions
• Master-slave
– One or more master processes generating tasks
– Allocate tasks to slave processes
– Allocation may be static or dynamic
• Pipeline/producer-consumer
– Pass a stream of data through a sequence of processes
– Each performs some operation on it
• Hybrid
– Apply multiple models hierarchically, or apply multiple models
in sequence to different phases
27