Munchen

Technische Universitat

Fundamental Algorithms
Chapter 3: Parallel Algorithms The PRAM Model
Michael Bader
Winter 2014/15

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 1
 Munchen
Technische Universitat

Example: Parallel Sorting

Definition
Sorting is required to order a given sequence of elements, or more
precisely:
Input : a sequence of n elements a1 , a2 , . . . , an
Output : a permutation (reordering) a10 , a20 , . . . , an0 of the input
sequence, such that a10 a20 an0 .

An naive(?) solution:
pairwise comparison of all elements
count wins for each element to obtain its position
use one processor for each comparison!

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 2
 Munchen
Technische Universitat

A (Naive?) Parallel Example: AccumulateSort

AccumulateSort ( A : Array [ 1 . . n ] ) {

Create Array P [ 1 . . n ] o f I n t e g e r ;
/ / a l l P [ i ]=0 a t s t a r t

f o r 1 <= i ,j <= n and i < j do i n p a r a l l e l {

i f A[ i ] > A[ j ]
then P [ i] : = P [ i ]+1
else P [ j] := P[ j ]+1;
}

f o r i from 1 to n do i n p a r a l l e l {
A [ P [ i ]+1 ] : = A [ i ] ;
}
}

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 3
 Munchen
Technische Universitat

AccumulateSort Discussion

Implementation:
do all n2 comparisons at once and in parallel

use n2 processors

count wins for each element; then move them to their

respective rank
complexity: TAS = (1) on n(n 1)/2 processors

Assumptions:
all read accesses to A can be done in parallel
increments of P[i] and P[j] can be done in parallel
second for-loop is executed after the first one (on all processors)
all moves A[ P[i] ] := A[i] happen in one atomic step
(no overwrites due to sequential execution)

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 4
 Munchen
Technische Universitat

Example: Parallel Searching

Definition (Search Problem)

Input: a set A of n elements A, and an element x A.
Output: The (smallest) index i {1, . . . , n} with x = A[i].

An immediate solution:
use n processors
on each processor: compare x with A[i]
return matching index/indices i

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 5
 Munchen
Technische Universitat

Simple Parallel Searching

ParSearch ( A : Array [ 1 . . n ] , x : Element ) : I n t e g e r {

f o r i from 1 to n do i n p a r a l l e l {
i f x = A [ i ] then r e t u r n i ;
}
}

Discussion:
Can all n processors access x simultaneously?
exclusive or concurrent read
What happens if more than one processor finds an x?
exclusive or concurrent write (of multiple returns)
general approach: parallelisation by competition

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 6
 Munchen
Technische Universitat

Towards Parallel Algorithms

First Problems and Questions:

parallel read access to variables possible?
parallel write access (or increments?) to variables possible?
are parallel/global copy statements realistic?
how do we synchronise parallel executions?

Reality vs. Theory:

on real hardware: probably lots of restrictions
(e.g., no parallel reads/writes; no global operations on or access
to memory)
in theory: if there were no such restrictions, how far can we get?
or: for different kinds of restrictions, how far can we get?

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 7
 Munchen
Technische Universitat

The PRAM Models

Shared Memory

P1 P2 P3 ... Pn

Central Control

Concurrent or Exclusive Read/Write Access:

EREW exclusive read, exclusive write
CREW concurrent read, exclusive write
ERCW exclusive read, concurrent write
CRCW concurrent read, concurrent write
M. Bader: Fundamental Algorithms
Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 8
 Munchen
Technische Universitat

Exclusive/Concurrent Read and Write Access

exclusive read concurrent read

X1 X3 X4 X
X6 Y
X2 X5

exclusive write concurrent write

X1 X3 X4 X
X6 Y
X2 X5

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 9
 Munchen
Technische Universitat

The PRAM Models (2)

Shared Memory

P1 P2 P3 ... Pn

Central Control

SIMD
Underlying principle for parallel hardware architecture:
strict single instruction, multiple data (SIMD)
All parallel instructions of a parallelized loop are performed
synchronously (applies even to simple if-statements)
M. Bader: Fundamental Algorithms
Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 10
 Munchen
Technische Universitat

Loops and If-Statements in PRAM Programs

Lockstep Execution of parallel for:
Parallel for-loops (i.e., with extension in parallel) are executed in lockstep.
Any instruction in a parallel for-loop is executed at the same time (and in sync) by all
involved processors.
If an instruction consists of several substeps, all substeps are executed in sync.
If an if-then-else statement appears in a parallel for-loop, all processors first evaluate the
comparison at the same time. Then, all processors on which the condition evaluates as true
execute the then branch. Finally, all processors on which the condition evaluates to false
execute the else branch.

Lockstep Example:
f o r i from 1 to n do i n p a r a l l e l {
i f U[ i ] > 0
then F [ i ] : = (U[ i ]U [ i 1]) / dx
else F [ i ] : = (U[ i +1]U [ i ] ) / dx
end i f
}

First, all processors perform the comparison U[i]>0

All processors where U[i]>0 then compute F[ i ]; note that first all processors read U[i ] and
then all processors read U[i1] (substeps!); hence, there is no concurrent read access!
Afterwards, the else-part is executed in the same manner by all processors with U[i]<=0

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 11
 Munchen
Technische Universitat

Parallel Search on an EREW PRAM

ToDos for exclusive read and exclusive write:

avoid exclusive access to x
replicate x for all processors (broadcast)
determine smallest index of all elements found:
determine minimum in parallel

Broadcast on the PRAM:

copy x into all elements of an array X[1..n]
note: each processor can only produce one copy per step

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 12
 Munchen
Technische Universitat

Broadcast on the PRAM Copy Scheme

5 5

5 5 5 5

5 5 5 5 5 5 5 5
M. Bader: Fundamental Algorithms
Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 13
 Munchen
Technische Universitat

Broadcast on the PRAM Implementation

BroadcastPRAM ( x : Element , A : Array [ 1 . . n ] ) {

/ / n assumed t o be 2 k
/ / Model : EREW PRAM

A[ 1 ] := x ;
f o r i from 0 to k1 do
f o r j from 2 i +1 to 2 ( i +1) do i n p a r a l l e l {
A [ j ] : = A [ j 2 i ] ;
}
}

Complexity:
T (n) = (log n) on n
2 processors

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 14
 Munchen
Technische Universitat

Minimum Search on the PRAM Binary Fan-In

4 7 3 9 5 6 10 8

4 3 5 8

3 5

3
M. Bader: Fundamental Algorithms
Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 15
 Munchen
Technische Universitat

Minimum on the PRAM Implementation

MinimumPRAM ( A : Array [ 1 . . n ] ) : I n t e g e r {
/ / n assumed t o be 2 k
/ / Model : EREW PRAM

f o r i from 1 to k do
f o r j from 1 to n / ( 2 i ) do i n p a r a l l e l {
i f A[ 2 j 1] < A[ 2 j ]
then A[ 2 j ] : = A[ 2 j 1 ] ;
end i f ;
A [ j ] : = A[ 2 j ] ;
}
return A [ 1 ] ;
}

n
Complexity: T (n) = (log n) on 2 processors

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 16
 Munchen
Technische Universitat

Binary Fan-In (2)

Comment Concerned about synchronous copy statement?
Modifiy stride!

4 7 3 9 5 6 10 8

4 3 5 8

3 5

3
M. Bader: Fundamental Algorithms
Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 17
 Munchen
Technische Universitat

Searching on the PRAM Parallel Implementation

SearchPRAM ( A : Array [ 1 . . n ] , x : Element ) : I n t e g e r {

/ / n assumed t o be 2 k
/ / Model : EREW PRAM

BroadcastPRAM ( x , X [ 1 . . n ] ) ;

f o r i from 1 to n do i n p a r a l l e l {
i f A[ i ] = X[ i ]
then X [ i ] := i ;
else X [ i ] : = n +1; / / ( i n v a l i d index )
end i f ;
}

r e t u r n MinimumPRAM ( X [ 1 . . n ] ) ;
}

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 18
 Munchen
Technische Universitat

The Prefix Problem

Definition (Prefix Problem)

Input: an array A of n elements ai .
Output: All terms a1 a2 ak for k = 1, . . . , n.
may be any associative operation.

Straightforward serial implementation:

P r e f i x ( A : Array [ 1 . . n ] ) {
/ / i n p l a c e computation :
f o r i from 2 to n do {
A [ i ] : = A [ i 1]A [ i ] ;
}

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 19
 Munchen
Technische Universitat

The Prefix Problem Divide and Conquer

Idea:
1. compute prefix problem for A1 , . . . , An/2
gives A1:1 , . . . , A1:n/2
2. compute prefix problem for An/2+1 , . . . , An
gives An/2+1:n/2+1 , . . . , An/2+1:n
3. multiply A1:n/2 with all An/2+1:n/2+1 , . . . , An/2+1:n
gives A1:n/2+1 , . . . , A1:n

Parallelism:
steps 1 and 2 can be computed in parallel (divide)
all multiplications in step 3 can be computed in parallel
recursive extension leads to parallel prefix scheme

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 20
 Munchen
Technische Universitat

Parallel Prefix Divide and Conquer

A1 A2 A3 A4 A5 A6 A7 A8

A1:2 A3 A3:4 A5:6 A7 A7:8

A1:3 A1:4 A5 A5:6 A5:7 A5:8

A1:5 A1:6 A1:7 A1:8

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 21
 Munchen
Technische Universitat

Parallel Prefix Scheme on a CREW PRAM

Additional Feature: In-Place Computation, Pin Elements to Cores

A1 A8 A2 A7 A3 A6 A4 A5

A1 A7:8 A1:2 A7 A3 A5:6 A3:4 A5

A1 A5:8 A1:2 A5:7 A1:3 A5:6 A1:4 A5

A1 A1:8 A1:2 A1:7 A1:3 A1:6 A1:4 A1:5

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 22
 Munchen
Technische Universitat

Outlook: Parallel Prefix on Distributed Memory

Consider scheme from previous slide:

A1 A8 A2 A7 A3 A6 A4 A5

A1 A7:8 A1:2 A7 A3 A5:6 A3:4 A5

A1 A5:8 A1:2 A5:7 A1:3 A5:6 A1:4 A5

A1 A1:8 A1:2 A1:7 A1:3 A1:6 A1:4 A1:5

Execution on Distributed Memory:

Each color corresponds to one compute node
Nodes cannot directly access matrices from a node with different colour
explicit data transfer (communication) required

Properties of the Distributed-Memory Parallel Prefix Scheme:

In-place computation; A[1:n] will overwrite A[n]; all A[j:n] stored on the same node
One of the two multiplied matrices is always local
Still, n/2 outgoing messages from A[1:n/2] in the last step (bottleneck!)

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 23
 Munchen
Technische Universitat

Parallel Prefix CREW PRAM Implementation

PrefixPRAM ( A : Array [ 1 . . n ] ) {
/ / n assumed t o be 2 k
/ / Model : CREW PRAM ( n / 2 p r o c e s s o r s )

f o r l from 0 to k1 do
f o r p from 2 l by 2 ( l +1) to n do i n p a r a l l e l
f o r j from 1 to 2 l do i n p a r a l l e l {
A [ p+ j ] : = A [ p ] A [ p+ j ] ;
}
}
Comments:
p- and j-loop together: n/2 multiplications per l-loop
concurrent read access to A[p] in the innermost loop

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 24
 Munchen
Technische Universitat

Parallel Prefix Scheme on an EREW PRAM

A1 A2 A3 A4 A5 A6 A7 A8

A1 A1:2 A2:3 A3:4 A4:5 A5:6 A6:7 A7:8

A1 A1:2 A1:3 A1:4 A2:5 A3:6 A4:7 A5:8

A1 A1:2 A1:3 A1:4 A1:5 A1:6 A1:7 A1:8

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 25
 Munchen
Technische Universitat

Parallel Prefix EREW PRAM Implementation

PrefixPRAM ( A : Array [ 1 . . n ] ) {
/ / n assumed t o be 2 k
/ / Model : EREW PRAM ( n1 p r o c e s s o r s )

f o r l from 0 to k1 do
f o r j from 2 l +1 to n do i n p a r a l l e l {
tmp [ j ] : = A [ j 2 l ] ;
A [ j ] : = tmp [ j ] A [ j ] ;
}
}
Comment:
all processors execute tmp[j] := A[j-2l] before multiplication!

M. Bader: Fundamental Algorithms

Chapter 3: Parallel Algorithms The PRAM Model, Winter 2014/15 26