Ch.

Ravi Singh RBCET, BLY CSE DEPARTMENT

Parallel Sorting Network:

A sorting network is a circuit that receives n input values (x0, x1,…..,xn) and permutes them to
produce n outputs (y0, y1, ………, yn) such that the output values are in increasing order.

The key component of these sorting networks is a comparator. A comparator is a device with
two input values x and y and two outputs x1 and y1. For an increasing comparator
x1=min(x,y) and y1=max(x,y).

x x1=min(x,y)
Comparator
y y1=max(x,y)

x + x1=min(x,y)

y + y1=max(x,y

Compare and exchange

In same way we can define a decreasing comparator.

A sorting network is an abstract mathematical model of a network of wires and comparator

modules that is used to sort a sequence of numbers. Each comparator connects two wires and
sorts the values by outputting the smaller value to one wire, and the larger to the other.

A sorting network consists of two items: comparators and wires. Each wire carries with it a
value, and each comparator has two input wires, and two output wires. When two values
enter a comparator, the comparator emits the lower value from the top wire, and the higher
value from the bottom wire. A network of wires and comparators that will correctly sort all
possible inputs into ascending order is called a sorting network.

A sorting network consists of two items: comparators and wires. Each wire carries with it a
value, and each comparator has two input wires, and two output wires. When two values
enter a comparator, the comparator emits the lower value from the top wire, and the higher
value from the bottom wire. A network of wires and comparators that will correctly sort all
possible inputs into ascending order is called a sorting network.

The full operation of a simple sorting network is shown below. It is easy to see why this
sorting network will correctly sort the inputs; note that the first four comparators will "sink"
the largest value to the bottom and "float" the smallest value to the top. The final comparator
simply sorts out the middle two wires.
A simple sorting network consisting of four wires
and five connectors

Demonstration of a comparator in a sorting network.

Insertion and selection Networks:

We can easily construct a network of any size recursively using the principles of insertion
and selection. Assuming we have a sorting network of size n, we can construct a network of
size n + 1 by "inserting" an additional number into the already sorted subnet (using the
principle behind insertion sort). We can also accomplish the same thing by first "selecting"
the lowest value from the inputs and then sort the remaining values recursively (using the
principle behind bubble sort).

A sorting network constructed recursively that first A sorting network constructed recursively that first sorts
sinks the largest value to the bottom and then sorts the first n wires, and then inserts the remaining value.
the remaining wires. Based on bubble sort Based on insertion sort
 The structures of these two sorting networks are very similar. A construction of the two
different variants, which collapses together comparators that can be performed
simultaneously shows that, in fact, they are identical.

BB When allowing for parallel

Bubble sorting networks
comparators, bubble sort and
Bub
Insertion sorting networks insertion sort are identical
Depth:
Maximum number of comparators on a path from an input wire to an output wire.
In case of bubble sort and insertion sort depth is O(2n-3).

Depth= 2n-3=2*6-3=9

It will return largest of n value. It will return largest of n-1 values and so on.

Efficient Networks:
The insertion network has a large depth of O(n) making it impractical. There are simple
networks which achieve depth O((log n)2) (hence size O(n (log n)2)), such as Batcher odd–
even mergesort, bitonic sort, Shell sort, and the Pairwise sorting network. These networks are
often used in practice.

Zero One Principle:

0-1 principle is used to check the validity of sorting network. The zero-one principle states
that a sorting network is valid if it can sort all 2n sequences of 0s and 1s. The principle has
been proven by a special case of the Bouricius's Theorem (Knuth, 1997) in 1954 by W. G.
Bouricius.
 Proof:
Suppose for contradiction that the network sorts all 2n-1 sequences of 0s and 1s correctly
but there is a sequence (a1, a2,……..,an) not correctly. That is for some ai<aj puts aj before
ai.
We define 0-1 function for some x as
F(x)= 0 if x<=ai
1 if x>ai

The network puts f(aj)=1 before f(ai)=0.

That shows contradiction from property of F(x) function. It means a sorting network sort
all 2n sequences of 0s and 1s correctly.

Linear Array and Even-Odd Transposition Sort:

The simplest way to interconnect n processors is one dimensional array. Here processor i
is linked to its two neighbor processors i-1 and i+1 through a communication line. Each
of the end processor has only one neighbor.
For sorting on a linear array we use odd-even transposition sort as described below

t=0

t=1

t=2

Even-Odd Transposition Sort

Algorithm Even-Odd Transposition Sort(A)

1. n=length(A)
2. for(t=0 to n/2 -2) { // n/2-1 times Even-Odd comparison are
performed
3. for(i=0 to n/2-1) {
4. Compare-Exchange(A[2i], A[2i+1]) } // Even Comparison
5. For(j=1 to n/2-1) {
6. Compare-Exchange(A[2j-1], A[2j]) } } // Odd Comparison
Algorithm Even-Odd Transposition using Merge Split Procedure:

1. Divide n element sequence among p processor. Each of p processor has n/p element in
its subsequence.
2. Apply Quick Sort on each subsequence in parallel.
3. Now each even numbered processor Pi merges the two sorted subsequences Si and Si
1
+1 into a sorted sequence Si . Then all odd numbered processors performed the same
process. This process is continuous until we get desired output.

Example:

Let A={8,2,5,10,1,7,3,12,6,11,4,9} and consider p=4.

P0 P1 P2 P3

{8,2,5} {10,1,7} {3,12,6} {11,4,9} Each processor applies Quick Sort on its subse
Subsequence

{2,5,8} {1,7,10} {3,6,12} {4,9,11}}

Even index
// Apply processors
Even perform merge operation
transposition

{P0 , P1} {P2, P3}

{1,2,5} {7,8,10} {3,4,6} {9,11,12}

Odd index processors perform merge operation

(P1 , P2)

{1,2,5} {3,4,6} {7,8,10} {9,11,12}

//Even index
Apply processors
Even perform merge operation
transposition

(P0 , P1) (P2, P3)

{1,2,3} {4,5,6} {7,8,9} {10,11,12}

Odd index processors perform merge
operation
(P1 , P2)

{1,2,3} {4,5,6} {7,8,9} {10,11,12} //stop, this is sorted sequence

Sorting on CRCW Model:
Assume that n2 processors (arranged in n*n order) are available on CRCW computer to sort
the array A= {a0, a1,……,an-1}. The concept of sorting by enumeration is used to sort the
given array. The position of each element in the sorted array is determined by computing
position[i], the number of elements smaller than it. If two elements ai and aj are equal then ai
is taken two be larger of two if i>j else aj is the larger. Once all the positions [i] have been
computed, ai is placed in position position[i] of the sorted array.

Enumeration Sort (A)

1. position[0,1,2,…….,n-1], sorted[0,1,2,------n-1]
2. spawn Pi, j where 0<=i<=n-1 and 0<=j<=n-1
3. for all Pi, j 0<=i<=n-1 and 0<=j<=n-1 // determine position[i] for
element A[i]
4. position[i]=0;
5. if{ A[i]>A[j] or (A[i]=A[j]) and i>j) }then
6. position[i]=1
7. end if
8. end for
9. for all Pi, 0 where 1<=i<=n-1 // Inserting elements in sorted array from
array A
10. do sorted [position[i]]=A[i]
11. end for
12. stop

Example:

Let A={5,2,4,5} we use 16 processors to sort this array :

A P0,0 P0,1 P0,2 P0,3 position sorted

5 5>5 (0) 5>2 (1) 5>4 (1) 5>5 (0) 2 2 If ai>aj then 1
P1,0 P1,1 P1,2 P1,3
2 2>5 (0) 2>2 (0) 2>4 (0) 2>5 (0) 0 4 Else if ai=aj and i>j
P2,0 P2,1 P2,2 P2,3 then 1
4 4>5 (0) 4>2 (1) 4>4 (0) 4>5 (0) 1 5
Else 0
P3,0 P3,1 P3,2 P3,3
5 5>5 (1) 5>2 (1) 5>4 (1) 5>5 (0) 3 5
After
After second
Processors containing elements first for for
Initially and current value of position loop loop
 Sorting on CREW Model:
Assume that p<=n (number of processors is less than the number of elements to be sorted.
Then proceed with the following two steps:

1. Distribute the given elements evenly among p processors and sort the subsequences
using QuickSort.
2. Now p sorted subsequences are merged pairwise, simultaneously using the procedure
CRCW Merge. The resulting subsequences are again merged pairwise and the process
continues until one sorted sequence of length n is obtained.

Example:

Let A={2,8,5,10,15,1,12,6,14,3,11,7,9,4,13,6} and p=4.

Initially A1 = {2,8,5,10} A2 = {15,1,12,6} A3 = {14,3,11,7} A4 = {9,4,13,6}

1 1 1 1 1 1 1 1
A 1 =A1 ,P1 =P1 A 2 =A2 ,P2 =P2 A 3 =A3 ,P3 =P3 A 4 =A4 ,P4 =P4

After A1 = {2,5,8,10} A2 = {1,6,12,15} A3 = {3,7,11,14} A4 = {4,6,9,13}

applying
quick sort

P12={P11 , P21} A22={A31 , A41} P22={P31 , P41}

A12={A12 ,A22}

A12 ={1,2,5,6,8,10,12,15} A22 ={3,4,6,7,9,11,13,14}

A13= {A12 , A22} P13={P12 , P22}

A13={1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
Algorithm:

CREW SORT (A, p)

1. Spawn Pi where 1<=i<=p

2. For all Pi where1<=i<=p
3. Do reads a distinct subsequence Ai of A of size n/p
4. QuickSort(Ai)
5. Ai1={Ai}
6. Pi1={Pi}
7. End for
8. u=1, v = p
9. while v>1 do // we have more than one subsequences
10. for all Pm where 1<=m<= v/2
11. Do Pmu+1 = {P2m-1u+1 , P2mu+1}
12. The processors in set Pmu+1 perform CREWMERGE as
13. Amu+1 = CREWMERGE{A2m-1u+1 , A2mu+1}
14. End for
15. if v is odd then
16. Pceil(v/2)u+1 = Pvu
17. Aceil(v/2)u+1=Avu
18. End if
19. u=u+1
20. v=v+1
21. end while
22. stop

Sorting on EREW Model:

Bitonic Merge Sort:

Bitonic Sequence: A bitonic sequence is a sequence of values a0,a1,…….,an-1 with the
property that are given below:
1. There exists an index i, where 0<=i<=n-1, such that a0 through ai is monotonically
increasing and ai through an-1 is monotonically decreasing.
Example:
{0,1,2,3,9,8,7,6,5,4}
From index value 0 to 4 is monotonically increasing order and index value 4 to 9 is
monotonically decreasing order.
2. There exists a cyclic shift (left or right shift) of indices so that the first condition is
satisfied.
Example:
8,9,2,1,0,4
Shift 4 values from right to left
0,4,8,9,2,1,0
This is a bitonic sequence so given sequence is also a bitonic sequence.
A bitonic sequence has two subsequences one is in increasing order and other is in
decreasing order.
a0<a1<……….<ai>ai+1>ai+2>……………..>an-1

Special Characteristics of a Bitonic Sequence:

If we perform a compare and exchange operation on ai with ai+n/2 for all 0<=i<n/2, where,
there are n numbers in the sequence, we get two bitonic sequences, where the numbers in one
bitonic sequence are less than the numbers in the other sequence.
Example:

3, 5, 8, 9, 7, 4, 2, 1

3, 5, 8, 9, 7, 4, 2, 1

Compare 3 with 7, Compare 5 with 4, Compare 8 with 2, Compare 9 with 1

We get the following two subsequences:
3, 4, 2, 1, 7, 5, 8, 9 Note: 7,5,8,9
Shift 1 value from right to left 5,8,9,7 this is a bitonic
sequence. So 7,5,8,9 is also a bitonic sequence.

Bitonic Sort Algorithm:

1. Assume we have an input array A of length n. Assume that the numbers in A

from A[o] to A[n/2-1] are in increasing order while the numbers from A[n/2] to
A[n-1] are in decreasing order. Then the numbers can be sorted by using
compare and exchange operations.
2. Compare and exchange moves smaller numbers of each pair to left and larger
numbers of each pair to right. Given a bitonic sequence A by recursively
performing compare and exchange operations we can sort the list A.
3. By merging the sorted sequence we can get the desired sorted array.
Example1: Increasing order decreasing order

3 5 8 9 10 12 14 20 95 90 60 40 35 23 18 0

// after 4th split we get sorted sequences so merge them.

// to use bitonic sort first we must have a bitonic sequence of n items.

Forming a Bitonic Sequence:

There is a very simple procedure to convert an unordered sequence into a bitonic
sequence.
Trivially, any two element sequence of numbers forms a bitonic sequence. But most
sequences are longer than two elements. So we inductively construct a bitonic sequence from
smaller bitonic sequences.
We start by forming 4 element bitonic sequences from consecutive two elements sequences.
Consider four elements in a sequence x0, x1, x2, x3. We sort x0, x1 into increasing order and x2,
x3 into decreasing order. We than merge the two pairs to form a 4 elements bitonic sequence.
Next we take two 4 element bitonic sequences sorting one in ascending order and other in
descending order and so on until we obtain the following bitonic sequence of the form.

n/2 Increasing order

n/2 Decreasing order

Example: Conversion of an unordered sequence into a bitonic sequence.

16 14 34 10 12 32 25 20 38 36 26 13 8 40 24 28

First four elements are in increasing order and next four are in
Bitonic Bitonic Bitonic

elements are in increasing order while next n/2 elements are

First eight elements are in increasing order and next eight
elements are in decreasing order. Stop because now n/2
A 4 8 16
first two elements are in increasing order and next two
16 14 10 10
14 16 14 12
34 34 16 14
elements in decreasing order

10 10 34 16
12 12 32 20

in decreasing order
32 32 25 25
25 25 20 32
20 20 12 34
38 36 13 40
36 38 26 38
26 26 36 36
decreasing order

13 13 38 28
8 8 40 26
40 40 28 24
24 28 24 13
28 24 8 8

Now we apply bitonic sort on sequence {10,12,14,16,20,25,32,34,40,38,36,28,26,24,13,8} in

the same way as we applied on above Example 1.

ODD-EVEN MERGE SORT:

Assume we have an array a[ ] of length n such that n= 2k. Assume that the numbers in a[ ]
from a[0] to a[n/2-1] are in increasing order while the numbers from a[n/2] to a[n-1] are also
in increasing order.

The entire array a[ ] can be sorted in increasing order by applying following procedure.
Let array a={10,12,14,16,20,25,32,34,8,13,24,26,28,36,38,40}

Array
a= 10 12 14 16 20 25 32 34 8 13 24 26 28 36 38 40
ae[] ao[]
10 14 20 32 8 24 28 38 12 16 25 34 13 26 36 40
ae[] ao[] ao[]
10 20 8 28 14 32 24 38 12 16 25 34 13 26 36 40
ae[] ao[] ao[] ao[]
10 8 20 28 14 32 24 38 12 16 25 34 13 26 36 40
merge ao[] ao[]
10 8 20 28 14 32 24 38 12 16 25 34 13 26 36 40
swap 1 and 2 if required ao[] ao[]
8 10 20 28 14 32 24 38 12 16 25 34 13 26 36 40
ae[] ao[] ao[]
8 10 20 28 14 24 32 38 12 16 25 34 13 26 36 40
merge ao[]
8 10 20 28 14 24 32 38 12 16 25 34 13 26 36 40
swap 1 and 2 if
required ao[]
8 10 20 28 14 24 32 38 12 16 25 34 13 26 36 40
first merge and swap 1 and 2 if required ao[]
8 10 14 20 24 28 32 38 12 25 13 36 16 34 26 40
ae[] ao[]
8 10 14 20 24 28 32 38 12 13 16 26 25 36 34 40
ae[] ao[] ao[]
8 10 14 20 24 28 32 38 12 16 13 26 25 36 34 40
first merge and swap
1 and 2 if required ao[]
8 10 14 20 24 28 32 38 12 13 16 26 25 36 34 40
ae[] ao[]
8 10 14 20 24 28 32 38 12 13 16 26 25 34 36 40
merge
8 10 14 20 24 28 32 38 12 13 16 26 25 34 36 40
swap 1 and 2 if
required
8 10 14 20 24 28 32 38 12 13 16 26 25 34 36 40

8 10 14 20 24 28 32 38 12 13 16 26 25 36 34 40
merge
8 10 12 13 14 16 20 24 25 26 28 32 34 36 38 40
swap 1 and 2 if required
a= 8 10 12 13 14 16 20 24 25 26 28 32 34 36 38 40
Parallel Merging:
The problem of merging can be defined as:
Let A={a1,a2,….ar} and B={b1,b2,……bs} be two sorted sequence in ascending order. We
merge A and B into one sorted sequence C={c1, c2,…………cr+s}, also sorted in ascending
order such that each element of sequence C belongs to either A or B.

Parallel Merging Networks: (odd-even merging networks)

This can be easily understood by the following figure:
2*2 odd-even merge network: // each set has two elements.

4*4 odd-even merge network: //each set has four elements.

Example:
Design a merging network for the two sorted sequences A={3,5,7,9} and B={2,4,6,8}.

Parallel merging networks for sequences A and B.

General form of n*n odd- even merging network:

L=low, H=high, A=element of A, B= element of B

Parallel Merging Algorithms on CREW/EREW/MCC:

We know that merging is the process of Turing two sorted sequences A[r] and B[s] into one
sorted sequence C[r+s]. For merging A and B into C we use the concept of ranking and we
consider all elements are distinct.
The rank of an element x in set A is denoted by rank(x, A), it defines the number of elements
less than x in set A.
Rank of an element x in set A and B is denoted by
rank(x,A,B)=rank(x,A)+rank(x,B)
First we determine ranks of all elements of set A and set B and then insert then into set C at
their appropriate positions.

Example:
Let A={1,2,3,4,5} and B={0,6,8,9,10}
rank(1,A,B)=1 // number of elements less than 1 in set A and set B.
rank(2,A,B)=2, rank(3,A,B)=3, rank(4,A,B)=4, rank(5,A,B)=5, rank(0,A,B)=0
rank(6,A,B)=6, rank(8,A,B)=7, rank(9,A,B)=8, rank(10,A,B)=9
Now we insert these elements into one sorted sequence C in ascending order, at their ranked
positions.
0 1 2 3 4 5 6 7 8 9

0 1 2 3 4 5 6 8 9 10

Merged array C

Example:
Using CREW/EREW merge procedure merge the following two subsequences
A={2,3,4,6,11,12,13,15,16,20,22,24} and
B={1,5,7,8,9,10,14,17,18,19,21,23} and consider number of processors are p=3

//First we divide these n elements among p processors such that each processors has n/p
distinct elements. After that we sort p subsequences in ascending order and then we apply
ranked procedure for merging these subsequences in parallel as described below.

p1 p2 p3
A1={2,3,4,6,11,12,13,15} A2={16,20,22,24, 1,5,7,8} A3={9,10,14,17,18,19,21,23}
sort A1 in ascending order sort A2 in ascending order sort A3 in ascending order
A1={2,3,4,6,11,12,13,15} A2={1,5,7,8,16,20,22,24} A3={9,10,14,17,18,19,21,23}
rank(2,A1,A2,A3)=1 rank(16,A1,A2,A3)=15 rank(9,A1,A2,A3)=8
rank(3,A1,A2,A3)=2 rank(20,A1,A2,A3)=19 rank(10,A1,A2,A3)=9
rank(4,A1,A2,A3)=3 rank(22,A1,A2,A3)=21 rank(14,A1,A2,A3)=13
rank(6,A1,A2,A3)=5 rank(24,A1,A2,A3)=23 rank(17,A1,A2,A3)=16
rank(11,A1,A2,A3)=10 rank(1,A1,A2,A3)=0 rank(18,A1,A2,A3)=17
rank(12,A1,A2,A3)=11 rank(5,A1,A2,A3)=4 rank(19,A1,A2,A3)=18
rank(13,A1,A2,A3)=12 rank(7,A1,A2,A3)=6 rank(21,A1,A2,A3)=20
rank(15,A1,A2,A3)=14 rank(8,A1,A2,A3)=7 rank(23,A1,A2,A3)=22
now we insert above values in array C at their ranked positions

C={1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24}
Finding Median of Two Sorted Sequences:
The median of two sorted sequences A and B is a pair (ax, by) where ax belongs to A and by
belongs to B.
The median pair (ax, by) satisfies the following properties:
1. Either ax or by is larger than {ceiling (m/2) -1} elements and smaller than floor
(m/2) elements.
2. If ax satisfies property 1 then
a. Either by is the largest element in B smaller than or equal to ax.
b. Or by is the smallest element in B larger than or equal to ax.
3. if by satisfies property 1 then
a. Either ax is the largest element in A smaller than or equal to by.
b. Or ax is the smallest element in A larger than or equal to by.

// m is total number of elements in set A and set B.

Algorithm:
Let us consider A(a1,a2,……….,ar) and B=(b1,b2,………………..,bs)

Median(A,B)
1. na // total number of elements in set A
2. nb // total number of elements in set B
3. lowa // low index value of set A
4. higha // high index value of set A
5. lowb // low index value of set B
6. highb // high index value of set B
7. set lowa=1, lowb=1, higha=r, highb=s, na=r, nb=s
8. while na>1 and nb>1{
9. u=lowa+ceiling{(higha-lowa-1)/2}
10. v= lowb+ceiling{(highb-lowb-1)/2}
11. w=min{floor(na/2),floor(nb/2)}
12. na=na-w
13. nb=nb-w
14. if au>=bv //element au is greater than or equal to element bv
15. {
16. higha=higha-w, lowb=lowb+w}
17. Else{ highb=highb-w, lowa=lowa+w}} //end of while loop.
18. return index value of (au-1,au,au+1) x (bv-1,bv,bv+1) satisfying properties of median pair.
Example:

Let A={10,11,12,13,14,15,16,17,18} and B ={3,4,5,6,7,8,19,20,21,22}

Initial step: na=9, nb=10, lowa=1, lowb=1, higha=9, highb=10

First iteration of while loop:
u=5, v=5, w=min(4,5)=4, na=5 and nb=6 here a5>b5 so higha=5 and low b=5, lowa=1,
highb=10.
Second iteration of while loop:
u=3, v=7, w=min(2,3)=2, na=3 and nb=4 here a3<b7 so lowa=3 and high b=8, higha=5, lowb=5.
Third iteration of while loop:
u=4, v=6, w=min(1,2)=1, na=2 and nb=3 here a4>b6 so higha=4 and low b=6, lowa=3, highb=8.
Fourth iteration of while loop:
u=3, v=7, w=min(1,1)=1,na=1 and nb=2 here a3<b7 so lowa=4 and high b=7, higha=5, lowb=5.
Now while loop terminated.
Now we have nine pairs in {11,12,13} x {8,9,20}. Only two pairs {(13,8),(13,19)} from these
nine pairs satisfy the properties of median pair. The above procedure return the index values
of (ax, by) for which ax+by is minimum.
so median procedure return (4,6) // index values of 13 and 8
Design Issues for Parallel Algorithms:
The methodology of design process focuses on main four distinct phases:
1. Partitioning
2. Communication
3. Agglomerations
4. Mapping
In the first and second phases, concurrency and scalability are involved and in the third and
fourth phases, performance and communication are involved. These phases are presented in
figure given below.

Problem

Defined tasks
partition

Tasks
communiction
communication

Agglomerated tasks
agglomeration

mapping mapping tasks

Processor 1 Processors 2

Different phases in design of parallel algorithm.

Notes: Block diagram of parallel sorting networks:
Shearsort Algorithm:
The Shearsort algorithm consists of an alternation of operations in which all columns are
Sorted in increasing order from top to bottom, and operations in which the even rows, that is
the rows with index 0,2,4,…..,n-2, are sorted from left to right while the odd rows with index
values 1,3,5,………..,n-1, are sorted from right to left.

Algorithm:
Let us consider a 2 dimensional mesh having n rows and m columns.
for i=1,..., 2(log n) + 1 do sequentially
if (i is odd)
then SORT_ALL_ROWS_SNAKELIKE // ( Figure, Phase 1,3,5)
else SORT_ALL_COLUMNS //( Figure, Phase 2,4)

Example:
Illustrate the operations of Shearsort algorithm on the following mesh.

// here number of rows n=4, so for loop is continue from i=1 to 2(log4) +1 // log4=2
// if i is odd then we sort odd rows from right to left ( ) and even rows from left
to right ( ).
// if i is even then we sort all columns from top to bottom.