PARALLEL

PROCESSING/PROGRAMMING
CATALIN BOJA
[email protected]
BUCHAREST UNIVERSITY OF ECONOMIC STUDIES
SORTING ALGORITHMS FOR PARALLEL
PROCESSING

• Bubble Sort
• Optimized Bubble Sort Algorithm
• Odd-Even Sort
• Merge sort
• Bitonic Sort
BUBBLE SORT

• Bubble sort is a O(N2) sorting algorithm.

• It is simple to understand and implement.
• It has terrible performance, even when compared to other O(N2) algorithms.
• So why discuss it?
• Easy to understand
• Easy to implement
• “Easy” to parallelize
• http://en.wikipedia.org/wiki/Bubble_sort
CLASSIC BUBBLE SORT

do { • Worst case scenario – generates N2 iterations

bool isOver = false;
• Goes many times through the array and check pairs of
for(int i = 1; i < n; i++) neighboring values
if(v[i-1] > v[i])
• The array is considered sorted after an array pass
{ that will not trigger any interchange
swap(&v[i-1],&v[i]);
• Each iteration is dependent on 2 array values (i and i-
isOver = true; 1)
}
• Because of each loop inter dependencies (you process
} while(!isOver); I and i-1 elements) is almost impossible to parallelize
OPTIMIZED BUBBLE SORT

do {
int new_n = 0;
• Reduces the number of iterations as
for(int i = 1; i < n; i++) n is moved closer to the beginning of
if(v[i-1] > v[i])
the array (as the rest of the array is
{
swap(&v[i-1],&v[i]); already sorted)
new_n = i;
} • Each iteration is still dependent on
n = new_n; 2 array values (i and i-1)
} while(n > 0);
• The dependencies are still there
ODD-EVEN SORT

• Parallelizable version of Bubble sort

• Requires N external passes through the array
• Each pass through the array analyzes either:
• Every pair of odd indexed elements and the preceding element, or
• Every pair of even indexed elements and the preceding element.
• Within each pass, elements that are not in order are swapped
• https://en.wikipedia.org/wiki/Odd%E2%80%93even_sort
ODD-EVEN SORT ALGORITHM – SOLUTION 1

for(int it = 0; it < N; it++) • As this algorithm is a bubble sort, its work

{
if(it%2==1){ complexity is O(N2).
for(int i = 2; i < N; i +=2)
if(v[i-1] > v[i]) • The step complexity of the algorithm is O(N)
swap(&v[i-1],&v[i]); as each inner loop may be executed in
} parallel.
else {
for(int i = 1; i < N; i +=2) • The results of each iteration of the outer loop
if(v[i-1] > v[i])
swap(&v[i-1],&v[i]); are dependent upon the previous iteration.

}
}
• There are N iterations in the outer loop, and
each inner loop consists of a full pass through
the array, requiring O(N) operations.
ODD-EVEN SORT ALGORITHM – SOLUTION 2

void OddEvenSort(int* v, int N)

{
int changeOddEven = 1, start = 0, i;
int temp;
while(changeOddEven || start) {
changeOddEven = 0;
for(i = start; i < N-1; i += 2) {
if(v[i] > v[i + 1]) {
int temp = v[i]; v[i] = v[i+1]; v[i+1] = temp;
changeOddEven = 1;
}
}
if(start == 0) start = 1;
else start = 0;
}
}
ODD-EVEN SORT PARALLELIZATION WITH OPENMP

for(int it = 0; it < N; n++) {

if(it%2==1){ • Swap operations from each iteration
#pragma omp parallel for private(i) \
shared(v)
are independent.
for(int i = 2; i < N; i+=2)
if(v[i-1] > v[i]) • Each odd or even step is
swap(&v[i-1],&v[i]);
} else { parallelized
#pragma omp parallel for private(i) \
shared(v)
for(i = 1; i < N; i+=2)
• Another solution would be to use
if(v[i-1] > v[i])
swap(&v[i-1],&v[i]);
sections and use only 2 parallel
} threads
}
MERGESORT

• Is a Divide & Conquer type algorithm (like QuickSort)

• Recursively splits the array in 2 halves until that is not possible (you get a
single value sub-array). Then, merges the halves by placing each element to
the right location. The resulted array will be sorted
• Has a complexity of O(N*LogN)
• https://en.wikipedia.org/wiki/Merge_sort
MERGESORT

Divide phase

Merge phase

https://en.wikipedia.org/wiki/Merge_sort
MERGESORT

Animation from https://www.thecrazyprogrammer.com/2014/03/c-program-for-implementation-of-merge-sort.html

BITONIC MERGESORT

• Based on the mathematical Bitonic sequence theory

• One of the fastest parallel sorting algorithms
• Is slower that MergeSort on single processor but faster on parallel
implementations
• Has a performance of O(N*log2N) for sequential implementation and
O(log2N) for parallel implementation
BITONIC MERGESORT

• A bitonic sequence is defined as a list with

no more than one LOCAL MAXIMUM and
no more than one LOCAL MINIMUM
• A bitonic sequence is a set of values in
which there are 2 sequences: one that
increases up to a point (bitonic point) and
after that is decreasing

From http://web.mst.edu/~ercal/387/slides/SLIDES-10-sorting.ppt
BITONIC MERGESORT

This is ok!
1 Local MAX; 1 Local MIN
The list is bitonic!

This is NOT bitonic

1 Local MAX; 2 Local MINs

From http://web.mst.edu/~ercal/387/slides/SLIDES-10-sorting.ppt
BITONIC MERGESORT

• The algorithm involves multiple steps in which you take the N input array and
create a Bitonic sequence; you star with small sequences (2, 4, 8, …) until you
get to the final sequence of N values
• After you obtain the Bitonic search you rearrange the values by comparing
them with their correspondent in the other half. You start with initial N/2 halfs
and you move down until you get to 2 elements sequences
 BITONIC MERGESORT

Phase I Phase II
https://www.geeksforgeeks.org/bitonic-sort/
BITONIC MERGESORT

The 2nd phase:

1. Divide the bitonic list into two equal
halves.
2. Compare-Exchange each item on the first
half with the corresponding item in the
second half
3. You get 2 bitonic sequences where the
numbers in one sequence are all less than
the numbers in the other sequence
4. Continue until you get the array sorted

18
SEARCHING ALGORITHMS

• Naïve Search • String searching

• Divide and Conquer: Binary Search • Naïve string-search algorithm
• Boyer–Moore string-search algorithm
• Graphs • Knuth–Morris–Pratt algorithm
• Depth-First Search
• Breadth-First Search
• Best-First Search
NAÏVE SEARCH

int search(char *t, int start, int end, char *p)

{ • Check each pattern char against
int count = 0;
int n = end - start + 1; each text char
int m = strlen(p);
for (int i = start; i <= end - m; i++)
{ • Complexity of O(n*m) for worst case
int j = 0;
for (j = 0; j < m; j++) scenario
if (t[i + j] != p[j]) break;

if (j == m)
count++;
}
return count;
}
BOYER–MOORE STRING-SEARCH ALGORITHM

• Reduces the number of comparisons by changing the shift step from 1 to m

(where m is the length of the pattern)
• Requires a preprocessing of the pattern in order to determine
• is an efficient string-searching algorithm that is the standard benchmark for
practical string-search literature
(https://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string-
search_algorithm)
BOYER–MOORE STRING-SEARCH ALGORITHM

• Complexity of O(n+m) only if the pattern is NOT in the text

• Complexity of O(nm) when the pattern occurs in the text
• Is very efficient for large patterns
• A shift is calculated by applying any of the two rules
• the bad character rule
• the good suffix rule
• https://people.ok.ubc.ca/ylucet/DS/BoyerMoore.html
KNUTH-MORRIS-PRATT

• Uses the same principle as Boyer–Moore: increases the shift step when it’s
clear that the pattern is not present
• Complexity of O(n+m) because requires the build of the table algorithm for
the given pattern
• https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_
algorithm
• https://people.ok.ubc.ca/ylucet/DS/KnuthMorrisPratt.html
GAME OF LIFE

• Life is an example of a cellular automaton - any system in which rules are applied to
cells and their neighbors in a regular grid
• Invented by the mathematician John Conway in 1970
(https://www.youtube.com/watch?v=E8kUJL04ELA)
• The rules are not arbitrary chosen. They provide a balance -> it’s hard to tell
whether a pattern will die out completely, form a stable population, or grow forever
• is a zero-player game, meaning that its evolution is determined by its initial state,
requiring no further input
GAME OF LIFE

• Life is played on an infinite grid of square cells

• A cell can be live or dead
• Every cell interacts with its eight neighbors (except border cells)
• Any live cell with fewer than two live neighbors dies, as if by underpopulation.
• Any live cell with two or three live neighbors lives on to the next generation.
• Any live cell with more than three live neighbors dies, as if by overpopulation.
• Any dead cell with exactly three live neighbors becomes a live cell, as if by reproduction.

• The rules are applied simultaneously to every cell in the seed; “births and deaths occur
simultaneously, and the discrete moment at which this happens is sometimes called a tick“,
https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life
GAME OF LIFE

• The game has a lot of patterns

(https://en.wikipedia.org/wiki/Con
way%27s_Game_of_Life#Examples
_of_patterns):
• Still lifes
• Oscillators
• Spaceships
GOOGLE PAGERANK

• Algorithm used initially by Google to rank Web pages b their popularity

• Defined by the Google co-founders Larry Page and Sergey Brin in 1997 as
part of a research project at Stanford University
(http://infolab.stanford.edu/~backrub/google.html)
• Idea similar to the “importance” of scientific papers based on the number of
references from other papers
GOOGLE PAGERANK

• Used 2000 in the Google Toolbar for Internet Explorer in order to show the
PageRank (logarithmic value) of the current Web page
• Generated a link spam frenzy
• Removed from Firefox in 2011 and in 2013 received the last update; in 2016
Google officially removed support for Toolbar Pagerank
GOOGLE PAGERANK

• If your page has a PageRank of 20, and there are 4.285.199.774 pages
indexed by Google, it follows that the odds that a "random surfer" is reading
your page right now are 20/4.285.199.77

https://searchengineland.com/rip-google-pagerank-retrospective-244286
GOOGLE PAGERANK

• Still used by Google, but with a far

less importance (the importance of
Web pages is given by its content
relevance)
• An interesting algorithm that can be
parallelized
GOOGLE PAGERANK

• We assume page A has pages T1…Tn which point to it (i.e., are citations). The
parameter d is a damping factor which can be set between 0 and 1. We usually set d to
0.85. There are more details about d in the next section. Also C(A) is defined as the
number of links going out of page A. The PageRank of a page A is given as follows:
PR(A) = (1‐d) + d (PR(T1)/C(T1) + … + PR(Tn)/C(Tn))
• Note that the PageRanks form a probability distribution over web pages, so the sum of
all web pages’ PageRanks will be one.
http://infolab.stanford.edu/~backrub/google.html
GOOGLE PAGERANK

• The PageRank (PR) is actually a probability that the user will eventually visit that
page
• All web pages PRs sum will be equal to 1
• A page PR is given by
• Number and quality of inbound links
• Number of outbound links
• The PR of each linking page
• The probability to randomly hit the page by a user (who just clicks links) – damping factor,
usually 0.85
GOOGLE PAGERANK

PR(A) = (1‐d) + d (PR(T1)/C(T1) + … + PR(Tn)/C(Tn))

• Ti – page that has a link to A
• PR(Ti) – PageRank value of page Ti
• C(Ti) – number of outgoing links from page Ti
• PR(Ti)/C(Ti) – measures the quality of the link; the PR of page Ti is spread evenly among all the pages to
which it has links
• D – damping factor used to model the situation the user will go to a random page; usually is 0.85; also used
to “damp down” the value of a link received from a page which is not directly linked; used to normalize the
sum
• 1-d – adds the value lost by multiplying the PRs sum with d;
GOOGLE PAGERANK

• How do you compute it ?

• Each page PR is dependent on the all other pages PR. Which do you compute first ?
PageRank or PR(A) can be calculated using a simple iterative algorithm, and corresponds
to the principal eigenvector of the normalized link matrix of the web.
http://infolab.stanford.edu/~backrub/google.html
• So we start computing PRs but we do it many times (multiple iterations) until we get
the desired precision
GOOGLE PAGERANK

1. We give al pages the same PR (aka probability to be accessed)

PR_1(Ti) = 1/N where N is the number of pages
2. Compute each page PR based on outbound links
PR_2(Ti) = (PR_1(T1)/C(T1) + … + PR_1(Tk)/C(Tk))
3. Take into consideration Dangling Pages = pages with no outbound links; so, they
give the same probability to reach an other page; they share their PR to all other
pages
PR_3(Ti) = PR_2(Ti) + SUM(DanglingPagesPR / N)
GOOGLE PAGERANK

4. Take into consideration Damping Factor

PR[i] = PR_3(Ti) * damping_factor + (1 - damping_factor) / page_cnt
5. Repeat last steps until you get desired precision (difference between current
iteration PR and the previous one)
GOOGLE PAGERANK

Influence of the dampening factor value (http://www.pagerank.dk/Pagerank-

formula/Damping-factor.htm)
• A low damping factor will make calculations easier as the iterations will quickly
converge.
• A low damping factor means that the PageRank will be greatly influenced by
PageRank received from external pages - rather than the internal link structure.
• A high damping factor will result in the site's total PageRank growing higher.
PageRank received from external pages will be passed around in the system.