Distributed & Parallel Algorithms
(CS 18301)
Coordinator: Dr. Ashish Kumar Maurya
Teacher : Shailendra Kumar Singh
By:- Shailendra Kumar Singh, CSE Department, MNNIT
1 Allahabad
Course/Subject Outline
Algorithms
Distributed Processing
Parallel Processing
Flynn’s classification and Amdahl’s law
Abstract Model of Parallel Computing
Parallel Random Access Machine
Topologies
Parallel Sorting and Searching
Parallel Matrix Operations
Graph and List Application
By:- Shailendra Kumar Singh, CSE Department, MNNIT
2 Allahabad
What are algorithms?
Informally, an algorithm is any well-defined
computational procedure that takes some
value, or set of values, as input and produces
some value, or set of values, as output.
An algorithm is thus a sequence of
computational steps that transform the input
into the output.
By:- Shailendra Kumar Singh, CSE Department, MNNIT
3 Allahabad
Why study algorithms ?
Algorithms help us to understand scalability.
Performance often draws the line between
what is feasible and what is impossible.
Algorithmic mathematics provides a language
for talking about program behavior.
By:- Shailendra Kumar Singh, CSE Department, MNNIT
4 Allahabad
Algorithm Analysis
Analyzing an algorithm means, predicting the resources
that the algorithm requires. Occasionally, resources
such as memory, communication bandwidth, or
computer hardware are of primary concern, but most
often it is computational time that we want to
measure.
Number of algorithms can be written to solve a single
problem. So by analyzing the algorithms, we can find
the best solution (algorithm) to the problem.
By:- Shailendra Kumar Singh, CSE Department, MNNIT
5 Allahabad
Order of growth
• Any algorithm is expected to work fast for any
input size.
• For smaller input size our algorithm will work fine
but for higher input size the execution time is
much higher.
• By increasing the size of n (input size) we can
analyze how well our algorithm works.
By:- Shailendra Kumar Singh, CSE Department, MNNIT
6 Allahabad
Cont..
Let input size, n=5 and we have to sort the list of
elements for e.g. 25,29,10,15,2.
• So for n=5 our algorithm will work fine but what if
n=5000?
• So our algorithm will take much longer time to sort
the elements or cause small delays to give the
result.
• How the behavior of algorithm changes with the no.
of inputs will give the analysis of the algorithm and
By:- Shailendra Kumar Singh, CSE Department, MNNIT
7 is Allahabad
called the Order of Growth.
Order of Growth
1. Big O (Oh)
2. Big Ω (Omega)
3. Big θ (Theta)
4. Small o
5. Small ω
By:- Shailendra Kumar Singh, CSE Department, MNNIT
8 Allahabad
What kinds of problems are solved by
algorithms?
Sorting
E commerce
Graph searching
…
By:- Shailendra Kumar Singh, CSE Department, MNNIT
9 Allahabad
Cont..
The problem of sorting
Input: Sequence 〈a1, a2, ..., an〉 of numbers.
Output: Permutation 〈a'1, a '2, ..., a'n〉
such that a'1≤ a'2≤ ...≤ a'n .
Example:
Input: 8 2 4 9 3 6
Output: 2 3 4 6 8 9
By:- Shailendra Kumar Singh, CSE Department, MNNIT
10 Allahabad
Insertion sort
Given array divided into two sub-arrays.
Pick first value from unsorted sub-array and store in
sorted sub-array.
Sorted
Unsorted
Example
By:- Shailendra Kumar Singh, CSE Department, MNNIT
11 Allahabad
Cont..
By:- Shailendra Kumar Singh, CSE Department, MNNIT
12 Allahabad
Cont..
By:- Shailendra Kumar Singh, CSE Department, MNNIT
13 Allahabad
Cont..
By:- Shailendra Kumar Singh, CSE Department, MNNIT
14 Allahabad
Cont..
By:- Shailendra Kumar Singh, CSE Department, MNNIT
15 Allahabad
Cont..
By:- Shailendra Kumar Singh, CSE Department, MNNIT
16 Allahabad
Cont..
By:- Shailendra Kumar Singh, CSE Department, MNNIT
17 Allahabad
Cont..
By:- Shailendra Kumar Singh, CSE Department, MNNIT
18 Allahabad
Cont..
By:- Shailendra Kumar Singh, CSE Department, MNNIT
19 Allahabad
Cont..
By:- Shailendra Kumar Singh, CSE Department, MNNIT
20 Allahabad
Insertion sort algorithm
By:- Shailendra Kumar Singh, CSE Department, MNNIT
21 Allahabad
Time Complexity
INSERTION-SORT (A) Cost Times
1 for j <- 2 to length[A] C1 2,3…(n+1)= n
2 do key <- A[j] C2 n-1
3 i <- j - 1 C3 n-1
4 while i > 0 and A[i] > key C4
5 do A[i + 1] <- A[i] C5
6 i <- i - 1 C6
7 A[i + 1] <- key
C7 n-1
By:- Shailendra Kumar Singh, CSE Department, MNNIT
22 Allahabad
Cont..
By:- Shailendra Kumar Singh, CSE Department, MNNIT
23 Allahabad
Cont..
By:- Shailendra Kumar Singh, CSE Department, MNNIT
24 Allahabad
Cont..
INSERTION-SORT (A) Cost Times
1 for j <- 2 to length[A] C1 2,3…(n+1)= n
2 do key <- A[j] C2 n-1
3 i <- j - 1 C3 n-1
4 while i > 0 and A[i] > key C4 (n-1), ((n-1)n)/2
5 do A[i + 1] <- A[i] C5 0 , ((n-1)n)/2
6 i <- i - 1 C6 0 , ((n-1)n)/2
7 A[i + 1] <- key
C7 n-1
By:- Shailendra Kumar Singh, CSE Department, MNNIT
25 Allahabad
Cont..
Best Worst
Cost Times
C1*n C1*n
C1 2,3…(n+1)= n
C2 n-1 C2*n-1 C2*n-1
C3 n-1 C3*n-1 C3*n-1
C4 (n-1), ((n-1)n)/2 C4*n-1 C4*(n^2-n)/2
C5 0 , ((n-1)n)/2 C5*0 C5*(n^2-n)/2
C6 0 , ((n-1)n)/2 C6*0 C6*(n^2-n)/2
C7 n-1 C7*n-1 C7*n-1
By:- Shailendra Kumar Singh, CSE Department, MNNIT
=an+b =a’n^2+b’n+c
26 Allahabad
Time & Space Complexity
Length of A=n
Best Case= an+b => n+1 => O(n)
Worst Case= a’n^2+b’n+c => n^2+n+1=> O(n^2)
eg. n=100, n^2=10,000
Space Complexity= 3 => O(1)
By:- Shailendra Kumar Singh, CSE Department, MNNIT
27 Allahabad
Big O
o It represents the maximum time that the algorithm takes for its
execution.
o Worst Case
o Upper bound
o Growth rate of f(n) is not more than growth rate of g(n).
there exist c>0 and n0>0 , n ≥ n0
f(n) ≤ c.g(n) => f(n) = O(g(n))
Eg. 2n=O(n) => 2n ≤ c.n; 2 ≤c;
2n ≤2*n => n ≥1 => n0=1
By:- Shailendra Kumar Singh, CSE Department, MNNIT
f(n)=2n+1 ≤ c.( ?) { c ≥3, n ≥1}
28 Allahabad
Big Ω
It represents the minimum time that the algorithm
takes for its execution.
Best case
Lower bound
Growth rate of f(n) is not less than growth rate of g(n).
there exist c>0 and n0>0 , n ≥ n0
f(n) ≥ c. g(n) f(n) = Ω (g(n))
Eg. F(n)=2n+1; 2n+1 ≥c.(?) 2n+1 ≥ c.(n) { c ≤ 2, n ≥0 }
By:- Shailendra Kumar Singh, CSE Department, MNNIT
29 Allahabad
Big θ
It represents the average time an algorithm takes for its
execution.
Average case
Exact time
Let c1 . g(n) and c2 . g(n) are two constant multiples of
g(n). c1, c2 and n0>0 , n ≥ n0
0 ≤ c1.g(n) ≤ f(n) ≤ c2. g(n) f(n) = θ(g(n))
Eg. c1.g(n) ≤ 2n+1≤ c2. g(n) c1.n ≤ 2n+1≤ c2.n
By:- Shailendra Kumar Singh, CSE Department, MNNIT
30 Allahabad
Small o and small ω
Small o
f(n) < c.g(n) => f(n) = o(g(n))
If for all c>0 and n0>0 , n ≥ n0
Eg. 2n=O(n) but 2n ≠ o(n);
2n < c.n 2n < 2.1 * n {c>2}; 2n=o(n^2) {c>0}
Small ω
f(n) > c. g(n) => f(n) = Ω (g(n))
For all c>0 and n0>0 , n ≥ n0
{ Iff g(n)=o(f(n)) } 2n=o(n^2) g(n)=2n, f(n)=n^2
f(n) > c. g(n) n^2 > c(2n) {c>0}
By:- Shailendra Kumar Singh, CSE Department, MNNIT
31 Allahabad