Design and Analysis of

Algorithms
4th Semester, CST Department
Theory Paper
Complexity asymptotic bounds
An approximate measure of the time taken for
the program to run
Stated as a function f(n) of the problem size
Problem size for sorting size of the array
Factorization of one integer its magnitude
Inequality in terms of known function g(n)
c1 2 g(n) for all n > n0
Complexity asymptotic bounds
Tight bound Big theta f(n) = (g(n))
c1 2 g(n)
Tight upper bound Big-O f(n) = O (g(n))
2 g(n)
Tight lower bound Big Omega f(n) = (g(n))
c1
Upper bound small-o f(n) = o (g(n))
f(n) < c2 g(n)
Lower bound small-omega f(n) = (g(n))
c1 g(n) < f(n)
Outline Syllabus different paradigms
Divide and conquer approach
Sorting algorithms
Dynamic programming approach
Matrix multiplication
Greedy strategy
Minimum spanning tree in graph
Backtracking strategy
Depth first and breadth first search in graph
 Outline syllabus -
some data structures and operations
Operation on Disjoint sets
Shortest path in graph
Operation on dynamic sets
Hashing
Operation on Polynomials
FFT algorithm
Operations in number theory
RSA algorithm
Outline syllabus advanced topics
Complexity theory
Randomized algorithms
Parallel algorithms
NP completeness
Approximation algorithms
 Divide and conquer strategy
DIVIDE into subproblems
CONQUER the subproblems solving recursively
COMBINE the solutions of subproblems

Consider a subproblems, each of size (1/b)

Time to divide = D(n), Time to combine = C(n)
T(n) = a T(n/b)+D(n)+C(n)
 Analogy in Merge Sort
DIVIDE: n-element sequence into n/2
elements in two subsequences
CONQUER: sort the two subsequences
recursively
COMBINE: MERGE the two subsequences

D(n) = (1), C(n) = (n), a=2 and b=2

T(n) = 2 T(n/2)+ (n) giving T(n) = (n lg n)
 MERGE SORT algorithm
MERGE_SORT (A,p,r)
if (p<r)
q = floor((p+r)/2)
MERGE_SORT(A,p,q)
MERGE_SORT(A,q+1,r)
MERGE(A,p,q,r)
Initial sequence: (5);(7);(6);(4);(1);(3);(2);(8)
Step 1 of conquer: (5,7) ; (4,6); (1,3); (2,8)
Step 2 of conquer: (4,5,6,7) ; (1,2,3,8)
Step 3 of conquer: (1,2,3,4,5,6,7,8)
 Quick Sort Algorithm
QUICK_SORT(A,p,r)
if (p<r)
q=PARTITION(A,p,r)
QUICK_SORT(A,p,q)
QUICK_SORT(A,q+1,r)
PARTITION(A,p,r)
x=A[p]
i=p-1, j=r+1
while TRUE
repeat j=j-
repeat i=i+1 until A[i
If i<j
exchange A[i], A[j]
Else
return (j)
 Solving recurrence
Substitution method
Guess a bound and then use mathematical
induction to prove the guess correct
Iteration method
Convert the recurrence into a summation and
then bound the summation
Master method
Provides bound for all recurrences of the form
T(n) = a T(n/b) + f(n)
 Substitution method
T(n)=2 T(n/2) + n Guess T(n) = O(n lg n)
Start by assuming this bound holds for T(n/2)
Hence lg (n/2)
0
Difficult for n=1 but holds for n=2 onwards with c
Substituting into the recurrence yields
lg (n/2)) + n
lg (n/2)) + n
= cn lg n cn lg 2 + n = cn lg n cn + n
lg n (proved by induction)
 Substitution method
lg n
Renaming m = lg n, T(2m) = 2 T(2m/2) + m
Next rename T(2m) = S(m) to get
S(m) = 2 S(m/2) + m
S(m) = O(m lg m) alike the earlier example.
Hence T(n)= T(2m) = S(m)=O(m lg m) = O(lg n lg lg n)
 Iteration method
T(n) = 3 T (n/4) + n ----- now iterate steps of this
= n + 3 ((n/4) + 3 T (n/16))
= n + 3 ((n/4) + 3 (n/16)+ 3T(n/64))
= n + 3(n/4)+9(n/16)+27 T(n/64)
So i-th term is 3i (n/4i)
Last term is 3^(log4 43))
Iteration hits n=1 when (n/4i) =1 or i > log4 n
Hence T(n) is a geometric series of (3/4) i
 Recursion tree
Convenient way to visualize the recursions

Consider T(n) = 2 T(n/2) + n2

At the topmost level, excess work is n2
The next level will need (n/2)2 + (n/2)2 = n2/2
The next level will need 4 (n/4)2 = n2/4
This continues to lg n levels, total work O(n2)
 Recursion tree
Consider T(n) = T(n/3) + T(2n/3) + n
Here at every level, work amount is n.
So it is important to find the height of this tree
Longest path from root to leaf is (2/3)k n =1
So the height of the tree is k = log 3/2 n
Solution of the recurrence is at most n log 3/2 n
This can be considered as O(n lg n)
Master theorem
Outline of proof
Terminating condition
Proof of first two cases
Proof of third case
Generalization
 Complexity and randomization
Running time of PARTITION on subarray of size
n will be (n)
Worst case: T(n) = T(n-1)+ (n) = (n^2)
Best case: T(n) = 2 T(n/2)+ (n) = (n lg n)
To find average case, we need to randomize.
RANDOMIZED_PARTITION(A,p,r)
i=RANDOM(p,r)
exchange A[p] with A[i]
return PARTITION(A,p,r)
 Average case complexity of
Randomized Quick Sort
T(n) = (1/n) (T(1)+T(n- n-1 T(q) +T(n-q)) + (n)
q=1

Now, (1/n)(T(1)+T(n-1))=(1/n)(O(1)+O(n^2))= (n)

Hence, due to symmetry wrt q,
n-1 T(k) + (n)
k=1
lg n + b
lg 2 lg n 1/8 n2 which is bounded

This establishes the (n lg n) bound.

 Lower bound for sorting algorithms
that employ comparisons
Result: Any decision tree that sorts n elements
has a height of h = (n lg n)
Proof:
There can be n! permutations of n elements.
Hence there are n! leaves of the decision tree
lg (n!)
Use Stirling approximation, n! > (n/e)^n
To get h > n lg n which leads to the result.
 Counting Sort and Radix Sort
COUNTING-SORT (A,B,k) A= [31 6 41 11 32 42 12 43] (array to be
sorted)
for i=1 to k
C= [2 0 2 3 0 1] (count)
C[i]=0
C= [2 2 4 7 7 8] (cumulative)
for j=1 to length[A] B= [- - - - - - - -] (initial)
C[A[j]]=C[A[j]]+1
for i=2 to k B= [- - - - - - 43 -] C= [2 2 4 6 7 8]
C[i]=C[i]+C[i-1] B= [- 12 - - - - 43 -] C=[1 2 4 6 7 8]
for j=length[A] downto 1 B= [- 12 - - - 42 43 -] C= [1 2 4 5 7 8]
B[C[A[J]]]=A[j] B= [- 12 - 32 - 42 43 -] C=[1 2 3 5 7 8]
C[A[j]]=C[A[j]]-1 B= [11 12 - 32 - 42 43 -] C=[0 2 3 5 7 8]
When k-O(n), T(n)=O(n) linear B= [11 12 32 41 42 43 -] C=[0 2 3 4 7 8]
Radix-Sort(A,d) B= [11 12 32 41 42 43 6] C=[0 2 3 4 7 7]
for i=1 to d B= [11 12 31 32 41 42 43 6] C=[0 2 2 4 7 7]
use stable counting sort
on digits from right to left, keeping B holds the sorted output
the other digits intact
 Linear time sorting Bucket sort
BUCKET-SORT (A)
n = length(A)
for i= 1 to n
insert A[i] into list B[floor(n A[i])]
for i=0 to n-1
sort list B[i] using insertion sort
Concatenate lists B[0], B[1], .. , B[n-1] together
 Bucket sort time complexity
Here ni is a random variable denoting |B[i]|, the size of
the ith bucket.
Insertion sort runs in quadratic time, so that expected
i^2)) summed over all n buckets
i.e. i= 0 to n-1.
Now probability that ni=k follows binomial distribution
with p=1/n since there are n elements and n buckets.
Hence E[ni]= np=1 and Var[ni]= np(1-p)=1-1/n
So, E[ni^2] = 2 1/n = (1).
Hence total expected time for bucket sort is O(n)
 Medians and order statistics
RANDOMIZED-SELECT(A,p,r,i)
if p==r return(A[p])
q=RANDOMIZED-PARTITION(A,p,r)
k=q-p+1
if i
return RANDOMIZED-SELECT(A,p,q,i)
else
return RANDOMIZED-SELECT(A,q+1,r,i-k)

RANDOMIZED_PARTITION(A,p,r)
i=RANDOM(p,r)
exchange A[p] with A[i]
return PARTITION(A,p,r)
 Time Complexity of Selection
To find an upper bound T(n) on expected time
- k,n-k)))+O(n)
-
[max(k,n-k) = k or n-k split at k=n/2]
In worst case, T(n-1) =O(n^2) so that T(n-1)/n = O(n)

To solve the recurrence- cn, and

noting that the sum over T(k) runs from n/2 to n;
-1) ½(n/2) ((n/2)-1)) + O(n)
cn picking c large enough
 Worst case linear time selection
1. Divide n elements into n/5 groups of 5 O(n)
elements each, n mod 5 in last group
2. Find median of each group using O(n) calls of
insertion sort and taking middlemost O(1) size set
element. For even size in last group, take insertion sorts
larger of the two medians.
3. Use SELECT recursively to find median x T(n/5)
of these n/5 medians
4. Partition input array around median of
medians x as the pivotal element, no of O(n)
elements on low side being k
5. Use SELECT recursively to find i-th T(7n/10 +6)
smallest element on low side if i i-
k) th element on high side if i > k
Time complexity median of medians
At least half of medians found in Step-
At least half of the groups contribute 3 elements >x,
except its own group and the last one i.e. at least
3(½(n/5)-2)
Hence no of elements > x or < x is at least = 3n/10 -6
Hence SELECT is called recursively in STEP 5 on at most
7n/10 + 6 elements

cn cn
One example case for SELECT