Design and Analysis of Algorithms

(203105301)
ushil Kumar Assistant ro essor
Computer Science and Engineering
CHAPTER-1
Introduction
 Outline
• Characteristics of algorithm.
• Analysis of algorithm: Asymptotic analysis of complexity bounds–
best, average and worst-case behavior;
• Performance measurements of Algorithm,
• Time and space trade-offs,
• Analysis of recursive algorithms through recurrence relations:
Substitution method
 Computational problems?
• A computational problem specifies an input-output relationship
• What does the input look like?
• What should the output be for each input?
• Example:
• Input: an integer number n
• Output: Is the number even?
• Example:
• Input: A list of names of people
• Output: The same list sorted alphabetically
 Problems and Solution as Algorithm
• For example, we need to solve a computational problem
“Convert a distance in kilometer to meter”
An algorithm specifies how to solve it,
I. 1. Read distance in kilometer
II. 2. Calculate distance-in-meter = distance-in-kilometer*1000
III. 3. Print distance-in-meter
 What is Algorithm?
• The word algorithm comes from the name of a Persian mathematician Abu
Ja’far Mohammed ibn-i Musa al Khowarizmi.
• Algorithm is a finite set of instructions used to accomplish particular task.

• An algorithm takes some value, or set of values, as input and produces

some value, or set of values, as output.

Image source : Google

 What is Algorithm?

Input Algorithm Output

Algorithm

Image source : Google

 Characteristics of algorithm

Input
• Input may be Zero or more

Output
• At least one output will be produced..

Definiteness
• Each instruction should be cleared and unambiguous.

Finiteness
• the steps should be finite.
 Characteristics of algorithm

Effectiven • each step must be performed in a finite amount of time

ess

Non –
• more than one interpretation should not be generated.
ambiguous

• produce an relevant and correct output For each input

Correctness produce.

• applicable to all homogeneous problems.

Generality
Analysis of algorithm

Image source : Google

 Why algorithm analysis?

• For one problem one or more solutions are available.

• Which one is better ? How can we choose?

• The analysis of an algorithm can help us in understanding of

solution in better way.

• Time & Space analysis

Image source : Google

 Order of Growth
• Any algorithm is expected to work fast for any input size.
• Smaller input size algorithm will work fine but for higher input size
execution time is much higher.
• So how the behavior of algorithm changes with the no. of inputs
will give the analysis of the algorithm and is called the Order of
Growth.

Image source : Google

 Algorithm falls under three types
• Minimum time required for
program execution
• Lower Bound
Algorithm Analysis

Best Case
• Average time required
Average Case for program execution
• Tight bound

Worst case • Maximum time required

for program execution
• Upper Bound
 Rate of Growth
• Rate at which the running time increases as a function of input is
called rate of growth.
Time Complexity Name Performance
1 Constant Best
logn Logarithmic Very good
n Linear Good
nlogn Linear Fair
Logarithmic
n2 Quadratic Acceptable
n3 Cubic Poor
2n Exponential Bad

Image source : Google

 Asymptotic Complexity

• Refers to defining the mathematical bound of its run-time performance.

• Running time of an algorithm as a function of input size n for large n.
• asymptotic means approaching a value or curve arbitrarily.
Asymptotic Notations
Ο Notation
• express the tight upper bound of an algorithm
• f(n)=O(g(n)) implies: Ο(g(n)) = { f(n) : there exists positive constants c>0 and n0
such that f(n) ≤ c.g(n) for all n > n0. }
 Asymptotic analysis
Ω Notation
• express the lower bound of an algorithm
• f(n)=  (g(n)) implies:  (g(n)) = { f(n) : there exists positive
constants c>0 and n0 such that f(n) ≥ g(n) for all n > n0. }
θ Notation
• express both the lower bound and the upper bound (tight bound)”
• f(n)=  (g(n)) implies:  (g(n)) = { f(n) : there exists positive
constants c1>0, c2>0 and n0 such that c1.g(n) ≤ f(n) ≤ c2.g(n) for all
n > n0. }
OR, if & only if f(n)=O(g(n)) and f(n)=  (g(n)) for all n > n0
Asymptotic analysis
 Asymptotic analysis
Examples on Big-O Notation(Upper bound/Worst case)
1)find upper bound for f(n)=2n+2
solution:2n+2<=4n, for all n>=1
so 2n+2=O(n) with c=4 and n0 =1

2)find upper bound for f(n)=2n2+1

solution:2n2+1<=3n2, for all n>=1
so 2n2+1=O(n2) with c=3 and n0 =1
 Asymptotic analysis
Examples on Big-O Notation(Upper bound/Worst case)

3)Find upper bound for f(n)=5n4+4n+1

Solution : 5n4+4n+1 <=7n4, for all n>=2
so 5n4+4n+1 =O(n4) with c=7 and n0 =2
 Now Try to solve below Questions
I. Find upper bound for f(n)=200

II. Find upper bound for f(n)=n3+n2

III. Show that 20n3=O(n4) for appropriate c and n0.

 Asymptotic analysis
Examples on Big- Ω Notation(Lower bound/ Best case)
1)find lower bound for f(n)=2n+2
solution:
2n<=2n+2, for all n>=1
so 2n+2= Ω(n) with c=2 and n0 =1

2)find lower bound for f(n)=2n2+1

solution:
n2<=2n2+1, for all n>=1
so 2n2+1= Ω(n2) with c=1 and n0 =1
 Asymptotic analysis
Examples on Big- Ω Notation(Lower bound/ Best case)
3)Find lower bound for f(n)=5n4+4n+1
Solution :
4n4 <=5n4+4n+1, for all n>=1
so 5n4+4n+1 = Ω(n4) with c=4 and n0 =1
 Now Try to solve below Questions

I. For n+54 = (lg n). Choose c and n0

II. Any linear function an + b is in (n). How?
 Asymptotic analysis
Examples on Big- θ Notation(Tight bound/ Average case)
1)find tight bound for f(n)=2n+2
solution:
2n<=2n+2<=4n, for all n>=1
so 2n+2= θ(n) with c1 =2, c2 =4 and n0 =1
2)find lower bound for f(n)=2n2+1
solution:
n2<=2n2+1<=3n2, for all n>=1
so 2n2+1= θ(n2) with c1 =1, c2 =3 and n0 =1
 Now Try to solve below Questions

I. Is 5n3  Q(n4) ??

II. How about 32n Q(3n)??

 Performance measurements of Algorithm
• Measure of the amount of time and/or space required by an algorithm for
an input of a given size (n).
• What effects run time of an algorithm?
I. computer used, hardware platform
II. representation of abstract data types (ADT's)
III. efficiency of compiler
IV. competence of implementer (programming skills)
V. complexity of underlying algorithm
VI. size of the input
Out of these above (V) and (VI) are generally most significant
 Time complexity and Space complexity

Amount of time required to

execute an algorithm

Time complexity
Complexity
Space complexity

Amount of space required

to execute an algorithm
Analysis of recursive algorithms through recurrence
Recurrence relations: relations
• Refer as an equation that recursively defines a sequence where the
next term is a function of the previous terms
• Can easily describe the runtime of recursive algorithms.
 Let’s create recurrence relation
Example:
f(n)
{
if (n == 0)
return 1;
else
return f(n - 1);
}
 Recurrence relation

• The base case(termination condition) is reached when n == 0. The

method performs one comparison. Thus, the number of operations
when n == 0, T(0), is some constant c.
• When n !=0, the method calls itself,
• using ONE recursive call, with a parameter n – 1.
 Recurrence relation

Therefore the recurrence relation is:

Recursive Case: T(n)=T(n-1),for n>0

Base Case: T(n)=c , for n=0 (Here T(n) is running time and n is input size)

Recurrence relation for factorial is T(n)=T(n-1) , for n>0

T(n)=c, for n=0
 Exercise
Find out recurrence relation for following algorithm.
I. int fib(int n)
{
if (n <= 1)
return n;
else
return fib(n-1) + fib(n-2);
}
 Exercise
Find out recurrence relation for following algorithm.
ii.) int A(int n) {
if (n == 1)
return 2;
else
return A (n / 2) + A( n / 2) + 5;
}
 Solving Recurrence Relation

I. Substitution Method
II. Iterative Method
III. Recurrence Tree Method
IV. Master’s Method
Solving Recurrence Relations- Substitution Method
Substitution Method
I. T(n)=T(n-1)+1, for n>0
T(n)=1, for n=0
Solution:
T(n)=T(n-1)+1 ...........................1)
find T(n-1) and put value in equation 1)
T(n-1)=T(n-2)+1 ..........................2)
T(n-2)=T(n-3)+1 .........................3)
put the value of equation 2) and 3) in 1)
T(n)=T(n-3)+3
Solving Recurrence Relations- Substitution Method
Substitution Method
continue for k times .now equation look like,
T(n)=T(n-k)+k
to reach base case ,n-k=0 so k=n
T(n)=T(n-n)+n
T(n)=1+n
so ,T(n)=O(n)
Solving Recurrence Relations- Substitution Method
Substitution Method
II. T(n)=T(n-1)+n, for n>1
T(n)=1, for n=1
Solution:
T(n)=T(n-1)+n ......................1)
T(n-1)=T(n-2)+(n-1)
T(n-2)=T(n-3)+(n-2)
Put the value in equation 1)
T(n)=T(n-2)+(n-1)+n
T(n)=T(n-3)+(n-2)+(n-1)+n .......................2)
Solving Recurrence Relations- Substitution Method
Substitution Method
continue for k times .now equation look like,
T(n)=T(n-(k+1))+(n-k)+.......(n-2)+(n-1)+n .....................3)
To reach base case ,n-(k+1)=1 so k=n-2
Put the value in equation 2)
T(n)=T(n-(n-2+1))+(n-(n-2))+.......(n-2)+(n-1)+n
T(n)=T(1)+2+.......(n-2)+(n-1)+n
T(n)=1+2+.......(n-2)+(n-1)+n
T(n)=(n(n+1))/2
So T(n)=O(n2)
Exercise
www.paruluniversity.ac.in
Design and analysis of Algorithms
Sushil Kumar

Assistant Professor CSE Department

 Unit-1

Recurrence tree method

Introduction
• It is Graphical Representation based method for finding the time complexity of recurrence equation.
• It is divide & conquer based approach.
• Each node of Tree represents the cost of single sub problem.
• And Each level cost will be sum up as a pre level cost.
• Now finally all pre level cost will be sum up to find Total cost of Problem.

• How To Derive the Tree?

• The root is indicated by last term of recurrence equation.

• Example: T (n) = 2T (n/2)+ n2

• The root will be - n2
• Now the problem will be divided in to sub problems.
• So 2T(n/2)= T(n/2)+T(n/2)
Introduction
• Other Example :T(n)=T(n/3)+T(2n/3)+n
• Here Root Will be: n
• And sub problem will be: T(n/3) & T(2n/3)

• Now Guess Example :T(n)=T(n/4) + T(n/2) + cn2

• Here Root Will Be: ?
• And Sub problems will be: ?
Deriving Recurrence Tree and Finding Cost

• Example: T(n) = T (n/3) + T (2n/3) + cn.

Steps For deriving Recurrence tree:
• Example: Recursion tree T(n) = T (n/3) + T (2n/3) + cn.
• Step-1
• Root Will be: ?
• Two sub problem will be : ?
• Step-2
• Let’s Start to divide the problem and find out pre-level cost (cost at each level)
• Tree with sub problem.
• Two sub problem per each given
node.(n/3),(2n/3).

• Up to we get T(1) at leaf.

Fig.1.2.1[1]:Recurrence Tree
• Step-3
• Lets sum up all the pre-level cost and find the final cost for the given equation.
• You can find that in given Tree:

• At Level-0
• Root will be n –(Pre-level cost at level-0 : Cn)
• Each node represent cost of sub problem it can be seen as(n/3),(2n/3)

• At level-1
• At first level Pre- level cost will be = (n/3)+(2n/3)=(3n/3)=n or cn (where c=1 in this case)
• At level-2
• At level-2 again (n/3) will be divided in to n/9 and 2n/9
• And (2n/3) will be divided in to 2n/9 and 4n/9
• Total cost at level 2(pre-level cost at level -2)= c(n/9+2n/9+2n/9+4n/9)=c9n/9=cn

• Find pre-level cost for all level (Which will eventually be: cn).

• For final cost lets sum up all pre-level cost. And Find upper bound.
Lets Analyse & Sum up
• Omit floor and ceiling functions for simplicity. As before, we let c represent the constant factor in
the O(n) term.
• When we add the values across the levels of the recursion tree, we get a value of cn for every
level.
• The longest path from the root to a leaf is n → (2/3)n → (2/3)2n → ··· → 1.Since (2/3)k n =
1 when k = log3/2 n, the height of the tree is log3/2 n.
• The solution to the recurrence to be at most the number of levels times the cost of each level, or
O(cn log3/2 n) = O(n lg n).

• The total cost is evenly distributed throughout the levels of the recursion tree. There is a
complication here: we have yet to consider the cost of the leaves. If this recursion tree were a
complete binary tree of height log3/2 n, there would be leaves.
• Since the cost of each leaf is a constant, the total cost of all leaves would then be , which is ω(n lg
n).
• This recursion tree is not a complete binary tree, however, and so it has fewer than leaves.
Moreover, as we go down from the root, more and more internal nodes are absent.
• Consequently, not all levels contribute a cost of exactly cn; levels toward the bottom contribute
less.

• This is just guess so verify it by substitution method.

Class Exercise
• Draw the recursion tree for
T(n) = 3T(n/4) + cn2.
where c is a constant, and provide a tight asymptotic bound on its solution. Verify your bound
by the substitution method.
Another Example
• Draw the recursion tree for
T (n) = 4T (⌊n/2⌋)+cn, where c is a constant, and provide a tight asymptotic bound on its
solution. Verify your bound by the substitution method.
Master Theorem
Terminology
• Direct way to find the Solution of given Recurrence equation.
• It can apply only where the recurrence equation, can be transferred in following form or is of this
type.
T(n) = aT(n/b) + f(n) where a >= 1 and b > 1 and f(n) is a asymptotic function .

There are following three cases:

1. If f(n) = Θ(nc) where c < Logba then T(n) = Θ(nLogba)

2. If f(n) = Θ(nc) where c = Logba then T(n) = Θ(ncLog n)
3.If f(n) = Θ(nc) where c > Logba then T(n) = Θ(f(n))
Cont...
• Examples of some standard algorithms whose time complexity can be evaluated using Master
Method.

• Merge Sort: T(n) = 2T(n/2) + Θ(n). It falls in case 2 as c is 1 and Logba is also 1. So the solution is
Θ(n Logn)
• Binary Search: T(n) = T(n/2) + Θ(1). It also falls in case 2 as c is 0 and Logba is also 0. So the
solution is Θ(Logn)
Cont...
• Master method is mainly derived from recurrence tree method. If we draw recurrence tree of
T(n) = aT(n/b) + f(n), we can see that the work done at root is f(n) and work done at all leaves is
Θ(nc) where c is Logba. And the height of recurrence tree is Logbn.

• In recurrence tree method, we calculate total work done. If the work done at leaves is
polynomially more, then leaves are the dominant part, and our result becomes the work done at
leaves (Case 1).

• If work done at leaves and root(f(n)) is asymptotically same, then our result becomes height
multiplied by work done at any level (Case 2).

• If work done at root(f(n)) is asymptotically more, then our result becomes work done at root
(Case 3).
Case 1
• Lets take. T (n) = 9T(n/3) + n.
In given recurrence,
we have find a = 9, b = 3, f (n) = n.
so Θ(nLogba)=Θ(n2) .
Since f(n)=n , where c= 1
we can apply case 1 of the master theorem and conclude that the solution is T (n) = Θ(n2).
Case 2
• Lets take. T (n) = T (2n/3) + 1.
In given recurrence,
we have find a = 1, b = 3/2, f (n) = 1.
so Θ(nLogba)=Θ(nlog3/21) =n0 =1
Since f(n)=1.
we can apply case 2 of the master theorem and conclude that the solution is T (n) = Θ(log n).
Case 3
• T(n) = 3T(n/4) + n lg n.
we have a = 3, b = 4, f (n) = n lg n, and Θ(nLogba)= Θ(n0.793) . Since f(n)= Θ(n0.793+c)
Where c=0.217 so case 3 can be apply so
T(n)= Θ(f(n)).

For sufficiently large n, a f(n/b) = 3(n/4) lg(n/4) ≤ (3/4)n lg n = cf (n) for c = 3/4. Consequently, by
case 3, the solution to the recurrence is T(n) = Θ(n lg n).
Where master method can not apply?
• T(n/2)=2T(n/2)+n lg n

• Even though it has the proper form: a=2,b=2,f(n)=n lg n, and nlogba =n

• Now f(n)=n lg n which is greater than nlogba =n
• This is actually not a polynomially larger. That is the ration is (n lg n)/n=lg n is asymptotically less
that nc for any c
• This case fall between case 2 and case 3
Class exercise
1. T (n) = 4T(n/2) + n.
2. T (n) = 4T(n/2) + n2.
3. T (n) = 4T(n/2) + n3.
4. Can the master method be applied to the recurrence T (n) = 4T(n/2) + n2 lg n? Why or why
not? Give an asymptotic upper bound for this recurrence.
References
1. Introduction to Algorithms, 3rd Edition (The MIT Press),Author: Thomas cormen, Charlie
Leiserson, Ronald Rivest.