INTRODUCTION TO

ANALYSIS OF ALGORITHM
2019-20
Even Sem Mrs. S. R. Patil
 Contents
2

Performance analysis , space and time complexity

Growth of function – Big –Oh ,Omega , Theta
notation
Mathematical background for algorithm analysis,
Analysis of selection sort , insertion sort.
Recurrences: -
The substitution method
Recursion tree method
Master method
 Definition of Algorithm
3

An algorithm is a set of rules for carrying out calculation either by

hand or on a machine.
OR
An algorithm is a finite step-by-step procedure to achieve a
required result.
OR
An algorithm is a sequence of computational steps that transform
the input into the output.
OR
An algorithm is a sequence of operations performed on data that
have to be organized in data structures.
 Algorithm must satisfy following criteria
4

Input: One or more quantities.

Output: at least one quantity.
Definiteness: each instruction must be clear
and unambiguous.
Finiteness: Algorithm must terminate after
finite steps.
Effectiveness: Each instruction must be
feasible.
 Problem Solving Using Computers
5

Problem:
Strategy:
Algorithm:
• Input:
• Output:
• Step:
Analysis:
• Correctness:
• Time & Space:
• Optimality:
Implementation:
Verification:
 Design and Analysis Fundamentals
6

The “Design” pertain to

The description of algorithm at an abstract
level by means of a pseudo language, and
Proof of correctness i.e., the algorithm solves
the given problem in all cases.

The “Analysis” deals with

performance evaluation (complexity analysis).
 Performance Analysis
7

• Amount of computer memory

required during the program execution,
Space as a function of the input size.
Complexity

• Running time of the program as a

Time function of the size of input.
complexity
 Time Complexity
8

Best Case: Minimum amount time required for

the execution of an algorithm.

Worst Case: The behavior of the algorithm w.r.t.

the worst possible case of the input instance.

Average Case: Expected behavior when the

input is randomly drawn from a given distribution.
 Algorithm
Algorithm bs(low, high, a, x)
9 //input: Array a of size n, key to search
//output: successful or unsuccessful search
1. If low>high
2. return -1
3. else
4. Set mid←(low+high)/2;
5. if a(mid)=x
6. return mid
7. else
8. If x>a[mid]
9. Call bs(mid+1,high,a,x)
10. otherwise
11. Call bs(low,mid-1,a,x)}
 Asymptotic Notations
10

Asymptotic definition
A line that continually approaches a given curve but
does not meet it at any finite distance.
 Asymptotic Notation
11

Big O Notation: (Upper Bound)

 Examples on Big O
12

Prove that
1. if f(n)=5n3+n2+6n+2 then f(n)=O(n3)
2. If f(n)=4n3+2n+3 then f(n)=O(n3)
3. if f(n)=15n3+n2+4 then f(n)=O(n4)
4. If f(n)=1000n2+50n then f(n)=O(n2)
5. If f(n)=n2+50n then f(n)=O(n2)
6. If f(n)=n2+50n then f(n)≠O(n)
 Asymptotic Notation
13

Ω-Notation: (Omega Notation) Lower Bound

 Examples on Ω-Notation
14

Prove that
1. if f(n)=15n3+n2+4 then f(n)=Ω (n3)
2. if f(n)=15n3+n2+4 then f(n)=Ω (n2)
3. if f(n)=15n3+n2+4 then f(n)=Ω (n4)
4. If f(n)=1000n2+50n then f(n)=Ω (n2)
5. If f(n)=n2+50n then f(n)=Ω (n)
 Asymptotic Notation
15

Θ-Notation: (Theta Notation) Tight Bound

 Examples on Θ-Notation
16

Prove that
1. if f(n)=15n3+n2+4 then f(n)=Θ (n3)
2. if f(n)=15n3+n2+4 then f(n)=Θ (n2)
3. If f(n)=1000n2+50n then f(n)=Θ (n2)
4. If f(n)=n2+50n then f(n)=Θ (n)
 Growth of Functions
17

O(1)<O(logn)<O(n)<O(n*logn)<O(n2)<O(n3)<2n<n!
 Mathematical Analysis of Nonrecursive
Algorithms
18

General plan for analyzing time complexity of non-recursive

algorithms:
1. Decide a parameter indicating an input size.
2. Indentify algorithm’s basic operation.
3. Check whether the no. of times the basic operation is
executed depends only on size of an input. If it also
depends on some additional property the worst case,
average case and best case efficiencies have to be
investigated separately.
4. Set up sum expressing the number of times the algorithm’s
basic operation is executed.
5. Using standard formula find closed form formula for the
count.
 Important summation formulas
19

n
1. ∑1 = n
i =1
n
n(n + 1)
2. ∑ i = 1 + 2 + ... + n =
i =1 2
n
n(n + 1)(2n + 1)
3. ∑ i = 1 + 2 + ... + n =
2 2 2 2

i =1 6
n
1 k +1
4. ∑ i = 1 + 2 + ... + n ≈
k k k k
n
i =1 k +1
n n +1
a −1
5. ∑ a = 1 + a + ... + a =
i n
where (a ≠ 1)
i =0 a −1
 Important summation formulas
20
n
6. ∑ 2 = 1 + 2 + 2 + 2 + .... + 2 = 2
i 2 3 n n +1
−1
i =0
n
7. ∑ i 2 = (n − 1)2i n +1
+2
i =0
n
8. ∑ lg i ≈ n lg n
i =0
p2
9.∑1 = p2 − p1 + 1
i = p1
 Example: Linear Loop
21

Algorithm maxelement ( A , n )
1. max ← A( 1 )
2. for i = 1 to n do
3. if A(i) > max then
4. max = A(i)
5. return max
 Example: Linear Loop
22

Algorithm maxelement ( A , n )
n
1. max ← A( 1 ) f ( n ) = ∑ 1 = n = Θ( n )
2. for i = 1 to n do i =1
i=
3. if A(i) > max then
4. max = A(i)
5. return max
 Examples: Dependent loop
23

Algorithm Sort( A, n )
n −1 n
1. for i = 1 to n − 1 do f ( n) = ∑ ∑1
2. for j = i + 1 to n do i =1 j =i +1

3. if A(i ) > A( j ) then

4. Swap A(i) ↔ A(j)
5. return
 n −1 n
f (n) = ∑ ∑1
i =1 j = i +1
24
n −1 n −1
= ∑ [( n ) − (i + 1) + 1] = ∑ ( n − i )
i =1 i =1
n −1 n −1
= ∑ (n) − ∑ i
i =1 i =1

( n )( n − 1)
= n ( n − 1) −
2
 1
= n ( n − 1) 1 − 
 2
n ( n − 1) n 2 − n
= = = Θ(n 2 )
2 2
 Example: Loops within Loop
25

Algorithm matrixmultiplication(a, b, n)
1. for i ←1 to n do
2. for j ← 1 to n do
3. c[i, j ] ← 0
4. for k ← 1 to n do
5. c[i, j ] ← c[i, j ] + a[i, k ] * b[k , j ]
 Example: Loops within Loop
26

n  n  n

f (n) = ∑  ∑ 1 +
 ∑ 2  
i =1  j =1  k =1 
n  n 
= ∑  ∑ (1 + 2 n )
 
i =1  j =1 
n  n n  n
= ∑  ∑ 1 + 2∑ n  =
  ∑ (n + 2 n )
2

i =1  j =1 j =1  i =1
n n
= ∑ n + 2∑ n
i =1 i =1
2 2 3
= n + 2n = Θ (n ) 3
 Example
27

Algorithm sum(n)
1. s ← 0
2. for i ←1 to n do
3. s ← s+i
4. return s
a. What does this algorithm compute?
b. What is its basic operation?
c. How many times is this basic operation executed?
d. What is its order of growth?
 Logarithmic loop
28

Algorithm ABC (n)

1. c ←n
2. while(c>1)
3. x ←x+1
4. c ← c/2
5. return
 Logarithmic loop
29

Algorithm ABC (n)

1. c ←n For n=8 loop executes 3 times
For n=16 loop executes 4 times
2. while(c>1) :
3. x ←x+1 :
4. c ← c/2 For n=2k loop executes k times
5. return So n=2k
k = log2 n
 Selection sort
30

Algorithm selection(a,n)
//input:
//output:

1. for pass←1 to n-1 do

2. min_index ← pass,
3. for i ← pass+1 to n
4. if(a(min_index)->a[i])
5. min_index ← i
6. Swap a(min_index) ↔ a(pass)
 Selection sort analysis (Best/Worst/
31
Average Case)
 n 
n −1
f (n) = ∑  ∑1 
Algorithm selection(a,n) pass =1  i = pass +1 
//input: n −1
//Output: = ∑ (n − pass − 1 + 1)
pass =1
1. for pass←1 to n-1 do
n −1
2. min_index ← pass = ∑ (n − pass )
pass =1
3. for i ← pass+1 to n n −1 n −1

4. if(a(min_index)>a[i]) = ∑ n − ∑ pass
pass =1 pass =1
5. min_index ← i
n(n − 1)
6. Swap a[min_index] ↔ a[pass] = n ( n − 1) −
2
[
= (n − 1). n − n ]2 = (n − 1).[1 + n 2]
= Θ( n 2 )
 Insertion sort analysis
32

Algorithm insertion (a,n)

1. for pass←1 to n-1 do
2. temp ← a[pass+1]
3. i ← pass
4. while(i>0 and a[i]>temp)
5. a[i+1] ← a[i]
6. i ← i-1
7. a[i+1] ← temp
 Insertion sort analysis
33

Algorithm insertion (a,n)

1. for pass←1 to n-1 do Best Case
n −1
2. temp ← a[pass+1] f (n) = ∑1 = (n − 1)
3. i ← pass pass =1

= Ω( n)
4. while(i>0 and a[i]>temp)
5. a[i+1] ← a[i]
6. i ← i-1
7. a[i+1] ← temp
 Insertion sort analysis
34

n −1
 pass 
f ( n ) = ∑  ∑ 1
pass =1  i =1 
n −1
 pass 
f ( n ) = ∑  ∑ 1
pass =1  i =1 
n −1
= ∑ ( pass )
pass =1

n ( n − 1)
= = n2 − n
2
= Ο ( n 2 ) or Θ ( n 2 )
 Mathematical Analysis of recursive Algorithms
35

General plan for analyzing time complexity of recursive

algorithms:
1. Decide on a parameter indicating an input size.
2. Indentify algorithm’s basic operation.
3. Check whether the no. of times the basic operation is
executed depends only on size of an input. If it also
depends on some additional property the worst case,
average case and best case efficiencies have to be
investigated separately.
4. Set up recurrence relation, with an appropriate initial
conditions, for the number of times the basic operation
is executed.
5. Solve the recurrence relation.
 Methods to solve Recurrence Relation
36

Substitution Method
Forward Substitution
Backward Substitution
Master Method
Recursion Tree Method
 Recursive algorithm
37

Algorithm fact(n)
1. if (n ←0 or 1)
2. return 1
3. else
4. return (n*fact(n-1))
 Recursive algorithm
38

Algorithm fact(n)
if n = 0 ,1 T(n) = 0
1. if (n ←0 or 1)
otherwise T (n) = T (n − 1) + 1
2. return 1
3. else For recursive call Time for
performing
4. return (n*fact(n-1)) fact(n-1)
multiplication

Solve recurrence relation by substitution method

 Backward Substitution
39

if n = 0,1 T(n) = 0
n >1 T (n) = T (n − 1) + 1
T (n) = T (n − 1) + 1
= T (n − 2) + 1 + 1
= T (n − 3) + 1 + 1 + 1
= :
= T (n − i ) + i
we know T (n) = 0 for n = 0, substitute n − i = 0 so n = i
= T ( 0) + n
= 0 + n = Θ( n )
 Forward Substitution
40

if n = 0,1 T(n) = 0
n >1 T (n) = T (n − 1) + 1
T (0) = T (1) = 0
T (2) = T (1) + 1 = 1
T (3) = T (2) + 1 = 2
T (4) = T (3) + 1 = 3
:
:
T ( n ) = n − 1 = Θ( n )
 Tower of Hanoi
41

Problem definition:
let A, B, C are 3 towers (pegs) and n disks(rings) are
placed on tower A in the descending order of their
sizes. The problem is to move all the disks from
tower A to tower C using tower B.
 Tower of Hanoi
42

Move disk 1 from A to C.

Move disk 2 from A to B. n-1 disks are moved from A to B using C

Move disk 1 from C to B.

Move disk 3 from A to C. nth disk is moved from A to C

Move disk 1 from B to A.

Move disk 2 from B to C. n-1 disks are moved from B to C using A

Move disk 1 from A to C.

 Tower of Hanoi
43

Algorithm Tower(n, A, C, B)
1. if(n>1)
2. Tower(n-1, A, B, C) //n-1 disks are moved from A to B using C

3. move nth ring from A to C

4. Tower(n-1, B, C, A) //n-1 disks are moved from B to C using A
 Recurrence Relation for Hanoi Analysis
44

if n = 1 T (n) = 1
otherwise T (n) = T (n − 1) + 1 + T (n − 1)
= 2T (n − 1) + 1
 Analysis of Tower of Hanoi Problem
45

if n = 1 T (n) = 1
otherwise T (n) = T (n − 1) + 1 + T (n − 1)
= 2T (n − 1) + 1 = 2i T (n − i ) + 2i − 1
T (n) = 2T (n − 1) + 1 we know T (1) = 1
= 2(2.T (n − 2) + 1) + 1 ∴ substitute n − i = 1 and i = n + 1
= 2 2.T (n − 2) + 2 + 1 = 2 2.T (n − 2) + 3
= 2 n +1T (1) + 2 n +1 − 1
= 2 2 (2.T (n − 3) + 1) + 2 + 1
= 2.2 n +1 − 1 = 4.2 n − 1
= 23 T (n − 3) + 2 2 + 2 + 1 = 23 T (n − 3) + 7
= Θ( 2 n )
= 23 (2.T (n − 4) + 1) + 2 2 + 2 + 1
= 2 4 T (n − 4) + 23 + 2 2 + 2 + 1 = 2 4 T (n − 4) + 15
=M
= 2 i T ( n − i ) + 2i − 1
 Examples on Substitution Method
46

1. T ( n ) = 1 for n = 1
= 2T (n − 1) for n > 1
2. T (1) = 0, T (n) = T ( n ) + 1
2
3. T (1) = 1, T (n) = 2T ( n ) + n
2
 Examples on Substitution Method
47

1. T (n) = 1 for n = 1
= 2T (n − 1) for n > 1
Solu : T (n) = 2T (n − 1)
= 2.(2T (n − 2)) = 2 2 T (n − 2)
= 2 2 (2T (n − 3)) = 23 T (n − 3)
:
:
= 2i T (n − i )
We know T(1) = 1, Substitute n-i = 1, i = n-1
T (n) = 2 n −1T (1)
= 2 n −1.1 = 2 n −1 = Θ(2 n )
 Examples
48

2. T (1) = 0
T ( n) = T ( n ) + 1
2
= T (n ) + 1 + 1
4
= T (n ) + 1 + 1 + 1
8
M
= T(n i)+i
2
we know T (1) = 0
∴ n i = 1 i.e. n = 2i , i = log 2 n
2
T (n) = T (1) + log 2 n
= 0 + log 2 n = Θ(log 2 n)
 = 2(2.T ( n ) + n ) + n
Examples 4 2
= 4T ( n ) + n + n
49 4
3. T (1) = 1 = 4(2.T ( n ) + n ) + 2n
8 4
T (n) = 2T ( n ) + n
2 = 8.T ( n ) + 3n
8
M
= 2i T ( n i ) + in
2
we know T (1) = 1
∴n = 1 i.e. n = 2i , i = log 2 n
2i
T (n) = 2i T (1) + log 2 n.n
= n + n log 2 n = Θ(n log 2 n)
 Examples
50

T (n) = T(n-1 ) + 5 for n > 1, T( 1 ) = 0

T(n) = 3.T(n-1 ) for n > 1, T( 1 ) = 4
T(n) = T(n-1 ) + n for n > 0, T( 0 ) = 0
T (n) = T ( n ) + 1 for n > 1, T (1) = 1 solve for n = 3k
3
 Master Theorem
51

Let a ≥ 1 and b > 1 be constants , let f(n) be function, and

let T(n) be defined on nonnegative integers by the recurrence
T (n) = aT (n / b) + f (n)
Then T(n) has the following asymptotic bounds :

1. if f(n) = O(n logb a − ε )for some constant ε > 0, then T(n) = Θ(n log b a )

2.if f(n) = Θ(n logb a ), then T(n) = Θ(n logb a log n)

3.if f(n) = Ω(n logb a + ε )for some constant ε > 0 and if af(n/b) ≤ cf(n)
for some constants c < 1 and all sufficiently large n, then T(n) = Θ(f(n)).
 Examples on Master Theorem
52

1.T (n) = 9T (n / 3) + n
2.T (n) = T (2n / 3) + 1
1 2
3.T (n) = T (n / 2) + n + n
2
4.T (n) = 2T (n / 4) + n + 42
3
5.T (n) = 3T (n / 2) + n + 1
4
6.T (n) = 3T (n / 4) + n log n
7.T (n) = 2T (n / 2) + n log n
8.T (n) = 2T (n / 2) + Θ(n)
9. T (n) = 7T (n / 2) + Θ(n 2 )
 Examples on Master Theorem
53

1.T (n) = 9T (n / 3) + n Ans :T (n) = Θ( n 2 )

2.T ( n) = T (2n / 3) + 1 Ans :T (n) = Θ( n log n)
1
3.T ( n) = T (n / 2) + n 2 + n
2
4.T ( n) = 2T (n / 4) + n + 42
3
5.T (n) = 3T (n / 2) + n + 1
4
6.T (n) = 3T (n / 4) + n log n Ans :T (n) = Θ( n log n)
7.T (n) = 2T (n / 2) + n log n Ans : T (n) = Θ( n log 2 n)
8.T (n) = 2T (n / 2) + Θ(n)
9. T (n) = 7T (n / 2) + Θ(n 2 )