Algorithms and Complexity

Problem: Find the average of two numbers

Input: two numbers // 23, 45

Program (algorithm):
Step 1: read two numbers// 23, 45
Step 2: sum the numbers// 68
Step 3: divide the sum by 2 // 34
Output: print output// 34

Efficiency of Algorithm:
Time Complexity and
Space Complexity

RAM model:
to find time complexity
count all basic operations (+,-,/,*,++,--,=, >,<, function call, access)

Example (complexity): Find the max number from n numbers

Input: n numbers
Output: 1 number
Algorithm (code):

int a[n] 0
int max = a[0] 1+ 1=2
for(i=1; i<n; i++){ 1+ n + (n-1)=2n
if(max < a[i]) max = a[i]  4*(n-1) = 4n-4
} 0
print(max) 1
…..
Time complexity (maximum case):
0+2+2n+4n-4+0+1 = 6n-1

Space complexity:
List of variables: a, n, max, i

4n[a]+4 [n]+4 [max]+4 [i] = 4n+12

If n=10 then
time complexity = 59 unit and space complexity = 52 bytes

Page 1 of 10
Asymptotic Notations
for the above algorithm, T(n)=6n-1, n>0
Other examples:
T(n)=4n+1, n>3
T(n)=3n^2+2n-1, n>15

There are 3 asymptotic notations:

Big O: T(n) will be of O(g(n)) if T(n)<=c.g(n) for n>=n0
Big Ω:
Big Θ:

Find Big O for T(n)=4n+1

T(n)=4n+1
T(n)<=4n+n, n>0
T(n)<=5n, n>=1 Here, c = 5 and g(n) = n, n0=1
So, T(n) is O(n)

Another way to do it
T(n)=4n+1
T(n)<=4n+0.5n, n>1
T(n)<=4.5n, n>=2 Here, c = 4.5 and g(n) = n, n0=2
So, T(n) is O(n)

Another way to do it
T(n)=4n+1
T(n)<=4n+0.2n, n>4
T(n)<=4.2n, n>=5 Here, c = 4.5 and g(n) = n, n0=2
So, T(n) is O(n)

Another way to do it
T(n)=4n+1
T(n)<4n + 2n, n>=1
T(n)<6n, n>=1 Here, c = 6 and g(n) = n, n0=1
So, T(n) is O(n)

Another way to do it
T(n)=4n+1
T(n)<4 n^2, n>1
Here, c = 4 and g(n) = n^2, n0=1
So, T(n) is O(n^2)

Page 2 of 10
Big O: More examples:
T(n)=3n^2+2n-1
T(n) < 3n^2+2n^2-1+1, n>0
T(n) < 5n^2, n>=1
Where c=5, g(n)=n^2, n0=1
So, T(n) is O(n^2)

T(n)=3n^2+2n+1
T(n) <= 3n^2+2n^2+n^2, n>=1
T(n) <= 6n^2, n>=1
Where c=6, g(n)=n^2, n0=1
So, T(n) is O(n^2)

Exercise: Find big O for T(n)=3n^2-2n+1

Worst Case and Best Case (example from sorting):

3, 5, 8, 4, 9, 11  3, 4, 5, 8, 9, 11 (few swapping) – 2ms
11, 9, 8, 4, 5, 3  3, 4, 5, 8, 9, 11 (more swapping) – 5ms
3, 4, 5, 8, 9, 11  3, 4, 5, 8, 9, 11 (zero swapping, best case)- 1ms
11, 9, 8, 5, 4, 3  3, 4, 5, 8, 9, 11 (maximum swapping, worst case) – 15ms

Big Omega:
T(n)=3n^2+2n+1
T(n) >3n^2, n>=1
Where c=3, g(n)=n^2, n0=1
So, T(n) is Ω(n^2)

Another way to do it
T(n)=3n^2+2n+1
T(n) > 3n, n>=1
Where c=3, g(n)=n, n0=1
So, T(n) is Ω(n)

Page 3 of 10
Big Theta:
T(n)=3n^2+2n+1
T(n) <= 6n^2, n>=1 (Big O)
T(n) > 3n^2, n>=1 (Big Ω)

We can write:
3n^2<T(n) <= 6n^2, n>=1
Where c1=3, c2=6, g(n)=n^2, n0=1
So, T(n) is Θ(n^2)

Next Class:
Find time complexity functions for bubble sort, linear search, binary search
Find asymptotic notations also for the above.

Complexity for Bubble Sort

Page 4 of 10
Merge Sort:

Counting Sort:
Input:
3 6 7 2 1 9 5 2

Count:
1 2 3 4 5 6 7 8 9

1 2 1 0 1 1 1 0 1

Count cumulative:
1 2 3 4 5 6 7 8 9

1 321 43 4 54 65 76 7 87

Output:
1 2 2 4 5 6 7 9

Page 5 of 10
 Divide and Conquer Algorithms
1. Repeated backward substitution
2. Substitution method
3. Recurrence Tree
4. Master Method

Repeated backward substitution

Merger Sort, T(n) = 2T(n/2) + n (recurrence relation)
Solve this by Repeated backward substitution method to find T(n):
T(n) = 2T(n/2) + n
= 2[2T(n/4) + n/2] + n
= 2^2 T(n/2^2) + n + n
= 2^2 [2T(n/2^3) + n/4] + 2n
= 2^3 T(n/2^3) + n + 2n
= 2^3 T(n/2^3) + 3n
= replace again
= 2^4 T(n/2^4) + 4n
…..
…..
….. after k substitution
T(n) = 2^k T(n/2^k) + kn
[n=2^k,
log n = log (2^k) = k log2 = k]
= n T(1) + (log n) n
T(n) = n log n + n

Big O
T(n) <= n log n + n log n
T(n) <= 2n log n, c=2, g(n)=n log n
T(n) is O(n log n)

Task: Do same for Binary Search [Lab Manual 2]

T(n) = 1.T(n/2) + 1

Substitution Method
T(n) = 4 T(n/2) + n

Guess: T(n) is O(n^3) => prove T(n) <= cn^3

T(n) = 4 T(n/2) + n
<= 4 c(n/2)^3 + n

Page 6 of 10
 = 0.5c n^3 + n
= c n^3 – 0.5c n^3 +n
= c n^3 – (0.5 n^3 -n)
<= cn^3 (proved)

Task: Prove that Merger Sort is O(n logn) using substitution method
Task: Prove that Merger Sort is O(n logn) using Recurrence Tree
Task: Prove that Binary Search is O(logn) using substitution method
Task: Prove that Binary Search is O(logn) using Recurrence Tree

Example of recurrence relation:

Pseudo code/code:
f(n):
if(n==0) return 1
else return n*f(n-1)

Recurrence Relation
f(n) = {
( )

T(n) = {
( )

Master Method:
T(n) = a T(n/b) + f(n)
Case 1: f(n) is O( )  T(n) is Θ( )
Case 2: f(n) is Θ( )  T(n) is Θ( )
Case 3: f(n) is Ω( )  T(n) is Θ( ( ))

Example: T(n) = 2 T(n/2) +n

a =2 b = 2 f(n) = n
Step 1: =n
Step 2: compare f(n) and  f(n) is Θ( )  case 2
Step 3: T(n) is Θ( )  T(n) is Θ( )

Page 7 of 10
Example: T(n) = T(n/2) +1
a =1 b = 2 f(n) = 1
Step 1: = =1
Step 2: compare f(n) and  case 2
Step 3: T(n) is Θ( )  T(n) is Θ( )

Example: T(n) = 9T(n/3) +n

a =9 b = 3 f(n) = n
Step 1: = =
Step 2: compare f(n) and  f(n) is O( ) case 1
Step 3: f(n) is O( )  T(n) is Θ( )  T(n) is Θ( )

Example: T(n) = 3T(n/4) +n logn

a =3 b = 4 f(n) = n log n
Step 1: = =
Step 2: compare f(n) and  f(n) is Ω( ) case 3
Step 3: f(n) is Ω( )  T(n) is Θ( ( ))  T(n) is Θ( )

Page 8 of 10
 Greedy Algorithms
Fractional Knapsack Problem:
W=16
ITEM WEIGHT VALUE Per unit value

i1 6 6 1

i2 10 2 0.2

i3 3 1 0.33

i4 5 8 1.6

i5 1 3 3

i6 3 5 1.67

Rearrange the item according to their per unit value:

i5, i6, i4, i1, i3, i2
i5: 1KG, Remaining 15, Profit: 3
i6: 3KG, Remaining 12, Profit: 3+5
i4: 5KG, Remaining 7, Profit: 3+5+8
i1: 6KG, Remaining 1, Profit: 3+5+8+6
i3: 1KG, Remaining 0, Profit: 3+5+8+6+0.33
i2: 0KG, Remaining 0, Profit: 3+5+8+6+0.33+0
Solution: [6, 0, 1, 5, 1, 3], Profit=22.33

Greedy approach will not give all possible optimal solution.[n logn]
If I want to find all possible solutions. [2^n]
Is it possible to get all possible solution in less than 2^n?

Page 9 of 10
i 1 2 3 4 5 6 7 8 9 10 11

Sj 1 5 0 3 6 5 3 12 8 2 8

fj 4 7 6 5 10 9 9 16 12 14 11

i 1 4 3 2 7 6 5 11 9 10 8

Sj 1 3 0 5 3 5 6 8 8 2 12

fj 4 5 6 7 9 9 10 11 12 14 16

Select:
Activity 1, Activity 4, Activity 3, Activity 2, Activity 7, Activity 6, Activity 5, Activity 11,
Activity 9, Activity 10, Activity 8

Page 10 of 10