Growth of Algorithms

Dr. Aimal Tariq Rextin

Adv Algo
 Example of Algorithm
A sequence of numbers

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Adv Algo
 Example of Algorithm
A sequence of numbers

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

The well-known Fibonacci sequence

Leonardo da Pisa (ca. 1170–ca. 1250)

son of Guglielmo “Bonaccio”
a.k.a. Leonardo Fibonacci

Adv Algo
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Adv Algo
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
Scheme
(define (F n)
(cond
((= n 0) 0)
((= n 1) 1)
(else (+ (F (- n 1)) (F (- n 2))))))

Adv Algo
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
Java
public class Fibonacci {
public static int F(int n) {
if (n == 0) {
return 0;
} else if (n == 1) {
return 1;
} else {
return F(n-1) + F(n-2);
} }
}

Adv Algo
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
C or C++
int F(int n) {
if (n == 0) {
return 0;
} else if (n == 1) {
return 1;
} else {
return F(n-1) + F(n-2);
}
}

Adv Algo
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
Ruby
def F(n)
case n
when 0
return 0
when 1
return 1
else
return F(n-1) + F(n-2)
end
end
Adv Algo
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
Python
def F(n):
if n == 0:
return 0
elif n == 1:
return 1
else:
return F(n-1) + F(n-2)

Adv Algo
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:

very concise C/C++ (or Java)

int F(int n) { return (n<2)?n:F(n-1)+F(n-2); }

Adv Algo
 The Fibonacci Sequence

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
“pseudo-code”
Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

Adv Algo
How The Algorithm Works?

Adv Algo
 Questions on Our First Algorithm

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

Adv Algo
 Questions on Our First Algorithm

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

1. Is the algorithm correct?

▶ for every valid input, does it terminate?
▶ if so, does it do the right thing?

Adv Algo
 Questions on Our First Algorithm

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

1. Is the algorithm correct?

▶ for every valid input, does it terminate?
▶ if so, does it do the right thing?

2. How much time does it take to complete?

Adv Algo
 Questions on Our First Algorithm

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

1. Is the algorithm correct?

▶ for every valid input, does it terminate?
▶ if so, does it do the right thing?

2. How much time does it take to complete?

3. Can we do better?
Adv Algo
Correctness

Adv Algo
 Correctness

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)


0
 if n = 0
Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Adv Algo
 Correctness

Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)


0
 if n = 0
Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


The algorithm is clearly correct

▶ assuming n ≥ 0
Adv Algo
 Performance
How long does it take?

Adv Algo
 Performance
How long does it take?

Let’s try it out. . .

Adv Algo
 Results

60
Ruby
Java
running time (seconds)

Python
40 C
C-wiz
C-gcc

20

0
20 25 30 35 40 45 50
n

Adv Algo
Comments

Adv Algo
 Comments
Different implementations perform differently
▶ simple language tricks don’t seem to pay off

Adv Algo
 Comments
Different implementations perform differently
▶ simple language tricks don’t seem to pay off

However, the differences do not seem to be substantial

▶ all implementations sooner or later seem to hit a wall. . .

Adv Algo
 Comments
Different implementations perform differently
▶ simple language tricks don’t seem to pay off

However, the differences do not seem to be substantial

▶ all implementations sooner or later seem to hit a wall. . .

Conclusion: the problem probably is with the algorithm

Adv Algo
Is the Problem The algorithm

Adv Algo
 Is the Problem The algorithm
we need a implementation independent way to measure time of
algorithms.

Adv Algo
 Is the Problem The algorithm
we need a implementation independent way to measure time of
algorithms.
Lets use Maths

Adv Algo
 Complexity of Our First Algorithm
We need a mathematical characterization of the performance of the
algorithm

We’ll call it the algorithm’s computational complexity

Adv Algo
 Complexity of Our First Algorithm
We need a mathematical characterization of the performance of the
algorithm

We’ll call it the algorithm’s computational complexity

Let T (n) be the number of basic steps needed to compute

Fibonacci(n)

Adv Algo
 Complexity of Our First Algorithm
We need a mathematical characterization of the performance of the
algorithm

We’ll call it the algorithm’s computational complexity

Let T (n) be the number of basic steps needed to compute

Fibonacci(n)
Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

Adv Algo
 Complexity of Our First Algorithm
We need a mathematical characterization of the performance of the
algorithm

We’ll call it the algorithm’s computational complexity

Let T (n) be the number of basic steps needed to compute

Fibonacci(n)
Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

T (0) = 2; T (1) = 3

Adv Algo
 Complexity of Our First Algorithm
We need a mathematical characterization of the performance of the
algorithm

We’ll call it the algorithm’s computational complexity

Let T (n) be the number of basic steps needed to compute

Fibonacci(n)
Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

T (0) = 2; T (1) = 3
T (n) = T (n − 1) + T (n − 2) + 4
Adv Algo
Complexity of Our First Algorithm (2-1)
Starting Point:
T (n) = T (n − 1) + T (n − 2) + 4 (Equation 1)
This form does not immediately reveal the complexity.

Adv Algo
Complexity of Our First Algorithm (2-1)
Starting Point:
T (n) = T (n − 1) + T (n − 2) + 4 (Equation 1)
This form does not immediately reveal the complexity.
A Simplifying Assumption:
T (n − 1) ≈ T (n − 2)

Adv Algo
Complexity of Our First Algorithm (2-1)
Starting Point:
T (n) = T (n − 1) + T (n − 2) + 4 (Equation 1)
This form does not immediately reveal the complexity.
A Simplifying Assumption:
T (n − 1) ≈ T (n − 2)

Then:
T (n) ≈ T (n − 1) + T (n − 1) + 4 = 2 T (n − 1) + 4.
Let c = 4. Hence,
T (n) = 2 T (n − 1) + c (Equation 2)

Adv Algo
Complexity of Our First Algorithm (2-1)
Starting Point:
T (n) = T (n − 1) + T (n − 2) + 4 (Equation 1)
This form does not immediately reveal the complexity.
A Simplifying Assumption:
T (n − 1) ≈ T (n − 2)

Then:
T (n) ≈ T (n − 1) + T (n − 1) + 4 = 2 T (n − 1) + 4.
Let c = 4. Hence,
T (n) = 2 T (n − 1) + c (Equation 2)
Expanding Further:
 
T (n) = 2 T (n−1)+c = 2 T (n−2)+T (n−3)+c +c = 4 T (n−2)+3c.
(3)
Adv Algo
Complexity of Our First Algorithm (2-2)
Continuing the Process:
 
T (n) = 4 T (n−2)+3c = 4 T (n−3)+T (n−4) +3c = 8 T (n−3)+7c.
(4)

Adv Algo
Complexity of Our First Algorithm (2-2)
Continuing the Process:
 
T (n) = 4 T (n−2)+3c = 4 T (n−3)+T (n−4) +3c = 8 T (n−3)+7c.
(4)

General Pattern:
T (n) = 2k T (n − k) + 2k − 1 c


(Equation 5)

Adv Algo
Complexity of Our First Algorithm (2-2)
Continuing the Process:
 
T (n) = 4 T (n−2)+3c = 4 T (n−3)+T (n−4) +3c = 8 T (n−3)+7c.
(4)

General Pattern:
T (n) = 2k T (n − k) + 2k − 1 c


(Equation 5)

Using T (0) = 2 :
If we let n − k = 0, then k = n. Thus,
T (n) = 2n T (0) + 2n − 1 c = 2n · 2 + (2n − 1)c.

Therefore,
T (n) ∝ 2n .

Adv Algo
Complexity of Our First Algorithm (2-2)
Continuing the Process:
 
T (n) = 4 T (n−2)+3c = 4 T (n−3)+T (n−4) +3c = 8 T (n−3)+7c.
(4)

General Pattern:
T (n) = 2k T (n − k) + 2k − 1 c


(Equation 5)

Using T (0) = 2 :
If we let n − k = 0, then k = n. Thus,
T (n) = 2n T (0) + 2n − 1 c = 2n · 2 + (2n − 1)c.

Therefore,
T (n) ∝ 2n .

Question: Can we do better?

Adv Algo
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Adv Algo
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Idea: we can build Fn from the ground up!

Adv Algo
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Idea: we can build Fn from the ground up!

SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else pprev = 0
6 prev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f
Adv Algo
 Results

60
Ruby
Scheme
running time (seconds)

Python
40 C
C-wiz
Java
C-gcc
(Python) SmartFibonacci
20

0
20 40 60 80 100 120 140 160 180 200
n

Adv Algo
 Review :The Fibonacci Sequence
The sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Adv Algo
 Review :The Fibonacci Sequence
The sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
“pseudo-code”
Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

Adv Algo
 Running Time
T (n) ∝ 2n i.e Running time is directly proportional an exponential
function
60
Ruby
Scheme
running time (seconds)

Python
40 C
C-wiz
Java
C-gcc
20

0
20 40 60 80 100 120 140 160 180 200
n Adv Algo
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Adv Algo
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Idea: we can build Fn from the ground up!

Adv Algo
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Idea: we can build Fn from the ground up!

SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else pprev = 0
6 prev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f
Adv Algo
 Results

60
Ruby
Scheme
running time (seconds)

Python
40 C
C-wiz
Java
C-gcc
(Python) SmartFibonacci
20

0
20 40 60 80 100 120 140 160 180 200
n

Adv Algo
Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) =

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = 6 + 6(n − 1)

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = 6 + 6(n − 1) = 6n

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = 6 + 6(n − 1) = 6n

The complexity of SmartFibonacci(n) is linear in n

Adv Algo
Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) =

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = c1 + c2 n

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = c1 + c2 n hence T (n) ∝ n

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = c1 + c2 n hence T (n) ∝ n

The complexity of SmartFibonacci(n) is linear in n

Adv Algo
 Results

60
Ruby
Scheme
running time (seconds)

Python
40 C
C-wiz
Java
C-gcc
(Python) SmartFibonacci
20

0
20 40 60 80 100 120 140 160 180 200
n

Adv Algo
 Slow vs. Fast Fibonacci
We informally characterized our two Fibonacci algorithms

Adv Algo
 Slow vs. Fast Fibonacci
We informally characterized our two Fibonacci algorithms

▶ Fibonacci is exponential in n

▶ SmartFibonacci is (almost) linear in n

Adv Algo
 Slow vs. Fast Fibonacci
We informally characterized our two Fibonacci algorithms

▶ Fibonacci is exponential in n

▶ SmartFibonacci is (almost) linear in n

How do we characterize the complexity of algorithms?

▶ in general

Adv Algo
 Slow vs. Fast Fibonacci
We informally characterized our two Fibonacci algorithms

▶ Fibonacci is exponential in n

▶ SmartFibonacci is (almost) linear in n

How do we characterize the complexity of algorithms?

▶ in general

▶ in a way that is specific to the algorithms

▶ but independent of implementation details

Adv Algo
 Order of Growth
We care about the order of growth or rate of growth of T (n).

Adv Algo
 Order of Growth
We care about the order of growth or rate of growth of T (n).
Ignoring lower-order terms:

T (n) = an2 + bn + c =⇒ T (n) ∼ n2 .

Adv Algo
 Order of Growth
We care about the order of growth or rate of growth of T (n).
Ignoring lower-order terms:

T (n) = an2 + bn + c =⇒ T (n) ∼ n2 .

Notation:
T (n) = Θ(n2 )
This is read as “T (n) is theta of n-squared”.

Adv Algo
 Assignment 0—Weight 10%
Median
How would you find the median for this input output pair:
Input Sorted array of numbers with size N
the median number

Adv Algo
 Assignment 0—Weight 10%
Median
How would you find the median for this input output pair:
Input Sorted array of numbers with size N
the median number

Adv Algo
 Assignment 0—Weight 10%
Median
How would you find the median for this input output pair:
Input Sorted array of numbers with size N
the median number

the median is A[⌈ N2 ⌉] if N is odd

and median is average of A[⌈ N2 ⌉] and A[⌈ N2 + 1⌉] if N is even

Adv Algo
 Assignment 0—Weight 10%
Median
How would you find the median for this input output pair:
Input Sorted array of numbers with size N
the median number

the median is A[⌈ N2 ⌉] if N is odd

and median is average of A[⌈ N2 ⌉] and A[⌈ N2 + 1⌉] if N is even
Constant time

Adv Algo