CSE102

Algorithmic Foundations
And Problem Solving
Algorithm efficiency: asymptotic analysis

Dr Yushi Li
Department of Computer Science
 Learning outcomes
Understand the notations of asymptotic analysis.
Able to carry out simple asymptotic analysis of
algorithms

2
 Time Complexity Analysis
How fast is the algorithm?
Code the algorithm and run the program,
then measure the running time

1. Depend on the speed of the computer

2. Waste time coding and testing if the
algorithm is slow

Identify some important operations/steps

and count how many times these
operations/steps needed to executed
3
 Time Complexity Analysis
How to measure efficiency?
Number of operations usually expressed in
terms of input size

 If we doubled the input size, how much

longer would the algorithm take?
If we trebled the input size, how much
longer would it take?

4
Finding the minimum...
 Finding min among 3 numbers
Input: 3 numbers a, b, c
Output: the min value of these 3 numbers
Algorithm: Input a, b, c

Y ab? N

Y ac? Y bc? N

Output a N Output b Output c

Output c 6
 Finding min among 3 numbers
Example 1
Input a, b, c
a=3, b=5, c=2
Y N
ab?

Y ac? Y bc? N

Output a N Output b Output c

Output c
2 is output
7
 Finding min among 3 numbers
Example 2
Input a, b, c
a=6, b=5, c=7
Y N
ab?

Y ac? Y bc? N

Output a N Output b Output c

Output c
5 is output
8
 Pseudo Code
Important operation: input a, b, c
if (a  b) then
comparison if (a  c) then
output a
How many comparison this else
output c
algorithm requires? else
if (b  c) then
output b
else
output c

It takes 2 comparisons

9
 Finding min among n numbers
Input a0, a1, ...
Any
Y a0  a1 ? N systematic
approach?
Y a0  a2 ? Y a1  a2 ?
N N
a0  a3 ? a1  a3 ?

a2  a3 ? a2  a3 ?

10
Finding min among n numbers
Input a[0], a[1], ..., a[n-1]

min = 0
i=1
i = i+1
i < n? N

Y
Output a[min]
Y
min = i a[i] < a[min]?
N
11
 Finding min among n numbers
input a[0], a[1], ..., a[n-1]
min = 0
i = 1
while (i < n) do Example
begin
if (a[i] < a[min]) then a[]={50,30,40,20,10}
min = i
i = i + 1 iteration min a[min]
end before 0 50
output a[min]
1 1 30
2 1 30
3 3 20
4 412 10
 Finding min among n numbers
 Time complexity: ?? comparisons n-1

input a[0], a[1], ..., a[n-1]

min = 0
i = 1
while (i < n) do
begin
if (a[i] < a[min]) then
min = i
i = i + 1
end
output a[min]

13
 Finding min using for-loop
Rewrite the above while-loop into a for-loop
input a[0], a[1], ..., a[n-1]
min = 0
for i = 1 to n-1 do
begin
if (a[i] < a[min]) then
min = i
end
output a[min]

14
 Why efficiency matters?
speed of computation by hardware has been
improved
efficiency still matters
ambition for computer applications grow with
computer power
demand a great increase in speed of computation

15
 Amount of data handled matches
speed increase?
When computation speed vastly increased, can we
handle much more data?
Suppose
• an algorithm takes n2 comparisons to sort n numbers
• we need 1 sec to sort 5 numbers (25 comparisons)
• computing speed increases by factor of 100
Using 1 sec, we can now perform 100x25 comparisons,
i.e., to sort 50 numbers
With 100 times speedup, only sort 10 times more
numbers!
16
 Time complexity
- Big O notation …
 Which algorithm is the fastest?
Consider a problem that can be solved by 5 algorithms
A1, A2, A3, A4, A5 using different number of operations
(time complexity).
f1(n) = 50n + 20 f2(n) = 10 n log2 n + 100
f3(n) = n2 - 3n + 6 f4(n) = 2n2
f5(n) = 2n/8 - n/4 + 2
n 1 2 4 8 16 32 64 128 256 512 1024 2048
f1(n) = 50n + 20 70 120 220 420 820 1620 3220 6420 12820 25620 51220 102420
f2(n) = 10 n log2n + 100 100 120 180 340 740 1700 3940 9060 20580 46180 102500 225380
f3(n) = n2 - 3n + 6 4 4 10 46 214 934 3910 16006 64774 3E+05 1E+06 4E+06
2
f4(n) = 2n 2 8 32 128 512 2048 8192 32768 131072 5E+05 2E+06 8E+06
f5(n) = 2n/8 - n/4 + 2 2 2 3 32 8190 5E+08 2E+18

f5(n) f3(n) f1(n)

Depends on the size of the problem! 18

 1000000

100000

10000
Time

1000

ff1(n)
1(n) == 50n
50n ++ 20
20
100 ff2(n)
2(n) == 10
10 nn log2n
log2n ++ 100
100
f3(n) = n2 - 3n + 6
f3(n) = n2 - 3n + 6
10 f4(n) = 2n2
f4(n) = 2n2
ff5(n) = 2n/8 - n/4 + 2
5(n) = 2 /8 - n/4 + 2
n

1
1 2 4 8 16 32 64 128 256 512 1024 2048 4096

19
 What do we observe?
There is huge difference between
– functions involving powers of n (e.g., n, n log n, n2,
called polynomial functions) and
– functions involving powering by n (e.g., 2n, called
exponential functions)
Among polynomial functions, those with same
order of power are more comparable
– e.g., f3(n) = n2 - 3n + 6 and f4(n) = 2n2

20
Growth of functions

21
Relative growth rate
2n
n2

log n
c
n
22
 Hierarchy of functions
We can define a hierarchy of functions each having a
greater order of magnitude than its predecessor:

1 log n n n2 n3 ... nk ... 2n

constant logarithmic polynomial exponential

 We can further refine the hierarchy by

inserting n log n between n and n2,
n2 log n between n2 and n3, and so on.
23
 Hierarchy of functions (2)
1 log n n n2 n3 ... nk ... 2n

constant logarithmic polynomial exponential

Important: as we move from left to right, successive

functions have greater order of magnitude than the
previous ones.
As n increases, the values of the later functions increase
more rapidly than the values of the earlier ones.
 Relative growth rates increase

24
 Hierarchy of functions (3)
1 log n n n2 n3 ... nk ... 2n

constant logarithmic polynomial exponential

Now, when we have a function, we can assign the

function to some function in the hierarchy:
– For example, f(n) = 2n3 + 5n2 + 4n + 7
The term with the highest power is 2n3.
The growth rate of f(n) is dominated by n3.
This concept is captured by Big-O notation

25
 Big-O notation
f(n) = O(g(n)) [read as f(n) is of order g(n)]
Roughly speaking, this means f(n) is at most a
constant times g(n) for all large n
Examples
– 2n3 = O(n3)
– 3n2 = O(n2)
– 2n log n = O(n log n)
– n3 + n2 = O(n3)
function on L.H.S and function on R.H.S are said to
have the same order of magnitude

26
 Big-O notation - formal definition

f(n) = O(g(n)): There exists a constant c and no

such that f(n)  c g(n) for all n > no

27
 f(n) = O(g(n) – Graphical illustration
7000
constant c & no c g(n)
6000
such that
5000 n>no, f(n)  c g(n)
f(n)
4000
Time

3000

2000

1000

0
n0
100 300 500 700 900 1100

input size (n)

28
 400
Example – n+60 = O(n)
constant c & no
350
such that 2n
300 n>no, f(n)  c g(n)

250 n + 60
Time

200

150 n

100

50 c=2, n0=60

0
n0
10 30 50 70 90 110 130 150 170 190

input size (n)

29
 Example – 2nlog2n+100n+1000 = O(n log n)
70000
constant c & no
60000 such that 3 n log n
n>no, f(n)  c g(n)
50000

40000
f(n)
Time

30000
c=3, n0=600
n log n
20000

10000

0
n0
100 300 500 700 900 1100 1300 1500 1700 1900

input size (n)

30
 600
f(n) = O(g(n))?
constant c & no c g(n)
such that n>no,
500
f(n)  c g(n)

400 f(n)
Time

300

200

100 value of f(n) for n<n0

does NOT matter

0
10 30 n0 50 70 90 110

input size (n)

31
 Which one is the fastest?
• Usually we are only interested in the asymptotic
time complexity, i.e., when n is large

O(log n) < O(n) < O(n) < O(n log n) < O(n2) < O(2n)

32
 Proof of order of magnitude
 Show that 2n2 + 4n is O(n2)
 Since
n  n2 for all n>1,
we have 2n2 + 4n  2n2 + 4n2  6n2 for all n>1.
 Therefore, by definition, 2n2 + 4n is O(n2).

 Show that n3 + n2 log n + n is O(n3)

 Sincen  n3 and log n  n for all n>1,
we have n2 log n  n3, and
n3 + n2 log n + n  3n3 for all n>1.
 Therefore, by definition, n3 + n2 log n + n is
O(n3). 33
 Exercise
Determine the order of magnitude of the following
functions.
1. n3 + 3n2 + 3
2. 4n2 log n + n3 + 5n2 + n
3. 2n2 + n2 log n
4. 6n2 + 2n

34
 Exercise (2)
Prove the order of magnitude
1. n3 + 3n2 + 3

2. 4n2 log n + n3 + 5n2 + n

35
 Exercise (2) cont’d
Prove the order of magnitude
3. 2n2 + n2 log n

4. 6n2 + 2n

36
 More Exercise
Are the followings correct?
1. n2log n + n3 + 3n2 + 3 O(n2)?
2. n + 1000 O(n)?
3. 6n20 + 2n O(n20)?
4. n3 + 5n2 log n + n O(n2 log n) ?

37
 Some algorithms we learnt
input n
Computing sum of the first n sum = 0
for i = 1 to n do
numbers begin
sum = sum + i
input n end
sum = n*(n+1)/2 output sum
output sum
O(?) O(?)

Finding the min min = 0

for i = 1 to n-1 do
value among n if (a[i] < a[min]) then
min = i
numbers output a[min]
O(?)

38
 More exercise
What is the time complexity of the following pseudo
code?
for i = 1 to 2n do
for j = 1 to n do
x = x + 1
O(?)

39