Complexity of Algorithms

(The growth of functions)

Chapter 3.2
Dr. Maria Tamoor

1 05/06/2024
 Complexity

In general, we are not so much interested in the time

and space complexity for small inputs.

For example, while the difference in time complexity

between linear and binary search is meaningless for a
sequence with n = 10, it is huge value for n = 230.

2 05/06/2024
 Complexity

For example, let us assume two algorithms A and B that

solve the same class of problems.
The time complexity of A is 5,000n, the one for B is
1.1n for an input with n elements.
For n = 10,
A requires 50,000 steps, but B only 3, so B seems to be
superior to A.
For n = 1000,
 however, A requires 5,000,000 steps, while B requires
2.51041 steps.
3 05/06/2024
 Complexity
Comparison: time complexity of algorithms A and B

Input Size Algorithm A Algorithm B

n 5,000n I_1.1n_I

10 50,000 3

100 500,000 13,781

1,000 5,000,000 2.5*1041

1,000,000 5*109 4.8*1041392

4 05/06/2024
 Complexity

This means that algorithm B cannot be used for large inputs,

while algorithm A is still feasible.

So what is important is the growth of the complexity

functions.
 What really matters in comparing the complexity of algorithms?

We only care about the behavior for large problems.

Even bad algorithms can be used to solve small problems.

The growth of time and space complexity with increasing input

size n is a suitable measure for the comparison of algorithms.

5 05/06/2024
 The Growth of Functions
The growth of functions is usually described using the
big-O notation.

Definition: Let f and g be functions from the integers or

the real numbers to the real numbers.
We say that f(x) is O(g(x)) if there are constants C and k
such that
|f(x)|  C|g(x)|
whenever x > k. This is read as “f (x) is big-oh of g(x).”

6 05/06/2024
7 05/06/2024
 Explanation
 Big O is also known as the upper bound of the function
 For example let f(x)=5x2+5
We have to look for g(X) such that it satisfies this condition
 |f(x)|  c|g(x)|
5x2+5 c|g(x)|
Where x>k
Estimate smallest possible g(x) with some c and k
Is this true 5x2+5 cx2 ??? YES if we use c= 6
It will become 5x2+5 6x2 but is this true if x=1? NO
If x=2, 25 24 NO
If x=3, 50 54 YES so for any x which will be greater 3, 5x 2+5
6x2 will be True
So with c=6 and k=2 we have got 5x 2+5 Cx2
8 05/06/2024
 Note: To establish that f (x) is O(g(x)) we need only
one pair of witnesses C and k that satisfies the
inequality |f (x)| ≤ C|g(x)| if x > k. However, this pair
C and k that meets the definition is never unique. If
such a pair exists, there exists infinity others.
To see this, note that if C and k are one pair of
witnesses, then any pair C´and k´, whereC < C´ and
k< k´ , is also a pair of witnesses, because |f (x)| ≤ C|
g(x)| ≤ C´ |g(x)| whenever x > k´ > k.

9 05/06/2024
 Notational Issues
Big-O notation is a way of comparing functions. Notation
unconventional:
EG: 3x 3 + 5x 2 – 9 = O (x 3)
Doesn’t mean
“3x 3 + 5x 2 – 9 equals the function O (x 3)”
Which actually means
“3x 3+5x 2 –9 is dominated by x 3”
Read as: “3x 3+5x 2 –9 is big-Oh of x 3”

10 L8
 Intuitive Notion of Big-O
Asymptotic notation captures behavior of functions for
large values of x.
EG: Dominant term of 3x 3+5x 2 –9 is x 3.
As x becomes larger and larger, other terms become
insignificant and only x 3 remains in the picture:

11 L8
 Let f(n)=3n+2 and g(n)=n
We have to check is f(n) is O(g(n))
If its true then it should be in the form
f(n)<=c(g(n))
3n+2<=cn
3n+2<=4n
C=4
And n>=2

12 05/06/2024
 The Growth of Functions

Example:
Show that f(x) = x2 + 2x + 1 is O(x2).

For x > 1 we have:

x2 + 2x + 1  x2 + 2x2 + x2
 x2 + 2x + 1  4x2

Therefore, for C = 4 and k = 1:

f(x)  Cx2 whenever x > k.

 f(x) is O(x2).

13 05/06/2024
14 05/06/2024
 Big-O Notation
Show that f(x) = 7x2 is O(x3).
First try x>7
so, let C = 1 and k = 7 as witnesses.

Could try x > 1. Then we have 7x2 < 7x3

so, C = 7 and k = 1 are also witnesses to f(x)
being O(x3). Note that f(x) is also O(x4), etc.

15
 Big-O Notation
To prove that a function f(x) is O(g(x))
Find values for k and C, not generally the smallest one,
larger values also work!!
It is sufficient to find a certain k and C that works

16
 Big-O Notation
Show that f(n) = n2 is or isn’t O(n).

Show that no pair of C and k exists such that

n2 ≤ Cn whenever n > k.

When n > 0, divide both sides of n2 ≤ Cn by n to get

n ≤ C. No matter what C and k are, n ≤ C will not
hold for all n with n > k.

17
 The Growth of Functions

Question: If f(x) is O(x2), is it also O(x3)?

Yes. x3 grows faster than x2, so x3 grows also faster than

f(x).

Therefore, we always have to find the smallest simple

function g(x) for which f(x) is O(g(x)).

18 05/06/2024
 We have seen in previous example which shows that
7x2 is O(x3). Is it also true that x3 is O(7x2)?

19 05/06/2024
 The Growth of Functions
“Popular” functions g(n) are
n log n, 1, 2n, n2, n!, n, n3, log n

Listed from slowest to fastest growth:

1
log n
n
n log n
n2
n3
2n
n!

20 05/06/2024
 Practice Questions
Function: f(x)=5x2+3+7
Question: What is the Big O complexity of f(x)?
Function: g(n)=2n+n3
Question: Determine the Big O complexity of g(n).
Function: h(n)=log(n)+n
Question: What is the Big O complexity of h(n)?
Function: j(n)=n! (factorial)
Question: Determine the Big O complexity of j(n).
Function: k(n)=n​+2n
Question: Find the Big O complexity of k(n).

21 05/06/2024
 Useful Rules for Big-O
For any polynomial f(x) = a xn + a xn-1 + … + a , where
n n-1 0
a0, a1, …, an are real numbers,
f(x) is O(xn).

If f1(x) is O(g1(x)) and f2(x) is O(g2(x)), then

(f1 + f2)(x) is O(max(g1(x), g2(x)))

If f1(x) is O(g(x)) and f2(x) is O(g(x)), then

(f1 + f2)(x) is O(g(x)).

If f1(x) is O(g1(x)) and f2(x) is O(g2(x)), then

(f1f2)(x) is O(g1(x) g2(x)).
22 05/06/2024
 Show that n < 2n whenever n is a positive integer.
Show that this inequality implies that n is O(2n), and
use this inequality to show that log n is O(n).
taking logarithms (base 2) of both sides of this
inequality shows that
logn < n.
It follows that
log n is O(n).

23 05/06/2024
 Give big-O estimates for the factorial function and the logarithm of the factorial
function, where
the factorial function f (n) = n! is defined by
n! = 1 ・ 2 ・ 3 ・ ・ ・ ・ ・ n
whenever n is a positive integer, and 0! = 1. For example,
1! = 1, 2! = 1 ・ 2 = 2, 3! = 1 ・ 2 ・ 3 = 6, 4! = 1 ・ 2 ・ 3 ・ 4 = 24.
Note that the function n! grows rapidly. For instance,
20! = 2,432,902,008,176,640,000.
A big-O estimate for n! can be obtained by noting that each term in the product does
not exceed n. Hence,
n! = 1 ・ 2 ・ 3 ・ ・ ・ ・ ・ n
≤n ・ n ・ n ・・・・・ n
= nn.
Taking logarithms of both sides of the inequality established for n!, we obtain log n!
≤ log nn = n log n.

24 05/06/2024
 f (x) = (x + 1) log(x2 + 1) + 3x2.
First, a big-O estimate for (x + 1) log(x2 + 1) will be
found. Note that (x + 1) is O(x). Furthermore, x2 + 1 ≤
2x2 when x > 1. Hence,
Log(x2+1)=log(2x2)=2log(x)
it follows that (x + 1) log(x2 + 1) is O(x log x).
Because 3x2 is O(x2),
We know that f(x) is O(max(x log x, x2)).
Because x log x ≤ x2, for x > 1, it follows
that f (x) is O(x2).

25 05/06/2024
 Give a big-O estimate for f (n) = 3n log(n!) + (n 2 + 3) log
n, where n is a positive integer.
From previous example we know that log(n!) is O(n log
n).
Using this estimate and the fact that 3n is O(n), Theorem
3 gives the estimate that 3n log(n!) is O(n2 log n).
n2 + 3 is O(n2).
Thus, from Theorem 3 it follows that (n2 + 3) log n is
O(n2 log n).
It shows that f (n) = 3n log(n!) + (n2 + 3) log n is O(n2
log n).
26 05/06/2024
 Find Big O
(n log n + n 2 )(n 3 + 2)
= n 4 log n + n 5 + 2n log n + 2n 2
Biggest term is n 5 so the function is O(n 5 ).

27 05/06/2024
 At each step two comparisons are made; i ≤ n and x  ai .
To end the loop, one comparison i ≤ n is made.

28 05/06/2024
 Big-Omega and Big-Theta Notation
Big O has limitations like it gives upper bound but
doesn’t give lower bound for the size of f (x) for large
x, for this, we use big-Omega (big-Ω) notation.
When we want to give both an upper and a lower
bound on the size of a function f(x), relative to a
reference function g(x),we use big-Theta (big-ϴ)
notation.

29 05/06/2024
 Big Omega
Big Omega T(N) is ( F(N) ) if there are positive
constants c and N0 such that
T(N) > cF( N )) when N > N0
Big O is similar to less than or equal, an upper bound.
Big Omega is similar to greater than or equal, a lower
bound.

30 05/06/2024
 Big theta
C1|g(x)| ≤ |f (x)| ≤ C2|g(x)|
Whenever x > k. The existence of the constants C1,
C2, and k tells us that f (x) is Ω(g(x)) and
that f (x) is O(g(x)), respectively.

31 05/06/2024
Relative Rates of Growth
Analysis Mathematical Relative
Type Expression Rates of
Growth
Big O T(N) = O( F(N) ) T(N) < F(N)

Big  T(N) = ( F(N) ) T(N) > F(N)

Big  T(N) = ( F(N) ) T(N) = F(N)

32
 Big Summary
Upper Bound – Use Big-Oh

Lower Bound – Use Big-Omega

Upper and Lower (or Order of Growth) –

Use Big-Theta

33