CS 253: Algorithms

Chapter 3

Growth of Functions

Credit: Dr. George Bebis

Analysis of Algorithms
Goal:
 To analyze and compare algorithms in terms of running time and memory
requirements (i.e. time and space complexity)

 In other words, how does the running time and space requirements change
as we increase the input size n ?

(sometimes we are also interested in the coding complexity)

 Input size (number of elements in the input)

◦ size of an array or a matrix

◦ # of bits in the binary representation of the input

◦ vertices and/or edges in a graph, etc.

2
Types of Analysis
 Worst case
◦ Provides an upper bound on running time
◦ An absolute guarantee that the algorithm would not run
longer, no matter what the inputs are

 Best case
◦ Provides a lower bound on running time
◦ Input is the one for which the algorithm runs the fastest

 Average case
◦ Provides a prediction about the running time
◦ Assumes that the input is random

Lower Bound ≤ Running Time ≤ Upper Bound

3
Computing the Running Time
 Measure the execution time ?
Not a good idea ! It varies for different
microprocessors!

 Count the number of statements executed?

Yes, but you need to be very careful!
High-level programming languages have statements which
require a large number of low-level machine language
instructions to execute (a function of the input size n).
For example, a subroutine call can not be counted as one
statement; it needs to be analyzed separately

 Associate a "cost" with each statement.

Find the "total cost“ by multiplying the cost with the total
number of times each statement is executed. 

(we have seen examples before)

4
Example
Algorithm X Cost

  sum = 0; c1
for(i=0; i<N; i++) c2
for(j=0; j<N; j++) c3
sum += arrY[i][j]; c4
------------

Total Cost = c1 + c2 * (N+1) + c3 * N *

(N+1) + c4 * N2

5
Asymptotic Analysis
 To compare two algorithms with running times f(n) and g(n), we need a
rough measure that characterizes how fast each function grows with
respect to n

 In other words, we are interested in how they behave asymptotically (i.e. for
large n) (called rate of growth)

Big O notation: asymptotic “less than” or “at most”:

f(n)=O(g(n)) implies: f(n) “≤” g(n)

 notation: asymptotic “greater than” or “at least”:

f(n)=  (g(n)) implies: f(n) “≥” g(n)

 notation: asymptotic “equality” or “exactly”:

f(n)=  (g(n)) implies: f(n) “=” g(n)

6
 Big-O Notation
 We say
fA(n) = 7n+18 is order n, or O (n)

It is, at most, roughly proportional to

n.

fB(n)
fB(n) = 3n2+5n +4 is order n2, or
O(n2). fA(n)

Function value 
It is, at most, roughly proportional to
n2 .

 Ingeneral, any O(n2) function is

faster- growing than any O(n)
function.

Increasing n 
7
More Examples …
 n4 + 100n2 + 10n + 50  O(n4)
10n3 + 2n2  O(n3)
n3 - n2  O(n3)

 constants
10 is O(1)
1273 is O(1)

what is the rate of growth for Algorithm X studied earlier (in Big O
notation)?

Total Time = c1 + c2*(N+1) + c2 * N*(N+1) + c3*N2

If c1, c2, c3 , and c4 are constants then Total Time =

O(N2)
8
Definition of Big O
 O-notation
Big-O example, graphically
Note that 30n+8 is O(n).
cn =31n
Can you find a c and n0
which can be used in the
formal definition of Big
O? 30n+8

Function value 
Youcan easily see that
30n+8 isn’t less than n
anywhere (n>0).
n
But it is less than
31n everywhere to
the right of n=8.

So, one possible (c , n0) n0=8

pair that can be used in
n
the formal definition:

c = 31, n0 = 8
30n+8  O(n)
Big-O Visualization
O(g(n)) is the set of
functions with
smaller or same
order of growth as
g(n)

11
No Uniqueness
 There is no unique set of values for n0 and c in proving
the asymptotic bounds

 Prove that 100n + 5 = O(n2)

(i) 100n + 5 ≤ 100n + n = 101n ≤ 101n 2 for all n ≥ 5

You may pick n0 = 5 and c = 101 to complete the proof.

(ii) 100n + 5 ≤ 100n + 5n = 105n ≤ 105n 2 for all n ≥ 1

You may pick n0 = 1 and c = 105 to complete the proof.

12
Definition of 

(g(n)) is the set of

functions with larger or
same order of growth as
g(n)
Examples
◦ 5n2 = (n)

 c, n0 such that: 0  cn  5n2  cn  5n2  c = 1 and n >

n0=1

◦ 100n + 5 ≠ (n2)

 c, n0 such that: 0  cn2  100n + 5

since 100n + 5  100n + 5n n1

cn2  105n  n(cn – 105)  0

Since n is positive  (cn – 105)  0  n  105/c

 contradiction: n cannot be smaller than a constant

Definition of 

-notation

(g(n)) is the set of

functions with the
same order of
growth as g(n)

15
Examples
◦ n2/2 –n/2 = (n2)
 ½ n2 - ½ n ≤ ½ n 2 n ≥ 0  c 2= ½

 ¼ n2 ≤ ½ n 2 - ½ n n ≥ 2  c 1= ¼

◦ n ≠ (n2): c1 n2 ≤ n ≤ c2 n2  only holds for: n ≤ 1/c1

◦ 6n3 ≠ (n2): c1 n2 ≤ 6n3 ≤ c2 n2

 only holds for: n ≤ c2 /6

◦ n ≠ (logn): c1 logn ≤ n ≤ c2 logn

 c2 ≥ n/logn,  n≥ n0 – impossible
Relations Between Different Sets

 Subset relations between order-of-growth sets.

RR
O( f ) ( f )
•f
( f )

17
Common orders of
magnitude

18
Common orders of magnitude

19
 Logarithms and properties
 In algorithm analysis we often use the
notation “log n” without specifying the base

Binary logarithm lg n  log 2 n log x y  y log x

Natural logarithm ln n  log e n log xy  log x  log y
x
log  log x  log y
y
lg k n  (lg n ) k a logb x  x logb a
lg lg n  lg(lg n ) log a x
log b x 
log a b
More Examples
 Foreach of the following pairs of functions, either f(n) is
O(g(n)), f(n) is Ω(g(n)), or f(n) = Θ(g(n)). Determine which
relationship is correct.

f(n) = (g(n))
◦ f(n) = log n2; g(n) = log n + 5 f(n) = (g(n))
◦ f(n) = n; g(n) = log n2 f(n) = O(g(n))
◦ f(n) = log log n; g(n) = log n f(n) = (g(n))
◦ f(n) = n; g(n) = log2 n f(n) = (g(n))
◦ f(n) = n log n + n; g(n) = log n f(n) = (g(n))
◦ f(n) = 10; g(n) = log 10 f(n) = (g(n))
◦ f(n) = 2n; g(n) = 10n2 f(n) = O(g(n))
◦ f(n) = 2n; g(n) = 3n
Properties
Theorem:
f(n) = (g(n))  f = O(g(n)) and f = (g(n))

 Transitivity:

◦ f(n) = (g(n)) and g(n) = (h(n))  f(n) = (h(n))

◦ Same for O and 
 Reflexivity:

◦ f(n) = (f(n))
◦ Same for O and 
 Symmetry:

◦ f(n) = (g(n)) if and only if g(n) = (f(n))

 Transpose symmetry:
◦ f(n) = O(g(n)) if and only if g(n) = (f(n))

22