Algorithms and Data Structures

CS 372

Asymptotic notations

(Based on slides by M. Nicolescu)

Algorithm Analysis
• The amount of resources used by the algorithm
– Space
– Computational time
• Running time:
– The number of primitive operations (steps) executed
before termination
• Order of growth
– The leading term of a formula
– Expresses the behavior of a function toward infinity

Asymptotic Notations

• A way to describe behavior of functions in the limit

– How we indicate running times of algorithms

– Describe the running time of an algorithm as n grows to ∞

• O notation: asymptotic “less than”: f(n) “≤” g(n)

• Ω notation: asymptotic “greater than”: f(n) “≥” g(n)

• Θ notation: asymptotic “equality”: f(n) “=” g(n)

1
 Logarithms
• In algorithm analysis we often use the notation “log n”
without specifying the base

Binary logarithm lg n = log2 n log x y = y log x

Natural logarithm ln n = loge n log xy = log x + log y
x
lg n = (lg n )
k k
log = log x − log y
y
lg lg n = lg(lg n )
loga x = loga b logb x

a logb x = x logb a

Asymptotic notations
• O-notation

• Intuitively: O(g(n)) = the set of

functions with a smaller or
same order of growth as g(n)

Examples

– 2n2 = O(n3): 2n2 ≤ cn3 ⇒ 2 ≤ cn ⇒ c = 1 and n0= 2

– n2 = O(n2): n2 ≤ cn2 ⇒ c ≥ 1 ⇒ c = 1 and n0= 1

– 1000n2+1000n = O(n2):

1000n2+1000n ≤ 1000n2+1000n2=2000n2 ⇒ c=2000 and n0 = 1

– n = O(n2): n ≤ cn2 ⇒ cn ≥ 1 ⇒ c = 1 and n0= 1

2
 Asymptotic notations (cont.)
• Ω - notation

• Intuitively: Ω(g(n)) = the set of

functions with a larger or same
order of growth as g(n)

Examples
– 5n2 = Ω(n)
∃ c, n0 such that: 0 ≤ cn ≤ 5n2 ⇒ cn ≤ 5n2 ⇒ c = 1 and n0 = 1

– 100n + 5 ≠ Ω(n2)
∃ c, n0 such that: 0 ≤ cn2 ≤ 100n + 5
100n + 5 ≤ 100n + 5n (∀ n ≥ 1) = 105n
cn2 ≤ 105n ⇒ n(cn – 105) ≤ 0
Since n is positive ⇒ cn – 105 ≤ 0 ⇒ n ≤ 105/c
⇒ contradiction: n cannot be smaller than a constant
– n = Ω(2n), n3 = Ω(n2), n = Ω(logn)
8

Asymptotic notations (cont.)

• Θ-notation

• Intuitively Θ(g(n)) = the set of

functions with the same order of
growth as g(n)

3
 Examples
– n2/2 –n/2 = Θ(n2)

• ½ n2 - ½ n ≤ ½ n2 ∀n ≥ 0 ⇒ c2= ½

• ½ n2 -½n≥½ n2 - ½ n * ½ n ( ∀n ≥ 2 ) = ¼ n2 ⇒ c1= ¼

– n ≠ Θ(n2): c1 n2 ≤ n ≤ c2 n2 ⇒ only holds for: n ≤ 1/c1

– 6n3 ≠ Θ(n2): c1 n2 ≤ 6n3 ≤ c2 n2 ⇒ only holds for: n ≤ c2 /6

– n ≠ Θ(logn): c1 logn ≤ n ≤ c2 logn

⇒ c2 ≥ n/logn, ∀ n≥ n0 – impossible

10

More on Asymptotic Notations

• There is no unique set of values for n0 and c in proving the
asymptotic bounds

• Prove that 100n + 5 = O(n2)

– 100n + 5 ≤ 100n + n = 101n ≤ 101n2

for all n ≥ 5

n0 = 5 and c = 101 is a solution

– 100n + 5 ≤ 100n + 5n = 105n ≤ 105n2

for all n ≥ 1

n0 = 1 and c = 105 is also a solution

Must find SOME constants c and n0 that satisfy the asymptotic notation relation
11

Comparisons of Functions
• Theorem:
f(n) = Θ(g(n)) ⇔ f = O(g(n)) and f = Ω(g(n))
• Transitivity:
– f(n) = Θ(g(n)) and g(n) = Θ(h(n)) ⇒ f(n) = Θ(h(n))
– Same for O and Ω
• Reflexivity:
– f(n) = Θ(f(n))
– Same for O and Ω
• Symmetry:
– f(n) = Θ(g(n)) if and only if g(n) = Θ(f(n))
• Transpose symmetry:
– f(n) = O(g(n)) if and only if g(n) = Ω(f(n))
12

4
Asymptotic Notations in Equations
• On the right-hand side
– Θ(n2) stands for some anonymous function in Θ(n2)
2n2 + 3n + 1 = 2n2 + Θ(n) means:
There exists a function f(n) ∈ Θ(n) such that
2n2 + 3n + 1 = 2n2 + f(n)
• On the left-hand side
2n2 + Θ(n) = Θ(n2)
No matter how the anonymous function is chosen on
the left-hand side, there is a way to choose the
anonymous function on the right-hand side to make
the equation valid.

13

Practical Complexity

250

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

14

Practical Complexity

500

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

15

5
 Practical Complexity

1000

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 3 5 7 9 11 13 15 17 19

16

Practical Complexity

5000

4000
f(n) = n
f(n) = log(n)
3000
f(n) = n log(n)
f(n) = n^2
2000 f(n) = n^3
f(n) = 2^n

1000

0
1 3 5 7 9 11 13 15 17 19

17

Asymptotic Notations - Examples

• For each of the following pairs of functions, either f(n) is
O(g(n)), f(n) is Ω(g(n)), or f(n) = Θ(g(n)). Determine
which relationship holds.
– f(n) = log n2; g(n) = log n + 5 f(n) = Θ (g(n))
– f(n) = n; g(n) = log n2 f(n) = Ω(g(n))
– f(n) = log log n; g(n) = log n f(n) = O(g(n))
– f(n) = n; g(n) = log2 n f(n) = Ω(g(n))
– f(n) = n log n + n; g(n) = log n f(n) = Ω(g(n))
– f(n) = 10; g(n) = log 10 f(n) = Θ(g(n))
– f(n) = 2n; g(n) = 10n2 f(n) = Ω(g(n))
– f(n) = 2n; g(n) = 3n f(n) = O(g(n))
18

6
 Other Asymptotic Notations
• A function f(n) is o(g(n)) if for any positive
constant c, there exists n0 such that
f(n) < c g(n) ∀ n ≥ n0

or lim (f(n)/g(n)) = 0
nض

19

Other Asymptotic Notations

• A function f(n) is ω(g(n)) if for any positive

constant c, there exists n0 such that
c g(n) < f(n) ∀ n ≥ n0

or lim (f(n)/g(n)) = ¶
nض

20

Intuitions
• Intuitively,

– o() is like < – ω() is like > – Θ() is like =

– O() is like ≤ – Ω() is like ≥

21

7
 Typical Running Time Functions
• Θ (1) (constant running time):
– Instructions are executed once or a few times
• Θ (logN) (logarithmic)
– A big problem is solved by cutting the original problem in smaller sizes,
by a constant fraction at each step
• Θ (N) (linear)
– A small amount of processing is done on each input element
• Θ (N logN)
– A problem is solved by dividing it into smaller problems, solving them
independently and combining the solution

22

Typical Running Time Functions

• Θ (N2 ) (quadratic)
– Typical for algorithms that process all pairs of data items (double nested
loops)

• Θ (N3 ) (cubic)
– Processing of triples of data (triple nested loops)

• Θ (NK ) (polynomial)

• Θ (2N ) (exponential)
– Few exponential algorithms are appropriate for practical use

23

Some Simple Summation Formulas

n
n( n + 1)
• Arithmetic series: ∑ k = 1 + 2 + ... + n =
k =1 2
n
x n +1 − 1
• Geometric series: ∑x
k =0
k
= 1 + x + x 2 + ... + x n =
x −1
(x ≠ 1)
∞
1
– Special case: x < 1: ∑x
k =0
k
=
1− x
n
1 1 1
• Harmonic series: ∑ k = 1 + 2 + ... + n
k =1
≈ ln n

• Other important formulas: ∑ lg k = lg(k!) ≈ n lg n

k =1

n
1
∑k
k =1
p
= 1p + 2 p + ... + n p ≈
p +1
n p +1

24