Analysis of Algorithms

Instructor: Dr. Ahmad Qawasmeh

Note: Slides are written by Dr. Monica Nicolescu at University of Nevada, Reno
 Algorithm Analysis
• The amount of resources used by the algorithm
– Space
– Computational time
• Running time:
– The number of primitive operations (steps) executed
before termination
• Order of growth
– The leading term of a formula
– Expresses the behavior of a function toward infinity

2
 Asymptotic Notations
• A way to describe behavior of functions in the limit

– How we indicate running times of algorithms

– Describe the running time of an algorithm as n grows to ∞

• O notation: asymptotic “less than”: f(n) “≤” g(n)

• Ω notation: asymptotic “greater than”: f(n) “≥” g(n)

• Θ notation: asymptotic “equality”: f(n) “=” g(n)

3
 Logarithms
• In algorithm analysis we often use the notation “log n”
without specifying the base

Binary logarithm lg n = log2 n logk n = (log n ) k

Natural logarithm ln n = loge n log log n = log(log n )
log x y = y log x
log xy = log x + log y
x
log = log x − log y
y
loga x = loga b logb x

a logb x = x logb a 4
 Asymptotic Notations - Examples
• For each of the following pairs of functions, either f(n) is
O(g(n)), f(n) is Ω(g(n)), or f(n) is Θ(g(n)). Determine
which relationship is correct.
– f(n) = log n2; g(n) = log n + 5 f(n) = Θ (g(n))
– f(n) = n; g(n) = log n2 f(n) = Ω(g(n))
– f(n) = log log n; g(n) = log n f(n) = O(g(n))
– f(n) = n; g(n) = log2 n f(n) = Ω(g(n))
– f(n) = n log n + n; g(n) = log n f(n) = Ω(g(n))
– f(n) = 10; g(n) = log 10 f(n) = Θ(g(n))
– f(n) = 2n; g(n) = 10n2 f(n) = Ω(g(n))
– f(n) = 2n; g(n) = 3n f(n) = O(g(n))
5
 Asymptotic notations
• O-notation

• Intuitively: O(g(n)) = the set of

functions with a smaller or
same order of growth as g(n)

6
 Examples

– 2n2 = O(n3): 2n2 ≤ cn3 ⇒ 2 ≤ cn ⇒ c = 1 and n0= 2

– n2 = O(n2): n2 ≤ cn2 ⇒ c ≥ 1 ⇒ c = 1 and n0= 1

– 1000n2+1000n = O(n2):

1000n2+1000n ≤ 1000n2+ 1000n2 = 2000n2

⇒ c=2000 and n0 = 1

– n = O(n2): n ≤ cn2 ⇒ cn ≥ 1 ⇒ c = 1 and n0= 1

7
 Asymptotic notations (cont.)
• Ω - notation

• Intuitively: Ω(g(n)) = the set of

functions with a larger or same
order of growth as g(n)

8
 Examples
– 5n2 = Ω(n)
∃ c, n0 such that: 0 ≤ cn ≤ 5n2 ⇒ cn ≤ 5n2 ⇒ c = 1 and n0 = 1

– 100n + 5 ≠ Ω(n2)
∃ c, n0 such that: 0 ≤ cn2 ≤ 100n + 5
100n + 5 ≤ 100n + 5n (∀ n ≥ 1) = 105n
cn2 ≤ 105n ⇒ n(cn – 105) ≤ 0
Since n is positive ⇒ cn – 105 ≤ 0 ⇒ n ≤ 105/c
⇒ contradiction: n cannot be smaller than a constant
– n = Ω(2n), n3 = Ω(n2), n = Ω(logn)
9
 Asymptotic notations (cont.)
• Θ-notation

• Intuitively Θ(g(n)) = the set of

functions with the same order of
growth as g(n)

10
 Examples
– n2/2 –n/2 = Θ(n2)

• ½ n2 - ½ n ≤ ½ n2 ∀n ≥ 0 ⇒ c2= ½

• ½ n2 - ½ n ≥ ½ n2 - ½ n * ½ n ( ∀n ≥ 2 ) = ¼ n2

⇒ c1= ¼

– n ≠ Θ(n2): c1 n2 ≤ n ≤ c2 n2

⇒ only holds for: n ≤ 1/c1

11
 Examples
– 6n3 ≠ Θ(n2): c1 n2 ≤ 6n3 ≤ c2 n2

⇒ only holds for: n ≤ c2 /6

– n ≠ Θ(logn): c1 logn ≤ n ≤ c2 logn

⇒ c2 ≥ n/logn, ∀ n≥ n0 – impossible

12
 More on Asymptotic Notations
• There is no unique set of values for n0 and c in proving the
asymptotic bounds

• Prove that 100n + 5 = O(n2)

– 100n + 5 ≤ 100n + n = 101n ≤ 101n2

for all n ≥ 5

n0 = 5 and c = 101 is a solution

– 100n + 5 ≤ 100n + 5n = 105n ≤ 105n2

for all n ≥ 1

n0 = 1 and c = 105 is also a solution

Must find SOME constants c and n0 that satisfy the asymptotic notation relation
13
 Comparisons of Functions
• Theorem:
f(n) = Θ(g(n)) ⇔ f = O(g(n)) and f = Ω(g(n))
• Transitivity:
– f(n) = Θ(g(n)) and g(n) = Θ(h(n)) ⇒ f(n) = Θ(h(n))
– Same for O and Ω
• Reflexivity:
– f(n) = Θ(f(n))
– Same for O and Ω
• Symmetry:
– f(n) = Θ(g(n)) if and only if g(n) = Θ(f(n))
• Transpose symmetry:
– f(n) = O(g(n)) if and only if g(n) = Ω(f(n))
14
Asymptotic Notations in Equations
• On the right-hand side
– Θ(n2) stands for some anonymous function in Θ(n2)
2n2 + 3n + 1 = 2n2 + Θ(n) means:
There exists a function f(n) ∈ Θ(n) such that
2n2 + 3n + 1 = 2n2 + f(n)
• On the left-hand side
2n2 + Θ(n) = Θ(n2)
No matter how the anonymous function is chosen on
the left-hand side, there is a way to choose the
anonymous function on the right-hand side to make
the equation valid.

15
Some Simple Summation Formulas
n
n( n + 1)
• Arithmetic series: ∑ k = 1 + 2 + ... + n =
k =1 2
n
x n +1 − 1
• Geometric series: ∑
k =0
x = 1 + x + x + ... + x =
k 2 n

x −1
(x ≠ 1)
∞
1
– Special case: x < 1: ∑ =
x k

k =0 1− x
n
1 1 1
• Harmonic series: ∑
k =1 k
= 1 +
2
+ ... +
n
≈ ln n

n
• Other important formulas: ∑ lg k ≈ n lg n
k =1

n
1
∑ k p = 1p + 2 p + ... + n p ≈
k =1 p +1
n p +1

16
 Mathematical Induction
• Used to prove a sequence of statements (S(1),

S(2), … S(n)) indexed by positive integers

• Proof:

– Basis step: prove that the statement is true for n = 1

– Inductive step: assume that S(n) is true and prove that

S(n+1) is true for all n ≥ 1

• Find case n “within” case n+1

17
 Example
• Prove that: 2n + 1 ≤ 2n for all n ≥ 3
• Basis step:
– n = 3: 2 ∗ 3 + 1 ≤ 23 ⇔ 7 ≤ 8 TRUE
• Inductive step:
– Assume inequality is true for n, and prove it for (n+1)
Assume: 2n + 1 ≤ 2n
Must prove: 2(n + 1) + 1 ≤ 2n+1
2(n + 1) + 1 = (2n + 1 ) + 2 ≤ 2n + 2 ≤
≤ 2n + 2n = 2n+1, since 2 ≤ 2n for n ≥ 1
18
More Examples

19
Readings

• Chapter 3
• Apendix A

20