L12: Analysis of Algorithms

 Reading: Rosen 3.1, 3.2, 3.3

Page 1
 Revisiting the Selection Sort Algorithm

 (1) for i = 1 to n - 1
 (2) for j = i + 1 to n
 (3) if ( A[i] > A[j] )
 (4) swap A[i] and A[j]
 (5) endif
 (6) endfor
 (7) endfor
An algorithm is a finite set of precise
instructions for performing a
computation or for solving a problem.

2
 How to Measure the Running Time?
 The real running time when executed on a computer?
 The number of times a particular line is executed (as
a function of input size 𝑛𝑛)?
 We counted line (3). Answer: 𝑛𝑛(𝑛𝑛 − 1)/2
 The total number of lines executed?
 The total number of machine instructions, but…
 Ignore lower order terms
 Ignore constant coefficients
 A constant is any quantity that doesn’t depend
on 𝑛𝑛.
 E.g., the running time of selection sort is Θ(𝑛𝑛2 )

3
 The Growth of Functions

 Which algorithm is better for large 𝑛𝑛?

 For Algorithm 1, 𝑇𝑇1 𝑛𝑛 = 3𝑛𝑛3 + 6𝑛𝑛2 − 4𝑛𝑛 + 17 = Θ(𝑛𝑛3 )
 For Algorithm 2, 𝑇𝑇2 𝑛𝑛 = 7𝑛𝑛2 − 8𝑛𝑛 + 20 = Θ(𝑛𝑛2 )
 Clearly, Algorithm 2 is better

4
 Big-Theta
 Definition: Let 𝑓𝑓 and 𝑔𝑔 be functions from the set of
positive real numbers to the set of positive real
numbers. We say that 𝑓𝑓(𝑥𝑥) is Θ(𝑔𝑔) if there are
constants 𝐶𝐶1 , 𝐶𝐶2 , and 𝑘𝑘 such that
𝐶𝐶1 𝑔𝑔 𝑥𝑥 ≤ 𝑓𝑓 𝑥𝑥 ≤ 𝐶𝐶2 𝑔𝑔 𝑥𝑥
whenever 𝑥𝑥 > 𝑘𝑘.
 This is read as “𝑓𝑓(𝑥𝑥) is big-Theta of 𝑔𝑔(𝑥𝑥)” or “𝑓𝑓(𝑥𝑥) is
asymptotically the same as 𝑔𝑔(𝑥𝑥).”
 Usually written as 𝑓𝑓 𝑛𝑛 = Θ 𝑔𝑔 𝑛𝑛 , although the more
mathematically correct way should be 𝑓𝑓 𝑛𝑛 ∈ Θ 𝑔𝑔 𝑛𝑛 .
 The constants 𝐶𝐶1 , 𝐶𝐶2 and 𝑘𝑘 are called witnesses to the
relationship. Only one pair of witnesses is needed.

5
 Using Definition to Derive Big-Theta

𝑇𝑇1 𝑛𝑛 = 3𝑛𝑛3 + 6𝑛𝑛2 − 4𝑛𝑛 + 17 = Θ(𝑛𝑛3 )

 Choose 𝐶𝐶1 = 2, 𝐶𝐶2 = 4

 Want a 𝑘𝑘 such that, when 𝑛𝑛 > 𝑘𝑘
2𝑛𝑛3 ≤ 3𝑛𝑛3 + 6𝑛𝑛2 − 4𝑛𝑛 + 17 ≤ 4𝑛𝑛3
−𝑛𝑛3 ≤ 6𝑛𝑛2 − 4𝑛𝑛 + 17 ≤ 𝑛𝑛3
 It’s clear that such a 𝑘𝑘 must exist, there is no need to
actually find it.

6
 Comparison of Algorithms
 𝑛𝑛 is big (big data!), so we are interested in
𝑇𝑇1 (𝑛𝑛)
lim
𝑛𝑛→∞ 𝑇𝑇2 (𝑛𝑛)

 Three cases:
𝑇𝑇1 (𝑛𝑛)
 lim = 0: Algorithm 1 is better
𝑛𝑛→∞ 𝑇𝑇2 (𝑛𝑛)
𝑇𝑇1 (𝑛𝑛)
 lim = ∞: Algorithm 2 is better
𝑛𝑛→∞ 𝑇𝑇2 (𝑛𝑛)
𝑇𝑇1 (𝑛𝑛) 𝑇𝑇1 (𝑛𝑛)
 lim = 𝐶𝐶 for some constant 0 < 𝐶𝐶 < ∞, or
𝑛𝑛→∞ 𝑇𝑇2 (𝑛𝑛) 𝑇𝑇2 (𝑛𝑛)
oscillates: Θ-notation cannot tell, need more careful
analysis.
 If 𝑇𝑇1 𝑛𝑛 = Θ 𝑔𝑔1 𝑛𝑛 , 𝑇𝑇2 𝑛𝑛 = Θ 𝑔𝑔2 𝑛𝑛 , it’s sufficient to
𝑔𝑔1 (𝑛𝑛)
consider lim
𝑛𝑛→∞ 𝑔𝑔2 (𝑛𝑛) 7
 Examples
log2 𝑛𝑛
 log10 𝑛𝑛 = = Θ(log 2 𝑛𝑛) = Θ(log 𝑛𝑛)
log2 10
9999
 99999999 =Θ 1
 210𝑛𝑛 is not Θ 2𝑛𝑛 , 3𝑛𝑛 is not Θ 2𝑛𝑛
 ∑𝑛𝑛𝑖𝑖=1 𝑖𝑖 2 ≤ 𝑛𝑛2 ⋅ 𝑛𝑛 ≤ 𝑛𝑛3
𝑛𝑛 2 𝑛𝑛 2 𝑛𝑛 1
∑𝑖𝑖=1 𝑖𝑖 ≥ ⋅ ≥ 𝑛𝑛3
2 2 8
So, ∑𝑛𝑛𝑖𝑖=1 𝑖𝑖 2 = Θ 𝑛𝑛3

Θ 𝑐𝑐 𝑛𝑛 , 𝑐𝑐 > 1
𝑛𝑛 𝑖𝑖 𝑐𝑐 𝑛𝑛 −1
 ∑𝑖𝑖=0 𝑐𝑐 = = � Θ 𝑛𝑛 , 𝑐𝑐 = 1 (geometric series)
𝑐𝑐−1
Θ 1 , 𝑐𝑐 < 1

8
 Solving Geometric Series

 Geometric series
𝑆𝑆(𝑛𝑛) = 1 + 𝑐𝑐 + 𝑐𝑐 2 + 𝑐𝑐 3 + ⋯ + 𝑐𝑐 𝑛𝑛
 Solving geometric series
𝑆𝑆 𝑛𝑛 + 1 = 𝑆𝑆 𝑛𝑛 ⋅ 𝑐𝑐 + 1
𝑆𝑆 𝑛𝑛 + 1 = 𝑆𝑆 𝑛𝑛 + 𝑐𝑐 𝑛𝑛+1
𝑆𝑆 𝑛𝑛 ⋅ 𝑐𝑐 + 1 = 𝑆𝑆 𝑛𝑛 + 𝑐𝑐 𝑛𝑛+1
𝑐𝑐 − 1 𝑆𝑆 𝑛𝑛 = 𝑐𝑐 𝑛𝑛+1 − 1
 If 𝑐𝑐 ≠ 1,
𝑐𝑐 𝑛𝑛+1 − 1
𝑆𝑆 𝑛𝑛 =
𝑐𝑐 − 1

9
 Examples

 log 𝑛𝑛! = log 𝑛𝑛 + log 𝑛𝑛 − 1 + ⋯ + log 1 ≤ 𝑛𝑛 log 𝑛𝑛

𝑛𝑛 𝑛𝑛 𝑛𝑛
log 𝑛𝑛! ≥ log 𝑛𝑛 + log 𝑛𝑛 − 1 + ⋯ + log ≥ log
2 2 2
So, log 𝑛𝑛! = Θ(𝑛𝑛 log 𝑛𝑛).
1
 ∑𝑛𝑛𝑖𝑖=1 = Θ log 𝑛𝑛 (harmonic series, derivation on board)
𝑖𝑖
 𝑐𝑐1𝑛𝑛 dominates 𝑛𝑛𝑐𝑐2 , which dominates log 𝑐𝑐3 𝑛𝑛,
for 𝑐𝑐1 > 1, 𝑐𝑐2 > 0, 𝑐𝑐3 > 0
 𝑛𝑛0.1 + log10 𝑛𝑛 = Θ 𝑛𝑛0.1
 1.1𝑛𝑛 + 𝑛𝑛100 = Θ(1.1𝑛𝑛 )

10
 Limitation of Big-Theta

 Some functions cannot be described by Θ

 Example:
the number of 1’s in the binary representation of 𝑛𝑛
 Oscillates between 1 and log 𝑛𝑛
 Some functions are hard to describe by Θ
 Example: 𝑛𝑛!
𝑛𝑛 𝑛𝑛
 It is known that 𝑛𝑛! = Θ 𝑛𝑛 (Stirling’s formula)
𝑒𝑒
but it’s very difficult to derive
 When used for analysis of algorithms (later)

11
 Big-Oh
 Definition: Let 𝑓𝑓 and 𝑔𝑔 be functions from the set of
positive real numbers to the set of positive real
numbers. We say that 𝑓𝑓(𝑥𝑥) is O(𝑔𝑔) if there are
constants 𝐶𝐶2 , and 𝑘𝑘 such that
𝑓𝑓 𝑥𝑥 ≤ 𝐶𝐶2 𝑔𝑔 𝑥𝑥
whenever 𝑥𝑥 > 𝑘𝑘.
 This is read as “𝑓𝑓(𝑥𝑥) is big-Oh of 𝑔𝑔(𝑥𝑥)” or “𝑓𝑓(𝑥𝑥) is
asymptotically dominated by 𝑔𝑔(𝑥𝑥).”
 Usually written as 𝑓𝑓 𝑛𝑛 = 𝑂𝑂 𝑔𝑔 𝑛𝑛 , although the more
mathematically correct way should be 𝑓𝑓 𝑛𝑛 ∈ 𝑂𝑂 𝑓𝑓 𝑛𝑛 .
 The constants 𝐶𝐶2 and 𝑘𝑘 are called witnesses to the
relationship. Only one pair of witnesses is needed.
12
 Big-Omega
 Definition: Let 𝑓𝑓 and 𝑔𝑔 be functions from the set of
positive real numbers to the set of positive real
numbers. We say that 𝑓𝑓(𝑥𝑥) is Ω(𝑔𝑔) if there are
constants 𝐶𝐶1 , and 𝑘𝑘 such that
𝐶𝐶1 𝑔𝑔 𝑥𝑥 ≤ 𝑓𝑓 𝑥𝑥
whenever 𝑥𝑥 > 𝑘𝑘.
 This is read as “𝑓𝑓(𝑥𝑥) is big-Omega of 𝑔𝑔(𝑥𝑥)” or “𝑓𝑓(𝑥𝑥)
asymptotically dominates 𝑔𝑔(𝑥𝑥).”
 Usually written as 𝑓𝑓 𝑛𝑛 = Ω 𝑔𝑔 𝑛𝑛 , although the more
mathematically correct way should be 𝑓𝑓 𝑛𝑛 ∈ Ω 𝑓𝑓 𝑛𝑛 .
 The constants 𝐶𝐶1 and 𝑘𝑘 are called witnesses to the
relationship. Only one pair of witnesses is needed.
 𝑓𝑓 𝑥𝑥 = Θ 𝑔𝑔 𝑥𝑥 iff 𝑓𝑓 𝑥𝑥 = O 𝑔𝑔 𝑥𝑥 and 𝑓𝑓 𝑥𝑥 = Ω 𝑔𝑔 𝑥𝑥
13
 Examples:

 𝑓𝑓 𝑛𝑛 = 32𝑛𝑛2 + 17𝑛𝑛 − 32.

 𝑓𝑓(𝑛𝑛) is 𝑂𝑂(𝑛𝑛2), 𝑂𝑂(𝑛𝑛3), Ω(𝑛𝑛2), Ω(𝑛𝑛), and Θ(𝑛𝑛2).
 𝑓𝑓(𝑛𝑛) is not 𝑂𝑂(𝑛𝑛), Ω(𝑛𝑛3), Θ(𝑛𝑛), or Θ(𝑛𝑛3).
 The number of 1’s in the binary representation of 𝑛𝑛 is
 𝑂𝑂(log 𝑛𝑛)
 Ω(1)
 𝑛𝑛!
 𝑛𝑛! ≤ 𝑛𝑛𝑛𝑛 = 𝑂𝑂(𝑛𝑛𝑛𝑛 )
𝑛𝑛 𝑛𝑛
𝑛𝑛 2 𝑛𝑛 2
 𝑛𝑛! ≥ =Ω
2 2

14
Insertion Sort
Insertion-Sort(𝐴𝐴):
for 𝑗𝑗 ← 2 to 𝑛𝑛 do
𝑘𝑘𝑘𝑘𝑘𝑘 ← 𝐴𝐴[𝑗𝑗]
𝑖𝑖 ← 𝑗𝑗 − 1
while 𝑖𝑖 ≥ 1 and 𝐴𝐴[𝑖𝑖] > 𝑘𝑘𝑘𝑘𝑘𝑘 do
𝐴𝐴[𝑖𝑖 + 1] ← 𝐴𝐴[𝑖𝑖]
𝑖𝑖 ← 𝑖𝑖 − 1
endwhile
𝐴𝐴 𝑖𝑖 + 1 ← 𝑘𝑘𝑘𝑘𝑘𝑘
endfor

sorted key unsorted

15
 Insertion Sort: Example
 1st iteration:
 (41825 )→(44825)→(14825)
 key = 1
 2nd iteration:
 (14825 )
 key = 8
 3rd iteration:
 (14825 )→(14885)→(14485)→(12485)
 key = 2
 4th iteration:
 (12485 )→(12488)→(12458)
 key = 5

16
 Analysis of Algorithms
 Question: What’s the running time of insertion sort if
the input array is already sorted?
 Answer: Θ 𝑛𝑛 .
 Question: What’s the running time of insertion sort if
the input array is inversely sorted?
 Answer: Θ 𝑛𝑛2 .
 Observation: The running time of an algorithm doesn’t
only depend on 𝑛𝑛, it also depends on the actual input!
 Question: Which input should we use for analyzing an
algorithm?
 Best-case? Average-case? Worst-case?

17
 Worst-case Analysis

 The default of algorithm analysis

 Especially easy when using big-Oh
 You don’t really have to find the worst-case input!
 Can easily conclude that insertion sort runs in time
𝑂𝑂 𝑛𝑛2 .
 How to show an algorithm has worst-case running
time Θ 𝑛𝑛2 ?
 Show that it’s 𝑂𝑂(𝑛𝑛2 )
 Show that it’s Ω(𝑛𝑛2 )
 Find an input such that the running time is Ω(𝑛𝑛2 )

18
 Example: Linear Search
 Problem: Given an array 𝐴𝐴 of unordered elements and
𝑥𝑥, find 𝑥𝑥 or report that 𝑥𝑥 doesn’t exist in 𝐴𝐴
 Algorithm:
procedure linear search(x:integer,
a1, a2, …,an: distinct integers)
i := 1
while (i ≤ n and x ≠ ai)
i := i + 1
if i ≤ n then location := i
else location := 0
return location

19
 Analysis of Linear Search

 Running time is 𝑂𝑂(𝑛𝑛)

 Obvious, since the loop has at most 𝑛𝑛 iterations.
 (Worst-case) running time is Ω(𝑛𝑛)
 When 𝑥𝑥 doesn’t exist in 𝐴𝐴, the loop has exactly 𝑛𝑛
iterations.
 Running time is Θ(𝑛𝑛)

20
 Example: Binary Search
 Problem: Given an array 𝐴𝐴 of ordered elements and 𝑥𝑥,
find 𝑥𝑥, or report that 𝑥𝑥 doesn’t exist in 𝐴𝐴
 Algorithm:
procedure binary search(x: integer, a1,a2,…, an: increasing
integers)
i := 1 {i is the left endpoint of interval}
j := n {j is right endpoint of interval}
while i < j
m := ⌊(i + j)/2⌋
if x > am then i := m + 1
else j := m
if x = ai then location := i
else location := 0
return location
21
 Analysis of Binary Search
 Assumption: 𝑛𝑛 = 2𝑘𝑘
 Running time is 𝑂𝑂 𝑘𝑘 = 𝑂𝑂(log 𝑛𝑛)
 The length of the range 𝑗𝑗 − 𝑖𝑖 + 1 decrease by half in
each iteration
 The algorithm terminates when 𝑖𝑖 ≥ 𝑗𝑗, i.e.,
𝑗𝑗 − 𝑖𝑖 + 1 ≤ 1
 (Worst-case) running time is Ω(log 𝑛𝑛)
 When 𝑥𝑥 doesn’t exist in 𝐴𝐴, the loop has exactly log 𝑛𝑛
iterations.
 Running time is Θ(log 𝑛𝑛)
 What if 𝑛𝑛 is not in the form of 2𝑘𝑘 ?
 Find 𝑘𝑘 such that 2𝑘𝑘 < 𝑛𝑛 < 2𝑘𝑘+1 . The running time
must be between Θ 𝑘𝑘 and Θ 𝑘𝑘 + 1 , which is Θ(𝑘𝑘).
22