Matrix Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16
Number of Bits in an Integer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Mathematical Analysis of Recursive Algorithms 18
Chapter 2: Fundamentals of the Analysis of Plan for Analyzing Recursive Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Factorial Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Algorithm Efficiency Recurrence for Factorial Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Solving the Factorial Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
The first rule of any technology used in a busi- Towers of Hanoi Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Recurrence for Towers of Hanoi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
ness is that automation applied to an efficient Solving the Towers of Hanoi Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . . 24
operation will magnify the efficiency. The sec- Number of Bits in an Integer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
ond is that automation applied to an inefficient Recurrence for BitCount Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Solving the BitCount Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
operation will magnify the inefficiency. (Bill
Fibonacci Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Gates) Recurrence for Fibonacci Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Approximating the Fibonacci Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Empirical Analysis of Algorithms 31
Plan for Empirical Analysis of Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . 31
Empirical Analysis of UniqueElements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Analysis Framework 2 Results of Empirical Analysis: Comparisons . . . . . . . . . . . . . . . . . . . . . . . . 33
Analysis of Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Results of Empirical Analysis: Timings. . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Efficiency as a Function of Input Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Algorithm Visualization 35
Measuring Running Time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Visualization of Sorting Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
worst/best/average. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Asymptotic Notation 6
Order of Growth. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Asymptotic Order of Growth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Illustration of Asymptotic Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Examples of Asymptotic Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Properties of Order of Growth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Basic Asymptotic Efficiency Classes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Mathematical Analysis of Nonrecursive Algorithms 12
Plan for Analyzing Nonrecursive Algorithms . . . . . . . . . . . . . . . . . . . . . . . . 12
Useful Summation Formulas and Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Finding the Maximum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Element Uniqueness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

1 2
Analysis Framework 2
Measuring Running Time
Analyze time efficiency by:
Analysis of Algorithms
 identifying the basic operation(s), the operation(s) contributing the most to
 Issues: running time,
– Correctness. Is the algorithm correct?  characterizing the number of times it is performed as a function of input size.
– Time Efficiency. How much time does the algorithm use? We can crudely estimate running time by T (n) ≈ cop ∗ C(n)
– Space Efficiency. How much extra space (in addition to the space needed
for the input and output) does the algorithm use?  T (n): running time as a function of n.
 cop : running time of a single basic operation.
 Approaches:
 C(n): number of basic operations as a function of n.
– Theoretical Analysis. Proof of correctness and big-Oh and related
CS 3343 Analysis of Algorithms Chapter 2: Slide – 4
notations.
– Empirical Analysis. Testing and measurement over a range of instances.
CS 3343 Analysis of Algorithms Chapter 2: Slide – 2
Worst-Case, Best-Case, and Average-Case
algorithm SequentialSearch(A[0..n − 1], K)
Efficiency as a Function of Input Size // Searches for a value in an array
// Input: An array A and a search key K
Typically, more time and space are needed to run an algorithm on bigger inputs // Output: The index where K is found or −1
(e.g., more numbers, longer strings, larger graphs). for i ← 0 to n − 1 do
if A[i] = K then return i
Analyze efficiency as a function of n=size of input.
return −1
 Searching/sorting. n=number of items in list.
 String processing. n = length of string(s).  Basic Operation: The comparison in the loop
 Matrix operations, n = dimension of matrix. n × n matrix has n2 elements.  Worst-Case: n comparisons
 Graph processing. nV = number of vertices and nE = number of edges.  Best-Case: 1 comparison
CS 3343 Analysis of Algorithms Chapter 2: Slide – 3  Average-Case: (n+1)/2 comparisons assuming each element equally likely to
be searched.
CS 3343 Analysis of Algorithms Chapter 2: Slide – 5

3 4
Asymptotic Notation 6
Illustration of Asymptotic Order
20
n + 3 sin n
Order of Growth 2n
15 n/2

functions on n
Typically, the basic operation count can be approximated as c g(n), where g(n) is
the order of growth, often some “simple” function such as: 10

0
0 5 10 15 20
n

Graph suggests (and we can prove) n + 3 sin n is:

O(n): n + 3 sin n ≤ 2n when n ≥ 3.
CS 3343 Analysis of Algorithms Chapter 2: Slide – 6 Ω(n): n + 3 sin n ≥ n/2 when n ≥ 6.
CS 3343 Analysis of Algorithms Chapter 2: Slide – 8

Asymptotic Order of Growth

Examples of Asymptotic Order
A way of comparing functions that ignores constant factors and small input sizes.
t(n) O(n) O(n2) O(n3 ) Ω(n)Ω(n2) Ω(n3)
 Big-Oh O(g(n)): functions ≤ c g(n). log2 n T T T F F F
t(n) ∈ O(g(n)) if there are positive constants c and n0 such that 10n + 5 T T T T F F
t(n) ≤ c g(n) for all n ≥ n0 n(n−1)/2 F T T T T F
(n + 1)3 F F T T T T
 Big-Omega Ω(g(n)): functions ≥ c g(n). 2n F F F T T T
t(n) ∈ Ω(g(n)) if there are positive constants c and n0 such that
t(n) ≥ c g(n) for all n ≥ n0 For example, 10n + 5 is Θ(n). Assuming n ≥ 5:
10n+5 ∈ O(n) because 10n+5 ≤ 10n+n = 11n.
 Big-Theta Θ(g(n)): functions ≈ c g(n). 10n + 5 ∈ Ω(n) because 10n + 5 ≥ 10n.
CS 3343 Analysis of Algorithms Chapter 2: Slide – 9
Both t(n) ∈ O(g(n)) and t(n) ∈ Ω(g(n)).
CS 3343 Analysis of Algorithms Chapter 2: Slide – 7

5 6
Properties of Order of Growth Mathematical Analysis of Nonrecursive Algorithms 12

 If f1(n) is order g1(n), and f2(n) is order g2(n), then f1 (n) + f2(n) is order
Plan for Analyzing Nonrecursive Algorithms
max(g1 (n), g2(n)).
 If b > 1, then logb n ∈ Θ(log n).  Decide on parameter n indicating input size.

 Polynomials of degree k ∈ Θ(nk ), that is, ak nk + ak−1nk−1 + · · · + a0 ∈ Θ(nk ).  Identify algorithm’s basic operation(s).

 Exponential functions an have different orders of growth for different a’s.  Determine worst, average, and best cases for input of size n.

 order log n < order nk where k > 0 < order an where a > 1 < order n! <  Set up a sum for the number of times the basic operation is executed.
order nn  Simplify the sum using standard formulas and rules (see Appendix A).
CS 3343 Analysis of Algorithms Chapter 2: Slide – 10 CS 3343 Analysis of Algorithms Chapter 2: Slide – 12

Basic Asymptotic Efficiency Classes Useful Summation Formulas and Rules

Class Name Example X
n
 1 = 1 + 1 + · · · + 1 = n ∈ Θ(n)
1 constant access array element
i=1
log n logarithmic binary search
Xn
n(n + 1)
n linear find median  i = 1 + 2 + ··· + n = ∈ Θ(n2 )
n log n “n-log-n” mergesort i=1
2
n2 quadratic insertion sort
X
n
nk+1
n3 cubic matrix multiplication  ik = 1 + 2 k + · · · + n k ≈ ∈ Θ(nk+1)
an exponential generating all subsets i=1
k+1
n! factorial generating all permutations X
n
an+1 − 1
CS 3343 Analysis of Algorithms Chapter 2: Slide – 11
 ai = 1 + a + · · · + an = ∈ Θ(an )
i=1
a−1
X
n X
n X
n X
n X
n
 (ai ± bi ) = ai ± bi c ai = c ai
i=1 i=1 i=1 i=1 i=1

CS 3343 Analysis of Algorithms Chapter 2: Slide – 13

7 8
Finding the Maximum Matrix Multiplication

algorithm MaxElement(A[0..n − 1]) algorithm MatrixMultiply(A, B : n × nmatrices)

// Returns the maximum value in an array // Multiplies matrix A times matrix B
// Input: A nonempty array A of real numbers // Input: Two n × n matrices A and B
// Output: The maximum value in A // Output: Matrix C = AB
maxval ← A[0] for i ← 0 to n − 1 do
for i ← 1 to n − 1 do for j ← 0 to n − 1 do
if A[i] > maxval then C[i, j] ← 0
maxval ← A[i] for k ← 0 to n − 1 do
return maxval C[i, j] ← C[i, j] + A[i, k] ∗ B[k, j]
return C
Basic Operation: comparison in loop
X
n−1
Basic Operation: inner loop multiplications
Performs 1 = n − 1 comparisons X n−1 X
n−1 X n−1

i=1 Performs 1 = n3 multiplications

CS 3343 Analysis of Algorithms Chapter 2: Slide – 14 i=0 j=0 k=0
CS 3343 Analysis of Algorithms Chapter 2: Slide – 16

Element Uniqueness
Number of Bits in an Integer
algorithm UniqueElements(A[0..n − 1])
// Returns true if all elements are different algorithm BitCount(m)
// Input: An array A // Input: A positive integer m
// Output: Returns true if there are no duplicates // Output: The number of bits to encode m
for i ← 0 to n − 2 do count ← 1
for j ← i + 1 to n − 1 do while m > 1 do
if A[i] = A[j] then return false count ← count + 1
return true m ← ⌊m/2⌋
return count
 Basic Operation: inner loop comparison n = length of m in bits
 Best/Worst-Case: from 1 to n(n − 1)/2 Basic Operation: The division by 2 in loop
X
n−2 X
n−1 Performs n − 1 operations because n = count.
n−(i+1) = (n−1)+(n−2)+· · ·+1 = k = n(n−1)/2 CS 3343 Analysis of Algorithms Chapter 2: Slide – 17
i=0 k=1
CS 3343 Analysis of Algorithms Chapter 2: Slide – 15

9 10
Mathematical Analysis of Recursive Algorithms 18
Solving the Factorial Recurrence
Make a reasonable guess.
Plan for Analyzing Recursive Algorithms
 Forward substitution. M(1) = M(0) + 1 = 1
 Decide on parameter n indicating input size.
M(2) = M(1) + 1 = 2
 Identify algorithm’s basic operation(s). M(3) = M(2) + 1 = 3
 Determine worst, average, and best cases for input of size n.  Backward substitution. M(n) = M(n − 1) + 1
= [M(n − 2) + 1] + 1 = M(n − 2) + 2
 Set up a recurrence relation expressing the basic operation count. = [M(n − 3) + 1] + 2 = M(n − 3) + 3
 Solve the recurrence (at least determine it’s order of growth). Prove M(n) = n by mathematical induction.
CS 3343 Analysis of Algorithms Chapter 2: Slide – 18
 Basis: if n = 0, then M(n) = M(0) = 0 = n
 Induction: if M(n − 1) = n − 1, then
Factorial Function M(n) = M(n − 1) + 1 = (n − 1) + 1 = n
CS 3343 Analysis of Algorithms Chapter 2: Slide – 21

algorithm Factorial(n)
// Computes n! recursively Towers of Hanoi Algorithm
// Input: A nonnegative integer n
// Output: The value of n! algorithm Towers(n, i, j)
if n = 0 then return 1 // Moves n disks from peg i to peg j
else return Factorial(n − 1) ∗ n // Input: Integers n > 0, 1 ≤ i, j ≤ 3, i 6= j
Input Size: Use number n (actually n has about log2 n bits) // Output: Specifies disk moves in correct order
Basic Operation: multiplication if n = 1 then
CS 3343 Analysis of Algorithms Chapter 2: Slide – 19 move disk 1 from peg i to peg j
else
Recurrence for Factorial Function Towers(n−1, i, 6−i−j)
move disk n from peg i to peg j
 Let M(n) = multiplication count to compute Factorial(n). Towers(n−1, 6−i−j, j)
 M(0) = 0 because no multiplications are performed to compute Factorial(0). Input Size: Use number n
 If n > 0, then Factorial(n) performs recursive call plus one multiplication. Basic Operation: moving a disk
CS 3343 Analysis of Algorithms Chapter 2: Slide – 22

M(n) = M(n − 1) + 1
to compute to multiply
Factorial(n − 1) Factorial(n − 1) by n

CS 3343 Analysis of Algorithms Chapter 2: Slide – 20

11 12
Recurrence for Towers of Hanoi Number of Bits in an Integer
 Let M(n) = move count to compute Towers(n, ·, ·).
algorithm BitCount(n)
 M(1) = 1 because 1 move is needed for Towers(1, ·, ·).
// Input: A positive integer n
 If n > 1, then Towers(n, ·, ·) performs 2 recursive calls plus one move. // Output: The number of bits to encode n
if m = 1 then return 1
M(n) = 2M(n − 1) + 1 else return BitCount(⌊n/2⌋) + 1
to compute to move
Towers(n−1, ·, ·) disk n Input Size: Use number n (actually n has about log2 n bits)
twice Basic Operation: division by 2
CS 3343 Analysis of Algorithms Chapter 2: Slide – 25
CS 3343 Analysis of Algorithms Chapter 2: Slide – 23

Recurrence for BitCount Function

Solving the Towers of Hanoi Recurrence
 Let D(n) = division count to compute BitCount(n).
Make a reasonable guess.
 D(1) = 0 because no divisions are performed to compute BitCount(1).
 Forward substitution. M(2) = 2M(1) + 1 = 3
 If n > 1, then BitCount(n) performs recursive call on ⌊n/2⌋ plus one division.
M(3) = 2M(2) + 1 = 7
M(4) = 2M(3) + 1 = 15 D(n) = D(⌊n/2⌋) + 1
 Backward substitution. M(n) = 2M(n−1) + 1 to compute to compute
= 2[2M(n − 2) + 1] + 1 = 4M(n − 2) + 3 BitCount(⌊n/2⌋) ⌊n/2⌋
= 4[2M(n − 3) + 1] + 2 = 8M(n − 3) + 7
Prove M(n) = 2n − 1 by mathematical induction. CS 3343 Analysis of Algorithms Chapter 2: Slide – 26

 Basis: if n = 1, then M(n) = 1 = 2n − 1

 Induction: if M(n − 1) = 2n−1 − 1, then
M(n) = 2M(n − 1) + 1 = 2 ∗ (2n−1 − 1) + 1 = (2n − 2) + 1 = 2n − 1
CS 3343 Analysis of Algorithms Chapter 2: Slide – 24

13 14
Solving the BitCount Recurrence Recurrence for Fibonacci Function
Make a reasonable guess using powers of 2.  Let A(n) = addition count to compute F (n).
 Forward substitution. D(2) = D(1) + 1 = 1  A(1) = A(0) = 0 because no additions are performed to compute F (0) or
D(4) = D(2) + 1 = 2 F (1).
D(8) = D(4) + 1 = 3  If n > 1, then F (n) performs two recursive calls plus one addition.
 Backward substitution. D(n) = D(n/2) + 1
= [D(n/4) + 1] + 1 = D(n/4) + 2 A(n) = A(n − 1) + A(n − 2) + 1
= [D(n/8) + 1] + 2 = D(n/8) + 3 to compute to compute to add F (n − 1)
F (n − 1) F (n − 2) and F (n − 2)
Prove D(2k ) = k by mathematical induction. CS 3343 Analysis of Algorithms Chapter 2: Slide – 29
k
 Basis: if k = 0, then D(2 ) = D(1) = 0 = k
 Induction: if D(2k−1) = k − 1, then D(2k ) = D(2k−1) + 1 = (k − 1) + 1 = k Approximating the Fibonacci Recurrence
CS 3343 Analysis of Algorithms Chapter 2: Slide – 27
Make a reasonable guess at a lower bound.
Fibonacci Function  Forward substitution. A(2) = 1, A(3) = 2, A(4) = 4, A(5) = 7, A(6) = 12.
 Backward substitution.
A(n) = A(n − 1) + A(n − 2) + 1
algorithm F (n) = [A(n − 2) + A(n − 3) + 1] + A(n − 2) + 1
// Computes nth Fibonacci number recursively = 2A(n − 2) + A(n − 3) + 2
// Input: A nonnegative integer n
// Output: The nth Fibonacci number Prove A(n) ≥ 2n/2 when n ≥ 4.
if n = 0 or n = 1 then return n  Basis: True for A(4) and A(5).
else return F (n − 1) + F (n − 2)  Induction: A(n) = 2A(n − 2) + A(n − 3) + 2
≥ 2A(n − 2) ≥ 2 ∗ 2(n−2)/2 = 2n/2
Input Size: Use number n (actually n has about log2 n bits) CS 3343 Analysis of Algorithms Chapter 2: Slide – 30
Basic Operation: addition in else statement
CS 3343 Analysis of Algorithms Chapter 2: Slide – 28

15 16
Empirical Analysis of Algorithms 31
Results of Empirical Analysis: Comparisons
500000
Plan for Empirical Analysis of Algorithms No Duplicates

Number of Comparisons
One Duplicate
400000
 Understand the experiment’s purpose. 300000
 Decide on the metric M and the measurement unit.
200000

 Decide on characteristics of input. 100000

 Prepare program implementing algorithm(s) and generating a sample of inputs. 0

0 200 400 600 800 1000
n
 Run the algorithm(s) on the sample and record the data.
 No duplicates line is n(n − 1)/2.
 Analyze the data.  One duplicate line is least squares fit, resulting in
CS 3343 Analysis of Algorithms Chapter 2: Slide – 31 0.24 ∗ n2 + 17 ∗ n − 2042 ≈ n2/4.
 Predict: Half the time with one duplicate.
Empirical Analysis of UniqueElements CS 3343 Analysis of Algorithms Chapter 2: Slide – 33

 Analyze UniqueElements, comparing arrays with unique elements vs. one

duplicate. Results of Empirical Analysis: Timings
 Measure number of comparisons on different input sizes, from 50 to 1000 by 11
10 No Duplicates
One Duplicate
50. 9

Time in Milliseconds
8
7
 For arrays with a duplicate, randomly choose a pair of positions that will have 6
the same value. Also, run 10 times for each input size. 5
4
3
 Prepare UniqueElementsExperiment.java. 2
1
0
 Run program, record data, and create graph. 0 200 400 600 800 1000
n
 Analyze the data.  No duplicates and one duplicate lines are:
CS 3343 Analysis of Algorithms Chapter 2: Slide – 32 −0.037 + 0.0016 ∗ n + 8.4 × 10−6 ∗ n2 .
−0.073 + 0.00047 ∗ n + 5.2 × 10−6 ∗ n2.
CS 3343 Analysis of Algorithms Chapter 2: Slide – 34

17 18
Algorithm Visualization 35

Visualization of Sorting Algorithms

http://homepages.dcc.ufmg.br/∼dorgival/applets/SortingPoints/SortingPoints.h

http://maven.smith.edu/∼thiebaut/java/sort/demo.html
CS 3343 Analysis of Algorithms Chapter 2: Slide – 35

19