A N A LY T I C C O M B I N AT O R I C S

PART O N E

1. Analysis
of
Algorithms
http:// aofa.cs.princeton.edu
Why Analyze an Algorithm?

1. Classify problems and algorithms by difficulty.

2. Predict performance, compare algorithms, tune parameters.

3. Better understand and improve implementations and algorithms.

Intellectual challenge: AofA is even more interesting than programming!

2
Analysis of Algorithms (Babbage, 1860s)

“As soon as an Analytic Engine exists, it will necessarily guide the future course of the science.
Whenever any result is sought by its aid, the question will arise—By what course of
calculation can these results be arrived at by the machine in the shortest time?”

— Charles Babbage (1864)

Analytic Engine

how many times do you

have to turn the crank?

3
Analysis of Algorithms (Turing (!), 1940s)

“It is convenient to have a measure of the amount of work involved in a computing

process, even though it be a very crude one. We may count up the number of times
that various elementary operations are applied in the whole process ...”
— Alan Turing (1947)

4
Analysis of Algorithms (Knuth, 1960s)

To analyze an algorithm:
•Develop a good implementation. D. E. Knuth

•Identify unknown quantities representing the basic operations.

•Determine the cost of each basic operation.
•Develop a realistic model for the input.
•Analyze the frequency of execution of the unknown quantities.
Σ
• Calculate the total running time: frequency(q) × cost(q)
q

BENEFITS:
Scientific foundation for AofA.
Can predict performance and compare algorithms.

DRAWBACKS:
Model may be unrealistic.
Too much detail in analysis.
5
Theory of Algorithms (AHU, 1970s; CLR, present day)

To address Knuth drawbacks:

Aho, Hopcroft
•Analyze worst-case cost and Ullman

[takes model out of the picture].

•Use O-notation for upper bound

[takes detail out of analysis].

Cormen, Leiserson,
Rivest, and Stein
•Classify algorithms by these costs.

BENEFIT: Enabled a new Age of Algorithm Design.

DRAWBACK: Cannot use to predict performance or compare algorithms.

(An elementary fact that is often overlooked!)

6
Example: Two sorting algorithms

Quicksort Mergesort
Worst-case number of compares: O(N 2) Worst-case number of compares: N log N
Classification O(N 2) Classification O(N log N)

BUT

Quicksort is twice as fast as Mergesort in practice and uses half the space

How do we know?
By analyzing both algorithms! (stay tuned)

Cannot use O- upper bounds to predict performance or compare algorithms.

7
Analytic combinatorics context

Drawbacks of Knuth approach:

• Model may be unrealistic.
• Too much detail in analysis.
Drawbacks of AHU/CLRS approach:
• Worst-case performance may not be relevant.
• Cannot use O- upper bounds to predict or compare.
Analytic combinatorics can provide:
• A calculus for developing models.
• General theorems that avoid detail in analysis.

AC Part I (this course):

• Underlying mathematics.
• Introduction to analytic combinatorics.
• Classical applications in AofA and combinatorics.
8
 A N A LY T I C C O M B I N AT O R I C S
PART O N E

1. Analysis of Algorithms
• History and motivation
• A scientific approach
• Example: Quicksort
OF
• Resources

http:// aofa.cs.princeton.edu

1a.AofA.History
 A N A LY T I C C O M B I N AT O R I C S
PART O N E

1. Analysis of Algorithms
• History and motivation
• A scientific approach
• Example: Quicksort
OF
• Resources

http:// aofa.cs.princeton.edu

1b.AofA.Scientific
Notation for theory of algorithms

“Big-Oh” notation for upper bounds

g(N) = o(f(N)) iff |g(N)/f(N)| is bounded from above as N → ∞

“Omega” notation for lower bounds

g(N) = Ω(f(N)) iff |g(N)/f(N)| is bounded from below as N → ∞

“Theta” notation for order of growth (“within a constant factor”)

g(N) = Θ(f(N)) iff g(N) = o(f(N)) and g(N) = Ω(f(N))

11
O-notation considered dangerous

How to predict performance (and to compare algorithms)?

N ot the scientific method: O-notation

Theorem: Running time is O(Nc) ✘

not at all useful for predicting performance

Scientific me thod calls for tilde-notation.

Hypothesis: Running time is ~aN c ✓

an effective path to predicting performance

O-notation is useful for many reasons, BUT

C o m m o n error: Thinking that O-notation is useful for predicting performance

12
Surely, we can do better

A typical exchange in Q&A

RS (in a talk): O-notation considered dangerous.

Cannot use it to predict performance.

Q: ?? O(N log N) surely beats O(N2)

RS: Not by the definition. O expresses upper bound.

Q: So, use Theta.

RS: Still (typically) bounding the worst case.

Is the input a worst case?

Q: (whispers to colleague) I’d use the Θ(N log N)

algorithm, wouldn’t you?

13
Galactic algorithms

R.J. Lipton: A galactic algorithm is one that will never be used.

Why? Any effect would never be noticed in this galaxy.

Ex. Chazelle’s linear-time triangulation algorithm

• theoretical tour-de-force
• too complicated to implement
• cost of implementing would exceed savings in this galaxy, anyway
One blogger’s conservative estimate:
75% SODA, 95% STOC/FOCS are galactic

OK for basic research to drive agenda, BUT

C o m m o n error: Thinking that a galactic algorithm is useful in practice.

14
Surely, we can do better

An actual exchange with a theoretical computer scientist:

TCS (in a talk): Algorithm A is bad.

Google should be interested in my new Algorithm B.

RS: What’s the matter with Algorithm A?

TCS: It is not optimal. It has an extra O(log log N) factor.

RS: But Algorithm B is very complicated, lg lg N is less

than 6 in this universe, and that is just an upper
bound. Algorithm A is certainly going to run 10 to
100 times faster in any conceivable real-world
situation. Why should Google care about Algorithm B?

TCS: Well, I like Algorithm B. I don’t care about Google.

15
Analysis of Algorithms (scientific approach)

Start with complete implementation suitable for application testing.

Analyze the algorithm by

• Identifying an abstract operation in the inner loop.
• Develop a realistic model for the input to the program.
• Analyze the frequency of execution C N of the op for input size N.
Hypothesize that the cost is ~aC N where a is a constant.

Validate the hypothesis by

• Developing generator for input according to model.
• Calculate a by running the program for large input.
• Run the program for larger inputs to check the analysis.
Sedgewick and Wayne
Algorithms, 4th edition
Validate the model by testing in application contexts. Section 1.4

Refine and repeat as necessary

16
Notation (revisited)

“Big-Oh” notation for upper bounds

g(N) = o(f(N)) iff |g(N)/f(N)| is bounded from above as N → ∞

“Omega” notation for lower bounds

g(N) = Ω(f(N)) iff |g(N)/f(N)| is bounded from below as N → ∞

“Theta” notation for order of growth (“within a constant factor”)

g(N) = Θ(f(N)) iff g(N) = o(f(N)) and g(N) = Ω(f(N))

for theory of algorithms

“Tilde” notation for asymptotic equivalence for analysis to predict performance

and to compare algorithms
g(N) ~ f(N) iff |g(N)/f(N)| → 1 as N → ∞
17
Components of algorithm analysis

Empirical
% java SortTest 1000000
• Run algorithm to solve real problem. 10 44.44
100 847.85
• Measure running time and/or 1000 12985.91
count operations. 10000 175771.70
100000 2218053.41
Challenge: need good implementation

Mathematical
Σ 1
• Develop mathematical model.
CN = N + 1 + (C + C N —k—1)
N k
• Analyze algorithm within model. 1≤k≤N
Challenge: need good model, need to do the math

Scientific % java QuickCheck 1000000

10 44.44 26.05
• Run algorithm to solve real problem. 100 847.85 721.03
1000 12985.91 11815.51
• Check for agreement with model. 10000 175771.70 164206.81
Challenge: need all of the above 100000 2218053.41 2102585.09
18
Potential drawbacks to the scientific approach

1. Model may not be realistic.

• A challenge in any scientific discipline.
• Advantage in CS: we can randomize to make the model apply.

➛
2. Math may be too difficult.
• A challenge in any scientific discipline (cf. statistical physics).
• A “calculus” for AofA is the motivation for this course!

3. Experiments may be too difficult.

• Not compared to other scientific disciplines.
• Can’t implement? Why analyze?

19
 A N A LY T I C C O M B I N AT O R I C S
PART O N E

1. Analysis of Algorithms
• History and motivation
• A scientific approach
• Example: Quicksort
OF
• Resources

http:// aofa.cs.princeton.edu

1b.AofA.Scientific
 A N A LY T I C C O M B I N AT O R I C S
PART O N E

1. Analysis of Algorithms
• History and motivation
• A scientific approach
• Example: Quicksort
OF
• Resources

http:// aofa.cs.princeton.edu

1c.AofA.Quicksort
Example: Quicksort
public class Quick
{
private static int partition(Comparable[] a, int lo, int hi)
{
int i = lo, j = hi+1;
while (true)
{
while (less(a[++i], a[lo])) if (i == hi) break;
while (less(a[lo], a[--j])) if (j == lo) break;
if (i >= j) break;
exch(a, i, j);
}
exch(a, lo, j);
return j;
}

private static void sort(Comparable[] a, int lo, int hi)

{
if (hi <= lo) return; Section 2.3
int j = partition(a, lo, hi);
sort(a, lo, j-1);
sort(a, j+1, hi);
}
}
http://algs4.cs.princeton.edu/23quicksort/Quick.java
23
Start: Preliminary decisions

Cost model
• running time? timing

• better approach: separate algorithm from implementation

• for sorting, associate compares with inner loop.
• Hypothesis: if number of compares is C, running time is ~aC

Input model counting

• assume input randomly ordered (easy to arrange)

• assume keys all different (not always easy to arrange)
Key question: Are models/assumptions realistic?

Stay tuned.

23
Setup: Relevant questions about quicksort

Assume array of size N with entries distinct and randomly ordered.

Q. How many compares to partition?

A. N+1

Q. What is the probability that the partitioning

item is the kth smallest?
A. 1 / N

Q. What is the size of the subarrays

in that case?
A. k − 1 and N − k

Q. Are the subarrays randomly

ordered after partitioning?
A. YES.

24
Main step: Formulate a mathematical problem

Recursive program and input model lead to a recurrence relation.

Assume array of size N with entries distinct and randomly ordered.

Let C N be the expected number of compares used by quicksort.

Σ 1
CN = N + 1 + (Ck —1 + C N —k)
N
1≤k≤N

compares for
for partitioning subarrays
when k is the
partitioning
probability element
k is the
partitioning
element
25
Simplifying the recurrence
Σ 1
CN = N + 1 + (Ck —1 + C N —k) C0 = 0
N
1≤k≤N

both sums are

C 0 + C 1 + ... + C N-1

Σ
2
Apply symmetry. CN = N + 1 + Ck —1
N
1≤k≤N

Σ
Multiply both sides by N. NCN = N(N + 1)+ 2 Ck—1
1≤k≤N

Subtract same formula for N−1. N C N —(N —1)C N—1 = 2N + 2C N—1

Collect terms.
NC N = (N + 1)CN—1 + 2N

26
Aside

Simplified recurrence gives efficient algorithm for computing result

Σ 1
CN = N + 1 + (Ck + C N —k—1) QUADRATIC time
N
O≤k≤N—1

⤷ c[0] = 0;
for (int N = 1; N <= maxN; N++)
{
c[N] = N+1;
for (int k = 0; k < N; k++)
c[N] += (c[k] + c[N-1-k])/N;
}

N C N = (N + 1)CN—1 + 2N LINEAR time

⤷ c[0] = 0;
for (int N = 1; N <= maxN; N++)
c[N] = (N+1)*c[N-1]/N + 2;

AofA: Finding a fast way to compute the running time of a program

27
Solving the recurrence

NC N = (N + 1)CN—1 + 2N

CN CN—1 2
Tricky (but key) step:
= +
divide by N(N+1) N +1 N N +1
CN CN—1 2 CN—2 2 2
= + = + +
N+1 N N+1 N —1 N N+1
Telescope.
C1 2 2 2
= + + .. . + +
2 3 N N +1
Σ 1
Simplify (ignore small terms). C N ~ 2N —2N
k
1≥k≥N
∫ ∞
1
Approximate with an C N ~ 2N ( dx + ц) —2N
integral (stay tuned) 1 x
= 2N ln N —2(1 —ц)N Euler’s constant ≐.57721

28
Finish: Validation (mathematical)
It is always worthwhile to check your math with your computer.

public class QuickCheck

{
public static void main(String[] args)
{
int maxN = Integer.parseInt(args[0]);
double[] c = new double[maxN+1];
c[0] = 0; NC N = (N + 1)CN—1 + 2N
for (int N = 1; N <= maxN; N++)
c[N] = (N+1)*c[N-1]/N + 2;

for (int N = 10; N <= maxN; N *= 10) 2N ln N —2(1 —ц)N

{
double approx = 2*N*Math.log(N) - 2*(1-.577215665)*N;
StdOut.printf("%10d %15.2f %15.2f\n", N, c[N], approx);
}
} % java QuickCheck 1000000
} 10 44.44 37.60
100 847.85 836.48
1000 12985.91 12969.94
10000 175771.70 175751.12
100000 2218053.41 2218028.23
1000000 26785482.23 26785452.45
29
Finish: Validation (checking the model)

It is always worthwhile to use your computer to check your model.

Example: Mean number of compares used by Quicksort for randomly ordered

distinct keys is 2N ln N —2(1 —ц)N

Experiment: Run code for randomly ordered distinct keys, count compares

2N ln N —2(1 —ц)N
1000 trials for each N
one grey dot for each trial
red dot: average for each N

✓
Observation: May be interested in distribution of costs
30
Quicksort compares: limiting distribution is not “normal”
see “Approximating the Limiting Quicksort Distribution.” by Fill and Janson (RSA 2001).

exact distribution
centered
(from recurrence)
on mean
for small N

empirical
validation
N = 1000

Bottom line:
• A great deal is known about the performance of Quicksort.
• AofA leads to intriguing new research problems.
31
Easy method to predict (approximate) performance

Hypothesis: Running time of Quicksort is ~aN ln N.

Experiment.
• Run for input size N. Observe running time.
• [Could solve for a.]
a(10N ) ln(10N ) = 10 + ln 10 = 10 + 1
• Predict time for 10N to increase by a factor of aN ln N ln N log 10 N

Example:
•Run quicksort 100 times for N = 100,000: Elapsed time: 4 seconds.
•Predict running time of 4 x 10.2 = 40.8 seconds for N = 1M.
•Observe running time of 41 seconds for N = 1M
•Confidently predict running time of 41 x 1000.5 = 11.4 hours for N = 1B.

Note: Best to also have accurate mathematical model. Why?

32
Validate-refine-analyze cycle

It is always worthwhile to validate your model in applications.

Quicksort: Validation ongoing for 50 years!

Example 1 (late 1970s): Sorting on the CRAY-1. as many times
as possible!
• Application: cryptography.
• Need to “sort the memory” 1M pseudo-random 64-bit words.
• Bottom line: analysis could predict running time to within 10−6 seconds.

Example 2 (1990s): UNIX system sort.

• Application: general-purpose.
• User app involving files with only a few distinct values performed poorly.
• Refinements: 3-way partitioning, 3-way string quicksort (see Algs4).
• Refined models (not simple): research ongoing. see“The number of symbol comparisons in QuickSort and QuickSelect.”
byVallee,Clement, Fill, and Flajolet (ICALP 2009).

Example 3 (2010s): Sorting for networking.

• Application: sort ~1B records ~1K characters each.
• Need to beat the competition or go out of business.
• Refinement: adapt to long stretches of equal chars (avoid excessive caching)
33
Double happiness

“People who analyze algorithms have double happiness. First of all

they experience the sheer beauty of elegant mathematical patterns
that surround elegant computational procedures. Then they
receive a practical payoff when their theories make it possible to
get other jobs done more quickly and more economically.”

— D. E.Knuth (1995)

34
 A N A LY T I C C O M B I N AT O R I C S
PART O N E

1. Analysis of Algorithms
• History and motivation
• A scientific approach
• Example: Quicksort
OF
• Resources

http:// aofa.cs.princeton.edu

1c.AofA.Quicksort
 A N A LY T I C C O M B I N AT O R I C S
PART O N E

1. Analysis of Algorithms
• History and motivation
• A scientific approach
• Example: Quicksort
OF
• Resources

http:// aofa.cs.princeton.edu

1d.AofA.Resources
Books
are the prime resources associated with this course.

First edition
Main (1995) Reference
text for
(2013) Algorithms

Text
for Reference
Part II for
Java

Reading the books is the best way to develop understanding.

37
Booksites
are web resources associated with the books.

http:// aofa.cs.princeton.edu

Surf the booksite to search for information, code, and data.

38
Extensive original research
is the basis for the material in this course.

Knuth's
collected
works

Flajolet's collected works

collected to appear 2014
Cambridge U. Press
works

research papers
and books
by hundreds
of others

A prime goal of this course: make this work accessible to you.

20,000+ pages of material (!) 40
More resources

Math typesetting

Symbolic math

Web references

40
Introduce, read, discuss

1. We introduce topics in lecture.

2. You read the book and do assignments before the next lecture.

3. We discuss reading and exercises online. [No assessments.]

The main resource in this class is YOU!

Goal: For you to learn quite a few things that you do not now know.

41
Exercises 1.14 and 1.15

How many recursive calls in Quicksort?

How many exchanges?

42
Exercises 1.17 and 1.18

Switch to insertion sort for small subarrays.

What choice of the threshold minimizes the number of compares?

43
Assignments for next lecture

1. Surf booksites
• http://aofa.cs.princeton.edu
• http://algs4.cs.princeton.edu

2. Start learning to use software.

• StdJava (from Algs4 booksite)
• TeX (optional: .html/MathJax)

3. Download Quicksort and predict performance on your computer.

4. Read pages 1-39 in text.

5. Write up solutions to Exercises 1.14, 1.15, 1.17, and 1.18.

44
 A N A LY T I C C O M B I N AT O R I C S
PART O N E

1. Analysis of Algorithms
• History and motivation
• A scientific approach
• Example: Quicksort
OF
• Resources

http:// aofa.cs.princeton.edu

1d.AofA.Resources
 A N A LY T I C C O M B I N AT O R I C S
PART O N E

1. Analysis
of
Algorithms
http:// aofa.cs.princeton.edu