Algorithms R OBERT S EDGEWICK | K EVIN W AYNE

1.4 A NALYSIS OF
A LGORITHMS
‣ introduction
‣ observations
‣ mathematical models
‣ order-of-growth classifications
‣ theory of algorithms
h t t p : / / a l g s 4. c s . p r i n c e t o n . e d u ‣ memory
 1.4 A NALYSIS OF
A LGORITHMS
‣ introduction
‣ observations
‣ mathematical models
‣ order-of-growth classifications
‣ theory of algorithms
h t t p : / / a l g s 4. c s . p r i n c e t o n . e d u ‣ memory
Running time

“ As soon as an Analytic Engine exists, it will necessarily guide the future

course of the science. Whenever any result is sought by its aid, the question
will arise—By what course of calculation can these results be arrived at by
the machine in the shortest time? ” — Charles Babbage (1864)

how many times do you

have to turn the crank?

Analytic Engine
4
Cast of characters

Programmer needs to develop

a working solution.

Student might play

any or all of these
Client wants to solve
roles someday.
problem efficiently.

Theoretician wants
to understand.

5
Reasons to analyze algorithms

Predict performance.

Compare algorithms. In this course

Provide guarantees.

Understand theoretical basis. In the next course

Primary practical reason: avoid performance bugs.

client gets poor performance because programmer

did not understand performance characteristics

6
Some algorithmic successes

Discrete Fourier transform.

・Break down waveform of N samples into periodic components.
・Applications: DVD, JPEG, MRI, astrophysics, ….
・Brute force: N steps.
2
Friedrich Gauss
・FFT algorithm: N log N steps, enables new technology. 1805

7
Some algorithmic successes

N-body simulation.
・Simulate gravitational interactions among N bodies.
・Brute force: N steps.
2

・Barnes-Hut algorithm: N log N steps, enables new research.

8
The challenge

Q. Will my program be able to solve a large practical input?

Why is my program so slow ? Why does it run out of memory ?

Insight. [Knuth 1970s] Use scientific method to understand performance.

9
Scientific method applied to analysis of algorithms

A framework for predicting performance and comparing algorithms.

Scientific method.
・Observe some feature of the natural world.
・Hypothesize a model that is consistent with the observations.
・Predict events using the hypothesis.
・Verify the predictions by making further observations.
・Validate by repeating until the hypothesis and observations agree.

Principles.
・Experiments must be reproducible.
・Hypotheses must be falsifiable.

Feature of the natural world. Computer itself.

10
 1.4 A NALYSIS OF
A LGORITHMS
‣ introduction
‣ observations
‣ mathematical models
‣ order-of-growth classifications
‣ theory of algorithms
h t t p : / / a l g s 4. c s . p r i n c e t o n . e d u ‣ memory
Example: 3-SUM

3-SUM. Given N distinct integers, how many triples sum to exactly zero?

a[i] a[j] a[k] sum

For the following 8 distinct integers
1
30 -40 -20 -10 40 0 10 5 30 -40 10 0
There are four such triples.
2 30 -20 -10 0

3 -40 40 0 0

-10 0 10 0
4

Context. Deeply related to problems in computational geometry.

13
3-SUM: brute-force algorithm

// ...

int ThreeSum(const vector<int>& a) {

int n = a.size();
int count = 0;
for (int i = 0; i < n; i++) {
for (int j = i + 1; j < n; j++) {
for (int k = j + 1; k < n; k++) {
if (a[i] + a[j] + a[k] == 0) { check each triple
count++;
}
}
}
}
return count;
}

int main() {
ifstream file("1Mints.txt");
vector<int> a;
int x;
while (file >> x) {
a.push_back(x);
}
cout << ThreeSum(a) << endl;
return 0;
}

14
Measuring the running time

Q. How to time a program? ThreeSum 1Kints.txt

A. Manual.

ThreeSum 2Kints.txt

ThreeSum 4Kints.txt

15
Measuring the running time

Q. How to time a program?

A. Automatic.

...
#include <ctime>
...

int main() {
ifstream file("1Mints.txt");
vector<int> a;
int x;
int c = 0;
while (file >> x && c++ < 2000) {
a.push_back(x);
}
clock_t start = clock();
cout << ThreeSum(a) << endl;
clock_t end = clock();
double elapsed = double(end - start) / CLOCKS_PER_SEC;
cout << "Time: " << elapsed << "s" << endl;
return 0; client code
}

16
Empirical analysis

Run the program for various input sizes and measure running time.

N time (seconds) †

250 0

500 0

1,000 0.1

2,000 0.8

4,000 6.4

8,000 51.1

16,000 ?

18
Data analysis

Standard plot. Plot running time T (N) vs. input size N.

T(N) is an exponential function of N.

19
Data analysis

Log-log plot. Plot running time T (N) vs. input size N using log-log scale.

lg(T (N)) = b lg N + c
b = 2.999
c = lg a = -33.2103

T (N) = a N b, where a = 2 c
3 orders
of magnitude

power law

Regression. Fit straight line through data points: a N b. slope

Hypothesis. The running time is about 1.006  10 –10  N 2.999 seconds.

Powe Rule Ref: https://geology.humboldt.edu/courses/geology531/531_handouts/equations_of_graphs.pdf 20
Prediction and validation

Hypothesis. The running time is about 1.006  10 –10  N 2.999 seconds.

"order of growth" of running

time is about N3 [stay tuned]
Predictions.
・51.0 seconds for N = 8,000.
・408.1 seconds for N = 16,000.

Observations.
N time (seconds) †

8,000 51.1

8,000 51

8,000 51.1

16,000 410.8

validates hypothesis!

22
Experimental algorithmics

Be careful to differentiate between:

Performance: how much time/memory/disk/... is actually
used when a program is run.This depends on the machine,
compiler, etc. as well as the code.

Complexity: how do the resource requirements of a program

or algorithm scale, i.e., what happens as the size of the
problem being solved gets larger.

Complexity affects performance but not the other way

around.
Experimental algorithmics

System independent effects.

・Algorithm. determines exponent
・Input data. in power law

System dependent effects. determines constant

・Hardware: CPU, memory, cache, … in power law

・Software: compiler, interpreter, garbage collector, …

・System: operating system, network, other apps, …

Bad news. Difficult to get precise measurements.

Good news. Much easier and cheaper than other sciences.

e.g., can run huge number of experiments

26
 1.4 A NALYSIS OF
A LGORITHMS
‣ introduction
‣ observations
‣ mathematical models
‣ order-of-growth classifications
‣ theory of algorithms
h t t p : / / a l g s 4. c s . p r i n c e t o n . e d u ‣ memory
Mathematical models for running time

Total running time:

The total running time of a program is determined by two primary factors:

• The cost of executing each statement
• The frequency of execution of each statement

・Need to analyze program to determine set of operations.

・Cost of execution depends on machine, compiler.
・Frequency depends on algorithm, input data.

31
Cost of basic operations

Challenge. How to estimate constants or the cost of common operations.

operation example nanoseconds †

integer add a+b 2.1

integer multiply a*b 2.4

integer divide a/b 5.4

floating-point add a+b 4.6

floating-point multiply a*b 4.2

floating-point divide a/b 13.5

sine Math.sin(theta) 91.3

arctangent Math.atan2(y, x) 129

... ... ...

† Running OS X on Macbook Pro 2.2GHz with 2GB RAM

32
Cost of basic operations

Observation. Most primitive operations take constant time.

operation example nanoseconds †

variable declaration int a c1

assignment statement a=b c2

integer compare a<b c3

array element access a[i] c4

array length a.length c5

1D array allocation new int[N] c6 N

2D array allocation new int[N][N] c7 N 2

Caveat. Non-primitive operations often take more than constant time.

novice mistake: abusive string concatenation

33
Example: 1-SUM

Q. How many instructions as a function of input size N ?

int count = 0;
for (int i = 0; i < N; i++)
if (a[i] == 0)
count++;

N array accesses

operation frequency

variable declaration 2

assignment statement 2

less than compare N+1

equal to compare N

array access N

increment N to 2 N

34
Example: 2-SUM

Q. How many instructions as a function of input size N ?

int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
if (a[i] + a[j] == 0)
count++;

operation frequency

variable declaration N+2

assignment statement N+2

less than compare ½ (N + 1) (N + 2)

equal to compare ½ N (N − 1)
tedious to count exactly
array access N (N − 1)

increment ½ N (N − 1) to N (N − 1)

37
Simplifying the calculations

“ It is convenient to have a measure of the amount of work involved

in a computing process, even though it be a very crude one. We may
count up the number of times that various elementary operations are
applied in the whole process and then given them various weights.
We might, for instance, count the number of additions, subtractions,
multiplications, divisions, recording of numbers, and extractions
of figures from tables. In the case of computing with matrices most
of the work consists of multiplications and writing down numbers,
and we shall therefore only attempt to count the number of
multiplications and recordings. ” — Alan Turing

38
Simplification 1: cost model

Cost model. Use some basic operation as a proxy for running time.

int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
if (a[i] + a[j] == 0)
count++;

operation frequency

variable declaration N+2

assignment statement N+2

less than compare ½ (N + 1) (N + 2)

equal to compare ½ N (N − 1)

array access N (N − 1) cost model = array accesses

increment ½ N (N − 1) to N (N − 1)

39
Simplification 2: tilde notation

・Estimate running time (or memory) as a function of input size N.

・Ignore lower order terms.
– when N is large, terms are negligible
– when N is small, we don't care

Ex. ⅙N3 - ½N 2 + ⅓ N ~ ⅙N3

discard lower-order terms

(e.g., N = 1000: 166.67 million vs. 166.17 million)

Technical definition. f(N) ~ g(N) means

40
Simplification 2: tilde notation

・Estimate running time (or memory) as a function of input size N.

・Ignore lower order terms.
– when N is large, terms are negligible
– when N is small, we don't care

operation frequency tilde notation

variable declaration N+2 ~N

assignment statement N+2 ~N

less than compare ½ (N + 1) (N + 2) ~½N2

equal to compare ½ N (N − 1) ~½N2

array access N (N − 1) ~N2

increment ½ N (N − 1) to N (N − 1) ~ ½ N 2 to ~ N 2

41
Example: 2-SUM

Q. Approximately how many array accesses as a function of input size N ?

int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
if (a[i] + a[j] == 0)
count++;

A. ~ N 2 array accesses.

Bottom line. Use cost model and tilde notation to simplify counts.
42
Example: 3-SUM

Q. Approximately how many array accesses as a function of input size N ?

int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
for (int k = j+1; k < N; k++)
if (a[i] + a[j] + a[k] == 0) "inner loop"
count++;

A. ~ ½ N 3 array accesses.

Bottom line. Use cost model and tilde notation to simplify counts.
43
Diversion: estimating a discrete sum

Q. How to estimate a discrete sum?

A. Replace the sum with an integral, and use calculus!

Ex 1. 1 + 2 + … + N.

Ex 2. 1k + 2k + … + N k.

Ex 3. 1 + 1/2 + 1/3 + … + 1/N.

Ex 4. 3-sum triple loop.

44
Estimating a discrete sum

Q. How to estimate a discrete sum?

A. Replace the sum with an integral, and use calculus!

Ex 4. 1 + ½ + ¼ + ⅛ + …

Caveat. Integral trick doesn't always work!

45
Estimating a discrete sum

Q. How to estimate a discrete sum?

A3. Use Maple or Wolfram Alpha.

wolframalpha.com

46
Mathematical models for running time

In principle, accurate mathematical models are available.

In practice,
・Formulas can be complicated.
・Advanced mathematics might be required.
・Exact models best left for experts.
costs (depend on machine, compiler)

TN = c1 A + c2 B + c3 C + c4 D + c5 E
A = array access
B = integer add
C = integer compare frequencies
D = increment (depend on algorithm, input)
E = variable assignment

Bottom line. We use approximate models in this course: T(N) ~ c N 3.

47
 1.4 A NALYSIS OF
A LGORITHMS
‣ introduction
‣ observations
‣ mathematical models
‣ order-of-growth classifications
‣ theory of algorithms
h t t p : / / a l g s 4. c s . p r i n c e t o n . e d u ‣ memory
Common order-of-growth classifications

Definition. If f (N) ~ c g(N) for some constant c > 0, then the order of growth
of f (N) is g(N).
・Ignores leading coefficient.
・Ignores lower-order terms.
Ex. The order of growth of the running time of this code is N 3.

int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
for (int k = j+1; k < N; k++)
if (a[i] + a[j] + a[k] == 0)
count++;

Typical usage. With running times.

where leading coefficient

depends on machine, compiler, ... 50
Common order-of-growth classifications

Good news. The set of functions

1, log N, N, N log N, N 2, N 3, and 2N
suffices to describe the order of growth of most common algorithms.

51
 Common order-of-growth classifications

order of
name typical code framework description example
growth

add two
1 constant a = b + c; statement
numbers

while (N > 1)
log N logarithmic divide in half binary search
{ N = N / 2; ... }

for (int i = 0; i < N; i++) find the

N linear loop
{ ... } maximum

divide
N log N linearithmic [see mergesort lecture] mergesort
and conquer

for (int i = 0; i < N; i++)

check all
N 2 quadratic for (int j = 0; j < N; j++) double loop
pairs
{ ... }

for (int i = 0; i < N; i++)

for (int j = 0; j < N; j++) check all
N3 cubic triple loop
for (int k = 0; k < N; k++) triples
{ ... }

exhaustive check all

2N exponential [recursive solution of Tower of Hanoi]
search subsets

52
Practical implications of order-of-growth

problem size solvable in minutes

growth
rate
1970s 1980s 1990s 2000s

1 any any any any

log N any any any any

tens of hundreds of
N millions billions
millions millions
hundreds of hundreds of
N log N millions millions
thousands millions
tens of
N2 hundreds thousand thousands
thousands

N3 hundred hundreds thousand thousands

2N 20 20s 20s 30

Bottom line. Need linear or linearithmic alg to keep pace with Moore's law.
53
Search Algorithm: Binary Search

int search(int A[], int size, int Num) {

int first = 0;
int last = size - 1;
int mid = first + (last - first) / 2;

while (first < last) {

if (A[mid] > Num)
last = mid - 1;
else if (A[mid] == Num)
return mid; // Num found at mid
else
first = mid + 1;
mid = first + (last - first) / 2;
}

if (A[mid] == Num)
return mid; // Num found at mid
else
return -1; // Num not found
}
Analyzing BinarySearch

• One comparison after loop

• First time through loop, toss half of array (2 comps)
• Second time, half remainder (1/4 original) 2 comps
• Third time, half remainder (1/8 original) 2 comps
• …

• Loop Iteration Remaining Elements

1 N/2
2 N/4
3 N/8
4 N/16
…
?? 1

How long to get to 1?

Analyzing Binary Search (cont)

• Looking at the problem in reverse, how long to double the

number until we get to N?
• N = 2X and solve for X

• log 2 N = log 2 2 𝑋 = X

• Two comparisons for each iteration, plus one comparison

at the end -- binary search takes in the worst case.
• Binary search in worst-case: O(log 2 𝑁)
• Sequential search in worst-case: O(N )
An N2 log N algorithm for 3-SUM
input
Algorithm. 30 -40 -20 -10 40 0 10 5
・Step 1: Sort the N (distinct) numbers.
sort
・Step 2: For each pair of numbers a[i] -40 -20 -10 0 5 10 30 40
and a[j], binary search for -(a[i] + a[j]).
binary search

(-40, -20) 60

Analysis. Order of growth is N 2 log N. (-40, -10) 50

・Step 1: N 2 with insertion sort. (-40, 0) 40

・Step 2: N 2 log N with binary search. (-40, 5) 35

(-40, 10) 30

⋮ ⋮

(-20, -10) 30

⋮ ⋮ only count if
a[i] < a[j] < a[k]
(-10, 0) 10
to avoid
⋮ ⋮ double counting
( 10, 30) -40
63
( 10, 40) -50

( 30, 40) -70

Comparing programs

Hypothesis. The sorting-based N 2 log N algorithm for 3-SUM is significantly

faster in practice than the brute-force N 3 algorithm.

N time (seconds) N time (seconds)

1,000 0.1 1,000 0.14

2,000 0.8 2,000 0.18

4,000 6.4 4,000 0.34

8,000 51.1 8,000 0.96

ThreeSum 16,000 3.67

32,000 14.88

64,000 59.16

ThreeSumFast

Guiding principle. Typically, better order of growth  faster in practice.

64
 1.4 A NALYSIS OF
A LGORITHMS
‣ introduction
‣ observations
‣ mathematical models
‣ order-of-growth classifications
‣ theory of algorithms
h t t p : / / a l g s 4. c s . p r i n c e t o n . e d u ‣ memory
Types of analyses

Best case. Lower bound on cost.

・Determined by “easiest” input.
・Provides a goal for all inputs.
Worst case. Upper bound on cost.
・Determined by “most difficult” input.
・Provides a guarantee for all inputs.
this course
Average case. Expected cost for random input.
・Need a model for “random” input.
・Provides a way to predict performance.

Ex 1. Array accesses for brute-force 3-SUM. Ex 2. Compares for binary search.

Best: ~ ½ N3 Best: ~ 1
Average: ~ ½ N3 Average: ~ lg N
Worst: ~ ½ N3 Worst: ~ lg N
67
Theory of algorithms

Goals.
・Establish “difficulty” of a problem.
・Develop “optimal” algorithms.
Approach.
・Suppress details in analysis: analyze “to within a constant factor.”
・Eliminate variability in input model: focus on the worst case.
Upper bound. Performance guarantee of algorithm for any input.
Lower bound. Proof that no algorithm can do better.
Optimal algorithm. Lower bound = upper bound (to within a constant factor).

69
Commonly-used notations in the theory of algorithms

notation provides example shorthand for used to

½ N2
Asymptotic order of 10 N 2 classify
Big Theta Θ(N2)
growth 5 N 2 + 22 N log N + 3N algorithms
⋮

10 N 2
100 N develop
Big Oh Θ(N2) and smaller O(N2)
22 N log N + 3 N upper bounds
⋮

½N2
N5 develop
Big Omega Θ(N2) and larger Ω(N2)
N 3 + 22 N log N + 3 N lower bounds
⋮
Theory of algorithms: example 1

Goals.
・Establish “difficulty” of a problem and develop “optimal” algorithms.
・Ex. 1-SUM = “Is there a 0 in the array? ”
Upper bound. A specific algorithm.
・Ex. Brute-force algorithm for 1-SUM: Look at every array entry.
・Running time of the optimal algorithm for 1-SUM is O(N).
Lower bound. Proof that no algorithm can do better.
・Ex. Have to examine all N entries (any unexamined one might be 0).
・Running time of the optimal algorithm for 1-SUM is Ω(N).
Optimal algorithm.
・Lower bound equals upper bound (to within a constant factor).
・Ex. Brute-force algorithm for 1-SUM is optimal: its running time is Θ(N).
71
Theory of algorithms: example 2

Goals.
・Establish “difficulty” of a problem and develop “optimal” algorithms.
・Ex. 3-SUM.
Upper bound. A specific algorithm.
・Ex. Brute-force algorithm for 3-SUM.
・Running time of the optimal algorithm for 3-SUM is O(N 3).

72
Theory of algorithms: example 2

Goals.
・Establish “difficulty” of a problem and develop “optimal” algorithms.
・Ex. 3-SUM.
Upper bound. A specific algorithm.
・Ex. Improved algorithm for 3-SUM.
・Running time of the optimal algorithm for 3-SUM is O(N 2 log N ).

Lower bound. Proof that no algorithm can do better.

・Ex. Have to examine all N entries to solve 3-SUM.
・Running time of the optimal algorithm for solving 3-SUM is Ω(N ).
Open problems.
・Optimal algorithm for 3-SUM?
・Subquadratic algorithm for 3-SUM?
・Quadratic lower bound for 3-SUM?
73
Algorithm design approach

Start.
・Develop an algorithm.
・Prove a lower bound.
Gap?
・Lower the upper bound (discover a new algorithm).
・Raise the lower bound (more difficult).
Golden Age of Algorithm Design.
・1970s-.
・Steadily decreasing upper bounds for many important problems.
・Many known optimal algorithms.
Caveats.
・Overly pessimistic to focus on worst case?
・Need better than “to within a constant factor” to predict performance.
74
Commonly-used notations in the theory of algorithms

notation provides example shorthand for used to

10 N 2 provide
Tilde leading term ~ 10 N 2 10 N 2 + 22 N log N approximate
10 N 2 + 2 N + 37 model

½ N2
asymptotic order classify
Big Theta Θ(N2) 10 N 2
of growth algorithms
5N 2+ 22 N log N + 3N

10 N 2
develop
Big Oh Θ(N2) and smaller O(N2) 100 N
upper bounds
22 N log N + 3 N

½N2
develop
Big Omega Θ(N2) and larger Ω(N2) N 5
lower bounds
N 3+ 22 N log N + 3 N

Common mistake. Interpreting big-Oh as an approximate model.

This course. Focus on approximate models: use Tilde-notation
75
 1.4 A NALYSIS OF
A LGORITHMS
‣ introduction
‣ observations
‣ mathematical models
‣ order-of-growth classifications
‣ theory of algorithms
h t t p : / / a l g s 4. c s . p r i n c e t o n . e d u ‣ memory
Basics

Bit. 0 or 1. NIST most computer scientists

Byte. 8 bits.
Megabyte (MB). 1 million or 220 bytes.
Gigabyte (GB). 1 billion or 230 bytes.

64-bit machine. We assume a 64-bit machine with 8-byte pointers.

・Can address more memory.
・Pointers use more space.

78
Typical memory usage for fundamental types and vectors

type bytes type bytes

bool 1 vector<char> N + 24

char 1 vector<int> 4 N + 24

int 4 vector<double> 8 N + 24

float 4
one-dimensional vectors

long 4

long long 8
type bytes
double 8
vector<vector<char>> ~1MN

fundamental types vector<vector<int>> ~4MN

vector<vector<double>> ~8MN

two-dimensional vectors
79
Typical memory usage for objects in C++

Virtual table pointer. 8 bytes for each virtual base class.

Padding. Each object uses a multiple of alignof() bytes.

struct S {
char a; // size: 1, alignment: 1 • Objects of type S can be
allocated at any address
char b; // size: 1, alignment: 1
because both S.a and S.b can
}; // sizeof(S) = 2, alignof(S) = 1 be allocated at any address.

struct X { • Objects of struct X must be

allocated at 4-byte boundaries
int n; // size: 4, alignment: 4
because X.n must be allocated
char c; // size: 1, alignment: 1 at 4-byte boundaries because
// three bytes padding int's alignment requirement is
}; // sizeof(X) = 8, alignof(X) = 4 (usually) 4

80
Typical memory usage summary

Total memory usage for a data type value:

・Primitive type: 4 bytes for int, 8 bytes for double, …
・Pointer: 8 bytes.
・vector: 24 bytes + memory for each array entry.
・Virtual object: 8 bytes per virtual base + memory instance variables.
・Padding: round up to multiple of alignof() bytes.

Shallow memory usage: Don't count referenced objects.

Deep memory usage: If array entry or instance variable is a reference,

count memory (recursively) for referenced object.

82
Turning the crank: summary

Empirical analysis.
・Execute program to perform experiments.
・Assume power law and formulate a hypothesis for running time.
・Model enables us to make predictions.

Mathematical analysis.
・Analyze algorithm to count frequency of operations.
・Use tilde notation to simplify analysis.
・Model enables us to explain behavior.

Scientific method.
・Mathematical model is independent of a particular system;
applies to machines not yet built.
・Empirical analysis is necessary to validate mathematical models
and to make predictions.

83