Datenstrukturen

und Algorithmen

Analysis of Algorithms
 1.4 A NALYSIS OF A LGORITHMS

h tt p : / / a l g s 4 . c s . p r i n c e t o n . e d u

2
 Running time
“ As soon as an Analytic Engine exists, it will necessarily guide the future
course of the science. Whenever any result is sought by its aid, the question
will arise—By what course of calculation can these results be arrived at by
the machine in the shortest time? ” — Charles Babbage (1864)

Analytic Engine
3
 Running time
“ As soon as an Analytic Engine exists, it will necessarily guide the future
course of the science. Whenever any result is sought by its aid, the question
will arise—By what course of calculation can these results be arrived at by
the machine in the shortest time? ” — Charles Babbage (1864)

how many times do you

have to turn the crank?

Analytic Engine
3
Cast of characters

4
Cast of characters
Programmer needs to develop
a working solution.

4
 Cast of characters
Programmer needs to develop
a working solution.

Client wants to solve

problem ef ciently.

4
fi
 Cast of characters
Programmer needs to develop
a working solution.

Client wants to solve

problem ef ciently.

Theoretician wants
to understand.

4
fi
 Cast of characters
Programmer needs to develop
a working solution.

Student might play

Client wants to solve any or all of these

problem ef ciently. roles someday.

Theoretician wants
to understand.

4
fi
Predict performance.

Compare algorithms. this course

Provide guarantees.

Understand theoretical basis. „Berechenbarkeit und Komplexität“

Primary practical reason: avoid performance issues.

client gets poor performance because programmer

did not understand performance characteristics

5
Discrete Fourier transform.
• Break down waveform of N samples into periodic components.

• Applications: DVD, JPEG, MRI, astrophysics, ….

• Brute force: N2 steps.

• FFT algorithm: N log N steps, enabled new technology.

time
quadratic
64T

32T

16T
linearithmic
8T
linear

size 1K 2K 4K 8K 6
 N-body simulation
• Simulate gravitational interactions among N bodies.

• Brute force: N 2 steps.

• Barnes-Hut algorithm: N log N steps, enabled new research.

7
 N-body simulation
• Simulate gravitational interactions among N bodies.

• Brute force: N 2 steps.

• Barnes-Hut algorithm: N log N steps, enabled new research.

7
 The challenge

Q: Will my program be able to solve a large practical input?

8
 The challenge
Why is my program so slow ?

Q: Will my program be able to solve a large practical input?

8
 The challenge
Why is my program so slow ? Why does it run out of memory ?

Q: Will my program be able to solve a large practical input?

8
 The challenge
Why is my program so slow ? Why does it run out of memory ?

Q: Will my program be able to solve a large practical input?

Insight: [Knuth 1970s] Use scienti c method to understand
performance.
8
fi
 Scienti c method

9
fi
 Scienti c method

• Observe some feature of the natural world.

9
fi
 Scienti c method

• Observe some feature of the natural world.

• Hypothesize a model that is consistent with the observations.

9
fi
 Scienti c method

• Observe some feature of the natural world.

• Hypothesize a model that is consistent with the observations.

• Predict events using the hypothesis.

9
fi
 Scienti c method

• Observe some feature of the natural world.

• Hypothesize a model that is consistent with the observations.

• Predict events using the hypothesis.

• Verify the predictions by making further observations.

9
fi
 Scienti c method

• Observe some feature of the natural world.

• Hypothesize a model that is consistent with the observations.

• Predict events using the hypothesis.

• Verify the predictions by making further observations.

• Validate by repeating until the hypothesis and observations agree.

9
fi
 Principles

• Experiments must be reproducible.

• Hypotheses must be falsi able.

10
fi
 Example: 3-Sum
Given N distinct integers, how many triples sum to exactly zero?

% cat 8ints.txt
a[i] a[j] a[k] sum
30 -40 -20 -10 40 0 10 5
30 -40 10 0

% python3 3Sum.py 8ints.txt 30 -20 -10 0

4
-40 40 0 0

-10 0 10 0

11
 Example: 3-Sum
Given N distinct integers, how many triples sum to exactly zero?

% cat 8ints.txt
a[i] a[j] a[k] sum
30 -40 -20 -10 40 0 10 5
30 -40 10 0

% python3 3Sum.py 8ints.txt 30 -20 -10 0

4
-40 40 0 0

-10 0 10 0

11
 3-Sum
brute-force algorithm
import sys
import DSA

def count(a):
N = len(a)
count = 0
for i in range(0,N):
for j in range(i+1,N):
for k in range(j+1,N):
if a[i] + a[j] + a[k] == 0: check each triple
count+=1
return count

f = DSA.In(sys.argv[1])
a = f.readAllInts()
print(count(a))

12
Q: How to time a program? Manual!
% python3 3Sum.py 1Kints.txt

tick tick tick

70
% python3 3Sum.py 2Kints.txt

tick tick tick tick tick tick tick tick

tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick

528
% python3 3Sum.py 4Kints.txt
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
% python3 3Sum.py 1Kints.txt
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
70 tick tick tick tick tick tick tick tick
% python3 3Sum.py 2Kints.txt tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
528 tick tick tick tick tick tick tick tick
% python3 3Sum.py 4Kints.txt tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick 4039
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick 13
 Automatic
class Stopwatch (part of DSA.py )

Stopwatch() create a new stopwatch

float : elapsedTime() time since creation (in seconds)

f = DSA.In(sys.argv[1])
a = f.readAllInts()
s = DSA.Stopwatch()
c = count(a)
time = s.elapsedTime()
print("elapsed time:",time,"seconds")
print(c)
client code

14
 Empirical analysis

Run the program for

various input sizes
and measure running
time.

15
 Empirical analysis

Run the program for

various input sizes
and measure running
time.

15
 Empirical analysis
N time (seconds) †

250 0

Run the program for 500 0

various input sizes 1.000 0,1

and measure running 2.000 0,8

time. 4.000 6,4

8.000 51,1

16.000 ?

16
Standard plot. Plot running time T(N) vs. input size N.

standard plot 50 log-log plot 51.2 straight line

of slope 3
25.6

40 12.8

running time T(N)

6.4

lg(T(N))
30 3.2

1.6

20 .8

.4

10 .2

.1

1K 2K 4K 8K 1K 2K 4K 8K
problem size N lg N
Analysis of experimental data (the running time of ThreeSum)

17
 Log-log plot: Plot running time T(N) vs. input size N using log-log scale.

log-log plot 51.2 straight line

of slope 3
25.6

lg (T (N)) = b ⋅ lg (N) + c
12.8

6.4
b = 2.999

lg(T(N))
3.2

1.6
c = − 33.2103
.8
b c
.4 T (N) = a ⋅ N , where a = 2
.2

.1

1K 2K 4K 8K 1K 2K 4K 8K
problem size N lg N
Analysis of experimental data (the running time of ThreeSum)
b
Regression: Fit straight line through data points: a ⋅ N .
−10 2.999
Hypothesis: The running time is about 1.006 ⋅ 10 ⋅N seconds.
18
 −10 2.999
Hypothesis: The running time is about 1.006 ⋅ 10 ⋅N seconds.

Predictions:

• 51.0 seconds for N = 8,000.

• 408.1 seconds for N = 16,000.

Observations: N time (seconds)

8.000 51,1

8.000 51

8.000 51,1

16.000 410,8

validates hypothesis!

19
 Doubling hypothesis
Run program, doubling the size of the input.
b
T (2N) a ⋅ (2N) b
= = 2
T (N) a⋅N b

Quick way to estimate b in a power-law relationship.

20
 b
Doubling Hypothesis: Running time is about a ⋅ N with b = lg(ratio)

N time (seconds) † ratio lg ratio

250 0 –

b
T (2N) a ⋅ (2N)
500 0 4,8 2,3
b
1.000 0,1 6,9 2,8 = = 2
T (N) a ⋅ Nb
2.000 0,8 7,7 2,9

4.000 6,4 8 3 lg (6.4 / 0.8) = 3.0

8.000 51,1 8 3

seems to converge to a constant b ≈ 3

Hypothesis: Running time is about 0.998 ⋅ 10 −10 3

⋅N

21
Experimental algorithmics
• System independent effects:
• Algorithm
determines exponent

• Input data in power law

determines
• System dependent effects. constant in power

law
• Hardware: CPU, memory, cache, …

• Software: compiler, VM, garbage collector, …

• System: operating system, network, other apps, …

22
Mathematical models
for running time

23
Donald Knuth’s approach
Total running time:
sum of cost * frequency for operations.

• Need to analyze program to Donald Knuth

determine set of operations. 1974 Turing Award

• Cost depends on machine, compiler.

• Frequency depends on algorithm, input data.

24
 Challenge: How to estimate
constants.
operation example nanoseconds †

integer add a + b 2,1

integer multiply a * b 2,4

integer divide a // b 5,4

oating-point add a + b 4,6

oating-point multiply a * b 4,2

oating-point divide a / b 13,5

sine math.sin(theta) 91,3

arctangent math.atan2(y, x) 129

... ... ...

† Running OS X on Macbook Pro 2.2GHz with 2GB RAM

25
fl
fl
fl
 Observation: Most primitive operations take
constant time.
operation example nanoseconds †

assignment statement a = b c2

integer compare a < b c3

array element access a[i] c4

array length len(a) c5

1D array allocation intArray(N) c6 N

Caveat: Non-primitive operations often take more than constant time.

e.g. "+ operator" (concatenate) two strings
26
 Example: 1-Sum
Q: How many instructions as a function of input size N ?

operation frequency

variable declaration 2
count = 0 assignment statement 2
for i in range(0,N):
if a[i] == 0: less than compare N+1
count+=1
equal to compare N

array access N

N array accesses increment N to 2 N

27
 2-Sum

count = 0
for i in range(0,N):
for j in range(i+1,N):
if a[i] + a[j] == 0:
count+=1

28
 2-Sum

count = 0
for i in range(0,N):
for j in range(i+1,N):
if a[i] + a[j] == 0:
count+=1

1
0 + 1 + 2 + . . . + (N 1) = N (N 1)
2 ⇥
N
=
2

28
 In nite Sum

https://www.youtube.com/watch?v=w-I6XTVZXww
29
fi
 2-Sum
operation frequency

count = 0 variable declaration N+2

for i in range(0,N):
for j in range(i+1,N): assignment statement N+2
if a[i] + a[j] == 0:
range accesses ½ (N + 1) (N + 2)
count+=1
equal to compare ½ N (N − 1)

array access N (N − 1)

increment ½ N (N − 1) to N (N − 1)

30
 2-Sum
operation frequency

count = 0 variable declaration N+2

for i in range(0,N):
for j in range(i+1,N): assignment statement N+2
if a[i] + a[j] == 0:
range accesses ½ (N + 1) (N + 2)
count+=1
equal to compare ½ N (N − 1)
1 array access N (N − 1)
0 + 1 + 2 + . . . + (N 1) = N (N 1)
2 ⇥
=
N increment ½ N (N − 1) to N (N − 1)
2

30
“It is convenient to have a measure of the amount of work involved in a computing process, even
though it be a very crude one. We may count up the number of times that various elementary operations
are applied in the whole process and then given them various weights. We might, for instance, count the
number of additions, subtractions, multiplications, divisions, recording of numbers, and extractions of
figures from tables. In the case of computing with matrices most of the work consists of multiplications
and writing down numbers, and we shall therefore only attempt to count the number of multiplications
and recordings.” — Alan Turing

ROUNDING-OFF ERRORS IN MATRIX PROCESSES

By A. M. TURING
{National Physical Laboratory, Teddington, Middlesex)
[Received 4 November 1947]
SUMMARY
A number of methods of solving sets of linear equations and inverting matrices
are discussed. The theory of the rounding-off errors involved is investigated for
some of the methods. In all cases examined, including the well-known 'Gauss
elimination process', it is found that the errors are normally quite moderate: no
exponential build-up need occur.
Included amongst the methods considered is a generalization of Choleski's method
which appears to have advantages over other known methods both as regards
accuracy and convenience. This method may also be regarded as a rearrangement
of the elimination process.

Downlo
THIS paper contains descriptions of a number of methods for solving sets 31
Simpli cation 1 ”Cost Model“

Use some basic operation as a proxy for

running time.

32
fi
 Cost model
count = 0
for i in range(0,N):
for j in range(i+1,N):
if a[i] + a[j] == 0:
count+=1
operation frequency

variable declaration N+2

assignment statement N+2

range accesses ½ (N + 1) (N + 2)

equal to compare ½ N (N − 1)

array access N (N − 1)
cost model = array accesses

increment ½ N (N − 1) to N (N − 1)
(we assume compiler or runtime do not

optimize any array accesses away!)

33
 Cost model
count = 0
for i in range(0,N):
for j in range(i+1,N):
if a[i] + a[j] == 0:
count+=1
operation frequency

1 variable declaration N+2

0 + 1 + 2 + . . . + (N 1) = N (N 1)
2 ⇥
=
N assignment statement N+2
2
range accesses ½ (N + 1) (N + 2)

equal to compare ½ N (N − 1)

array access N (N − 1)
cost model = array accesses

increment ½ N (N − 1) to N (N − 1)
(we assume compiler or runtime do not

optimize any array accesses away!)

33
 Simpli cation 2 ”Tilde Notation“

Estimate running time (or memory) as a function of input size N.

Ignore lower order terms.

(when N is large, terms are negligible when N is small, we don't care)

34
fi
 Tilde Notation

N 3/6

Ex 1: ⅙ N 3 + 20 N + 16 ~ ⅙N3
166,666,667 N 3/6 ! N 2/2 + N /3
Ex 2: ⅙ N 3 + 100 N 4/3 + 56 ~ ⅙N3
166,167,000
Ex 3: ⅙N3 - ½N 2 + ⅓ N ~ ⅙N3
N 1,000
Leading-term approximation

Technical de nition. f(N) ~ g(N) means

35
fi
 Tilde Notation

N 3/6

Ex 1: ⅙ N 3 + 20 N + 16 ~ ⅙N3
166,666,667 N 3/6 ! N 2/2 + N /3
Ex 2: ⅙ N 3 + 100 N 4/3 + 56 ~ ⅙N3
166,167,000
Ex 3: ⅙N3 - ½N 2 + ⅓ N ~ ⅙N3
N 1,000
Leading-term approximation

f (N)
Technical de nition. f(N) ~ g(N) means lim = 1
N → ∞ g(N)

35
fi
 Tilde Notation
operation frequency tilde notation

variable declaration N+2 ~N

assignment statement N+2 ~N

range accesses ½ (N + 1) (N + 2) ~½N2

equal to compare ½ N (N − 1) ~½N2

array access N (N − 1) ~N2

increment ½ N (N − 1) to N (N − 1) ~ ½ N 2 to ~ N 2

36
 2-Sum
Q: Approximately how many array
accesses as a function of input size N ?

count = 0
for i in range(0,N):
for j in range(i+1,N):
if a[i] + a[j] == 0: "inner loop"
count+=1

A: ~ N2 array accesses.
37
 2-Sum
Q: Approximately how many array
accesses as a function of input size N ?

count = 0
for i in range(0,N):
for j in range(i+1,N):
if a[i] + a[j] == 0: "inner loop"
count+=1

1
0 + 1 + 2 + . . . + (N 1) = N (N 1)
2 ⇥
N
=
2

A: ~ N2 array accesses.
37
 Bottom line:
Use cost model and tilde
notation to simplify counts!

38
 3-Sum
Q: Approximately how many array accesses
as a function of input size N ?
count = 0
for i in range(0,N):
for j in range(i+1,N):
for k in range(j+1,N):
if a[i] + a[j] + a[k] == 0: "inner loop"
count+=1

(3)
N N (N − 1) (N − 2) 1 3
= ∼ N
3! 6

A: ~ ½N 3 array accesses.
39
Q: How to estimate a discrete sum?
• A1: Take a discrete mathematics course.
• A2: Replace the sum with an integral, and use calculus
(doesn’t always work)!
N N
1 2
∑ ∫x=1
Ex 1. 1 + 2 + … + N. i∼ x dx ∼ N
i=1
2

N N
1
∫x=1
ik ∼ x k dx ∼ N k+1
Ex 2. 1k + 2k + … + N k. ∑ k + 1
i=1

N N
1 1
∑i ∫x=1 x
Ex 3. 1 + 1/2 + 1/3 + … + 1/N. ∼ dx ∼ ln N
i=1
40
A3: Use Maple, Wolfram Alpha etc.

41
A3: Use Maple, Wolfram Alpha etc.

wolframalpha.com

41
A3: Use Maple, Wolfram Alpha etc.

wolframalpha.com

[wayne:nobel.princeton.edu] > maple15

|\^/| Maple 15 (X86 64 LINUX)
._|\| |/|_. Copyright (c) Maplesoft, a division of Waterloo Maple Inc. 2011
\ MAPLE / All rights reserved. Maple is a trademark of
<____ ____> Waterloo Maple Inc.
| Type ? for help.
> factor(sum(sum(sum(1, k=j+1..N), j = i+1..N), i = 1..N));

N (N - 1) (N - 2)
-----------------
6
41
• In principle, accurate mathematical models are available.
• In practice,
• Formulas can be complicated.
• Advanced mathematics might be required.
• Exact models best left for experts.
• Bottom line: We use approximate models in this course:
T(N) ~ c N .
3

42
 Order-of-growth
Classi cations

43
fi
 Common order-of-growth
classi cations
De nition. If f(N) ~ c g(N) for some constant c > 0, then the order
of growth of f(N) is g(N).
• Ignores leading coef cient.
• Ignores lower-order terms.

44
fi
fi
fi
 Example
count = 0
for i in range(0,N):
for j in range(i+1,N):
for k in range(j+1,N):
if a[i] + a[j] + a[k] == 0:
count+=1

The order of growth of the running time of this code is N 3.

45
 time
Common Classes
200T

100T
logarithmic
constant
Good news: The set of functions: 1, log N, N, N log N, N 2, N 3, and 2N
suf ces to describe the order of growth of most common algorithms.
100K 200K 500K
size

log-log plot
512T

exponential

ic

ic
cubi

hm
rat

r
rit
d

ea
qua

ea

lin
lin
64T
time

8T

4T

2T
logarithmic
T
constant

1K 2K 4K 8K size 512K

Typical orders of growth 46

fi
 order of
name typical code framework description example T(2N) / T(N)
growth
add two
1 constant a = b + c statement 1
numbers

while N > 1:
log N logarithmic divide in half binary search ~1
N = N // 2 ...

for i in range(0,N): nd the

N linear loop 2
... maximum
divide
N log N linearithmic [see mergesort lecture] mergesort ~2
and conquer
for i in range(0,N):
N2 quadratic for j in range(0,N): double loop check all pairs 4
...
for i in range(0,N):
for j in range(0,N):
N 3 cubic triple loop check all triples 8
for k in range(0,N):
...
exhaustive check all
2N exponential [see combinatorial search lecture] T(N)
search subsets
47
fi
growth problem size solvable in minutes time to process millions of inputs

rate 1970s 1980s 1990s 2000s 1970s 1980s 1990s 2000s

1 "any" "any" "any" "any" "instant" "instant" "instant" "instant"

log N "any" "any" "any" "any" "instant" "instant" "instant" "instant"

tens of hundreds of
N millions billions minutes seconds second "instant"
millions millions

hundreds of hundreds of tens of

N log N millions millions hour minutes seconds
thousands millions seconds

tens of
N2 hundreds thousand thousands decades years months weeks
thousands

N3 hundred hundreds thousand thousands "never" "never" "never" millennia

48
 Binary search
• Goal: Given a sorted array and a key, nd index of the key
in the array?
• Idea: Compare key against middle entry.
• Too small, go left.
• Too big, go right.
• Equal, found.

49
fi
 Binary search demo

successful search for 33

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo hi

50
 Binary search demo

successful search for 33

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo mid hi

50
 Binary search demo

successful search for 33

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo hi

51
 Binary search demo

successful search for 33

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo mid hi

51
 Binary search demo

successful search for 33

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo hi

52
 Binary search demo

successful search for 33

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo mid hi

52
 Binary search demo

lo = hi
successful search for 33

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

53
 Binary search demo

lo = hi
successful search for 33

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

mid

return 4

53
 Binary search demo

unsuccessful search for 34

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo hi

54
 Binary search demo

unsuccessful search for 34

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo mid hi

54
 Binary search demo

unsuccessful search for 34

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo hi

55
 Binary search demo

unsuccessful search for 34

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo mid hi

55
 Binary search demo

unsuccessful search for 34

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo hi

56
 Binary search demo

unsuccessful search for 34

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

lo mid hi

56
 Binary search demo

lo = hi
unsuccessful search for 34

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

57
 Binary search demo

lo = hi
unsuccessful search for 34

6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

mid

return -1

57
 Binary search implementation

Trivial to implement?

58
 Binary search implementation

Trivial to implement?
• First binary search published in 1946.

58
 Binary search implementation

Trivial to implement?
• First binary search published in 1946.
• First bug-free one in 1962.

58
 Binary search implementation

Trivial to implement?
• First binary search published in 1946.
• First bug-free one in 1962.
• Bug in Java's Arrays.binarySearch() discovered in 2006.

58
 Binary search: Python3
def binSearch(a,e):
lo = 0
hi = len(a)-1
while hi >= lo:
mid = lo+(hi-lo)//2
if e > a[mid]:
lo = mid+1
elif e < a[mid]:
hi = mid-1
else:
return mid
return -1

Invariant: If key appears in the array a[], then a[lo] ≤ key ≤ a[hi].
59
 Mathematical analysis
Proposition: Binary search uses at most 1 + lg N key compares to search
in a sorted array of size N.

Def.: T(N ) = # key compares to binary search a sorted subarray of

size ≤ N.

Binary search recurrence:

T (N ) ≤ T (N // 2) + 1 for N > 1, with T (1) = 1.

60
 Proof Sketch
T (N) ≤ T (N // 2) + 1 [ given ]

61
 Proof Sketch
T (N) ≤ T (N // 2) + 1 [ given ]
≤ T (N // 4) + 1 + 1 [ apply recurrence to rst term ]

61
fi
 Proof Sketch
T (N) ≤ T (N // 2) + 1 [ given ]
≤ T (N // 4) + 1 + 1 [ apply recurrence to rst term ]
≤ T (N // 8) + 1 + 1 + 1 [ apply recurrence to rst term ]

61
fi
fi
 Proof Sketch
T (N) ≤ T (N // 2) + 1 [ given ]
≤ T (N // 4) + 1 + 1 [ apply recurrence to rst term ]
≤ T (N // 8) + 1 + 1 + 1 [ apply recurrence to rst term ]
⋮

61
fi
fi
 Proof Sketch
T (N) ≤ T (N // 2) + 1 [ given ]
≤ T (N // 4) + 1 + 1 [ apply recurrence to rst term ]
≤ T (N // 8) + 1 + 1 + 1 [ apply recurrence to rst term ]
⋮
≤ T (N // N) + 1 + 1 + … + 1 [ stop applying, T(1) = 1 ]

61
fi
fi
 Proof Sketch
T (N) ≤ T (N // 2) + 1 [ given ]
≤ T (N // 4) + 1 + 1 [ apply recurrence to rst term ]
≤ T (N // 8) + 1 + 1 + 1 [ apply recurrence to rst term ]
⋮
≤ T (N // N) + 1 + 1 + … + 1 [ stop applying, T(1) = 1 ]
= 1 + lg N

61
fi
fi
N 2 log N algorithm for 3-Sum
input

30 -40 -20 -10 40 0 10 5

• Step 1: Sort the N (distinct) numbers.

• Step 2: For each pair of numbers a[i]

and a[j], binary search for -(a[i] + a[j]).

62
N 2 log N algorithm for 3-Sum
input

30 -40 -20 -10 40 0 10 5

sort

-40 -20 -10 0 5 10 30 40

• Step 1: Sort the N (distinct) numbers.

• Step 2: For each pair of numbers a[i]

and a[j], binary search for -(a[i] + a[j]).

62
N 2 log N algorithm for 3-Sum
binary search
sort (-40, -20) 60

-40 -20 -10 0 5 10 30 40 (-40, -10) 50

(-40, 0) 40
(-40, 5) 35
(-40, 10) 30
⋮ ⋮

(-20, -10) 30
• Step 1: Sort the N (distinct) numbers. only count if
⋮ ⋮
a[i] < a[j] < a[k]
• Step 2: For each pair of numbers a[i] (-10, 0) 10 to avoid
and a[j], binary search for -(a[i] + a[j]).
⋮ ⋮ double counting

( 10, 30) -40

( 10, 40) -50
( 30, 40) -70

62
 Analysis

• Order of growth is N 2 log N.

• Step 1: N 2 with insertion sort (details later).
• Step 2: N 2 log N with binary search.

Remark: Can achieve N 2 by modifying binary search step.

63
Speed Comparison
N time (seconds) N time (seconds)

1.000 0,1 1.000 0,14

2.000 0,8 2.000 0,18

4.000 6,4 4.000 0,34

8.000 51,1 8.000 0,96

3Sum.py 16.000 3,67

32.000 14,88

64.000 59,16

3Sum-fast.py

64
Theory of algorithms

65
 Types of analyses
• Best case
Lower bound on cost

• Worst case
Upper bound on cost

• Average case
Expected cost for random input
Ex 1. Array accesses for brute-force 3-SUM. Ex 2. Compares for binary search.
Best: ~ ½ N3 Best: ~ 1
Average: ~ ½ N3 Average: ~ lg N
Worst: ~ ½ N3 Worst: ~ lg N
66
 Types of analyses
• Best case
Lower bound on cost

• Worst case
Upper bound on cost
this course

• Average case
Expected cost for random input
Ex 1. Array accesses for brute-force 3-SUM. Ex 2. Compares for binary search.
Best: ~ ½ N3 Best: ~ 1
Average: ~ ½ N3 Average: ~ lg N
Worst: ~ ½ N3 Worst: ~ lg N
66
 Theory of algorithms
• Goals:

• Establish “dif culty” of a problem.

• Develop “optimal” algorithms.

• Approach:

• Suppress details in analysis: analyze “to within a constant factor.”

• Eliminate variability in input model: focus on the worst case.

67
fi
 Theory of algorithms

• Upper bound:
Performance guarantee of algorithm for any input.

• Lower bound:
Proof that no algorithm can do better.

• Optimal algorithm:
Lower bound = upper bound (to within a constant factor).

68
Commonly-used notations in the theory of
algorithms
notation provides example shorthand for used to

½ N2
asymptotic 10 N 2 classify
Big Theta Θ(N2)
order of growth 5 N 2 + 22 N log N + 3N algorithms
⋮

10 N 2
100 N develop
Big Oh Θ(N2) and smaller O(N2)
22 N log N + 3 N upper bounds
⋮

½N2
N5 develop
Big Omega Θ(N2) and larger Ω(N2)
N 3 + 22 N log N + 3 N lower bounds
⋮
69
 Example 1

1-Sum
“Is there a 0 in the array?”

70
• Upper bound: A speci c algorithm.

• Ex. Brute-force algorithm for 1-Sum: Look at every array entry.

• Running time of the optimal algorithm for 1-Sum is O(N).

71
fi
• Upper bound: A speci c algorithm.

• Ex. Brute-force algorithm for 1-Sum: Look at every array entry.

• Running time of the optimal algorithm for 1-Sum is O(N).

• Lower bound: Proof that no algorithm can do better.

• Ex. Have to examine all N entries

(any unexamined one might be 0).

• Running time of the optimal algorithm for 1-Sum is Ω(N).

• Optimal algorithm.

71
fi
• Upper bound: A speci c algorithm.

• Ex. Brute-force algorithm for 1-Sum: Look at every array entry.

• Running time of the optimal algorithm for 1-Sum is O(N).

• Lower bound: Proof that no algorithm can do better.

• Ex. Have to examine all N entries

(any unexamined one might be 0).

• Running time of the optimal algorithm for 1-Sum is Ω(N).

• Optimal algorithm.

• Lower bound equals upper bound (to within a constant factor).

• Brute-force algorithm for 1-Sum is optimal:

its running time is Θ(N).

71
fi
Example 2

3-Sum
72
• Upper bound: A speci c algorithm.

• Ex. Brute-force algorithm for 3-Sum.

• Running time of the optimal algorithm for 3-Sum is O(N3).

73
fi
• Upper bound: A speci c algorithm.

• Ex. Improved algorithm for 3-Sum.

• Running time of the optimal algorithm for 3-Sum is O(N2 log N ).

74
fi
• Upper bound: A speci c algorithm.

• Ex. Improved algorithm for 3-Sum.

• Running time of the optimal algorithm for 3-Sum is O(N2 log N ).

• Lower bound: Proof that no algorithm can do better.

• Ex. Have to examine all N entries to solve 3-Sum.

• Running time of the optimal algorithm for solving 3-Sum is Ω(N ).

75
fi
• Upper bound: A speci c algorithm.

• Ex. Improved algorithm for 3-Sum.

• Running time of the optimal algorithm for 3-Sum is O(N2 log N ).

• Lower bound: Proof that no algorithm can do better.

• Ex. Have to examine all N entries to solve 3-Sum.

• Running time of the optimal algorithm for solving 3-Sum is Ω(N ).

• Open problems:

• Optimal algorithm for 3-Sum? Subquadratic algorithm for 3-Sum?

Quadratic lower bound for 3-Sum?

75
fi
Algorithm Design

76
Algorithm Design
Develop a new Algorithm

76
Algorithm Design
Develop a new Algorithm

Prove a lower bound

76
Algorithm Design
Develop a new Algorithm

Gap?

Prove a lower bound

76
Algorithm Design
Lower the upper bound
Develop a new Algorithm (discover a new algorithm).

Gap?

Prove a lower bound

76
 Algorithm Design
Lower the upper bound
Develop a new Algorithm (discover a new algorithm).

Gap?

Raise the lower bound

Prove a lower bound (more dif cult).

76
fi
Golden Age of Algorithm Design

• 1970s
• Steadily decreasing upper bounds for many important problems.
• Many known optimal algorithms.

77
‑
 Caveats

• Overly pessimistic to focus on worst case?

• Often we need better than “to within a constant factor”
to predict performance.

78
Commonly-used notations in the theory of
algorithms
notation provides example shorthand for used to

10 N 2
provide
Tilde leading term ~ 10 N 2 10 N 2 + 22 N log N
approximate model
10 N 2 + 2 N + 37

½ N2
asymptotic classify
Big Theta Θ(N2) 10 N 2
order of growth algorithms
5 N 2 + 22 N log N + 3N

10 N 2
develop
Big Oh Θ(N2) and smaller O(N2) 100 N
upper bounds
22 N log N + 3 N

½N2
develop
Big Omega Θ(N2) and larger Ω(N2) N5
lower bounds
N 3+ 22 N log N + 3 N
79
Turning the crank: summary

80
 Summary
Empirical analysis.
• Execute program to perform
experiments.
• Assume power law and formulate a
hypothesis for running time.
• Model enables us to make predictions.

81
 Summary
Mathematical analysis.
• Analyze algorithm to count frequency of
operations.
• Use tilde notation to simplify analysis.
• Model enables us to explain behavior.

82
 Summary
Scienti c method.
• Mathematical model is independent of a
particular system; applies to machines not
yet built.
• Empirical analysis is necessary to validate mathematical
models and to make predictions.

83
fi
 R EADING L IST :

2.1 E LEMENTARY S ORTS

h tt p : / / a l g s 4 . c s . p r i n c e t o n . e d u

84