Introduction to Algorithms

6.046J/18.401J/SMA5503

Lecture 1
Prof. Charles E. Leiserson

Welcome to Introduction to
Algorithms, Fall 2001
Handouts
1.
2.
3.
4.
5.
6.

Day 1

Course Information
Calendar
Registration (MIT students only)
References
Objectives and Outcomes
Diagnostic Survey

Introduction to Algorithms

L1.2

Course information
1.
2.
3.
4.
5.
6.
7.

Staff
Distance learning
Prerequisites
Lectures
Recitations
Handouts
Textbook (CLRS)

8. Website
9. Extra help
10.Registration (MIT only)
11.Problem sets
12.Describing algorithms
13.Grading policy
14.Collaboration policy

Course information handout

Day 1

Introduction to Algorithms

L1.3

Analysis of algorithms
The theoretical study of computer-program
performance and resource usage.
Whats more important than performance?
modularity
user-friendliness
correctness
programmer time
maintainability
simplicity
functionality
extensibility
robustness
reliability
Day 1

Introduction to Algorithms

L1.4

Why study algorithms and

performance?
Algorithms help us to understand scalability.
Performance often draws the line between what
is feasible and what is impossible.
Algorithmic mathematics provides a language
for talking about program behavior.
The lessons of program performance generalize
to other computing resources.
Speed is fun!
Day 1

Introduction to Algorithms

L1.5

The problem of sorting

Input: sequence a1, a2, , an of numbers.
Output: permutation a'1, a'2, , a'n such
that a'1 a'2 a'n .
Example:
Input: 8 2 4 9 3 6
Output: 2 3 4 6 8 9
Day 1

Introduction to Algorithms

L1.6

Insertion sort

pseudocode

INSERTION-SORT (A, n)
A[1 . . n]
for j 2 to n
do key A[ j]
ij1
while i > 0 and A[i] > key
do A[i+1] A[i]
ii1
A[i+1] = key
j

A:
sorted
Day 1

key
Introduction to Algorithms

L1.7

Example of insertion sort

8

Day 1

Introduction to Algorithms

L1.8

Example of insertion sort

8

Day 1

Introduction to Algorithms

L1.9

Example of insertion sort

Day 1

Introduction to Algorithms

L1.10

Example of insertion sort

Day 1

Introduction to Algorithms

L1.11

Example of insertion sort

Day 1

Introduction to Algorithms

L1.12

Example of insertion sort

Day 1

Introduction to Algorithms

L1.13

Example of insertion sort

Day 1

Introduction to Algorithms

L1.14

Example of insertion sort

Day 1

Introduction to Algorithms

L1.15

Example of insertion sort

Day 1

Introduction to Algorithms

L1.16

Example of insertion sort

Day 1

Introduction to Algorithms

L1.17

Example of insertion sort

Day 1

9 done

Introduction to Algorithms

L1.18

Running time
The running time depends on the input: an
already sorted sequence is easier to sort.
Parameterize the running time by the size of
the input, since short sequences are easier to
sort than long ones.
Generally, we seek upper bounds on the
running time, because everybody likes a
guarantee.

Day 1

Introduction to Algorithms

L1.19

Kinds of analyses
Worst-case: (usually)
T(n) = maximum time of algorithm
on any input of size n.
Average-case: (sometimes)
T(n) = expected time of algorithm
over all inputs of size n.
Need assumption of statistical
distribution of inputs.
Best-case: (bogus)
Cheat with a slow algorithm that
works fast on some input.
Day 1

Introduction to Algorithms

L1.20

Machine-independent time
What is insertion sorts worst-case time?
It depends on the speed of our computer:
relative speed (on the same machine),
absolute speed (on different machines).
BIG IDEA:
Ignore machine-dependent constants.
Look at growth of T(n) as n .
Asymptotic Analysis
Day 1

Introduction to Algorithms

L1.21

-notation
Math:

(g(n)) = { f (n) : there exist positive constants c1, c2, and

n0 such that 0 c1 g(n) f (n) c2 g(n)
for all n n0 }

Engineering:
Drop low-order terms; ignore leading constants.
Example: 3n3 + 90n2 5n + 6046 = (n3)

Day 1

Introduction to Algorithms

L1.22

Asymptotic performance
When n gets large enough, a (n2) algorithm
always beats a (n3) algorithm.

T(n)

n
Day 1

n0

We shouldnt ignore
asymptotically slower
algorithms, however.
Real-world design
situations often call for a
careful balancing of
engineering objectives.
Asymptotic analysis is a
useful tool to help to
structure our thinking.

Introduction to Algorithms

L1.23

Insertion sort analysis

Worst case: Input reverse sorted.

T ( n) =

2)
(

(
j
)
=

[arithmetic series]

j =2

Average case: All permutations equally likely.

T ( n) =

( j / 2) = (n 2 )
j =2

Is insertion sort a fast sorting algorithm?

Moderately so, for small n.
Not at all, for large n.
Day 1

Introduction to Algorithms

L1.24

Merge sort
MERGE-SORT A[1 . . n]
1. If n = 1, done.
2. Recursively sort A[ 1 . . n/2 ]
and A[ n/2+1 . . n ] .
3. Merge the 2 sorted lists.
Key subroutine: MERGE

Day 1

Introduction to Algorithms

L1.25

Merging two sorted arrays

20 12
13 11
7

Day 1

Introduction to Algorithms

L1.26

Merging two sorted arrays

20 12
13 11
7

1
1

Day 1

Introduction to Algorithms

L1.27

Merging two sorted arrays

20 12

20 12

13 11

13 11

Day 1

Introduction to Algorithms

L1.28

Merging two sorted arrays

20 12

20 12

13 11

13 11

Day 1

Introduction to Algorithms

L1.29

Merging two sorted arrays

20 12

20 12

20 12

13 11

13 11

13 11

Day 1

Introduction to Algorithms

L1.30

Merging two sorted arrays

20 12

20 12

20 12

13 11

13 11

13 11

Day 1

Introduction to Algorithms

L1.31

Merging two sorted arrays

20 12

20 12

20 12

20 12

13 11

13 11

13 11

13 11

Day 1

Introduction to Algorithms

L1.32

Merging two sorted arrays

20 12

20 12

20 12

20 12

13 11

13 11

13 11

13 11

Day 1

Introduction to Algorithms

L1.33

Merging two sorted arrays

20 12

20 12

20 12

20 12

20 12

13 11

13 11

13 11

13 11

13 11

Day 1

Introduction to Algorithms

L1.34

Merging two sorted arrays

20 12

20 12

20 12

20 12

20 12

13 11

13 11

13 11

13 11

13 11

Day 1

Introduction to Algorithms

11

L1.35

Merging two sorted arrays

20 12

20 12

20 12

20 12

20 12

20 12

13 11

13 11

13 11

13 11

13 11

13

Day 1

Introduction to Algorithms

11

L1.36

Merging two sorted arrays

20 12

20 12

20 12

20 12

20 12

20 12

13 11

13 11

13 11

13 11

13 11

13

Day 1

Introduction to Algorithms

11

12

L1.37

Merging two sorted arrays

20 12

20 12

20 12

20 12

20 12

20 12

13 11

13 11

13 11

13 11

13 11

13

11

12

Time = (n) to merge a total

of n elements (linear time).
Day 1

Introduction to Algorithms

L1.38

Analyzing merge sort

T(n)
MERGE-SORT A[1 . . n]
(1)
1. If n = 1, done.
2T(n/2) 2. Recursively sort A[ 1 . . n/2 ]
Abuse
and A[ n/2+1 . . n ] .
(n)
3. Merge the 2 sorted lists
Sloppiness: Should be T( n/2 ) + T( n/2 ) ,
but it turns out not to matter asymptotically.
Day 1

Introduction to Algorithms

L1.39

Recurrence for merge sort

T(n) =

(1) if n = 1;
2T(n/2) + (n) if n > 1.

We shall usually omit stating the base

case when T(n) = (1) for sufficiently
small n, but only when it has no effect on
the asymptotic solution to the recurrence.
CLRS and Lecture 2 provide several ways
to find a good upper bound on T(n).
Day 1

Introduction to Algorithms

L1.40

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.

Day 1

Introduction to Algorithms

L1.41

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
T(n)

Day 1

Introduction to Algorithms

L1.42

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
T(n/2)

T(n/2)

Day 1

Introduction to Algorithms

L1.43

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/2

cn/2
T(n/4)

Day 1

T(n/4)

T(n/4)

Introduction to Algorithms

T(n/4)

L1.44

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/2

cn/2
cn/4

cn/4

cn/4

cn/4
(1)
Day 1

Introduction to Algorithms

L1.45

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/2

cn/2
cn/4

cn/4

cn/4

h = lg n cn/4
(1)
Day 1

Introduction to Algorithms

L1.46

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn
cn/2

cn/2
cn/4

cn/4

cn/4

h = lg n cn/4
(1)
Day 1

Introduction to Algorithms

L1.47

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn
cn/2

cn/2
cn/4

cn/4

cn/4

h = lg n cn/4

cn

(1)
Day 1

Introduction to Algorithms

L1.48

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn
cn/2

cn/2
cn/4

cn/4

cn/4

cn

h = lg n cn/4

cn

(1)
Day 1

Introduction to Algorithms

L1.49

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn
cn/2

cn/2
cn/4

cn/4

(1)
Day 1

cn/4

cn

h = lg n cn/4

cn

#leaves = n

(n)

Introduction to Algorithms

L1.50

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn
cn/2

cn/2
cn/4

cn/4

(1)

cn/4

cn

h = lg n cn/4

cn

#leaves = n

(n)
Total = (n lg n)

Day 1

Introduction to Algorithms

L1.51

Conclusions
(n lg n) grows more slowly than (n2).
Therefore, merge sort asymptotically
beats insertion sort in the worst case.
In practice, merge sort beats insertion
sort for n > 30 or so.
Go test it out for yourself!

Day 1

Introduction to Algorithms

L1.52

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 2
Prof. Erik Demaine

Solving recurrences
The analysis of merge sort from
Lecture 1 required us to solve a
recurrence.
Recurrences are like solving integrals,
differential equations, etc.
o Learn a few tricks.
Lecture 3: Applications of recurrences.

Day 3

Introduction to Algorithms

L2.2

Substitution method
The most general method:
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.
Example: T(n) = 4T(n/2) + n
[Assume that T(1) = (1).]
Guess O(n3) . (Prove O and separately.)
Assume that T(k) ck3 for k < n .
Prove T(n) cn3 by induction.
Day 3

Introduction to Algorithms

L2.3

Example of substitution
T (n) = 4T (n / 2) + n
4c ( n / 2 ) 3 + n
= ( c / 2) n 3 + n
desired residual
= cn3 ((c / 2)n3 n)
cn3 desired
whenever (c/2)n3 n 0, for example,
if c 2 and n 1.
residual
Day 3

Introduction to Algorithms

L2.4

Example (continued)
We must also handle the initial conditions,
that is, ground the induction with base
cases.
Base: T(n) = (1) for all n < n0, where n0
is a suitable constant.
For 1 n < n0, we have (1) cn3, if we
pick c big enough.
This bound is not tight!
Day 3

Introduction to Algorithms

L2.5

A tighter upper bound?

We shall prove that T(n) = O(n2).
Assume that T(k) ck2 for k < n:
T (n) = 4T (n / 2) + n
4cn 2 + n
= O(n) Wrong! We must prove the I.H.

= cn 2 ( n) [ desired residual ]
cn 2
for no choice of c > 0. Lose!
Day 3

Introduction to Algorithms

L2.6

A tighter upper bound!

IDEA: Strengthen the inductive hypothesis.
Subtract a low-order term.
Inductive hypothesis: T(k) c1k2 c2k for k < n.
T (n) = 4T (n / 2) + n
4(c1 (n / 2) 2 c2 (n / 2) + n
= c1n 2 2c2 n + n
= c1n 2 c2 n (c2 n n)
c1n 2 c2 n if c2 > 1.

Pick c1 big enough to handle the initial conditions.

Day 3

Introduction to Algorithms

L2.7

Recursion-tree method
A recursion tree models the costs (time) of a
recursive execution of an algorithm.
The recursion tree method is good for
generating guesses for the substitution method.
The recursion-tree method can be unreliable,
just like any method that uses ellipses ().
The recursion-tree method promotes intuition,
however.
Day 3

Introduction to Algorithms

L2.8

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:

Day 3

Introduction to Algorithms

L2.9

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:
T(n)

Day 3

Introduction to Algorithms

L2.10

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
T(n/4)

Day 3

T(n/2)

Introduction to Algorithms

L2.11

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
T(n/16)

Day 3

T(n/8)

(n/2)2
T(n/8)

Introduction to Algorithms

T(n/4)

L2.12

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/8)2

(n/8)2

(n/4)2

(n/16)2

(n/2)2

(1)
Day 3

Introduction to Algorithms

L2.13

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/8)2

(n/2)2
(n/8)2

(n/4)2

(n/16)2

n2

(1)
Day 3

Introduction to Algorithms

L2.14

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/8)2

(n/8)2

5 n2
16

(n/4)2

(n/16)2

(n/2)2

n2

(1)
Day 3

Introduction to Algorithms

L2.15

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:
n2

(n/8)2

(n/8)2

(n/16)2

(n/2)2
(n/4)2

5 n2
16
25 n 2
256

(n/4)2

n2

(1)
Day 3

Introduction to Algorithms

L2.16

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/2)2
(n/8)2

(n/8)2

(1)
Day 3

(n/4)2

( ) +( )

2
5
5
1 + 16 + 16

Total = n
= (n2)

Introduction to Algorithms

5 n2
16
25 n 2
256

(n/4)2
(n/16)2

n2

5 3
16

+L
geometric series
L2.17

The master method

The master method applies to recurrences of
the form
T(n) = a T(n/b) + f (n) ,
where a 1, b > 1, and f is asymptotically
positive.

Day 3

Introduction to Algorithms

L2.18

Three common cases

Compare f (n) with nlogba:
1. f (n) = O(nlogba ) for some constant > 0.
f (n) grows polynomially slower than nlogba
(by an n factor).
Solution: T(n) = (nlogba) .
2. f (n) = (nlogba lgkn) for some constant k 0.
f (n) and nlogba grow at similar rates.
Solution: T(n) = (nlogba lgk+1n) .
Day 3

Introduction to Algorithms

L2.19

Three common cases (cont.)

Compare f (n) with nlogba:
3. f (n) = (nlogba + ) for some constant > 0.
f (n) grows polynomially faster than nlogba (by
an n factor),
and f (n) satisfies the regularity condition that
a f (n/b) c f (n) for some constant c < 1.
Solution: T(n) = ( f (n)) .

Day 3

Introduction to Algorithms

L2.20

Examples
Ex. T(n) = 4T(n/2) + n
a = 4, b = 2 nlogba = n2; f (n) = n.
CASE 1: f (n) = O(n2 ) for = 1.
T(n) = (n2).
Ex. T(n) = 4T(n/2) + n2
a = 4, b = 2 nlogba = n2; f (n) = n2.
CASE 2: f (n) = (n2lg0n), that is, k = 0.
T(n) = (n2lg n).
Day 3

Introduction to Algorithms

L2.21

Examples
Ex. T(n) = 4T(n/2) + n3
a = 4, b = 2 nlogba = n2; f (n) = n3.
CASE 3: f (n) = (n2 + ) for = 1
and 4(cn/2)3 cn3 (reg. cond.) for c = 1/2.
T(n) = (n3).
Ex. T(n) = 4T(n/2) + n2/lg n
a = 4, b = 2 nlogba = n2; f (n) = n2/lg n.
Master method does not apply. In particular,
for every constant > 0, we have n = (lg n).
Day 3

Introduction to Algorithms

L2.22

General method (Akra-Bazzi)

k

T (n) = aiT (n / bi ) + f (n)

i =1

Let p be the unique solution to

p
(
a
/b
i i ) = 1.
k

i =1

Then, the answers are the same as for the

master method, but with np instead of nlogba.
(Akra and Bazzi also prove an even more
general result.)
Day 3

Introduction to Algorithms

L2.23

Idea of master theorem

f (n)

a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)

(1)
Day 3

#leaves = ah
= alogbn
= nlogba
Introduction to Algorithms

f (n)
a f (n/b)
a2 f (n/b2)

Recursion tree:

nlogba (1)

L2.24

Idea of master theorem

f (n)

a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
C
CASE
ASE 1:
1: The
The weight
weight increases
increases
geometrically
geometrically from
from the
the root
root to
to the
the
(1) leaves.
leaves. The
The leaves
leaves hold
hold aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight.
Day 3

Introduction to Algorithms

f (n)
a f (n/b)
a2 f (n/b2)

Recursion tree:

nlogba (1)
(nlogba)
L2.25

Idea of master theorem

Recursion tree:

f (n)

(1)
Day 3

a f (n/b)
a2 f (n/b2)

a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)

f (n)

C
CASE
ASE 2:
2: (k
(k == 0)
0) The
The weight
weight
isis approximately
approximately the
the same
same on
on
each
each of
of the
the log
logbbnn levels.
levels.
Introduction to Algorithms

nlogba (1)
(nlogbalg n)
L2.26

Idea of master theorem

f (n)

a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
C
CASE
ASE 3:
3: The
The weight
weight decreases
decreases
geometrically
geometrically from
from the
the root
root to
to the
the
(1) leaves.
leaves. The
The root
root holds
holds aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight.
Day 3

Introduction to Algorithms

f (n)
a f (n/b)
a2 f (n/b2)

Recursion tree:

nlogba (1)
( f (n))
L2.27

Conclusion

Next time: applying the master method.

For proof of master theorem, see CLRS.

Day 3

Introduction to Algorithms

L2.28

Appendix: geometric series

n +1
x

1
for x 1
1 + x + x2 + L + xn =
1 x

1
1+ x + x +L =
for |x| < 1
1 x
2

Return to last
slide viewed.

Day 3

Introduction to Algorithms

L2.29

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 3
Prof. Erik Demaine

The divide-and-conquer
design paradigm
1. Divide the problem (instance)
into subproblems.
2. Conquer the subproblems by
solving them recursively.
3. Combine subproblem solutions.

Day 4

Introduction to Algorithms

L3.2

Example: merge sort

1. Divide: Trivial.
2. Conquer: Recursively sort 2 subarrays.
3. Combine: Linear-time merge.
T(n) = 2 T(n/2) + O(n)
# subproblems
subproblem size
Day 4

Introduction to Algorithms

work dividing
and combining
L3.3

Master theorem (reprise)

T(n) = a T(n/b) + f (n)
CASE 1: f (n) = O(nlogba )
T(n) = (nlogba) .
CASE 2: f (n) = (nlogba lgkn)
T(n) = (nlogba lgk+1n) .
CASE 3: f (n) = (nlogba + ) and a f (n/b) c f (n)
T(n) = ( f (n)) .
Merge sort: a = 2, b = 2 nlogba = n
CASE 2 (k = 0) T(n) = (n lg n) .
Day 4

Introduction to Algorithms

L3.4

Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
Day 4

Introduction to Algorithms

12

15
L3.5

Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
Day 4

Introduction to Algorithms

12

15
L3.6

Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
Day 4

Introduction to Algorithms

12

15
L3.7

Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
Day 4

Introduction to Algorithms

12

15
L3.8

Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
Day 4

Introduction to Algorithms

12

15
L3.9

Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
Day 4

Introduction to Algorithms

12

15
L3.10

Recurrence for binary search

T(n) = 1 T(n/2) + (1)
# subproblems
subproblem size

work dividing
and combining

nlogba = nlog21 = n0 = 1 CASE 2 (k = 0)

T(n) = (lg n) .
Day 4

Introduction to Algorithms

L3.11

Powering a number
Problem: Compute a n, where n N.
Naive algorithm: (n).
Divide-and-conquer algorithm:
an

a n/2 a n/2
a (n1)/2 a (n1)/2 a

if n is even;
if n is odd.

T(n) = T(n/2) + (1) T(n) = (lg n) .

Day 4

Introduction to Algorithms

L3.12

Fibonacci numbers
Recursive definition:
0
if n = 0;
if n = 1;
Fn = 1
Fn1 + Fn2 if n 2.
0

8 13 21 34 L

Naive recursive algorithm: ( n)

(exponential time), where = (1 + 5) / 2
is the golden ratio.
Day 4

Introduction to Algorithms

L3.13

Computing Fibonacci
numbers
Naive recursive squaring:
Fn = n/ 5 rounded to the nearest integer.
Recursive squaring: (lg n) time.
This method is unreliable, since floating-point
arithmetic is prone to round-off errors.
Bottom-up:
Compute F0, F1, F2, , Fn in order, forming
each number by summing the two previous.
Running time: (n).
Day 4

Introduction to Algorithms

L3.14

Recursive squaring
Fn +1
Theorem:
Fn

Fn 1 1
.
=

Fn 1 1 0

Algorithm: Recursive squaring.

Time = (lg n) .

Proof of theorem. (Induction on n.)

1
F
F
2
1 1 1
Base (n = 1):
.
=

F1 F0 1 0
Day 4

Introduction to Algorithms

L3.15

Recursive squaring
Inductive step (n 2):

Fn +1
F
n

Day 4

.
Fn Fn
Fn 1 1 1

Fn 1 Fn 1 Fn 2 1 0
n1
1 1
1 1
=

1 0
1 0
n
1 1
=

1
0

Introduction to Algorithms

L3.16

Matrix multiplication
Input: A = [aij], B = [bij].
Output: C = [cij] = A B.
c11 c12
c c
21 22
M M
c c
n1 n 2

L c1n a11 a12

L c2 n a21 a22
=
M
O M M
L cnn an1 an 2

i, j = 1, 2, , n.

L a1n b11 b12

L a2 n b21 b22

O M M M
L ann bn1 bn 2

L b1n
L b2 n

O M
L bnn

cij = aik bkj

k =1

Day 4

Introduction to Algorithms

L3.17

Standard algorithm
for i 1 to n
do for j 1 to n
do cij 0
for k 1 to n
do cij cij + aik bkj

Running time = (n3)

Day 4

Introduction to Algorithms

L3.18

Divide-and-conquer algorithm
IDEA:
nn matrix = 22 matrix of (n/2)(n/2) submatrices:
r s a b e f
t u = c d g h

C
r
s
t
u

= ae + bg
= af + bh
= ce + dh
= cf + dg
Day 4

8 mults of (n/2)(n/2) submatrices

4 adds of (n/2)(n/2) submatrices
Introduction to Algorithms

L3.19

Analysis of D&C algorithm

T(n) = 8 T(n/2) + (n2)
# submatrices
submatrix size

work adding
submatrices

nlogba = nlog28 = n3 CASE 1 T(n) = (n3).

No better than the ordinary algorithm.
Day 4

Introduction to Algorithms

L3.20

Strassens idea
Multiply 22 matrices with only 7 recursive mults.
P1 = a ( f h)
P2 = (a + b) h
P3 = (c + d) e
P4 = d (g e)
P5 = (a + d) (e + h)
P6 = (b d) (g + h)
P7 = (a c) (e + f )
Day 4

r
s
t
u

= P5 + P4 P2 + P6
= P1 + P2
= P3 + P4
= P5 + P1 P3 P7

77 mults,
mults, 18
18 adds/subs.
adds/subs.
Note:
Note: No
No reliance
reliance on
on
commutativity
commutativity of
of mult!
mult!

Introduction to Algorithms

L3.21

Strassens idea
Multiply 22 matrices with only 7 recursive mults.
P1 = a ( f h)
P2 = (a + b) h
P3 = (c + d) e
P4 = d (g e)
P5 = (a + d) (e + h)
P6 = (b d) (g + h)
P7 = (a c) (e + f )
Day 4

r = P5 + P4 P2 + P6
= (a + d) (e + h)
+ d (g e) (a + b) h
+ (b d) (g + h)
= ae + ah + de + dh
+ dg de ah bh
+ bg + bh dg dh
= ae + bg

Introduction to Algorithms

L3.22

Strassens algorithm
1. Divide: Partition A and B into
(n/2)(n/2) submatrices. Form terms
to be multiplied using + and .
2. Conquer: Perform 7 multiplications of
(n/2)(n/2) submatrices recursively.
3. Combine: Form C using + and on
(n/2)(n/2) submatrices.

T(n) = 7 T(n/2) + (n2)

Day 4

Introduction to Algorithms

L3.23

Analysis of Strassen
T(n) = 7 T(n/2) + (n2)
nlogba = nlog27 n2.81 CASE 1 T(n) = (nlg 7).
The number 2.81 may not seem much smaller than
3, but because the difference is in the exponent, the
impact on running time is significant. In fact,
Strassens algorithm beats the ordinary algorithm
on todays machines for n 30 or so.
Best to date (of theoretical interest only): (n2.376L).
Day 4

Introduction to Algorithms

L3.24

VLSI layout
Problem: Embed a complete binary tree
with n leaves in a grid using minimal area.

W(n)
H(n)
H(n) = H(n/2) + (1) W(n) = 2 W(n/2) + (1)
= (lg n)
= (n)
Area = (n lg n)
Day 4

Introduction to Algorithms

L3.25

H-tree embedding
L(n)
L(n) = 2 L(n/4) + (1)
= ( n )

L(n)

Area = (n)
L(n/4) (1) L(n/4)
Day 4

Introduction to Algorithms

L3.26

Conclusion
Divide and conquer is just one of several
powerful techniques for algorithm design.
Divide-and-conquer algorithms can be
analyzed using recurrences and the master
method (so practice this math).
Can lead to more efficient algorithms

Day 4

Introduction to Algorithms

L3.27

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 4
Prof. Charles E. Leiserson

Quicksort
Proposed by C.A.R. Hoare in 1962.
Divide-and-conquer algorithm.
Sorts in place (like insertion sort, but not
like merge sort).
Very practical (with tuning).

Day 6

Introduction to Algorithms

L4.2

Divide and conquer

Quicksort an n-element array:
1. Divide: Partition the array into two subarrays
around a pivot x such that elements in lower
subarray x elements in upper subarray.
xx
xx
xx
2. Conquer: Recursively sort the two subarrays.
3. Combine: Trivial.
Key: Linear-time partitioning subroutine.
Day 6

Introduction to Algorithms

L4.3

Partitioning subroutine
PARTITION(A, p, q) A[ p . . q]
x A[ p]
pivot = A[ p]
Running
Running time
time
ip
== O(n)
O(n) for
for nn
for j p + 1 to q
elements.
elements.
do if A[ j] x
then i i + 1
exchange A[i] A[ j]
exchange A[ p] A[i]
return i

Invariant: xx
p
Day 6

xx

xx
i

Introduction to Algorithms

??
j

q
L4.4

Example of partitioning
66 10
10 13
13 55
i
j

Day 6

88

Introduction to Algorithms

33

22 11
11

L4.5

Example of partitioning
66 10
10 13
13 55
i
j

Day 6

88

Introduction to Algorithms

33

22 11
11

L4.6

Example of partitioning
66 10
10 13
13 55
i
j

Day 6

88

Introduction to Algorithms

33

22 11
11

L4.7

Example of partitioning
66 10
10 13
13 55
66

Day 6

88

33

22 11
11

55 13
13 10
10 88
i
j

33

22 11
11

Introduction to Algorithms

L4.8

Example of partitioning
66 10
10 13
13 55
66

Day 6

88

33

22 11
11

55 13
13 10
10 88
i
j

33

22 11
11

Introduction to Algorithms

L4.9

Example of partitioning
66 10
10 13
13 55
66

Day 6

88

33

22 11
11

55 13
13 10
10 88
i

33
j

22 11
11

Introduction to Algorithms

L4.10

Example of partitioning
66 10
10 13
13 55

Day 6

88

33

22 11
11

66

55 13
13 10
10 88

33

22 11
11

66

55

33 10
10 88 13
13 22 11
11
i
j

Introduction to Algorithms

L4.11

Example of partitioning
66 10
10 13
13 55

Day 6

88

33

22 11
11

66

55 13
13 10
10 88

33

22 11
11

66

55

33 10
10 88 13
13 22 11
11
i
j

Introduction to Algorithms

L4.12

Example of partitioning
66 10
10 13
13 55

Day 6

88

33

22 11
11

66

55 13
13 10
10 88

33

22 11
11

66

55

33 10
10 88 13
13 22 11
11

66

55

33

22
i

88 13
13 10
10 11
11
j

Introduction to Algorithms

L4.13

Example of partitioning
66 10
10 13
13 55

Day 6

88

33

22 11
11

66

55 13
13 10
10 88

33

22 11
11

66

55

33 10
10 88 13
13 22 11
11

66

55

33

22
i

88 13
13 10
10 11
11
j

Introduction to Algorithms

L4.14

Example of partitioning
66 10
10 13
13 55

Day 6

88

33

22 11
11

66

55 13
13 10
10 88

33

22 11
11

66

55

33 10
10 88 13
13 22 11
11

66

55

33

22
i

88 13
13 10
10 11
11

Introduction to Algorithms

L4.15

Example of partitioning
66 10
10 13
13 55

Day 6

88

33

22 11
11

66

55 13
13 10
10 88

33

22 11
11

66

55

33 10
10 88 13
13 22 11
11

66

55

33

22

88 13
13 10
10 11
11

22

55

33

66
i

88 13
13 10
10 11
11

Introduction to Algorithms

L4.16

Pseudocode for quicksort

QUICKSORT(A, p, r)
if p < r
then q PARTITION(A, p, r)
QUICKSORT(A, p, q1)
QUICKSORT(A, q+1, r)
Initial call: QUICKSORT(A, 1, n)

Day 6

Introduction to Algorithms

L4.17

Analysis of quicksort
Assume all input elements are distinct.
In practice, there are better partitioning
algorithms for when duplicate input
elements may exist.
Let T(n) = worst-case running time on
an array of n elements.

Day 6

Introduction to Algorithms

L4.18

Worst-case of quicksort
Input sorted or reverse sorted.
Partition around min or max element.
One side of partition always has no elements.

T (n) = T (0) + T (n 1) + (n)

= (1) + T (n 1) + (n)
= T (n 1) + (n)
= ( n 2 )
Day 6

(arithmetic series)

Introduction to Algorithms

L4.19

Worst-case recursion tree

T(n) = T(0) + T(n1) + cn

Day 6

Introduction to Algorithms

L4.20

Worst-case recursion tree

T(n) = T(0) + T(n1) + cn
T(n)

Day 6

Introduction to Algorithms

L4.21

Worst-case recursion tree

T(n) = T(0) + T(n1) + cn
cn
T(0) T(n1)

Day 6

Introduction to Algorithms

L4.22

Worst-case recursion tree

T(n) = T(0) + T(n1) + cn
cn
T(0) c(n1)
T(0) T(n2)

Day 6

Introduction to Algorithms

L4.23

Worst-case recursion tree

T(n) = T(0) + T(n1) + cn
cn
T(0) c(n1)
T(0) c(n2)
T(0)

O
(1)

Day 6

Introduction to Algorithms

L4.24

Worst-case recursion tree

T(n) = T(0) + T(n1) + cn
cn
T(0) c(n1)
T(0) c(n2)
T(0)

n
k = (n 2 )
k =1
O
(1)

Day 6

Introduction to Algorithms

L4.25

Worst-case recursion tree

T(n) = T(0) + T(n1) + cn
cn
(1) c(n1)
h=n

(1) c(n2)
(1)

n
k = (n 2 )
k =1
T(n) = (n) + (n2)
= (n2)

(1)
Day 6

Introduction to Algorithms

L4.26

Best-case analysis
(For intuition only!)

If were lucky, PARTITION splits the array evenly:

T(n) = 2T(n/2) + (n)
(same as merge sort)
= (n lg n)
What if the split is

1 9
always 10 : 10 ?

T (n) = T (101 n ) + T (109 n ) + (n)

What is the solution to this recurrence?
Day 6

Introduction to Algorithms

L4.27

Analysis of almost-best case

T (n)

Day 6

Introduction to Algorithms

L4.28

Analysis of almost-best case

cn

T (101 n )

Day 6

T (109 n )

Introduction to Algorithms

L4.29

Analysis of almost-best case

cn
1
10

cn

9
10

cn

9
9
81
1
T (100
n ) T (100
n ) T (100
n )T (100
n)

Day 6

Introduction to Algorithms

L4.30

Analysis of almost-best case

1
10

cn

9
100

9
10

cn

cn

log10/9n
9
81
cn
cn
100
100

(1)

O(n)
O(n) leaves
leaves

1
100

cn

cn
cn
cn

cn

(1)
Day 6

Introduction to Algorithms

L4.31

Analysis of almost-best case

1
10

9
100

9
10

cn

cn

log10/9n
9
81
cn
cn
100
100

(1)
(n lg n)
Lucky!
Day 6

O(n)
O(n) leaves
leaves

log10n
1
cn
100

cn

cn
cn
cn

cn

(1)
cn log10n T(n) cn log10/9n + (n)
Introduction to Algorithms

L4.32

More intuition
Suppose we alternate lucky, unlucky,
lucky, unlucky, lucky, .
L(n) = 2U(n/2) + (n) lucky
U(n) = L(n 1) + (n) unlucky
Solving:
L(n) = 2(L(n/2 1) + (n/2)) + (n)
= 2L(n/2 1) + (n)
= (n lg n) Lucky!
How can we make sure we are usually lucky?
Day 6

Introduction to Algorithms

L4.33

Randomized quicksort
IDEA: Partition around a random element.
Running time is independent of the input
order.
No assumptions need to be made about
the input distribution.
No specific input elicits the worst-case
behavior.
The worst case is determined only by the
output of a random-number generator.
Day 6

Introduction to Algorithms

L4.34

Randomized quicksort
analysis
Let T(n) = the random variable for the running
time of randomized quicksort on an input of size
n, assuming random numbers are independent.
For k = 0, 1, , n1, define the indicator
random variable
Xk =

1 if PARTITION generates a k : nk1 split,

0 otherwise.

E[Xk] = Pr{Xk = 1} = 1/n, since all splits are

equally likely, assuming elements are distinct.
Day 6

Introduction to Algorithms

L4.35

Analysis (continued)
T(n) =

T(0) + T(n1) + (n) if 0 : n1 split,

T(1) + T(n2) + (n) if 1 : n2 split,
M
T(n1) + T(0) + (n) if n1 : 0 split,
n 1

X k (T (k ) + T (n k 1) + (n)) .

k =0

Day 6

Introduction to Algorithms

L4.36

Calculating expectation
n 1

E[T (n)] = E X k (T (k ) + T (n k 1) + (n) )

k =0

Take expectations of both sides.

Day 6

Introduction to Algorithms

L4.37

Calculating expectation
n 1

E[T (n)] = E X k (T (k ) + T (n k 1) + (n) )

k =0

n 1

E[ X k (T (k ) + T (n k 1) + (n) )]

k =0

Linearity of expectation.

Day 6

Introduction to Algorithms

L4.38

Calculating expectation
n 1

E[T (n)] = E X k (T (k ) + T (n k 1) + (n) )

k =0

=
=

n 1

E[ X k (T (k ) + T (n k 1) + (n) )]

k =0
n 1

E[ X k ] E[T (k ) + T (n k 1) + (n)]

k =0

Independence of Xk from other random

choices.

Day 6

Introduction to Algorithms

L4.39

Calculating expectation
n 1

E[T (n)] = E X k (T (k ) + T (n k 1) + (n) )

k =0

=
=

n 1

E[ X k (T (k ) + T (n k 1) + (n) )]

k =0
n 1

E[ X k ] E[T (k ) + T (n k 1) + (n)]

k =0
n 1

n 1

n 1

= 1 E [T (k )] + 1 E [T (n k 1)] + 1 (n)
n k =0
n k =0
n k =0

Linearity of expectation; E[Xk] = 1/n .

Day 6

Introduction to Algorithms

L4.40

Calculating expectation

n 1
E[T (n)] = E X k (T (k ) + T ( n k 1) + (n) )

k =0
=
=

n 1

E[ X k (T (k ) + T (n k 1) + (n) )]

k =0
n 1

E[ X k ] E[T (k ) + T (n k 1) + (n)]

k =0
n 1

n 1

n 1

= 1 E [T (k )] + 1 E [T (n k 1)] + 1 (n)
n k =0
n k =0
n k =0
n 1

= 2 E [T (k )] + (n)
n k =1
Day 6

Summations have
identical terms.

Introduction to Algorithms

L4.41

Hairy recurrence
n 1

E[T (n)] = 2 E [T (k )] + (n)

n k =2
(The k = 0, 1 terms can be absorbed in the (n).)
Prove: E[T(n)] an lg n for constant a > 0 .
Choose a large enough so that an lg n
dominates E[T(n)] for sufficiently small n 2.
n 1

Use fact:

1 n 2 lg n 1n 2
k
lg
k

(exercise).

2
8

k =2
Day 6

Introduction to Algorithms

L4.42

Substitution method
n 1

E [T (n)] 2 ak lg k + (n)
n k =2
Substitute inductive hypothesis.

Day 6

Introduction to Algorithms

L4.43

Substitution method
n 1

E [T (n)] 2 ak lg k + (n)
n k =2
2a 1 n 2 lg n 1 n 2 + (n)
n 2
8
Use fact.

Day 6

Introduction to Algorithms

L4.44

Substitution method
n 1

E [T (n)] 2 ak lg k + (n)
n k =2
2a 1 n 2 lg n 1 n 2 + (n)
n 2
8
= an lg n an (n)
4

Express as desired residual.

Day 6

Introduction to Algorithms

L4.45

Substitution method
n 1

E [T (n)] 2 ak lg k + (n)
n k =2
= 2a 1 n 2 lg n 1 n 2 + (n)
n 2
8
= an lg n an (n)
4

an lg n ,
if a is chosen large enough so that
an/4 dominates the (n).
Day 6

Introduction to Algorithms

L4.46

Quicksort in practice
Quicksort is a great general-purpose
sorting algorithm.
Quicksort is typically over twice as fast
as merge sort.
Quicksort can benefit substantially from
code tuning.
Quicksort behaves well even with
caching and virtual memory.
Day 6

Introduction to Algorithms

L4.47

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 5
Prof. Erik Demaine

How fast can we sort?

All the sorting algorithms we have seen so far
are comparison sorts: only use comparisons to
determine the relative order of elements.
E.g., insertion sort, merge sort, quicksort,
heapsort.
The best worst-case running time that weve
seen for comparison sorting is O(n lg n) .
Is O(n lg n) the best we can do?
Decision trees can help us answer this question.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.2

Decision-tree example
Sort a1, a2, , an

1:2
1:2

2:3
2:3
123
123

1:3
1:3
213
213

1:3
1:3
132
132

312
312

2:3
2:3
231
231

321
321

Each internal node is labeled i:j for i, j {1, 2,, n}.

The left subtree shows subsequent comparisons if ai aj.
The right subtree shows subsequent comparisons if ai aj.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.3

Decision-tree example
Sort a1, a2, a3
= 9, 4, 6 :

1:2
1:2

94

2:3
2:3

123
123

1:3
1:3
213
213

1:3
1:3
132
132

312
312

2:3
2:3
231
231

321
321

Each internal node is labeled i:j for i, j {1, 2,, n}.

The left subtree shows subsequent comparisons if ai aj.
The right subtree shows subsequent comparisons if ai aj.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.4

Decision-tree example
Sort a1, a2, a3
= 9, 4, 6 :

1:2
1:2
2:3
2:3

123
123

1:3
1:3
213
213

1:3
1:3
132
132

312
312

96
2:3
2:3

231
231

321
321

Each internal node is labeled i:j for i, j {1, 2,, n}.

The left subtree shows subsequent comparisons if ai aj.
The right subtree shows subsequent comparisons if ai aj.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.5

Decision-tree example
Sort a1, a2, a3
= 9, 4, 6 :

1:2
1:2
2:3
2:3

123
123

1:3
1:3
213
213 4 6 2:3
2:3

1:3
1:3
132
132

312
312

231
231

321
321

Each internal node is labeled i:j for i, j {1, 2,, n}.

The left subtree shows subsequent comparisons if ai aj.
The right subtree shows subsequent comparisons if ai aj.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.6

Decision-tree example
Sort a1, a2, a3
= 9, 4, 6 :

1:2
1:2
2:3
2:3

123
123

1:3
1:3
213
213

1:3
1:3
132
132

312
312

2:3
2:3
231
231

321
321

469
Each leaf contains a permutation (1), (2),, (n) to
indicate that the ordering a(1) a(2) L a(n) has been
established.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.7

Decision-tree model
A decision tree can model the execution of
any comparison sort:
One tree for each input size n.
View the algorithm as splitting whenever
it compares two elements.
The tree contains the comparisons along
all possible instruction traces.
The running time of the algorithm = the
length of the path taken.
Worst-case running time = height of tree.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.8

Lower bound for decisiontree sorting

Theorem. Any decision tree that can sort n
elements must have height (n lg n) .
Proof. The tree must contain n! leaves, since
there are n! possible permutations. A height-h
binary tree has 2h leaves. Thus, n! 2h .
h lg(n!)
(lg is mono. increasing)
(Stirlings formula)
lg ((n/e)n)
= n lg n n lg e
= (n lg n) .
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.9

Lower bound for comparison

sorting
Corollary. Heapsort and merge sort are
asymptotically optimal comparison sorting
algorithms.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.10

Sorting in linear time

Counting sort: No comparisons between elements.
Input: A[1 . . n], where A[ j]{1, 2, , k} .
Output: B[1 . . n], sorted.
Auxiliary storage: C[1 . . k] .

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.11

Counting sort
for i 1 to k
do C[i] 0
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
for i 2 to k
do C[i] C[i] + C[i1]
C[i] = |{key i}|
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.12

Counting-sort example
A:

44

11

33

44

33

C:

B:

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.13

Loop 1
A:

44

11

33

44

33

C:

00

00

00

00

B:
for i 1 to k
do C[i] 0
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.14

Loop 2
A:

44

11

33

44

33

C:

00

00

00

11

B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.15

Loop 2
A:

44

11

33

44

33

C:

11

00

00

11

B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.16

Loop 2
A:

44

11

33

44

33

C:

11

00

11

11

B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.17

Loop 2
A:

44

11

33

44

33

C:

11

00

11

22

B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.18

Loop 2
A:

44

11

33

44

33

C:

11

00

22

22

B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.19

Loop 3
A:

44

11

33

44

33

B:

C:

11

00

22

22

C':

11

11

22

22

for i 2 to k
do C[i] C[i] + C[i1]
2001 by Charles E. Leiserson

Introduction to Algorithms

C[i] = |{key i}|

Day 8

L5.20

Loop 3
A:

44

11

33

44

33

B:

C:

11

00

22

22

C':

11

11

33

22

for i 2 to k
do C[i] C[i] + C[i1]
2001 by Charles E. Leiserson

Introduction to Algorithms

C[i] = |{key i}|

Day 8

L5.21

Loop 3
A:

44

11

33

44

33

B:

C:

11

00

22

22

C':

11

11

33

55

for i 2 to k
do C[i] C[i] + C[i1]
2001 by Charles E. Leiserson

Introduction to Algorithms

C[i] = |{key i}|

Day 8

L5.22

Loop 4
A:
B:

44

11

33

44

33

33

C:

11

11

33

55

C':

11

11

22

55

for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.23

Loop 4
A:
B:

44

11

33

44

33
44

33

C:

11

11

22

55

C':

11

11

22

44

for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.24

Loop 4
A:
B:

44

11

33

44

33

33

33

44

C:

11

11

22

44

C':

11

11

11

44

for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.25

Loop 4
1

A:

44

11

33

44

33

B:

11

33

33

44

C:

11

11

11

44

C':

00

11

11

44

for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.26

Loop 4
1

A:

44

11

33

44

33

B:

11

33

33

44

44

C:

00

11

11

44

C':

00

11

11

33

for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.27

Analysis
(k)
(n)
(k)
(n)

for i 1 to k
do C[i] 0
for j 1 to n
do C[A[ j]] C[A[ j]] + 1
for i 2 to k
do C[i] C[i] + C[i1]
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1

(n + k)
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.28

Running time
If k = O(n), then counting sort takes (n) time.
But, sorting takes (n lg n) time!
Wheres the fallacy?
Answer:
Comparison sorting takes (n lg n) time.
Counting sort is not a comparison sort.
In fact, not a single comparison between
elements occurs!
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.29

Stable sorting
Counting sort is a stable sort: it preserves
the input order among equal elements.
A:

44

11

33

44

33

B:

11

33

33

44

44

Exercise: What other sorts have this property?

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.30

Radix sort
Origin: Herman Holleriths card-sorting
machine for the 1890 U.S. Census. (See
Appendix .)
Digit-by-digit sort.
Holleriths original (bad) idea: sort on
most-significant digit first.
Good idea: Sort on least-significant digit
first with auxiliary stable sort.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.31

Operation of radix sort

329
457
657
839
436
720
355
2001 by Charles E. Leiserson

720
355
436
457
657
329
839

720
329
436
839
355
457
657

Introduction to Algorithms

329
355
436
457
657
720
839
Day 8

L5.32

Correctness of radix sort

Induction on digit position
Assume that the numbers
are sorted by their low-order
t 1 digits.
Sort on digit t

2001 by Charles E. Leiserson

Introduction to Algorithms

720
329
436
839
355
457
657

329
355
436
457
657
720
839
Day 8

L5.33

Correctness of radix sort

Induction on digit position
Assume that the numbers
are sorted by their low-order
t 1 digits.
Sort on digit t
Two numbers that differ in
digit t are correctly sorted.

2001 by Charles E. Leiserson

Introduction to Algorithms

720
329
436
839
355
457
657

329
355
436
457
657
720
839
Day 8

L5.34

Correctness of radix sort

Induction on digit position
Assume that the numbers
are sorted by their low-order
t 1 digits.
Sort on digit t
Two numbers that differ in
digit t are correctly sorted.
Two numbers equal in digit t
are put in the same order as
the input correct order.
2001 by Charles E. Leiserson

Introduction to Algorithms

720
329
436
839
355
457
657

329
355
436
457
657
720
839
Day 8

L5.35

Analysis of radix sort

Assume counting sort is the auxiliary stable sort.
Sort n computer words of b bits each.
Each word can be viewed as having b/r base-2r
digits.
8
8
8
8
Example: 32-bit word
r = 8 b/r = 4 passes of counting sort on
base-28 digits; or r = 16 b/r = 2 passes of
counting sort on base-216 digits.
How many passes should we make?
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.36

Analysis (continued)
Recall: Counting sort takes (n + k) time to
sort n numbers in the range from 0 to k 1.
If each b-bit word is broken into r-bit pieces,
each pass of counting sort takes (n + 2r) time.
Since there are b/r passes, we have

T (n, b) = b (n + 2 r ) .

r
Choose r to minimize T(n, b):
Increasing r means fewer passes, but as
r >> lg n, the time grows exponentially.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.37

Choosing r
T (n, b) = b (n + 2 r )

r
Minimize T(n, b) by differentiating and setting to 0.
Or, just observe that we dont want 2r >> n, and
theres no harm asymptotically in choosing r as
large as possible subject to this constraint.
Choosing r = lg n implies T(n, b) = (bn/lg n) .
For numbers in the range from 0 to n d 1, we
have b = d lg n radix sort runs in (d n) time.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.38

Conclusions
In practice, radix sort is fast for large inputs, as
well as simple to code and maintain.
Example (32-bit numbers):
At most 3 passes when sorting 2000 numbers.
Merge sort and quicksort do at least lg 2000 =
11 passes.
Downside: Unlike quicksort, radix sort displays
little locality of reference, and thus a well-tuned
quicksort fares better on modern processors,
which feature steep memory hierarchies.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.39

Appendix: Punched-card
technology
Herman Hollerith (1860-1929)
Punched cards
Holleriths tabulating system
Operation of the sorter
Origin of radix sort
Modern IBM card
Web resources on punchedcard technology
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.40

Herman Hollerith
(1860-1929)
The 1880 U.S. Census took almost
10 years to process.
While a lecturer at MIT, Hollerith
prototyped punched-card technology.
His machines, including a card sorter, allowed
the 1890 census total to be reported in 6 weeks.
He founded the Tabulating Machine Company in
1911, which merged with other companies in 1924
to form International Business Machines.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.41

Punched cards
Punched card = data record.
Hole = value.
Algorithm = machine + human operator.
Replica of punch
card from the
1900 U.S. census:
[Howells 2000]

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.42

Holleriths
tabulating
system

See figure from

[Howells 2000] .

Pantograph card
punch
Hand-press reader
Dial counters
Sorting box

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.43

Operation of the sorter

An operator inserts a card into
the press.
Pins on the press reach through
the punched holes to make
electrical contact with mercuryfilled cups beneath the card.
Whenever a particular digit
value is punched, the lid of the
corresponding sorting bin lifts.
The operator deposits the card
into the bin and closes the lid.
When all cards have been processed, the front panel is opened, and
the cards are collected in order, yielding one pass of a stable sort.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.44

Origin of radix sort

Holleriths original 1889 patent alludes to a mostsignificant-digit-first radix sort:
The most complicated combinations can readily be
counted with comparatively few counters or relays by first
assorting the cards according to the first items entering
into the combinations, then reassorting each group
according to the second item entering into the combination,
and so on, and finally counting on a few counters the last
item of the combination for each group of cards.

Least-significant-digit-first radix sort seems to be

a folk invention originated by machine operators.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.45

Modern IBM card

One character per column.

See examples on the WWW Virtual Punch-Card Server.

.

So, thats why text windows have 80 columns!

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.46

Web resources on punchedcard technology

Doug Joness punched card index
Biography of Herman Hollerith
The 1890 U.S. Census
Early history of IBM
Pictures of Holleriths inventions
Holleriths patent application (borrowed
from Gordon Bells CyberMuseum)
Impact of punched cards on U.S. history
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 8

L5.47

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 6
Prof. Erik Demaine

Order statistics
Select the ith smallest of n elements (the
element with rank i).
i = 1: minimum;
i = n: maximum;
i = (n+1)/2 or (n+1)/2: median.
Naive algorithm: Sort and index ith element.
Worst-case running time = (n lg n) + (1)
= (n lg n),
using merge sort or heapsort (not quicksort).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.2

Randomized divide-andconquer algorithm

RAND-SELECT(A, p, q, i) ith smallest of A[ p . . q]
if p = q then return A[ p]
r RAND-PARTITION(A, p, q)
krp+1
k = rank(A[r])
if i = k then return A[ r]
if i < k
then return RAND-SELECT(A, p, r 1, i )
else return RAND-SELECT(A, r + 1, q, i k )
k
A[r]
A[r]
A[r]
A[r]
p
r
q
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.3

Example
Select the i = 7th smallest:
66 10
10 13
13 55
pivot
Partition:
22 55

33

66

88

33

22 11
11

i=7

88 13
13 10
10 11
11

k=4

Select the 7 4 = 3rd smallest recursively.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.4

Intuition for analysis

(All our analyses today assume that all elements
are distinct.)
Lucky:
n log10 / 9 1 = n 0 = 1
T(n) = T(9n/10) + (n)
CASE 3
= (n)
Unlucky:
arithmetic series
T(n) = T(n 1) + (n)
= (n2)
Worse than sorting!
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.5

Analysis of expected time

The analysis follows that of randomized
quicksort, but its a little different.
Let T(n) = the random variable for the running
time of RAND-SELECT on an input of size n,
assuming random numbers are independent.
For k = 0, 1, , n1, define the indicator
random variable
1 if PARTITION generates a k : nk1 split,
Xk =
0 otherwise.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.6

Analysis (continued)
To obtain an upper bound, assume that the ith
element always falls in the larger side of the
partition:
T(max{0, n1}) + (n) if 0 : n1 split,
T(max{1, n2}) + (n) if 1 : n2 split,
T(n) =
M
T(max{n1, 0}) + (n) if n1 : 0 split,
=

n 1

X k (T (max{k , n k 1}) + (n)) .

k =0

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.7

Calculating expectation
n 1

E[T (n)] = E X k (T (max{k , n k 1}) + (n) )

k =0

Take expectations of both sides.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.8

Calculating expectation
n 1

E[T (n)] = E X k (T (max{k , n k 1}) + (n) )

k =0

n 1

E[ X k (T (max{k , n k 1}) + (n) )]

k =0

Linearity of expectation.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.9

Calculating expectation
n 1

E[T (n)] = E X k (T (max{k , n k 1}) + (n) )

k =0

=
=

n 1

E[ X k (T (max{k , n k 1}) + (n) )]

k =0
n 1

E[ X k ] E[T (max{k , n k 1}) + (n)]

k =0

Independence of Xk from other random

choices.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.10

Calculating expectation
n 1

E[T (n)] = E X k (T (max{k , n k 1}) + (n) )

k =0

=
=

n 1

E[ X k (T (max{k , n k 1}) + (n) )]

k =0
n 1

E[ X k ] E[T (max{k , n k 1}) + (n)]

k =0
n 1

n 1

= 1 E [T (max{k , n k 1})] + 1 (n)

n k =0
n k =0

Linearity of expectation; E[Xk] = 1/n .

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.11

Calculating expectation

n 1
E[T (n)] = E X k (T (max{k , n k 1}) + (n) )

k =0
=
=

n 1

E[ X k (T (max{k , n k 1}) + (n) )]

k =0
n 1

E[ X k ] E[T (max{k , n k 1}) + (n)]

k =0
n 1

n 1

= 1 E [T (max{k , n k 1})] + 1 (n)

n k =0
n k =0
n 1

2 E [T (k )] + (n)
n k = n / 2
2001 by Charles E. Leiserson

Upper terms
appear twice.

Introduction to Algorithms

Day 9

L6.12

Hairy recurrence
(But not quite as hairy as the quicksort one.)
n 1

E[T (n)] = 2 E [T (k )] + (n)

n k= n/2

Prove: E[T(n)] cn for constant c > 0 .
The constant c can be chosen large enough
so that E[T(n)] cn for the base cases.
n 1

3n 2
k

8 (exercise).
Use fact:
k = n / 2
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.13

Substitution method
n 1

E [T (n)] 2 ck + (n)
n k= n/2

Substitute inductive hypothesis.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.14

Substitution method
n 1

E [T (n)] 2 ck + (n)
n k= n/2

2c 3 n 2 + (n)
n 8
Use fact.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.15

Substitution method
n 1

E [T (n)] 2 ck + (n)
n k= n/2

2c 3 n 2 + (n)
n 8
= cn cn (n)

4
Express as desired residual.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.16

Substitution method
n 1

E [T (n)] 2 ck + (n)
n k= n/2

2c 3 n 2 + (n)
n 8
= cn cn (n)
4

cn ,
if c is chosen large enough so
that cn/4 dominates the (n).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.17

Summary of randomized
order-statistic selection
Works fast: linear expected time.
Excellent algorithm in practice.
But, the worst case is very bad: (n2).
Q. Is there an algorithm that runs in linear
time in the worst case?
A. Yes, due to Blum, Floyd, Pratt, Rivest,
and Tarjan [1973].
IDEA: Generate a good pivot recursively.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.18

Worst-case linear-time order

statistics
SELECT(i, n)
1. Divide the n elements into groups of 5. Find
the median of each 5-element group by rote.
2. Recursively SELECT the median x of the n/5
group medians to be the pivot.
3. Partition around the pivot x. Let k = rank(x).
4. if i = k then return x
elseif i < k
then recursively SELECT the ith
smallest element in the lower part
else recursively SELECT the (ik)th
smallest element in the upper part
2001 by Charles E. Leiserson

Introduction to Algorithms

Same as
RANDSELECT

Day 9

L6.19

Choosing the pivot

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.20

Choosing the pivot

1. Divide the n elements into groups of 5.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.21

Choosing the pivot

1. Divide the n elements into groups of 5. Find lesser

the median of each 5-element group by rote.
greater
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.22

Choosing the pivot

1. Divide the n elements into groups of 5. Find lesser

the median of each 5-element group by rote.
2. Recursively SELECT the median x of the n/5
group medians to be the pivot.
greater
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.23

Analysis

At least half the group medians are x, which

is at least n/5 /2 = n/10 group medians.

lesser

greater
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.24

Analysis

(Assume all elements are distinct.)

At least half the group medians are x, which

is at least n/5 /2 = n/10 group medians.
Therefore, at least 3 n/10 elements are x.

lesser

greater
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.25

Analysis

(Assume all elements are distinct.)

At least half the group medians are x, which

is at least n/5 /2 = n/10 group medians.
Therefore, at least 3 n/10 elements are x.
Similarly, at least 3 n/10 elements are x.
2001 by Charles E. Leiserson

Introduction to Algorithms

lesser

greater
Day 9

L6.26

Minor simplification
For n 50, we have 3 n/10 n/4.
Therefore, for n 50 the recursive call to
SELECT in Step 4 is executed recursively
on 3n/4 elements.
Thus, the recurrence for running time
can assume that Step 4 takes time
T(3n/4) in the worst case.
For n < 50, we know that the worst-case
time is T(n) = (1).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.27

Developing the recurrence

T(n)
(n)
T(n/5)
(n)

T(3n/4)

SELECT(i, n)
1. Divide the n elements into groups of 5. Find
the median of each 5-element group by rote.
2. Recursively SELECT the median x of the n/5
group medians to be the pivot.
3. Partition around the pivot x. Let k = rank(x).
4. if i = k then return x
elseif i < k
then recursively SELECT the ith
smallest element in the lower part
else recursively SELECT the (ik)th
smallest element in the upper part

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.28

Solving the recurrence

T (n) = T 1 n + T 3 n + (n)
4
5
T (n) 1 cn + 3 cn + (n)
4
5
= 19 cn + (n)
20
= cn 1 cn (n)
20

cn ,
if c is chosen large enough to handle both the
(n) and the initial conditions.

Substitution:
T(n) cn

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.29

Conclusions
Since the work at each level of recursion
is a constant fraction (19/20) smaller, the
work per level is a geometric series
dominated by the linear work at the root.
In practice, this algorithm runs slowly,
because the constant in front of n is large.
The randomized algorithm is far more
practical.
Exercise: Why not divide into groups of 3?
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 9

L6.30

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 7
Prof. Charles E. Leiserson

Symbol-table problem
Symbol table T holding n records:
x

record

key[x]
key[x]
Other fields
containing
satellite data

Operations on T:
INSERT(T, x)
DELETE(T, x)
SEARCH(T, k)

How should the data structure T be organized?

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.2

Direct-access table
IDEA: Suppose that the set of keys is K {0,
1, , m1}, and keys are distinct. Set up an
array T[0 . . m1]:
x
if k K and key[x] = k,
T[k] =
NIL otherwise.
Then, operations take (1) time.
Problem: The range of keys can be large:
64-bit numbers (which represent
18,446,744,073,709,551,616 different keys),
character strings (even larger!).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.3

Hash functions
Solution: Use a hash function h to map the
universe U of all keys into
T
{0, 1, , m1}:
0
k1

K
k2

k4

h(k1)
h(k4)

k5

h(k2) = h(k5)

k3

h(k3)

m1

When a record to be inserted maps to an already

As
each key
h maps
it to a slot of T.
occupied
slotisininserted,
T, a collision
occurs.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.4

Resolving collisions by
chaining
Records in the same slot are linked into a list.
T

49
49

86
86

52
52

h(49) = h(86) = h(52) = i

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.5

Analysis of chaining
We make the assumption of simple uniform
hashing:
Each key k K of keys is equally likely to
be hashed to any slot of table T, independent
of where other keys are hashed.
Let n be the number of keys in the table, and
let m be the number of slots.
Define the load factor of T to be
= n/m
= average number of keys per slot.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.6

Search cost
Expected time to search for a record with
a given key = (1 + ).
apply hash
function and
access slot

search
the list

Expected search time = (1) if = O(1),

or equivalently, if n = O(m).

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.7

Choosing a hash function

The assumption of simple uniform hashing
is hard to guarantee, but several common
techniques tend to work well in practice as
long as their deficiencies can be avoided.
Desirata:
A good hash function should distribute the
keys uniformly into the slots of the table.
Regularity in the key distribution should
not affect this uniformity.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.8

Division method
Assume all keys are integers, and define
h(k) = k mod m.
Deficiency: Dont pick an m that has a small
divisor d. A preponderance of keys that are
congruent modulo d can adversely affect
uniformity.
Extreme deficiency: If m = 2r, then the hash
doesnt even depend on all the bits of k:
If k = 10110001110110102 and r = 6, then
h(k) = 0110102 .
h(k)
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.9

Division method (continued)

h(k) = k mod m.
Pick m to be a prime not too close to a power
of 2 or 10 and not otherwise used prominently
in the computing environment.
Annoyance:
Sometimes, making the table size a prime is
inconvenient.
But, this method is popular, although the next
method well see is usually superior.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.10

Multiplication method
Assume that all keys are integers, m = 2r, and our
computer has w-bit words. Define
h(k) = (Ak mod 2w) rsh (w r),
where rsh is the bit-wise right-shift operator
and A is an odd integer in the range 2w1 < A < 2w.
Dont pick A too close to 2w.
Multiplication modulo 2w is fast.
The rsh operator is fast.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.11

Multiplication method
example
h(k) = (Ak mod 2w) rsh (w r)
Suppose that m = 8 = 23 and that our computer
has w = 7-bit words:
1011001 =A

1101011 =k
10010100110011
A
h(k)

0
7 1

5 4 3

Introduction to Algorithms

.
2A

Modular wheel
2001 by Charles E. Leiserson

3A

Day 11

L7.12

Dot-product method
Randomized strategy:
Let m be prime. Decompose key k into r + 1
digits, each with value in the set {0, 1, , m1}.
That is, let k = k0, k1, , km1, where 0 ki < m.
Pick a = a0, a1, , am1 where each ai is chosen
randomly from {0, 1, , m1}.

Define ha (k ) =

ai ki mod m.

i =0

Excellent in practice, but expensive to compute.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.13

Resolving collisions by open

addressing
No storage is used outside of the hash table itself.
Insertion systematically probes the table until an
empty slot is found.
The hash function depends on both the key and
probe number:
h : U {0, 1, , m1} {0, 1, , m1}.
The probe sequence h(k,0), h(k,1), , h(k,m1)
should be a permutation of {0, 1, , m1}.
The table may fill up, and deletion is difficult (but
not impossible).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.14

Example of open addressing

Insert key k = 496:

0. Probe h(496,0)

586
133
204

collision

481
m1

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.15

Example of open addressing

Insert key k = 496:

0. Probe h(496,0)
1. Probe h(496,1)

586
133

collision

204
481
m1

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.16

Example of open addressing

Insert key k = 496:

0. Probe h(496,0)
1. Probe h(496,1)
2. Probe h(496,2)

586
133
204
496
481

insertion
m1

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.17

Example of open addressing

Search for key k = 496:
0. Probe h(496,0)
1. Probe h(496,1)
2. Probe h(496,2)

586
133
204
496
481

Search uses the same probe

sequence, terminating sucm1
cessfully if it finds the key
and unsuccessfully if it encounters an empty slot.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.18

Probing strategies
Linear probing:
Given an ordinary hash function h(k), linear
probing uses the hash function
h(k,i) = (h(k) + i) mod m.
This method, though simple, suffers from primary
clustering, where long runs of occupied slots build
up, increasing the average search time. Moreover,
the long runs of occupied slots tend to get longer.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.19

Probing strategies
Double hashing
Given two ordinary hash functions h1(k) and h2(k),
double hashing uses the hash function
h(k,i) = (h1(k) + ih2(k)) mod m.
This method generally produces excellent results,
but h2(k) must be relatively prime to m. One way
is to make m a power of 2 and design h2(k) to
produce only odd numbers.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.20

Analysis of open addressing

We make the assumption of uniform hashing:
Each key is equally likely to have any one of
the m! permutations as its probe sequence.
Theorem. Given an open-addressed hash
table with load factor = n/m < 1, the
expected number of probes in an unsuccessful
search is at most 1/(1).

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.21

Proof of the theorem

Proof.
At least one probe is always necessary.
With probability n/m, the first probe hits an
occupied slot, and a second probe is necessary.
With probability (n1)/(m1), the second probe
hits an occupied slot, and a third probe is
necessary.
With probability (n2)/(m2), the third probe
hits an occupied slot, etc.

n
i
n
<
= for i = 1, 2, , n.
Observe that
mi m
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.22

Proof (continued)
Therefore, the expected number of probes is

2
n
n
1
n
1
1 + 1 +
L
L 1 +
1 +
m m 1 m 2 m n + 1
1 + (1 + (1 + (L (1 + )L)))
1+ + 2 +3 +L
=

i =0

= 1 .
1
2001 by Charles E. Leiserson

The textbook has a

more rigorous proof.
Introduction to Algorithms

Day 11

L7.23

Implications of the theorem

If is constant, then accessing an openaddressed hash table takes constant time.
If the table is half full, then the expected
number of probes is 1/(10.5) = 2.
If the table is 90% full, then the expected
number of probes is 1/(10.9) = 10.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 11

L7.24

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 8
Prof. Charles E. Leiserson

A weakness of hashing
Problem: For any hash function h, a set
of keys exists that can cause the average
access time of a hash table to skyrocket.
An adversary can pick all keys from
{k U : h(k) = i} for some slot i.
IDEA: Choose the hash function at random,
independently of the keys.
Even if an adversary can see your code,
he or she cannot find a bad set of keys,
since he or she doesnt know exactly
which hash function will be chosen.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 12

L8.2

Universal hashing
Definition. Let U be a universe of keys, and
let H be a finite collection of hash functions,
each mapping U to {0, 1, , m1}. We say
H is universal if for all x, y U, where x y,
we have |{h H : h(x) = h(y)}| = |H|/m.
That is, the chance
of a collision
between x and y is
1/m if we choose h
randomly from H.
2001 by Charles E. Leiserson

{h : h(x) = h(y)}

Introduction to Algorithms

|H|
m
Day 12

L8.3

Universality is good
Theorem. Let h be a hash function chosen
(uniformly) at random from a universal set H

of hash functions. Suppose h is used to hash

n arbitrary keys into the m slots of a table T.
Then, for a given key x, we have
E[#collisions with x] < n/m.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 12

L8.4

Proof of theorem
Proof. Let Cx be the random variable denoting
the total number of collisions of keys in T with
x, and let
1 if h(x) = h(y),
cxy =
0 otherwise.
Note: E[cxy] = 1/m and C x =

2001 by Charles E. Leiserson

Introduction to Algorithms

cxy .

yT {x}

Day 12

L8.5

Proof (continued)

E[C x ] = E c xy
yT { x}

2001 by Charles E. Leiserson

Take expectation
of both sides.

Introduction to Algorithms

Day 12

L8.6

Proof (continued)

E[C x ] = E c xy
yT { x}
=

E[cxy ]

yT { x}

2001 by Charles E. Leiserson

Take expectation
of both sides.
Linearity of
expectation.

Introduction to Algorithms

Day 12

L8.7

Proof (continued)

E[C x ] = E c xy
yT { x}
=

E[cxy ]

Linearity of
expectation.

1/ m

E[cxy] = 1/m.

yT { x}

Take expectation
of both sides.

yT { x}

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 12

L8.8

Proof (continued)

E[C x ] = E c xy
yT { x}
=

E[cxy ]

Linearity of
expectation.

1/ m

E[cxy] = 1/m.

yT { x}

Take expectation
of both sides.

yT { x}

= n 1 .
m
2001 by Charles E. Leiserson

Algebra.
Introduction to Algorithms

Day 12

L8.9

Constructing a set of
universal hash functions
Let m be prime. Decompose key k into r + 1
digits, each with value in the set {0, 1, , m1}.
That is, let k = k0, k1, , kr, where 0 ki < m.
Randomized strategy:
Pick a = a0, a1, , ar where each ai is chosen
randomly from {0, 1, , m1}.
r

Dot
product,
Define ha (k ) = ai ki mod m .
modulo
m
i =0
How big is H = {ha}? |H| = mr + 1. REMEMBER
THIS!

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 12

L8.10

Universality of dot-product
hash functions
Theorem. The set H = {ha} is universal.

Proof. Suppose that x = x0, x1, , xr and y =

y0, y1, , yr be distinct keys. Thus, they differ
in at least one digit position, wlog position 0.
For how many ha Hdo x and y collide?
We must have ha(x) = ha(y), which implies that
r

i =0

i =0

ai xi ai yi
2001 by Charles E. Leiserson

(mod m) .

Introduction to Algorithms

Day 12

L8.11

Proof (continued)
Equivalently, we have
r

ai ( xi yi ) 0

(mod m)

i =0

or
r
a0 ( x0 y0 ) + ai ( xi yi ) 0

(mod m) ,

i =1

which implies that

a0 ( x0 y0 ) ai ( xi yi )

(mod m) .

i =1

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 12

L8.12

Fact from number theory

Theorem. Let m be prime. For any z Zm
such that z 0, there exists a unique z1 Zm
such that
z z1 1 (mod m).
Example: m = 7.

1 2 3 4 5 6

z1

1 4 5 2 3 6

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 12

L8.13

Back to the proof

We have

a0 ( x0 y0 ) ai ( xi yi )

(mod m) ,

i =1

and since x0 y0 , an inverse (x0 y0 )1 must exist,

which implies that

a0 ai ( xi yi ) ( x0 y0 ) 1

i =1

(mod m) .

Thus, for any choices of a1, a2, , ar, exactly

one choice of a0 causes x and y to collide.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 12

L8.14

Proof (completed)
Q. How many has cause x and y to collide?
A. There are m choices for each of a1, a2, , ar ,
but once these are chosen, exactly one choice
for a0 causes x and y to collide, namely

1
a0 = ai ( xi yi ) ( x0 y0 ) mod m .

i =1

Thus, the number

of hras that cause x and y
r
to collide is m 1 = m = |H|/m.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 12

L8.15

Perfect hashing
Given a set of n keys, construct a static hash
table of size m = O(n) such that SEARCH takes
(1) time in the worst case.
IDEA: Twolevel scheme
with universal
hashing at
both levels.
No collisions
at level 2!
2001 by Charles E. Leiserson

T
0
1 44 31
31
2
3
4 11 00
00
5
6 99 86
86
m a

S1
14
27
1427
S4
26
26

h31(14) = h31(27) = 1
S6

40
22
40 37
37
22
0 1 2 3 4 5 6 7 8

Introduction to Algorithms

Day 12

L8.16

Collisions at level 2
Theorem. Let H be a class of universal hash
functions for a table of size m = n2. Then, if we
use a random h H to hash n keys into the table,
the expected number of collisions is at most 1/2.
Proof. By the definition of universality, the
probability that 2 given keys in the table collide
n
2
under h is 1/m = 1/n . Since there are (2 ) pairs
of keys that can possibly collide, the expected
number of collisions is
n 1
n(n 1) 1
2 < 1.
2 =
2
2
n
2 n
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 12

L8.17

No collisions at level 2
Corollary. The probability of no collisions
is at least 1/2.

Proof. Markovs inequality says that for any

nonnegative random variable X, we have
Pr{X t} E[X]/t.
Applying this inequality with t = 1, we find
that the probability of 1 or more collisions is
at most 1/2.
Thus, just by testing random hash functions
in H, well quickly find one that works.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 12

L8.18

Analysis of storage
For the level-1 hash table T, choose m = n, and
let ni be random variable for the number of keys
that hash to slot i in T. By using ni2 slots for the
level-2 hash table Si, the expected total storage
required for the two-level scheme is therefore
m1

2
E (ni ) = (n) ,
i =0

since the analysis is identical to the analysis from

recitation of the expected running time of bucket
sort. (For a probability bound, apply Markov.)
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 12

L8.19

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 9
Prof. Charles E. Leiserson

Binary-search-tree sort
T
Create an empty BST
for i = 1 to n
do TREE-INSERT(T, A[i])
Perform an inorder tree walk of T.
Example:
A = [3 1 8 2 6 7 5]

33
11

Tree-walk time = O(n),

but how long does it
take to build the BST?
2001 by Charles E. Leiserson

88
22

Introduction to Algorithms

66
55

77
Day 17

L9.2

Analysis of BST sort

BST sort performs the same comparisons as
quicksort, but in a different order!
3 1 8 2 6 7 5
1 2

8 6 7 5
2

675
5

The expected time to build the tree is asymptotically the same as the running time of quicksort.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.3

Node depth
The depth of a node = the number of comparisons
made during TREE-INSERT. Assuming all input
permutations are equally likely, we have
Average node depth
n

1
= E (# comparisons to insert node i )
n i =1

= 1 O(n lg n)
n
= O(lg n) .
2001 by Charles E. Leiserson

(quicksort analysis)

Introduction to Algorithms

Day 17

L9.4

Expected tree height

But, average node depth of a randomly built
BST = O(lg n) does not necessarily mean that its
expected height is also O(lg n) (although it is).
Example.

lg n
Ave. depth 1 n lg n + n n
n
2
= O(lg n)
2001 by Charles E. Leiserson

Introduction to Algorithms

h= n

Day 17

L9.5

Height of a randomly built

binary search tree
Outline of the analysis:
Prove Jensens inequality, which says that
f(E[X]) E[f(X)] for any convex function f and
random variable X.
Analyze the exponential height of a randomly
built BST on n nodes, which is the random
variable Yn = 2Xn, where Xn is the random
variable denoting the height of the BST.
Prove that 2E[Xn] E[2Xn ] = E[Yn] = O(n3),
and hence that E[Xn] = O(lg n).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.6

Convex functions
A function f : R R is convex if for all
, 0 such that + = 1, we have
f(x + y) f(x) + f(y)
for all x,y R.
f
f(y)

f(x) + f(y)
f(x)
f(x + y)
x
2001 by Charles E. Leiserson

x + y

Introduction to Algorithms

y
Day 17

L9.7

Convexity lemma
Lemma. Let f : R R be a convex function,
and let {1, 2 , , n} be a set of nonnegative
constants such that k k = 1. Then, for any set
{x1, x2, , xn} of real numbers, we have

n
n
f k xk k f ( xk ) .
k =1
k =1
Proof. By induction on n. For n = 1, we have
1 = 1, and hence f(1x1) 1f(x1) trivially.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.8

Proof (continued)
Inductive step:
n

f k xk =
k =1

n 1

k
f n xn + (1 n )
xk

k =11 n

Algebra.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.9

Proof (continued)
Inductive step:
n 1

k
f n xn + (1 n )
xk

k =11 n
n1 k

n f ( xn ) + (1 n ) f
xk
k =11 n

f k xk =

k =1

Convexity.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.10

Proof (continued)
Inductive step:
n 1

k
f n xn + (1 n )
xk

k =11 n
n1 k

n f ( xn ) + (1 n ) f
xk
k =11 n

n
f k xk =
k =1

n 1

k
n f ( xn ) + (1 n )
f ( xk )
k =11 n

Induction.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.11

Proof (continued)
Inductive step:
n 1

k
f n xn + (1 n )
xk

k =11 n
n1 k

n f ( xn ) + (1 n ) f
xk
k =11 n

n
f k xk =

k =1

n 1

k
n f ( xn ) + (1 n )
f ( xk )
k =11 n
n

= k f ( xk ) .
k =1

2001 by Charles E. Leiserson

Introduction to Algorithms

Algebra.
Day 17

L9.12

Jensens inequality
Lemma. Let f be a convex function, and let X
be a random variable. Then, f (E[X]) E[ f (X)].
Proof.

f ( E[ X ]) = f k Pr{ X = k}

k =

Definition of expectation.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.13

Jensens inequality
Lemma. Let f be a convex function, and let X
be a random variable. Then, f (E[X]) E[ f (X)].
Proof.

f ( E[ X ]) = f k Pr{ X = k}
k =

f (k ) Pr{X = k}

k =

Convexity lemma (generalized).

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.14

Jensens inequality
Lemma. Let f be a convex function, and let X
be a random variable. Then, f (E[X]) E[ f (X)].
Proof.

f ( E[ X ]) = f k Pr{ X = k}

k =

f (k ) Pr{X = k}

k =

= E[ f ( X )] .

Tricky step, but truethink about it.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.15

Analysis of BST height

Let Xn be the random variable denoting
the height of a randomly built binary
search tree on n nodes, and let Yn = 2Xn
be its exponential height.
If the root of the tree has rank k, then
Xn = 1 + max{Xk1, Xnk} ,
since each of the left and right subtrees
of the root are randomly built. Hence,
we have
Yn = 2 max{Yk1, Ynk} .
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.16

Analysis (continued)
Define the indicator random variable Znk as
1 if the root has rank k,
Znk =
0 otherwise.
Thus, Pr{Znk = 1} = E[Znk] = 1/n, and
n

Yn = Z nk (2 max{Yk 1 , Ynk }) .
k =1

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.17

Exponential height recurrence

n

E [Yn ] = E Z nk (2 max{Yk 1 , Ynk })

k =1

Take expectation of both sides.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.18

Exponential height recurrence

n
E [Yn ] = E Z nk (2 max{Yk 1 , Ynk })

k =1
n

= E [Z nk (2 max{Yk 1 , Ynk })]

k =1

Linearity of expectation.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.19

Exponential height recurrence

n

E [Yn ] = E Z nk (2 max{Yk 1 , Ynk })

k =1
n

= E [Z nk (2 max{Yk 1 , Ynk })]

k =1
n

= 2 E[ Z nk ] E[max{Yk 1 , Ynk }]
k =1

Independence of the rank of the root

from the ranks of subtree roots.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.20

Exponential height recurrence

n

E [Yn ] = E Z nk (2 max{Yk 1 , Ynk })

k =1
n

= E [Z nk (2 max{Yk 1 , Ynk })]

k =1
n

= 2 E[ Z nk ] E[max{Yk 1 , Ynk }]
k =1
n

2 E[Yk 1 + Ynk ]
n k =1

The max of two nonnegative numbers

is at most their sum, and E[Znk] = 1/n.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.21

Exponential height recurrence

n

E[Yn ] = E Z nk (2 max{Yk 1 , Ynk })

k =1
n

= E[Z nk (2 max{Yk 1 , Ynk })]

k =1
n

= 2 E[ Z nk ] E[max{Yk 1 , Ynk }]
k =1
n

2 E[Yk 1 + Ynk ]
n k =1
n 1

= 4 E[Yk ]
n k =0
2001 by Charles E. Leiserson

Each term appears

twice, and reindex.

Introduction to Algorithms

Day 17

L9.22

Solving the recurrence

Use substitution to
show that E[Yn] cn3
for some positive
constant c, which we
can pick sufficiently
large to handle the
initial conditions.

2001 by Charles E. Leiserson

n 1

E [Yn ] = 4 E[Yk ]
n k =0

Introduction to Algorithms

Day 17

L9.23

Solving the recurrence

Use substitution to
show that E[Yn] cn3
for some positive
constant c, which we
can pick sufficiently
large to handle the
initial conditions.

2001 by Charles E. Leiserson

n 1

E [Yn ] = 4 E[Yk ]
n k =0
n 1

4 ck 3
n k =0

Substitution.

Introduction to Algorithms

Day 17

L9.24

Solving the recurrence

Use substitution to
show that E[Yn] cn3
for some positive
constant c, which we
can pick sufficiently
large to handle the
initial conditions.

n 1

E [Yn ] = 4 E[Yk ]
n k =0
n 1

4 ck 3
n k =0
n 3
4
c
x dx
n 0

Integral method.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.25

Solving the recurrence

Use substitution to
show that E[Yn] cn3
for some positive
constant c, which we
can pick sufficiently
large to handle the
initial conditions.

n 1

E [Yn ] = 4 E[Yk ]
n k =0
n 1

4 ck 3
n k =0
n 3
4
c
x dx
n 0
4

4
c
n
=
n 4

Solve the integral.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.26

Solving the recurrence

Use substitution to
show that E[Yn] cn3
for some positive
constant c, which we
can pick sufficiently
large to handle the
initial conditions.

2001 by Charles E. Leiserson

n 1

E [Yn ] = 4 E[Yk ]
n k =0

Introduction to Algorithms

n 1

4 ck 3
n k =0
n 3
c
4
x dx
n 0
4

c
n
4
=
n 4
= cn3. Algebra.
Day 17

L9.27

The grand finale

Putting it all together, we have
2E[Xn] E[2Xn ]
Jensens inequality, since
f(x) = 2x is convex.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.28

The grand finale

Putting it all together, we have
2E[Xn] E[2Xn ]
= E[Yn]
Definition.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.29

The grand finale

Putting it all together, we have
2E[Xn] E[2Xn ]
= E[Yn]
cn3 .
What we just showed.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.30

The grand finale

Putting it all together, we have
2E[Xn] E[2Xn ]
= E[Yn]
cn3 .
Taking the lg of both sides yields
E[Xn] 3 lg n +O(1) .

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.31

Post mortem
Q. Does the analysis have to be this hard?
Q. Why bother with analyzing exponential
height?
Q. Why not just develop the recurrence on
Xn = 1 + max{Xk1, Xnk}
directly?
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.32

Post mortem (continued)

A. The inequality
max{a, b} a + b .
provides a poor upper bound, since the RHS
approaches the LHS slowly as |a b| increases.
The bound
max{2a, 2b} 2a + 2b
allows the RHS to approach the LHS far more
quickly as |a b| increases. By using the
convexity of f(x) = 2x via Jensens inequality,
we can manipulate the sum of exponentials,
resulting in a tight analysis.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.33

Thought exercises
See what happens when you try to do the
analysis on Xn directly.
Try to understand better why the proof
uses an exponential. Will a quadratic do?
See if you can find a simpler argument.
(This argument is a little simpler than the
one in the bookI hope its correct!)

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 17

L9.34

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 10
Prof. Erik Demaine

Balanced search trees

Balanced search tree: A search-tree data
structure for which a height of O(lg n) is
guaranteed when implementing a dynamic
set of n items.

Examples:

2001 by Charles E. Leiserson

AVL trees
2-3 trees
2-3-4 trees
B-trees
Red-black trees

Introduction to Algorithms

Day 18

L10.2

Red-black trees
This data structure requires an extra onebit color field in each node.
Red-black properties:
1. Every node is either red or black.
2. The root and leaves (NILs) are black.
3. If a node is red, then its parent is black.
4. All simple paths from any node x to a
descendant leaf have the same number
of black nodes = black-height(x).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.3

Example of a red-black tree

77
33
NIL

18
18
NIL

10
10
88

11
11

NIL NIL NIL NIL

2001 by Charles E. Leiserson

h=4

22
22

Introduction to Algorithms

NIL

26
26
NIL

NIL

Day 18

L10.4

Example of a red-black tree

77
33
NIL

18
18
NIL

10
10
88

22
22
11
11

NIL

NIL NIL NIL NIL

26
26
NIL

NIL

1. Every node is either red or black.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.5

Example of a red-black tree

77
33
NIL

18
18
NIL

10
10
88

22
22
11
11

NIL NIL NIL NIL

NIL

26
26
NIL

NIL

2. The root and leaves (NILs) are black.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.6

Example of a red-black tree

77
33
NIL

18
18
NIL

10
10
88

22
22
11
11

NIL NIL NIL NIL

NIL

26
26
NIL

NIL

3. If a node is red, then its parent is black.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.7

Example of a red-black tree

77 bh = 2
33
NIL

18
18 bh = 2
NIL

bh = 1 10
10
bh = 1

88

22
22
11
11

bh = 0 NIL NIL NIL NIL

NIL

26
26
NIL

NIL

4. All simple paths from any node x to a

descendant leaf have the same number of
black nodes = black-height(x).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.8

Height of a red-black tree

Theorem. A red-black tree with n keys has height
h 2 lg(n + 1).
Proof. (The book uses induction. Read carefully.)
INTUITION:
Merge red nodes
into their black
parents.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.9

Height of a red-black tree

Theorem. A red-black tree with n keys has height
h 2 lg(n + 1).
Proof. (The book uses induction. Read carefully.)
INTUITION:
Merge red nodes
into their black
parents.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.10

Height of a red-black tree

Theorem. A red-black tree with n keys has height
h 2 lg(n + 1).
Proof. (The book uses induction. Read carefully.)
INTUITION:
Merge red nodes
into their black
parents.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.11

Height of a red-black tree

Theorem. A red-black tree with n keys has height
h 2 lg(n + 1).
Proof. (The book uses induction. Read carefully.)
INTUITION:
Merge red nodes
into their black
parents.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.12

Height of a red-black tree

Theorem. A red-black tree with n keys has height
h 2 lg(n + 1).
Proof. (The book uses induction. Read carefully.)
INTUITION:
Merge red nodes
into their black
parents.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.13

Height of a red-black tree

Theorem. A red-black tree with n keys has height
h 2 lg(n + 1).
Proof. (The book uses induction. Read carefully.)
INTUITION:
Merge red nodes
h
into their black
parents.
This process produces a tree in which each node
has 2, 3, or 4 children.
The 2-3-4 tree has uniform depth h of leaves.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.14

Proof (continued)
We have
h h/2, since
at most half
the leaves on any path
are red.

The number of leaves

in each tree is n + 1
n + 1 2h'
lg(n + 1) h' h/2
h 2 lg(n + 1).

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.15

Query operations
Corollary. The queries SEARCH, MIN,
MAX, SUCCESSOR, and PREDECESSOR
all run in O(lg n) time on a red-black
tree with n nodes.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.16

Modifying operations
The operations INSERT and DELETE cause
modifications to the red-black tree:
the operation itself,
color changes,
restructuring the links of the tree via
rotations.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.17

Rotations
RIGHT-ROTATE(B)

BB

LEFT-ROTATE(A)

AA

AA

BB

Rotations maintain the inorder ordering of keys:

a , b , c a A b B c.
A rotation can be performed in O(1) time.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.18

Insertion into a red-black tree

IDEA: Insert x in tree. Color x red. Only redblack property 3 might be violated. Move the
violation up the tree by recoloring until it can
be fixed with rotations and recoloring.
Example:

77
33

18
18
10
10
88

2001 by Charles E. Leiserson

Introduction to Algorithms

22
22
11
11

26
26

Day 18

L10.19

Insertion into a red-black tree

IDEA: Insert x in tree. Color x red. Only redblack property 3 might be violated. Move the
violation up the tree by recoloring until it can
be fixed with rotations and recoloring.
Example:
33
Insert x =15.
Recolor, moving the
violation up the tree.

77
18
18
10
10
88

22
22
11
11

26
26

15
15
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.20

Insertion into a red-black tree

IDEA: Insert x in tree. Color x red. Only redblack property 3 might be violated. Move the
violation up the tree by recoloring until it can
be fixed with rotations and recoloring.
Example:
33
Insert x =15.
Recolor, moving the
violation up the tree.
RIGHT-ROTATE(18).
2001 by Charles E. Leiserson

77
18
18
10
10
88

Introduction to Algorithms

22
22
11
11

26
26

15
15
Day 18

L10.21

Insertion into a red-black tree

IDEA: Insert x in tree. Color x red. Only redblack property 3 might be violated. Move the
violation up the tree by recoloring until it can
be fixed with rotations and recoloring.
77

Example:
33
Insert x =15.
88
Recolor, moving the
violation up the tree.
RIGHT-ROTATE(18).
LEFT-ROTATE(7) and recolor.
2001 by Charles E. Leiserson

Introduction to Algorithms

10
10
18
18
11
11

22
22

15
15

26
26
Day 18

L10.22

Insertion into a red-black tree

IDEA: Insert x in tree. Color x red. Only redblack property 3 might be violated. Move the
violation up the tree by recoloring until it can
be fixed with rotations and recoloring.
Example:
77
Insert x =15.
Recolor, moving the 33
88
violation up the tree.
RIGHT-ROTATE(18).
LEFT-ROTATE(7) and recolor.
2001 by Charles E. Leiserson

Introduction to Algorithms

10
10
18
18
11
11

22
22

15
15

26
26

Day 18

L10.23

Pseudocode
RB-INSERT(T, x)
TREE-INSERT(T, x)
color[x] RED only RB property 3 can be violated
while x root[T] and color[p[x]] = RED
do if p[x] = left[p[p[x]]
then y right[p[p[x]]
y = aunt/uncle of x
if color[y] = RED
then Case 1
else if x = right[p[x]]
then Case 2 Case 2 falls into Case 3
Case 3
else then clause with left and right swapped
color[root[T]] BLACK
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.24

Graphical notation
Let
All

denote a subtree with a black root.

s have the same black-height.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.25

Case 1
Recolor

CC
DD

AA

2001 by Charles E. Leiserson

new x
DD

AA

BB

(Or, children of
A are swapped.)

CC

BB

Push Cs black onto

A and D, and recurse,
since Cs parent may
be red.
Introduction to Algorithms

Day 18

L10.26

Case 2
CC
AA

LEFT-ROTATE(A)

CC

BB

BB

AA

Transform to Case 3.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.27

Case 3
CC
BB

RIGHT-ROTATE(C)
y
AA

AA

BB
CC

Done! No more
violations of RB
property 3 are
possible.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.28

Analysis
Go up the tree performing Case 1, which only
recolors nodes.
If Case 2 or Case 3 occurs, perform 1 or 2
rotations, and terminate.
Running time: O(lg n) with O(1) rotations.
RB-DELETE same asymptotic running time
and number of rotations as RB-INSERT (see
textbook).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 18

L10.29

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 11
Prof. Erik Demaine

Dynamic order statistics

OS-SELECT(i, S): returns the i th smallest element
in the dynamic set S.
OS-RANK(x, S): returns the rank of x S in the
sorted order of Ss elements.
IDEA: Use a red-black tree for the set S, but keep
subtree sizes in the nodes.
Notation for nodes:
2001 by Charles E. Leiserson

key
key
size
size

Introduction to Algorithms

Day 20

L11.2

Example of an OS-tree
M
M
99
CC
55

PP
33

AA
11

FF
33
DD
11

NN
11

QQ
11

HH
11

size[x] = size[left[x]] + size[right[x]] + 1

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.3

Selection
Implementation trick: Use a sentinel
(dummy record) for NIL such that size[NIL] = 0.
OS-SELECT(x, i) ith smallest element in the
subtree rooted at x
k size[left[x]] + 1 k = rank(x)
if i = k then return x
if i < k
then return OS-SELECT(left[x], i )
else return OS-SELECT(right[x], i k )
(OS-RANK is in the textbook.)
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.4

Example
OS-SELECT(root, 5)
i=5
k=6
i=5
k=2

M
M
99

CC
55

PP
33

AA
11

FF
33
DD
11

i=3
k=2
HH
11

NN
11

QQ
11

i=1
k=1

Running time = O(h) = O(lg n) for red-black trees.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.5

Data structure maintenance

Q. Why not keep the ranks themselves
in the nodes instead of subtree sizes?
A. They are hard to maintain when the
red-black tree is modified.
Modifying operations: INSERT and DELETE.
Strategy: Update subtree sizes when
inserting or deleting.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.6

Example of insertion
INSERT(K)

M
M
10
910
9

CC
6565

PP
33

AA
11

FF
4343
DD
11

NN
11

QQ
11

HH
2121
KK
11

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.7

Handling rebalancing
Dont forget that RB-INSERT and RB-DELETE may
also need to modify the red-black tree in order to
maintain balance.
Recolorings: no effect on subtree sizes.
Rotations: fix up subtree sizes in O(1) time.

Example:

EE
16
16

CC
11
11
7

CC
16
16
4

EE
88

7
3

RB-INSERT and RB-DELETE still run in O(lg n) time.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.8

Data-structure augmentation
Methodology: (e.g., order-statistics trees)
1. Choose an underlying data structure (redblack trees).
2. Determine additional information to be
stored in the data structure (subtree sizes).
3. Verify that this information can be
maintained for modifying operations (RBINSERT, RB-DELETE dont forget rotations).
4. Develop new dynamic-set operations that use
the information (OS-SELECT and OS-RANK).
These steps are guidelines, not rigid rules.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.9

Interval trees
Goal: To maintain a dynamic set of intervals,
such as time intervals.
i = [7, 10]
low[i] = 7
5
4

10 = high[i]
11
17
15

19
18 22

23

Query: For a given query interval i, find an

interval in the set that overlaps i.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.10

Following the methodology

1. Choose an underlying data structure.
Red-black tree keyed on low (left) endpoint.
2. Determine additional information to be
stored in the data structure.
Store in each node x the largest value m[x]
in the subtree rooted at x, as well as the
interval int[x] corresponding to the key.
int
int
m
m
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.11

Example interval tree

17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88

22,23
22,23
23
23
15,18
15,18
18
18

7,10
7,10
10
10

2001 by Charles E. Leiserson

m[x] = max

Introduction to Algorithms

high[int[x]]
m[left[x]]
m[right[x]]
Day 20

L11.12

Modifying operations
3. Verify that this information can be maintained
for modifying operations.
INSERT: Fix ms on the way down.
Rotations Fixup = O(1) time per rotation:
11,15
11,15
30
30
6,20
6,20
30
30
30
30

6,20
6,20
30
30
19
19

11,15
11,15
19
19

30
30

14
14

14
14

19
19

Total INSERT time = O(lg n); DELETE similar.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.13

New operations
4. Develop new dynamic-set operations that use
the information.
INTERVAL-SEARCH(i)
x root
while x NIL and (low[i] > high[int[x]]
or low[int[x]] > high[i])
do i and int[x] dont overlap
if left[x] NIL and low[i] m[left[x]]
then x left[x]
else x right[x]
return x
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.14

Example 1: INTERVAL-SEARCH([14,16])
x

17,19
17,19
23
23

5,11
5,11
18
18
4,8
4,8
88

22,23
22,23
23
23
15,18
15,18
18
18

7,10
7,10
10
10

2001 by Charles E. Leiserson

x root
[14,16] and [17,19] dont overlap
14 18 x left[x]
Introduction to Algorithms

Day 20

L11.15

Example 1: INTERVAL-SEARCH([14,16])
17,19
17,19
23
23

5,11
5,11
18
18

4,8
4,8
88

22,23
22,23
23
23
15,18
15,18
18
18

7,10
7,10
10
10

2001 by Charles E. Leiserson

[14,16] and [5,11] dont overlap

14 > 8 x right[x]
Introduction to Algorithms

Day 20

L11.16

Example 1: INTERVAL-SEARCH([14,16])
17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88

22,23
22,23
23
23

x
7,10
7,10
10
10

2001 by Charles E. Leiserson

15,18
15,18
18
18

[14,16] and [15,18] overlap

return [15,18]
Introduction to Algorithms

Day 20

L11.17

Example 2: INTERVAL-SEARCH([12,14])
x

17,19
17,19
23
23

5,11
5,11
18
18
4,8
4,8
88

22,23
22,23
23
23
15,18
15,18
18
18

7,10
7,10
10
10

2001 by Charles E. Leiserson

x root
[12,14] and [17,19] dont overlap
12 18 x left[x]
Introduction to Algorithms

Day 20

L11.18

Example 2: INTERVAL-SEARCH([12,14])
17,19
17,19
23
23

5,11
5,11
18
18

4,8
4,8
88

22,23
22,23
23
23
15,18
15,18
18
18

7,10
7,10
10
10

2001 by Charles E. Leiserson

[12,14] and [5,11] dont overlap

12 > 8 x right[x]
Introduction to Algorithms

Day 20

L11.19

Example 2: INTERVAL-SEARCH([12,14])
17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88

22,23
22,23
23
23

x
7,10
7,10
10
10

2001 by Charles E. Leiserson

15,18
15,18
18
18

[12,14] and [15,18] dont overlap

12 > 10 x right[x]
Introduction to Algorithms

Day 20

L11.20

Example 2: INTERVAL-SEARCH([12,14])
17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88

22,23
22,23
23
23
15,18
15,18
18
18

7,10
7,10
10
10

2001 by Charles E. Leiserson

x
x = NIL no interval that
overlaps [12,14] exists
Introduction to Algorithms

Day 20

L11.21

Analysis
Time = O(h) = O(lg n), since INTERVAL-SEARCH
does constant work at each level as it follows a
simple path down the tree.
List all overlapping intervals:
Search, list, delete, repeat.
Insert them all again at the end.
Time = O(k lg n), where k is the total number of
overlapping intervals.
This is an output-sensitive bound.
Best algorithm to date: O(k + lg n).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.22

Correctness
Theorem. Let L be the set of intervals in the
left subtree of node x, and let R be the set of
intervals in xs right subtree.
If the search goes right, then
{ i L : i overlaps i } = .
If the search goes left, then
{i L : i overlaps i } =
{i R : i overlaps i } = .
In other words, its always safe to take only 1
of the 2 children: well either find something,
or nothing was to be found.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.23

Correctness proof
Proof. Suppose first that the search goes right.
If left[x] = NIL, then were done, since L = .
Otherwise, the code dictates that we must have
low[i] > m[left[x]]. The value m[left[x]]
corresponds to the right endpoint of some
interval j L, and no other interval in L can
have a larger right endpoint than high( j).
i
L
high( j) = m[left[x]]

low(i)

Therefore, {i L : i overlaps i } = .
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 20

L11.24

Proof (continued)
Suppose that the search goes left, and assume that
{i L : i overlaps i } = .
Then, the code dictates that low[i] m[left[x]] =
high[ j] for some j L.
Since j L, it does not overlap i, and hence
high[i] < low[ j].
But, the binary-search-tree property implies that
for all i R, we have low[ j] low[i].
But then {i R : i overlaps i } = .
i
2001 by Charles E. Leiserson

j
i
Introduction to Algorithms

Day 20

L11.25

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 12
Prof. Erik Demaine

Computational geometry
Algorithms for solving geometric problems
in 2D and higher.
Fundamental objects:

point

line segment

line

Basic structures:

point set
2001 by Erik D. Demaine

polygon
Introduction to Algorithms

Day 21

L12.2

Computational geometry
Algorithms for solving geometric problems
in 2D and higher.
Fundamental objects:

point

line segment

line

Basic structures:

triangulation
2001 by Erik D. Demaine

convex hull
Introduction to Algorithms

Day 21

L12.3

Orthogonal range searching

Input: n points in d dimensions
E.g., representing a database of n records
each with d numeric fields
Query: Axis-aligned box (in 2D, a rectangle)
Report on the points inside the box:
Are there any points?
How many are there?
List the points.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.4

Orthogonal range searching

Input: n points in d dimensions
Query: Axis-aligned box (in 2D, a rectangle)
Report on the points inside the box
Goal: Preprocess points into a data structure
to support fast queries
Primary goal: Static data structure
In 1D, we will also obtain a
dynamic data structure
supporting insert and delete
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.5

1D range searching
In 1D, the query is an interval:
First solution using ideas we know:
Interval trees
Represent each point x by the interval [x, x].
Obtain a dynamic structure that can list
k answers in a query in O(k lg n) time.

2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.6

1D range searching
In 1D, the query is an interval:
Second solution using ideas we know:
Sort the points and store them in an array
Solve query by binary search on endpoints.
Obtain a static structure that can list
k answers in a query in O(k + lg n) time.
Goal: Obtain a dynamic structure that can list
k answers in a query in O(k + lg n) time.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.7

1D range searching
In 1D, the query is an interval:
New solution that extends to higher dimensions:
Balanced binary search tree
New organization principle:
Store points in the leaves of the tree.
Internal nodes store copies of the leaves
to satisfy binary search property:
Node x stores in key[x] the maximum
key of any leaf in the left subtree of x.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.8

Example of a 1D range tree

17
17

11
66

88 12
12 14
14

2001 by Erik D. Demaine

43
43
26
35 41
41 42
42
26 35

Introduction to Algorithms

59
61
59 61

Day 21

L12.9

Example of a 1D range tree

xx

17
17
88

42
42

11

14
14
66

11
66

12
12
88 12
12 14
14

2001 by Erik D. Demaine

35
35
17
17

26
26

43
43
41
41

26
35 41
41 42
42
26 35

Introduction to Algorithms

>x

43
43

59
59
59
61
59 61

Day 21

L12.10

Example of a 1D range query

xx

17
17
88

42
42

11

14
14
66

11
66

12
12
88 12
12 14
14

35
35
17
17

26
26

>x

43
43
41
41

26
35 41
41 42
42
26 35

43
43

59
59
59
61
59 61

RANGE-QUERY([7, 41])
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.11

General 1D range query

root

split node

2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.12

Pseudocode, part 1:
Find the split node
1D-RANGE-QUERY(T, [x1, x2])
w root[T]
while w is not a leaf and (x2 key[w] or key[w] < x1)
do if x2 key[w]
then w left[w]
else w right[w]
w is now the split node
[traverse left and right from w and report relevant subtrees]

2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.13

Pseudocode, part 2: Traverse

left and right from split node
1D-RANGE-QUERY(T, [x1, x2])
[find the split node]
w is now the split node
if w is a leaf
then output the leaf w if x1 key[w] x2
else v left[w]
Left traversal
while v is not a leaf
do if x1 key[v]
then output the subtree rooted at right[v]
v left[v]
else v right[v]
output the leaf v if x1 key[v] x2
[symmetrically for right traversal]
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.14

Analysis of 1D-RANGE-QUERY
Query time: Answer to range query represented
by O(lg n) subtrees found in O(lg n) time.
Thus:
Can test for points in interval in O(lg n) time.
Can count points in interval in O(lg n) time
if we augment the tree with subtree sizes.
Can report the first k points in
interval in O(k + lg n) time.
Space: O(n)
Preprocessing time: O(n lg n)
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.15

2D range trees
Store a primary 1D range tree for all the points
based on x-coordinate.
Thus in O(lg n) time we can find O(lg n) subtrees
representing the points with proper x-coordinate.
How to restrict to points with proper y-coordinate?

2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.16

2D range trees
Idea: In primary 1D range tree of x-coordinate,
every node stores a secondary 1D range tree
based on y-coordinate for all points in the subtree
of the node. Recursively search within each.

2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.17

Analysis of 2D range trees

Query time: In O(lg2 n) = O((lg n)2) time, we can
represent answer to range query by O(lg2 n) subtrees.
Total cost for reporting k points: O(k + (lg n)2).
Space: The secondary trees at each level of the
primary tree together store a copy of the points.
Also, each point is present in each secondary
tree along the path from the leaf to the root.
Either way, we obtain that the space is O(n lg n).
Preprocessing time: O(n lg n)
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.18

d-dimensional range trees

Each node of the secondary y-structure stores
a tertiary z-structure representing the points
in the subtree rooted at the node, etc.
Query time: O(k + lgd n) to report k points.
Space: O(n lgd 1 n)
Preprocessing time: O(n lgd 1 n)
Best data structure to date:
Query time: O(k + lgd 1 n) to report k points.
Space: O(n (lg n / lg lg n)d 1)
Preprocessing time: O(n lgd 1 n)
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.19

Primitive operations:
Crossproduct
Given two vectors v1 = (x1, y1) and v2 = (x2, y2),
is their counterclockwise angle
convex (< 180),
v2
v1
v
reflex (> 180), or
v2
1

borderline (0 or 180)? convex

reflex
Crossproduct v1 v2 = x1 y2 y1 x2
= |v1| |v2| sin .
Thus, sign(v1 v2) = sign(sin ) > 0 if convex,
< 0 if reflex,
= 0 if borderline.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.20

Primitive operations:
Orientation test
Given three points p1, p2, p3 are they
in clockwise (cw) order,
in counterclockwise (ccw) order, or
collinear?
(p2 p1) (p3 p1)
> 0 if ccw
< 0 if cw
p2
= 0 if collinear
p1
cw
p1

p3
p2
p1
collinear
p3
ccw

p3
2001 by Erik D. Demaine

Introduction to Algorithms

p2
Day 21

L12.21

Primitive operations:
Sidedness test
Given three points p1, p2, p3 are they
in clockwise (cw) order,
in counterclockwise (ccw) order, or
collinear?
Let L be the oriented line from p1 to p2.
Equivalently, is the point p3
right of L,
p2
left of L, or
p1
cw
p1
on L?
p

p3
p2
p1
collinear
p3
ccw
p2

2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.22

Line-segment intersection
Given n line segments, does any pair intersect?
Obvious algorithm: O(n2).
e
d
a
c
f

b
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.23

Sweep-line algorithm
Sweep a vertical line from left to right
(conceptually replacing x-coordinate with time).
Maintain dynamic set S of segments
that intersect the sweep line, ordered
(tentatively) by y-coordinate of intersection.
Order changes when
new segment is encountered, segment
existing segment finishes, or endpoints
two segments cross
Key event points are therefore segment endpoints.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.24

d
a a
a c c
a b b b

e
d
c
b

d
c
b

e
e d
d b
b f

e
b
d
f

b
e e
d d
f f

e
d
a
c
f

b
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.25

Sweep-line algorithm
Process event points in order by sorting segment
endpoints by x-coordinate and looping through:
For a left endpoint of segment s:
Add segment s to dynamic set S.
Check for intersection between s
and its neighbors in S.
For a right endpoint of segment s:
Remove segment s from dynamic set S.
Check for intersection between
the neighbors of s in S.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.26

Analysis
Use red-black tree to store dynamic set S.
Total running time: O(n lg n).

2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.27

Correctness
Theorem: If there is an intersection,
the algorithm finds it.
Proof: Let X be the leftmost intersection point.
Assume for simplicity that
only two segments s1, s2 pass through X, and
no two points have the same x-coordinate.
At some point before we reach X,
s1 and s2 become consecutive in the order of S.
Either initially consecutive when s1 or s2 inserted,
or became consecutive when another deleted.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 21

L12.28

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 13
Prof. Erik Demaine

Fixed-universe
successor problem
Goal: Maintain a dynamic subset S of size n
of the universe U = {0, 1, , u 1} of size u
subject to these operations:
INSERT(x U \ S): Add x to S.
DELETE(x S): Remove x from S.
SUCCESSOR(x U): Find the next element in S
larger than any element x of the universe U.
PREDECESSOR(x U): Find the previous
element in S smaller than x.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.2

Solutions to fixed-universe
successor problem
Goal: Maintain a dynamic subset S of size n
of the universe U = {0, 1, , u 1} of size u
subject to INSERT, DELETE, SUCCESSOR, PREDECESSOR.
Balanced search trees can implement operations in
O(lg n) time, without fixed-universe assumption.
In 1975, Peter van Emde Boas solved this problem
in O(lg lg u) time per operation.
If u is only polynomial in n, that is, u = O(nc),
then O(lg lg n) time per operation-exponential speedup!
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.3

O(lg lg u)?!
Where could a bound of O(lg lg u) arise?
Binary search over O(lg u) things
T(u) = T( u ) + O(1)
T(lg u) = T((lg u)/2) + O(1)
= O(lg lg u)

2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.4

(1) Starting point: Bit vector

Bit vector v stores, for each x U,
1 if x S
vx = 0 if x S
Example: u = 16; n = 4; S = {1, 9, 10, 15}.
0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 1
0 1 2

3 4 5 6 7

8 9 10 11 12 13 14 15

Insert/Delete run in O(1) time.

Successor/Predecessor run in O(u) worst-case time.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.5

(2) Split universe into widgets

Carve universe of size u into u widgets
W0, W1, , W u 1 each of size u .
Example: u = 16, u = 4 .
W0

W1

W2

W3

0 1 0 0

0 0 0 0 0 1 1 0

0 0 0 1

0 1 2

4 5 6 7

12 13 14 15

2001 by Erik D. Demaine

8 9 10 11

Introduction to Algorithms

Day 23

L12.6

(2) Split universe into widgets

Carve universe of size u into u widgets
W0, W1, , W u 1 each of size u .
W0 represents 0, 1, , u 1 U;
W1 represents u , u + 1, , 2 u 1 U;
:
Wi represents i u , i u + 1, , (i + 1) u 1 U;
:
W u 1 represents u u , u u + 1 , , u 1 U.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.7

(2) Split universe into widgets

x=9

Define high(x) 0 and low(x) 0

1 0 0 1
so that x = high(x) u + low(x).
That is, if we write x U in binary, high(x) low(x)
=2
=1
high(x) is the high-order half of the bits,
and low(x) is the low-order half of the bits.
For x U, high(x) is index of widget containing x
and low(x) is the index of x within that widget.
W0

W1

W2

W3

0 1 0 0

0 0 0 0 0 1 1 0

0 0 0 1

0 1 2

4 5 6 7

12 13 14 15

2001 by Erik D. Demaine

8 9 10 11

Introduction to Algorithms

Day 23

L12.8

(2) Split universe into widgets

INSERT(x)
insert x into widget Whigh(x) at position low(x).
mark Whigh(x) as nonempty.

Running time T(n) = O(1).

2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.9

(2) Split universe into widgets

SUCCESSOR(x)
look for successor of x within widget Whigh(x)
starting after position low(x).
if successor found
then return it
else find smallest i > high(x)
for which Wi is nonempty.
return smallest element in Wi

O( u )

O( u )
O( u )

Running time T(u) = O( u ).

2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.10

Revelation
SUCCESSOR(x)
look for successor of x within widget Whigh(x)
starting after position low(x).
if successor found
then return it
else find smallest i > high(x)
for which Wi is nonempty.
return smallest element in Wi

2001 by Erik D. Demaine

Introduction to Algorithms

recursive
successor

recursive
successor
recursive
successor

Day 23

L12.11

(3) Recursion
Represent universe by widget of size u.
Recursively split each widget W of size |W|
into W subwidgets sub[W][0], sub[W][1], ,
sub[W][ W 1 ] each of size W .
Store a summary widget summary[W] of size W
representing which subwidgets are nonempty.
W
summary[W] sub[W][0] sub[W][1]

W
2001 by Erik D. Demaine

W
Introduction to Algorithms

sub[W][ W 1 ]

W
Day 23

L12.12

(3) Recursion
Define high(x) 0 and low(x) 0
so that x = high(x) W + low(x).
INSERT(x, W)
if sub[W][high(x)] is empty
then INSERT(high(x), summary[W])
INSERT(low(x), sub[W][high(x)])

Running time T(u) = 2 T( u ) + O(1)

T(lg u) = 2 T((lg u) / 2) + O(1)
= O(lg u) .
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.13

(3) Recursion
SUCCESSOR(x, W)
j SUCCESSOR(low(x), sub[W][high(x)])
if j <
then return high(x) W + j
else i SUCCESSOR(high(x), summary[W])
j SUCCESSOR( , sub[W][i])

T( u )
T( u )
T( u )

return i W + j

Running time T(u) = 3 T( u ) + O(1)

T(lg u) = 3 T((lg u) / 2) + O(1)
= O((lg u) lg 3) .
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.14

Improvements
Need to reduce INSERT and SUCCESSOR
down to 1 recursive call each.
1 call: T(u) = 1 T( u ) + O(1)
= O(lg lg n)
2 calls: T(u) = 2 T( u ) + O(1)
= O(lg n)
3 calls: T(u) = 3 T( u ) + O(1)
= O((lg u) lg 3)
Were closer to this goal than it may seem!
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.15

Recursive calls in successor

If x has a successor within sub[W][high(x)],
then there is only 1 recursive call to SUCCESSOR.
Otherwise, there are 3 recursive calls:
SUCCESSOR(low(x), sub[W][high(x)])
discovers that sub[W][high(x)] hasnt successor.
SUCCESSOR(high(x), summary[W])
finds next nonempty subwidget sub[W][i].
SUCCESSOR( , sub[W][i])
finds smallest element in subwidget sub[W][i].
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.16

Reducing recursive calls

in successor
If x has no successor within sub[W][high(x)],
there are 3 recursive calls:
SUCCESSOR(low(x), sub[W][high(x)])
discovers that sub[W][high(x)] hasnt successor.
Could be determined using the maximum
value in the subwidget sub[W][high(x)].
SUCCESSOR(high(x), summary[W])
finds next nonempty subwidget sub[W][i].
SUCCESSOR( , sub[W][i])
finds minimum element in subwidget sub[W][i].
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.17

(4) Improved successor

INSERT(x, W)
if sub[W][high(x)] is empty
then INSERT(high(x), summary[W])
INSERT(low(x), sub[W][high(x)])
if x < min[W] then min[W] x
new (augmentation)
if x > max[W] then max[W] x

Running time T(u) = 2 T( u ) + O(1)

T(lg u) = 2 T((lg u) / 2) + O(1)
= O(lg u) .
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.18

(4) Improved successor

SUCCESSOR(x, W)
if low(x) < max[sub[W][high(x)]]
then j SUCCESSOR(low(x), sub[W][high(x)])
return high(x) W + j
else i SUCCESSOR(high(x), summary[W])
j min[sub[W][i]]

T( u )
T( u )

return i W + j

Running time T(u) = 1 T( u ) + O(1)

= O(lg lg u) .
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.19

Recursive calls in insert

If sub[W][high(x)] is already in summary[W],
then there is only 1 recursive call to INSERT.
Otherwise, there are 2 recursive calls:
INSERT(high(x), summary[W])
INSERT(low(x), sub[W][high(x)])
Idea:We know that sub[W][high(x)]) is empty.
Avoid second recursive call by specially
storing a widget containing just 1 element.
Specifically, do not store min recursively.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.20

(5) Improved insert

INSERT(x, W)
if x < min[W] then exchange x min[W]
if sub[W][high(x)] is nonempty, that is,
min[sub[W][high(x)] NIL
then INSERT(low(x), sub[W][high(x)])
else min[sub[W][high(x)]] low(x)
INSERT(high(x), summary[W])
if x > max[W] then max[W] x

Running time T(u) = 1 T( u ) + O(1)

= O(lg lg u) .
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.21

(5) Improved insert

SUCCESSOR(x, W)
if x < min[W] then return min[W] new
if low(x) < max[sub[W][high(x)]]
then j SUCCESSOR(low(x), sub[W][high(x)])
return high(x) W + j
else i SUCCESSOR(high(x), summary[W])
j min[sub[W][i]]

T( u )
T( u )

return i W + j

Running time T(u) = 1 T( u ) + O(1)

= O(lg lg u) .
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.22

Deletion
DELETE(x, W)
if min[W] = NIL or x < min[W] then return
if x = min[W]
then i min[summary[W]]
x i W + min[sub[W][i]]
min[W] x
DELETE(low(x), sub[W][high(x)])
if sub[W][high(x)] is now empty, that is,
min[sub[W][high(x)] = NIL
then DELETE(high(x), summary[W])
(in this case, the first recursive call was cheap)
2001 by Erik D. Demaine

Introduction to Algorithms

Day 23

L12.23

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 14
Prof. Charles E. Leiserson

How large should a hash

table be?
Goal: Make the table as small as possible, but
large enough so that it wont overflow (or
otherwise become inefficient).
Problem: What if we dont know the proper size
in advance?
Solution: Dynamic tables.
IDEA: Whenever the table overflows, grow it
by allocating (via malloc or new) a new, larger
table. Move all items from the old table into the
new one, and free the storage for the old table.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.2

Example of a dynamic table

1. INSERT
2. INSERT

overflow

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.3

Example of a dynamic table

1. INSERT
2. INSERT

overflow

2001 by Charles E. Leiserson

Introduction to Algorithms

11

Day 24

L14.4

Example of a dynamic table

1. INSERT
2. INSERT

2001 by Charles E. Leiserson

11
2

Introduction to Algorithms

Day 24

L14.5

Example of a dynamic table

1. INSERT
2. INSERT
3. INSERT

2001 by Charles E. Leiserson

11
22
overflow

Introduction to Algorithms

Day 24

L14.6

Example of a dynamic table

1. INSERT
2. INSERT
3. INSERT

2001 by Charles E. Leiserson

1
2
overflow

Introduction to Algorithms

Day 24

L14.7

Example of a dynamic table

1. INSERT
2. INSERT
3. INSERT

2001 by Charles E. Leiserson

1
2

Introduction to Algorithms

Day 24

L14.8

Example of a dynamic table

1.
2.
3.
4.

INSERT
INSERT
INSERT
INSERT

2001 by Charles E. Leiserson

1
2
3
4

Introduction to Algorithms

Day 24

L14.9

Example of a dynamic table

1.
2.
3.
4.
5.

INSERT
INSERT
INSERT
INSERT
INSERT

2001 by Charles E. Leiserson

1
2
3
4
overflow

Introduction to Algorithms

Day 24

L14.10

Example of a dynamic table

1.
2.
3.
4.
5.

INSERT
INSERT
INSERT
INSERT
INSERT

2001 by Charles E. Leiserson

1
2
3
4
overflow

Introduction to Algorithms

Day 24

L14.11

Example of a dynamic table

1.
2.
3.
4.
5.

INSERT
INSERT
INSERT
INSERT
INSERT

2001 by Charles E. Leiserson

1
2
3
4

Introduction to Algorithms

Day 24

L14.12

Example of a dynamic table

1.
2.
3.
4.
5.
6.
7.

INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT

2001 by Charles E. Leiserson

1
2
3
4
5
6
7

Introduction to Algorithms

Day 24

L14.13

Worst-case analysis
Consider a sequence of n insertions. The
worst-case time to execute one insertion is
(n). Therefore, the worst-case time for n
insertions is n (n) = (n2).
WRONG! In fact, the worst-case cost for
n insertions is only (n) (n2).
Lets see why.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.14

Tighter analysis
Let ci = the cost of the i th insertion
i if i 1 is an exact power of 2,
=
1 otherwise.
i

sizei

16 16

ci

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

10
1

L14.15

Tighter analysis
Let ci = the cost of the i th insertion
i if i 1 is an exact power of 2,
=
1 otherwise.
i

sizei

16 16

1
1

1
2

1
4

1
8

ci

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

10
1

L14.16

Tighter analysis (continued)

n

Cost of n insertions = ci
i =1

n+

lg( n 1)

2j

j =0

3n
= ( n ) .
Thus, the average cost of each dynamic-table
operation is (n)/n = (1).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.17

Amortized analysis
An amortized analysis is any strategy for
analyzing a sequence of operations to
show that the average cost per operation is
small, even though a single operation
within the sequence might be expensive.
Even though were taking averages, however,
probability is not involved!
An amortized analysis guarantees the
average performance of each operation in
the worst case.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.18

Types of amortized analyses

Three common amortization arguments:
the aggregate method,
the accounting method,
the potential method.
Weve just seen an aggregate analysis.
The aggregate method, though simple, lacks the
precision of the other two methods. In particular,
the accounting and potential methods allow a
specific amortized cost to be allocated to each
operation.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.19

Accounting method
Charge i th operation a fictitious amortized cost
i, where $1 pays for 1 unit of work (i.e., time).
This fee is consumed to perform the operation.
Any amount not immediately consumed is stored
in the bank for use by subsequent operations.
The bank balance must not go negative! We
must ensure that n
n
ci ci
i =1

i =1

for all n.
Thus, the total amortized costs provide an upper
bound on the total true costs.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.20

Accounting analysis of
dynamic tables
Charge an amortized cost of i = $3 for the i th
insertion.
$1 pays for the immediate insertion.
$2 is stored for later table doubling.
When the table doubles, $1 pays to move a
recent item, and $1 pays to move an old item.
Example:
$0
$0 $0
$0 $0
$0 $2
$2 $2
$2 $2 $2 overflow
$0 $0

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.21

Accounting analysis of
dynamic tables
Charge an amortized cost of i = $3 for the i th
insertion.
$1 pays for the immediate insertion.
$2 is stored for later table doubling.
When the table doubles, $1 pays to move a
recent item, and $1 pays to move an old item.
Example:
overflow
$0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0
$0 $0
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.22

Accounting analysis of
dynamic tables
Charge an amortized cost of i = $3 for the i th
insertion.
$1 pays for the immediate insertion.
$2 is stored for later table doubling.
When the table doubles, $1 pays to move a
recent item, and $1 pays to move an old item.
Example:

$0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $2 $2 $2
$0 $0
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.23

Accounting analysis
(continued)
Key invariant: Bank balance never drops below 0.
Thus, the sum of the amortized costs provides an
upper bound on the sum of the true costs.
i

sizei

16 16

ci

2* 3

banki

10

*Okay, so I lied. The first operation costs only $2, not $3.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.24

Potential method
IDEA: View the bank account as the potential
energy ( la physics) of the dynamic set.
Framework:
Start with an initial data structure D0.
Operation i transforms Di1 to Di.
The cost of operation i is ci.
Define a potential function : {Di} R,
such that (D0 ) = 0 and (Di ) 0 for all i.
The amortized cost i with respect to is
defined to be i = ci + (Di) (Di1).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.25

Understanding potentials
i = ci + (Di) (Di1)
potential difference i

If i > 0, then i > ci. Operation i stores

work in the data structure for later use.
If i < 0, then i < ci. The data structure
delivers up stored work to help pay for
operation i.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.26

The amortized costs bound

the true costs
The total amortized cost of n operations is
n

i =1

i =1

ci = (ci + ( Di ) ( Di1 ))
Summing both sides.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.27

The amortized costs bound

the true costs
The total amortized cost of n operations is
n

i =1

i =1
n

ci = (ci + ( Di ) ( Di1 ))
= ci + ( Dn ) ( D0 )
i =1

The series telescopes.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.28

The amortized costs bound

the true costs
The total amortized cost of n operations is
n

i =1

i =1
n

ci = (ci + ( Di ) ( Di1 ))
= ci + ( Dn ) ( D0 )
i =1
n

ci
i =1

2001 by Charles E. Leiserson

since (Dn) 0 and

(D0 ) = 0.

Introduction to Algorithms

Day 24

L14.29

Potential analysis of table

doubling
Define the potential of the table after the ith
insertion by (Di) = 2i 2lg i. (Assume that
2lg 0 = 0.)
Note:
(D0 ) = 0,
(Di) 0 for all i.
Example:

= 26 23 = 4

$0
$0 $0
$0 $0
$0 $2
$2 $2
$2
$0 $0

accounting method)

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.30

Calculation of amortized costs

The amortized cost of the i th insertion is
i = ci + (Di) (Di1)
=

i + (2i 2lg i) (2(i 1) 2lg (i1))

if i 1 is an exact power of 2,
1 + (2i 2lg i) (2(i 1) 2lg (i1))
otherwise.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.31

Calculation (Case 1)
Case 1: i 1 is an exact power of 2.

i = i + (2i 2lg i) (2(i 1) 2lg (i1))

= i + 2 (2lg i 2lg (i1))

= i + 2 (2(i 1) (i 1))
= i + 2 2i + 2 + i 1
=3

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.32

Calculation (Case 2)
Case 2: i 1 is not an exact power of 2.

i = 1 + (2i 2lg i) (2(i 1) 2lg (i1))

= 1 + 2 (2lg i 2lg (i1))
=3

Therefore, n insertions cost (n) in the worst case.

Exercise: Fix the bug in this analysis to show that
the amortized cost of the first insertion is only 2.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.33

Conclusions
Amortized costs can provide a clean abstraction
of data-structure performance.
Any of the analysis methods can be used when
an amortized analysis is called for, but each
method has some situations where it is arguably
the simplest.
Different schemes may work for assigning
amortized costs in the accounting method, or
potentials in the potential method, sometimes
yielding radically different bounds.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 24

L14.34

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 15
Prof. Charles E. Leiserson

Dynamic programming
Design technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)
Given two sequences x[1 . . m] and y[1 . . n], find
a longest subsequence common to them both.
a not the
x: A B
C
B
D A B
BCBA =
LCS(x, y)
y: B
D C
A B
A
functional notation,
but not a function
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 26

L15.2

Brute-force LCS algorithm

Check every subsequence of x[1 . . m] to see
if it is also a subsequence of y[1 . . n].
Analysis
Checking = O(n) time per subsequence.
2m subsequences of x (each bit-vector of
length m determines a distinct subsequence
of x).
Worst-case running time = O(n2m)
= exponential time.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 26

L15.3

Towards a better algorithm

Simplification:
1. Look at the length of a longest-common
subsequence.
2. Extend the algorithm to find the LCS itself.
Notation: Denote the length of a sequence s
by | s |.
Strategy: Consider prefixes of x and y.
Define c[i, j] = | LCS(x[1 . . i], y[1 . . j]) |.
Then, c[m, n] = | LCS(x, y) |.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 26

L15.4

Recursive formulation
Theorem.

c[i1, j1] + 1
if x[i] = y[j],
c[i, j] = max{c[i1, j], c[i, j1]} otherwise.
Proof. Case x[i] = y[ j]:
x:

1
1

2
2

y:

L
j

Let z[1 . . k] = LCS(x[1 . . i], y[1 . . j]), where c[i, j]

= k. Then, z[k] = x[i], or else z could be extended.
Thus, z[1 . . k1] is CS of x[1 . . i1] and y[1 . . j1].
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 26

L15.5

Proof (continued)
Claim: z[1 . . k1] = LCS(x[1 . . i1], y[1 . . j1]).
Suppose w is a longer CS of x[1 . . i1] and
y[1 . . j1], that is, | w | > k1. Then, cut and
paste: w || z[k] (w concatenated with z[k]) is a
common subsequence of x[1 . . i] and y[1 . . j]
with | w || z[k] | > k. Contradiction, proving the
claim.
Thus, c[i1, j1] = k1, which implies that c[i, j]
= c[i1, j1] + 1.
Other cases are similar.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 26

L15.6

Dynamic-programming
hallmark #1
Optimal substructure
An optimal solution to a problem
(instance) contains optimal
solutions to subproblems.
If z = LCS(x, y), then any prefix of z is
an LCS of a prefix of x and a prefix of y.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 26

L15.7

Recursive algorithm for LCS

LCS(x, y, i, j)
if x[i] = y[ j]
then c[i, j] LCS(x, y, i1, j1) + 1
else c[i, j] max{ LCS(x, y, i1, j),
LCS(x, y, i, j1)}
Worst-case: x[i] y[ j], in which case the
algorithm evaluates two subproblems, each
with only one parameter decremented.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 26

L15.8

Recursion tree
m = 3, n = 4:

3,4
3,4

2,4
2,4
1,4
1,4

3,3
3,3

same
subproblem
2,3
2,3

1,3
1,3

3,2
3,2

2,3
2,3
2,2
2,2

1,3
1,3

m+n

2,2
2,2

Height = m + n work potentially exponential.,

but were solving subproblems already solved!
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 26

L15.9

Dynamic-programming
hallmark #2
Overlapping subproblems
A recursive solution contains a
small number of distinct
subproblems repeated many times.
The number of distinct LCS subproblems for
two strings of lengths m and n is only mn.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 26

L15.10

Memoization algorithm
Memoization: After computing a solution to a
subproblem, store it in a table. Subsequent calls
check the table to avoid redoing work.
LCS(x, y, i, j)
if c[i, j] = NIL
then if x[i] = y[j]
then c[i, j] LCS(x, y, i1, j1) + 1
else c[i, j] max{ LCS(x, y, i1, j),
LCS(x, y, i, j1)}

same
as
before

Time = (mn) = constant work per table entry.

Space = (mn).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 26

L15.11

Dynamic-programming
algorithm
IDEA:
Compute the
table bottom-up.
Time = (mn).

A B C B D
00 00 00 00 00 00
B 00 00 11 11 11 11
D 00 00 11 11 11 22
C 00 00 11
A 00 11 11
B 00 11 22
A 00 11 22

2001 by Charles E. Leiserson

Introduction to Algorithms

A B
00 00
11 11

22 22
22 22 22 22 22
22 22 22 33 33
22 33 33 33 44
22 33 33 44 44
Day 26

L15.12

Dynamic-programming
algorithm
IDEA:
Compute the
table bottom-up.
Time = (mn).
Reconstruct
LCS by tracing
backwards.
Space = (mn).
Exercise:
O(min{m, n}).
2001 by Charles E. Leiserson

A B C B D
00 00 00 00 00 00
B 00 00 11 11 11 11
D 00 00 11 11 11 22
C 00 00 11
A 00 11 11
B 00 11 22
A 00 11 22
Introduction to Algorithms

A B
00 00
11 11

22 22
22 22 22 22 22
22 22 22 33 33
22 33 33 33 44
22 33 33 44 44
Day 26

L15.13

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 16
Prof. Charles E. Leiserson

Graphs (review)
Definition. A directed graph (digraph)
G = (V, E) is an ordered pair consisting of
a set V of vertices (singular: vertex),
a set E V V of edges.
In an undirected graph G = (V, E), the edge
set E consists of unordered pairs of vertices.
In either case, we have | E | = O(V 2). Moreover,
if G is connected, then | E | | V | 1, which
implies that lg | E | = (lg V).
(Review CLRS, Appendix B.)
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.2

Adjacency-matrix
representation
The adjacency matrix of a graph G = (V, E), where
V = {1, 2, , n}, is the matrix A[1 . . n, 1 . . n]
given by
1 if (i, j) E,
A[i, j] =
0 if (i, j) E.
22

11

33

44

2001 by Charles E. Leiserson

A 1 2 3 4
1 0 1 1 0
2 0 0 1 0
3 0 0 0 0
4 0 0 1 0
Introduction to Algorithms

(V 2) storage
dense
representation.
Day 27

L16.3

Adjacency-list representation
An adjacency list of a vertex v V is the list Adj[v]
of vertices adjacent to v.
Adj[1] = {2, 3}
22
11
Adj[2] = {3}
Adj[3] = {}
Adj[4] = {3}

33
44
For undirected graphs, | Adj[v] | = degree(v).
For digraphs, | Adj[v] | = out-degree(v).
Handshaking Lemma: vV = 2 |E| for undirected
graphs adjacency lists use (V + E) storage
a sparse representation (for either type of graph).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.4

Minimum spanning trees

Input: A connected, undirected graph G = (V, E)
with weight function w : E R.
For simplicity, assume that all edge weights are
distinct. (CLRS covers the general case.)
Output: A spanning tree T a tree that connects
all vertices of minimum weight:
w(T ) = w(u , v) .
(u ,v )T

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.5

Example of MST
6

12
9

5
14

8
3

2001 by Charles E. Leiserson

15

10

Introduction to Algorithms

Day 27

L16.6

Optimal substructure
MST T:
(Other edges of G
are not shown.)

u
T1

T2
v

Remove any edge (u, v) T. Then, T is partitioned

into two subtrees T1 and T2.
Theorem. The subtree T1 is an MST of G1 = (V1, E1),
the subgraph of G induced by the vertices of T1:
V1 = vertices of T1,
E1 = { (x, y) E : x, y V1 }.
Similarly for T2.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.7

Proof of optimal substructure

Proof. Cut and paste:
w(T) = w(u, v) + w(T1) + w(T2).
If T1 were a lower-weight spanning tree than T1 for
G1, then T = {(u, v)} T1 T2 would be a
lower-weight spanning tree than T for G.
Do we also have overlapping subproblems?
Yes.
Great, then dynamic programming may work!
Yes, but MST exhibits another powerful property
which leads to an even more efficient algorithm.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.8

Hallmark for greedy

algorithms
Greedy-choice property
A locally optimal choice
is globally optimal.
Theorem. Let T be the MST of G = (V, E),
and let A V. Suppose that (u, v) E is the
least-weight edge connecting A to V A.
Then, (u, v) T.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.9

Proof of theorem
Proof. Suppose (u, v) T. Cut and paste.
T:

v
A
VA

2001 by Charles E. Leiserson

u
(u, v) = least-weight edge
connecting A to V A

Introduction to Algorithms

Day 27

L16.10

Proof of theorem
Proof. Suppose (u, v) T. Cut and paste.
T:

v
A
VA

u
(u, v) = least-weight edge
connecting A to V A

Consider the unique simple path from u to v in T.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.11

Proof of theorem
Proof. Suppose (u, v) T. Cut and paste.
T:

v
A
VA

u
(u, v) = least-weight edge
connecting A to V A

Consider the unique simple path from u to v in T.

Swap (u, v) with the first edge on this path that
connects a vertex in A to a vertex in V A.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.12

Proof of theorem
Proof. Suppose (u, v) T. Cut and paste.
T :
A
VA

v
u
(u, v) = least-weight edge
connecting A to V A

Consider the unique simple path from u to v in T.

Swap (u, v) with the first edge on this path that
connects a vertex in A to a vertex in V A.
A lighter-weight spanning tree than T results.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.13

Prims algorithm
IDEA: Maintain V A as a priority queue Q. Key
each vertex in Q with the weight of the leastweight edge connecting it to a vertex in A.
QV
key[v] for all v V
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
do if v Q and w(u, v) < key[v]
then key[v] w(u, v)
DECREASE-KEY
[v] u

At the end, {(v, [v])} forms the MST.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.14

Example of Prims algorithm

A
VA

14

8
3

2001 by Charles E. Leiserson

12
9

7
15
00

10

Introduction to Algorithms

Day 27

L16.15

Example of Prims algorithm

A
VA

14

8
3

2001 by Charles E. Leiserson

12
9

7
15
00

10

Introduction to Algorithms

Day 27

L16.16

Example of Prims algorithm

A
VA

14

77
7

8
3

2001 by Charles E. Leiserson

12

00
10
10

9
15

15
15

10

Introduction to Algorithms

Day 27

L16.17

Example of Prims algorithm

A
VA

14

77
7

8
3

2001 by Charles E. Leiserson

12

00
10
10

9
15

15
15

10

Introduction to Algorithms

Day 27

L16.18

Example of Prims algorithm

A
VA

12
12

55
14

77
7

8
3

2001 by Charles E. Leiserson

12

00
10
10

9
15

99
15
15

10

Introduction to Algorithms

Day 27

L16.19

Example of Prims algorithm

A
VA

12
12

55
14

77
7

8
3

2001 by Charles E. Leiserson

12

00
10
10

9
15

99
15
15

10

Introduction to Algorithms

Day 27

L16.20

Example of Prims algorithm

A
VA

66

55
14
14
14

77
7

8
3

2001 by Charles E. Leiserson

12

00
88

9
15

99
15
15

10

Introduction to Algorithms

Day 27

L16.21

Example of Prims algorithm

A
VA

66

55
14
14
14

77
7

8
3

2001 by Charles E. Leiserson

12

00
88

9
15

99
15
15

10

Introduction to Algorithms

Day 27

L16.22

Example of Prims algorithm

A
VA

66

55
14
14
14

77
7

8
3

2001 by Charles E. Leiserson

12

00
88

9
15

99
15
15

10

Introduction to Algorithms

Day 27

L16.23

Example of Prims algorithm

A
VA

66

55
14
3

77
7

33

2001 by Charles E. Leiserson

12

00
88

9
15

99
15
15

10

Introduction to Algorithms

Day 27

L16.24

Example of Prims algorithm

A
VA

66

55
14
3

77
7

33

2001 by Charles E. Leiserson

12

00
88

9
15

99
15
15

10

Introduction to Algorithms

Day 27

L16.25

Example of Prims algorithm

A
VA

66

55
14
3

77
7

33

2001 by Charles E. Leiserson

12

00
88

9
15

99
15
15

10

Introduction to Algorithms

Day 27

L16.26

Example of Prims algorithm

A
VA

66

55
14
3

77
7

33

2001 by Charles E. Leiserson

12

00
88

9
15

99
15
15

10

Introduction to Algorithms

Day 27

L16.27

Analysis of Prim
QV
(V)
key[v] for all v V
total
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
|V |
do if v Q and w(u, v) < key[v]
times degree(u)
times
then key[v] w(u, v)
[v] u
Handshaking Lemma (E) implicit DECREASE-KEYs.

Time = (V)TEXTRACT-MIN + (E)TDECREASE-KEY

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.28

Analysis of Prim (continued)

Time = (V)TEXTRACT-MIN + (E)TDECREASE-KEY
Q

TEXTRACT-MIN TDECREASE-KEY

Total

array

O(V)

O(1)

O(V2)

binary
heap

O(lg V)

O(lg V)

O(E lg V)

Fibonacci O(lg V)
heap
amortized
2001 by Charles E. Leiserson

O(1)
O(E + V lg V)
amortized worst case

Introduction to Algorithms

Day 27

L16.29

MST algorithms
Kruskals algorithm (see CLRS):
Uses the disjoint-set data structure (Lecture 20).
Running time = O(E lg V).
Best to date:
Karger, Klein, and Tarjan [1993].
Randomized algorithm.
O(V + E) expected time.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 27

L16.30

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 17
Prof. Erik Demaine

Paths in graphs
Consider a digraph G = (V, E) with edge-weight
function w : E R. The weight of path p = v1
v2 L vk is defined to be
k 1

w( p ) = w(vi , vi +1 ) .
i =1

Example:

vv11

vv22

2001 by Charles E. Leiserson

vv33

vv44

Introduction to Algorithms

vv55
w(p) = 2
Day 29

L17.2

Shortest paths
A shortest path from u to v is a path of
minimum weight from u to v. The shortestpath weight from u to v is defined as
(u, v) = min{w(p) : p is a path from u to v}.
Note: (u, v) = if no path from u to v exists.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.3

Optimal substructure
Theorem. A subpath of a shortest path is a
shortest path.

Proof. Cut and paste:

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.4

Triangle inequality
Theorem. For all u, v, x V, we have
(u, v) (u, x) + (x, v).

Proof.
(u, v)

uu
(u, x)

vv
(x, v)

xx
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.5

Well-definedness of shortest
paths
If a graph G contains a negative-weight cycle,
then some shortest paths may not exist.
Example:

<0

uu
2001 by Charles E. Leiserson

vv
Introduction to Algorithms

Day 29

L17.6

Single-source shortest paths

Problem. From a given source vertex s V, find
the shortest-path weights (s, v) for all v V.
If all edge weights w(u, v) are nonnegative, all
shortest-path weights must exist.
IDEA: Greedy.
1. Maintain a set S of vertices whose shortestpath distances from s are known.
2. At each step add to S the vertex v V S
whose distance estimate from s is minimal.
3. Update the distance estimates of vertices
adjacent to v.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.7

Dijkstras algorithm
d[s] 0
for each v V {s}
do d[v]
S
QV
Q is a priority queue maintaining V S
while Q
do u EXTRACT-MIN(Q)
S S {u}
for each v Adj[u]
relaxation
do if d[v] > d[u] + w(u, v)
then d[v] d[u] + w(u, v)
step

Implicit DECREASE-KEY
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.8

Example of Dijkstras
algorithm
Graph with
nonnegative
edge weights:

10

AA

1 4
3

2001 by Charles E. Leiserson

BB

Introduction to Algorithms

CC

2
8

D
D
7 9

EE

Day 29

L17.9

Example of Dijkstras
algorithm

BB

Initialize:
10

0 AA
Q: A B C D E
0

1 4
3

CC

2
8

D
D
7 9

EE

S: {}
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.10

Example of Dijkstras
algorithm
A EXTRACT-MIN(Q):
10

0 AA
Q: A B C D E
0

BB

1 4
3

CC

2
8

D
D
7 9

EE

S: { A }
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.11

Example of Dijkstras
algorithm
Relax all edges leaving A:
10

0 AA
Q: A B C D E
0

10

10
BB

1 4
3

CC
3

2
8

D
D
7 9

EE

S: { A }
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.12

Example of Dijkstras
algorithm
C EXTRACT-MIN(Q):
10

0 AA
Q: A B C D E
0

10

10
BB

1 4
3

CC
3

2
8

D
D
7 9

EE

S: { A, C }
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.13

Example of Dijkstras
algorithm
Relax all edges leaving C:
10

0 AA
Q: A B C D E
0

10
7

2001 by Charles E. Leiserson

11

7
BB

1 4
3

CC
3

2
8

11
D
D
7 9

EE
5

S: { A, C }

Introduction to Algorithms

Day 29

L17.14

Example of Dijkstras
algorithm
E EXTRACT-MIN(Q):
10

0 AA
Q: A B C D E
0

10
7

2001 by Charles E. Leiserson

11

7
BB

1 4
3

CC
3

2
8

11
D
D
7 9

EE
5

S: { A, C, E }

Introduction to Algorithms

Day 29

L17.15

Example of Dijkstras
algorithm
Relax all edges leaving E:
10

0 AA
Q: A B C D E
0

10
7
7

2001 by Charles E. Leiserson

11
11

7
BB

1 4
3

CC
3

2
8

11
D
D
7 9

EE
5

S: { A, C, E }

Introduction to Algorithms

Day 29

L17.16

Example of Dijkstras
algorithm
B EXTRACT-MIN(Q):
10

0 AA
Q: A B C D E
0

10
7
7

2001 by Charles E. Leiserson

11
11

7
BB

1 4
3

CC
3

2
8

11
D
D
7 9

EE
5

S: { A, C, E, B }

Introduction to Algorithms

Day 29

L17.17

Example of Dijkstras
algorithm
Relax all edges leaving B:
10

0 AA
Q: A B C D E
0

10
7
7

2001 by Charles E. Leiserson

11
11
9

7
BB

1 4
3

CC
3

9
D
D

2
8

7 9

EE
5

S: { A, C, E, B }

Introduction to Algorithms

Day 29

L17.18

Example of Dijkstras
algorithm
D EXTRACT-MIN(Q):
10

0 AA
Q: A B C D E
0

10
7
7

2001 by Charles E. Leiserson

11
11
9

7
BB

1 4
3

CC
3

2
8

9
D
D
7 9

EE
5

S: { A, C, E, B, D }

Introduction to Algorithms

Day 29

L17.19

Correctness Part I
Lemma. Initializing d[s] 0 and d[v] for all
v V {s} establishes d[v] (s, v) for all v V,
and this invariant is maintained over any sequence
of relaxation steps.
Proof. Suppose not. Let v be the first vertex for
which d[v] < (s, v), and let u be the vertex that
caused d[v] to change: d[v] = d[u] + w(u, v). Then,
d[v] < (s, v)
supposition
(s, u) + (u, v) triangle inequality
(s,u) + w(u, v) sh. path specific path
d[u] + w(u, v)
v is first violation
Contradiction.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.20

Correctness Part II
Theorem. Dijkstras algorithm terminates with
d[v] = (s, v) for all v V.
Proof. It suffices to show that d[v] = (s, v) for every
v V when v is added to S. Suppose u is the first
vertex added to S for which d[u] (s, u). Let y be the
first vertex in V S along a shortest path from s to u,
and let x be its predecessor:

uu
S, just before
adding u.
2001 by Charles E. Leiserson

ss
Introduction to Algorithms

xx

yy
Day 29

L17.21

Correctness Part II
(continued)
S
ss

uu
xx

yy

Since u is the first vertex violating the claimed invariant,

we have d[x] = (s, x). Since subpaths of shortest paths
are shortest paths, it follows that d[y] was set to (s, x) +
w(x, y) = (s, y) when (x, y) was relaxed just after x was
added to S. Consequently, we have d[y] = (s, y) (s, u)
d[u]. But, d[u] d[y] by our choice of u, and hence d[y]
= (s, y) = (s, u) = d[u]. Contradiction.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.22

Analysis of Dijkstra
|V |
times

while Q
do u EXTRACT-MIN(Q)
S S {u}
for each v Adj[u]
degree(u)
do if d[v] > d[u] + w(u, v)
times
then d[v] d[u] + w(u, v)

Handshaking Lemma (E) implicit DECREASE-KEYs.

Time = (V)TEXTRACT-MIN + (E)TDECREASE-KEY

Note: Same formula as in the analysis of Prims
minimum spanning tree algorithm.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.23

Analysis of Dijkstra
(continued)
Time = (V)TEXTRACT-MIN + (E)TDECREASE-KEY
Q

TEXTRACT-MIN TDECREASE-KEY

Total

array

O(V)

O(1)

O(V2)

binary
heap

O(lg V)

O(lg V)

O(E lg V)

Fibonacci O(lg V)
heap
amortized
2001 by Charles E. Leiserson

O(1)
O(E + V lg V)
amortized worst case

Introduction to Algorithms

Day 29

L17.24

Unweighted graphs
Suppose w(u, v) = 1 for all (u, v) E. Can the
code for Dijkstra be improved?
Use a simple FIFO queue instead of a priority
queue.
Breadth-first search
while Q
do u DEQUEUE(Q)
for each v Adj[u]
do if d[v] =
then d[v] d[u] + 1
ENQUEUE(Q, v)

Analysis: Time = O(V + E).

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.25

Example of breadth-first
search
aa

ff

hh

dd
bb

gg
ee

ii

cc
Q:
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.26

Example of breadth-first
search
0

aa

ff

hh

dd
bb

gg
ee

ii

cc
0

Q: a
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.27

Example of breadth-first
search
0

aa

ff

hh

dd
1

bb

gg
ee

ii

cc
1 1

Q: a b d
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.28

Example of breadth-first
search
0

aa

ff

hh

dd
1

bb

gg
ee

cc

ii

2
1 2 2

Q: a b d c e
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.29

Example of breadth-first
search
0

aa

ff

hh

dd
1

bb

gg
ee

cc

ii

2
2 2

Q: a b d c e
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.30

Example of breadth-first
search
0

aa

ff

hh

dd
1

bb

gg
ee

cc

ii

2
2

Q: a b d c e
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.31

Example of breadth-first
search
0

aa

dd
1

bb
cc

ff

1
3

hh

gg

ee

ii

3
3 3

Q: a b d c e g i
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.32

Example of breadth-first
search
4
0

aa

dd
1

bb
cc

ff

1
3

hh

gg

ee

ii

3
3 4

Q: a b d c e g i f
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.33

Example of breadth-first
search
0

aa

dd
1

bb
cc

ff

hh

gg

ee

ii

3
4 4

Q: a b d c e g i f h
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.34

Example of breadth-first
search
0

aa

dd
1

bb
cc

ff

hh

gg

ee

ii

3
4

Q: a b d c e g i f h
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.35

Example of breadth-first
search
0

aa

dd
1

bb
cc

ff

hh

gg

ee

ii

Q: a b d c e g i f h
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.36

Example of breadth-first
search
0

aa

dd
1

bb
cc

ff

hh

gg

ee

ii

Q: a b d c e g i f h
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.37

Correctness of BFS
while Q
do u DEQUEUE(Q)
for each v Adj[u]
do if d[v] =
then d[v] d[u] + 1
ENQUEUE(Q, v)

Key idea:
The FIFO Q in breadth-first search mimics
the priority queue Q in Dijkstra.
Invariant: v comes after u in Q implies that
d[v] = d[u] or d[v] = d[u] + 1.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 29

L17.38

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 18
Prof. Erik Demaine

Negative-weight cycles
Recall: If a graph G = (V, E) contains a negativeweight cycle, then some shortest paths may not exist.

Example:
<0
uu

vv

Bellman-Ford algorithm: Finds all shortest-path

lengths from a source s V to all v V or
determines that a negative-weight cycle exists.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 31

L18.2

Bellman-Ford algorithm
d[s] 0
for each v V {s}
do d[v]

initialization

for i 1 to | V | 1
do for each edge (u, v) E
do if d[v] > d[u] + w(u, v)
then d[v] d[u] + w(u, v)

relaxation
step

for each edge (u, v) E

do if d[v] > d[u] + w(u, v)
then report that a negative-weight cycle exists

At the end, d[v] = (s, v). Time = O(VE).

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 31

L18.3

Example of Bellman-Ford

BB

AA
4

CC

2001 by Charles E. Leiserson

1
5

EE

D
D

A B C D E
0

Introduction to Algorithms

Day 31

L18.4

Example of Bellman-Ford
1

BB

AA
4

CC

2001 by Charles E. Leiserson

1
5

EE

D
D

A B C D E
0
0 1

Introduction to Algorithms

Day 31

L18.5

Example of Bellman-Ford
1

BB

AA
4

CC
4

2001 by Charles E. Leiserson

1
5

EE

D
D

A
0
0
0

1
1

Introduction to Algorithms

Day 31

L18.6

Example of Bellman-Ford

1
BB

AA
4

CC
4
2

2001 by Charles E. Leiserson

1
5

EE

D
D

Introduction to Algorithms

A
0
0
0
0

1
1
1

4
2

Day 31

L18.7

Example of Bellman-Ford
1

BB

AA
4

CC
2

2001 by Charles E. Leiserson

1
5

EE

D
D

Introduction to Algorithms

A
0
0
0
0

1
1
1

4
2

Day 31

L18.8

Example of Bellman-Ford
1

BB

AA
4

CC
2

2001 by Charles E. Leiserson

1
5

EE

D
D

Introduction to Algorithms

A
0
0
0
0
0

1
1
1
1

4
2
2

Day 31

L18.9

Example of Bellman-Ford

1
BB

AA
4

CC
2

2001 by Charles E. Leiserson

1
5

EE

D
D
1

Introduction to Algorithms

A
0
0
0
0
0
0

1
1
1
1
1

4
2
2
2

Day 31

1
1

L18.10

Example of Bellman-Ford
1

BB

AA
4

CC
2

2001 by Charles E. Leiserson

1
5

EE

D
D
2
1

Introduction to Algorithms

A
0
0
0
0
0
0
0

1
1
1
1
1
1

4
2
2
2
2

1
2
Day 31

1
1
1
L18.11

Example of Bellman-Ford
1

BB

AA
4

EE
3

CC 5 D
D
2
1
2
Note: Values decrease
monotonically.

2001 by Charles E. Leiserson

Introduction to Algorithms

A
0
0
0
0
0
0
0

1
1
1
1
1
1

4
2
2
2
2

1
2
Day 31

1
1
1
L18.12

Correctness
Theorem. If G = (V, E) contains no negativeweight cycles, then after the Bellman-Ford
algorithm executes, d[v] = (s, v) for all v V.
Proof. Let v V be any vertex, and consider a shortest
path p from s to v with the minimum number of edges.

s
p: vv0
0

vv11

vv22

vv33

vvkk

Since p is a shortest path, we have

(s, vi) = (s, vi1) + w(vi1, vi) .
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 31

L18.13

Correctness (continued)
s
p: vv0
0

vv11

vv22

vv33

v
vvkk

Initially, d[v0] = 0 = (s, v0), and d[s] is unchanged by

subsequent relaxations (because of the lemma from
Lecture 17 that d[v] (s, v)).
After 1 pass through E, we have d[v1] = (s, v1).
After 2 passes through E, we have d[v2] = (s, v2).
M
After k passes through E, we have d[vk] = (s, vk).
Since G contains no negative-weight cycles, p is simple.
Longest simple path has |V| 1 edges.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 31

L18.14

Detection of negative-weight
cycles
Corollary. If a value d[v] fails to converge after
|V| 1 passes, there exists a negative-weight
cycle in G reachable from s.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 31

L18.15

DAG shortest paths

If the graph is a directed acyclic graph (DAG), we first
topologically sort the vertices.
Determine f : V {1, 2, , | V |} such that (u, v) E
f (u) < f (v).
O(V + E) time using depth-first search.

11
s 22

44
33

55

77
66

88
99

Walk through the vertices u V in this order, relaxing

the edges in Adj[u], thereby obtaining the shortest paths
from s in a total of O(V + E) time.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 31

L18.16

Linear programming
Let A be an mn matrix, b be an m-vector, and c
be an n-vector. Find an n-vector x that maximizes
cTx subject to Ax b, or determine that no such
solution exists.
n
m

.
A

2001 by Charles E. Leiserson

maximizing

x b
Introduction to Algorithms

cT

x
Day 31

L18.17

Linear-programming
algorithms
Algorithms for the general problem
Simplex methods practical, but worst-case
exponential time.
Ellipsoid algorithm polynomial time, but
slow in practice.
Interior-point methods polynomial time and
competes with simplex.
Feasibility problem: No optimization criterion.
Just find x such that Ax b.
In general, just as hard as ordinary LP.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 31

L18.18

Solving a system of difference

constraints
Linear programming where each row of A contains
exactly one 1, one 1, and the rest 0s.
Example:
Solution:
x1 = 3
x1 x2 3
x2 = 0
xj xi wij
x2 x3 2
x3 = 2
x1 x3 2
Constraint graph:
xj xi wij
2001 by Charles E. Leiserson

vvii

wij

Introduction to Algorithms

vvjj

(The A
matrix has
dimensions
|E | |V |.)
Day 31

L18.19

Unsatisfiable constraints
Theorem. If the constraint graph contains
a negative-weight cycle, then the system of
differences is unsatisfiable.
Proof. Suppose that the negative-weight cycle is
v1 v2 L vk v1. Then, we have
x2 x1
x3 x2

w12
w23
M

xk xk1 wk1, k
x1 xk wk1
0
2001 by Charles E. Leiserson

weight of cycle
<0
Introduction to Algorithms

Therefore, no
values for the xi
can satisfy the
constraints.
Day 31

L18.20

Satisfying the constraints

Theorem. Suppose no negative-weight cycle
exists in the constraint graph. Then, the
constraints are satisfiable.

Proof. Add a new vertex s to V with a 0-weight edge

to each vertex vi V.

vv11

vv44
vv77

2001 by Charles E. Leiserson

vv99
vv33

Introduction to Algorithms

Note:
No negative-weight
cycles introduced
shortest paths exist.

Day 31

L18.21

Proof (continued)
Claim: The assignment xi = (s, vi) solves the constraints.
Consider any constraint xj xi wij, and consider the
shortest paths from s to vj and vi:

ss

(s, vi)
(s, vj)

vvii
wij

vvjj

The triangle inequality gives us (s,vj) (s, vi) + wij.

Since xi = (s, vi) and xj = (s, vj), the constraint xj xi
wij is satisfied.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 31

L18.22

Bellman-Ford and linear

programming
Corollary. The Bellman-Ford algorithm can
solve a system of m difference constraints on n
variables in O(m n) time.
Single-source shortest paths is a simple LP
problem.
In fact, Bellman-Ford maximizes x1 + x2 + L + xn
subject to the constraints xj xi wij and xi 0
(exercise).
Bellman-Ford also minimizes maxi{xi} mini{xi}
(exercise).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 31

L18.23

Application to VLSI layout

compaction
Integrated
-circuit
features:
minimum separation
Problem: Compact (in one dimension) the
space between the features of a VLSI layout
without bringing any features too close together.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 31

L18.24

VLSI layout compaction

d1
11

x1

x2

x2 x1 d 1 +
Bellman-Ford minimizes maxi{xi} mini{xi},
which compacts the layout in the x-dimension.
Constraint:

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 31

L18.25

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 19
Prof. Erik Demaine

Shortest paths
Single-source shortest paths
Nonnegative edge weights

Dijkstras algorithm: O(E + V lg V)

General

Bellman-Ford: O(VE)

DAG

One pass of Bellman-Ford: O(V + E)

All-pairs shortest paths

Nonnegative edge weights

Dijkstras algorithm |V| times: O(VE + V 2 lg V)

General

Three algorithms today.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.2

All-pairs shortest paths

Input: Digraph G = (V, E), where |V | = n, with
edge-weight function w : E R.
Output: n n matrix of shortest-path lengths
(i, j) for all i, j V.
IDEA #1:
Run Bellman-Ford once from each vertex.
Time = O(V 2E).
Dense graph O(V 4) time.
Good first try!
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.3

Dynamic programming
Consider the n n adjacency matrix A = (aij)
of the digraph, and define
dij(m) = weight of a shortest path from
i to j that uses at most m edges.
Claim: We have
0 if i = j,
(0)
dij =
if i j;
and for m = 1, 2, , n 1,
dij(m) = mink{dik(m1) + akj }.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.4

Proof of claim

ks

dij(m) = mink{dik(m1) + akj }

es
g
d
1e

ii
Relaxation!

s
e
g
d
e
1
m
m
1
edg
es

for k 1 to n
do if dij > dik + akj
then dij dik + akj

jj
M

m 1 edges

Note: No negative-weight cycles implies

(i, j) = dij (n1) = dij (n) = dij (n+1) = L
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.5

Matrix multiplication
Compute C = A B, where C, A, and B are n n
matrices:
n
cij = aik bkj .
k =1

Time = (n3) using the standard algorithm.

What if we map + min and +?
cij = mink {aik + bkj}.
Thus, D(m) = D(m1) A.
Identity matrix = I =
2001 by Charles E. Leiserson

0
0
0
0

Introduction to Algorithms

= D0 = (dij(0)).
Day 32

L19.6

Matrix multiplication
(continued)
The (min, +) multiplication is associative, and
with the real numbers, it forms an algebraic
structure called a closed semiring.
Consequently, we can compute
D(1) = D(0) A = A1
D(2) = D(1) A = A2
M
M
D(n1) = D(n2) A = An1 ,
yielding D(n1) = ((i, j)).
Time = (nn3) = (n4). No better than n B-F.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.7

Improved matrix
multiplication algorithm
Repeated squaring: A2k = Ak Ak.
lg(n1)
2
4
2
.
Compute A , A , , A

O(lg n) squarings
Note: An1 = An = An+1 = L.
Time = (n3 lg n).
To detect negative-weight cycles, check the
diagonal for negative values in O(n) additional
time.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.8

Floyd-Warshall algorithm
Also dynamic programming, but faster!
Define cij(k) = weight of a shortest path from i
to j with intermediate vertices
belonging to the set {1, 2, , k}.

ii

kk

kk

kk

kk

jj

Thus, (i, j) = cij(n). Also, cij(0) = aij .

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.9

Floyd-Warshall recurrence
cij(k) = mink {cij(k1), cik(k1) + ckj(k1)}
cik
ii

(k1)

cij(k1)

ckj(k1)
jj

intermediate vertices in {1, 2, , k}

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.10

Pseudocode for FloydWarshall

for k 1 to n
do for i 1 to n
do for j 1 to n
do if cij > cik + ckj
then cij cik + ckj

relaxation

Notes:
Okay to omit superscripts, since extra relaxations
cant hurt.
Runs in (n3) time.
Simple to code.
Efficient in practice.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.11

Transitive closure of a
directed graph
Compute tij =

1 if there exists a path from i to j,

0 otherwise.

IDEA: Use Floyd-Warshall, but with (, ) instead

of (min, +):

tij(k) = tij(k1) (tik(k1) tkj(k1)).

Time = (n3).

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.12

Graph reweighting
Theorem. Given a label h(v) for each v V, reweight
each edge (u, v) E by
(u, v) = w(u, v) + h(u) h(v).
Then, all paths between the same two vertices are
reweighted by the same amount.
Proof. Let p = v1 v2 L vk be a path in the graph.
k 1

Then, we have w ( p ) = w ( vi ,vi +1 )

=
=

i =1
k 1

( w(vi ,vi +1 )+ h (vi )h (vi +1 ) )

i =1
k 1

w(vi ,vi +1 ) + h (v1 ) h (vk )

i =1

= w( p ) + h ( v1 ) h ( v k ) .
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.13

Johnsons algorithm
1. Find a vertex labeling h such that (u, v) 0 for all
(u, v) E by using Bellman-Ford to solve the
difference constraints
h(v) h(u) w(u, v),
or determine that a negative-weight cycle exists.
Time = O(V E).
2. Run Dijkstras algorithm from each vertex using .
Time = O(V E + V 2 lg V).
3. Reweight each shortest-path length (p) to produce
the shortest-path lengths w(p) of the original graph.
Time = O(V 2).

Total time = O(V E + V 2 lg V).

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 32

L19.14

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 20
Prof. Erik Demaine

Disjoint-set data structure

(Union-Find)
Problem: Maintain a dynamic collection of
pairwise-disjoint sets S = {S1, S2, , Sr}.
Each set Si has one element distinguished as the
representative element, rep[Si].
Must support 3 operations:
MAKE-SET(x): adds new set {x} to S
with rep[{x}] = x (for any x Si for all i).
UNION(x, y): replaces sets Sx, Sy with Sx Sy
in S for any x, y in distinct sets Sx, Sy .
FIND-SET(x): returns representative rep[Sx]
of set Sx containing element x.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.2

Simple linked-list solution

Store each set Si = {x1, x2, , xk} as an (unordered)
doubly linked list. Define representative element
rep[Si] to be the front of the list, x1.
Si :

x1
rep[Si]

x2

xk

MAKE-SET(x) initializes x as a lone node. (1)

FIND-SET(x) walks left in the list containing x
(n)
until it reaches the front of the list.
UNION(x, y) concatenates the lists containing
x and y, leaving rep. as FIND-SET[x].
(n)
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.3

Simple balanced-tree solution

Store each set Si = {x1, x2, , xk} as a balanced tree
(ignoring keys). Define representative element
rep[Si] to be the root of the tree.
MAKE-SET(x) initializes x
(1)
as a lone node.
FIND-SET(x) walks up the
tree containing x until it
reaches the root.
(lg n)
UNION(x, y) concatenates
the trees containing x and y,
(lg n)
changing rep.
2001 by Erik D. Demaine

Introduction to Algorithms

Si = {x1, x2, x3, x4, x5}

rep[Si] x1

x4
x2

x3
x5
Day 33

L20.4

Plan of attack
We will build a simple disjoint-union data structure
that, in an amortized sense, performs significantly
better than (lg n) per op., even better than
(lg lg n), (lg lg lg n), etc., but not quite (1).
To reach this goal, we will introduce two key tricks.
Each trick converts a trivial (n) solution into a
simple (lg n) amortized solution. Together, the
two tricks yield a much better solution.
First trick arises in an augmented linked list.
Second trick arises in a tree structure.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.5

Augmented linked-list solution

Store set Si = {x1, x2, , xk} as unordered doubly
linked list. Define rep[Si] to be front of list, x1.
Each element xj also stores pointer rep[xj] to rep[Si].
rep

Si :

x1
rep[Si]

x2

xk

FIND-SET(x) returns rep[x].

(1)
UNION(x, y) concatenates the lists containing
x and y, and updates the rep pointers for
(n)
all elements in the list containing y.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.6

Example of
augmented linked-list solution
Each element xj stores pointer rep[xj] to rep[Si].
UNION(x, y)
concatenates the lists containing x and y, and
updates the rep pointers for all elements in the
list containing y.
rep

Sx :

x1
rep[Sx]

x2

rep

Sy :
2001 by Erik D. Demaine

y1
rep[Sy]

Introduction to Algorithms

y2

y3
Day 33

L20.7

Example of
augmented linked-list solution
Each element xj stores pointer rep[xj] to rep[Si].
UNION(x, y)
concatenates the lists containing x and y, and
updates the rep pointers for all elements in the
list containing y.
rep
Sx Sy :
x1
rep[Sx]

x2

rep
y1
rep[Sy]

2001 by Erik D. Demaine

Introduction to Algorithms

y2

y3
Day 33

L20.8

Example of
augmented linked-list solution
Each element xj stores pointer rep[xj] to rep[Si].
UNION(x, y)
concatenates the lists containing x and y, and
updates the rep pointers for all elements in the
list containing y.
rep
Sx Sy :
x1
rep[Sx Sy]

x2
y1

2001 by Erik D. Demaine

Introduction to Algorithms

y2

y3
Day 33

L20.9

Alternative concatenation
UNION(x, y) could instead
concatenate the lists containing y and x, and
update the rep pointers for all elements in the
list containing x.
rep
rep

Sy :

y1
rep[Sy]

2001 by Erik D. Demaine

y2

Sx :

x1
rep[Sx]

x2

y3
Introduction to Algorithms

Day 33

L20.10

Alternative concatenation
UNION(x, y) could instead
concatenate the lists containing y and x, and
update the rep pointers for all elements in the
list containing x.
rep

Sx Sy :
y1
rep[Sy]
2001 by Erik D. Demaine

x1
rep[Sx]

rep
y2

x2

y3
Introduction to Algorithms

Day 33

L20.11

Alternative concatenation
UNION(x, y) could instead
concatenate the lists containing y and x, and
update the rep pointers for all elements in the
list containing x.
rep

Sx Sy :
y1
rep[Sx Sy]
2001 by Erik D. Demaine

x1

rep
y2

x2

y3
Introduction to Algorithms

Day 33

L20.12

Trick 1: Smaller into larger

To save work, concatenate smaller list onto the end
of the larger list. Cost = (length of smaller list).
Augment list to store its weight (# elements).
Let n denote the overall number of elements
(equivalently, the number of MAKE-SET operations).
Let m denote the total number of operations.
Let f denote the number of FIND-SET operations.
Theorem: Cost of all UNIONs is O(n lg n).
Corollary: Total cost is O(m + n lg n).
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.13

Analysis of Trick 1
To save work, concatenate smaller list onto the end
of the larger list. Cost = (1 + length of smaller list).
Theorem: Total cost of UNIONs is O(n lg n).
Proof. Monitor an element x and set Sx containing it.
After initial MAKE-SET(x), weight[Sx] = 1. Each
time Sx is united with set Sy, weight[Sy] weight[Sx],
pay 1 to update rep[x], and weight[Sx] at least
doubles (increasing by weight[Sy]). Each time Sy is
united with smaller set Sy, pay nothing, and
weight[Sx] only increases. Thus pay lg n for x.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.14

Representing sets as trees

Store each set Si = {x1, x2, , xk} as an unordered,
potentially unbalanced, not necessarily binary tree,
storing only parent pointers. rep[Si] is the tree root.
MAKE-SET(x) initializes x
Si = {x1, x2, x3, x4, x5 , x6}
(1)
as a lone node.
rep[Si] x1
FIND-SET(x) walks up the
tree containing x until it
x4
x3
reaches the root. (depth[x])
UNION(x, y) concatenates
the trees containing x and y
x2 x5 x6
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.15

Trick 1 adapted to trees

UNION(x, y) can use a simple concatenation strategy:
Make root FIND-SET(y) a child of root FIND-SET(x).
FIND-SET(y) = FIND-SET(x).
x1
We can adapt Trick 1
to this context also:
x
x
y1
4
3
Merge tree with smaller
weight into tree with
x2 x5 x6
y4 y3
larger weight.
Height of tree increases only when its size y
2
doubles, so height is logarithmic in weight.
Thus total cost is O(m + f lg n).
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

y5
L20.16

Trick 2: Path compression

When we execute a FIND-SET operation and walk
up a path p to the root, we know the representative
for all the nodes on path p.
x1
Path compression makes
x4
x3 y1
all of those nodes direct
children of the root.
x2 x5 x6
y4 y3
Cost of FIND-SET(x)
is still (depth[x]).
FIND-SET(y2) y2 y5
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.17

Trick 2: Path compression

When we execute a FIND-SET operation and walk
up a path p to the root, we know the representative
for all the nodes on path p.
x1
Path compression makes
x4
x3 y1
all of those nodes direct
children of the root.
x2 x5 x6
y4 y3
Cost of FIND-SET(x)
is still (depth[x]).
FIND-SET(y2) y2 y5
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.18

Trick 2: Path compression

When we execute a FIND-SET operation and walk
up a path p to the root, we know the representative
for all the nodes on path p.
x1
Path compression makes
x4
x3 y1 y2 y3
all of those nodes direct
children of the root.
x2 x5 x6 y4
y5
Cost of FIND-SET(x)
is still (depth[x]).
FIND-SET(y2)
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.19

Analysis of Trick 2 alone

Theorem: Total cost of FIND-SETs is O(m lg n).
Proof: Amortization by potential function.
The weight of a node x is # nodes in its subtree.
Define (x1, , xn) = i lg weight[xi].
UNION(xi, xj) increases potential of root FIND-SET(xi)
by at most lg weight[root FIND-SET(xj)] lg n.
Each step down p c made by FIND-SET(xi),
except the first, moves cs subtree out of ps subtree.
Thus if weight[c] weight[p], decreases by 1,
paying for the step down. There can be at most lg n
steps p c for which weight[c] < weight[p].
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.20

Analysis of Trick 2 alone

Theorem: If all UNION operations occur before
all FIND-SET operations, then total cost is O(m).
Proof: If a FIND-SET operation traverses a path
with k nodes, costing O(k) time, then k 2 nodes
are made new children of the root. This change
can happen only once for each of the n elements,
so the total cost of FIND-SET is O(f + n).

2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.21

Ackermanns function A
j + 1 if k = 0,
Define Ak ( j ) = ( j +1)
Ak 1 ( j ) if k 1. iterate j+1 times
A0(1) = 2
A0(j) = j + 1
A1(1) = 3
A1(j) ~ 2 j
A2(1) = 7
A2(j) ~ 2j 2j > 2j
j
A3(1) = 2047
2
.
2

..

.2

2
A3(j) > 2
2
A4(j) is a lot bigger. A4(1) > 2

..

2047

2048

Define (n) = min {k : Ak(1) n} 4 for practical n.

2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.22

Analysis of Tricks 1 + 2
Theorem: In general, total cost is O(m (n)).
(long, tricky proof see Section 21.4 of CLRS)

2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.23

Application:
Dynamic connectivity
Suppose a graph is given to us incrementally by
ADD-VERTEX(v)
ADD-EDGE(u, v)
and we want to support connectivity queries:
CONNECTED(u, v):
Are u and v in the same connected component?
For example, we want to maintain a spanning forest,
so we check whether each new edge connects a
previously disconnected pair of vertices.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.24

Application:
Dynamic connectivity
Sets of vertices represent connected components.
Suppose a graph is given to us incrementally by
ADD-VERTEX(v) MAKE-SET(v)
ADD-EDGE(u, v) if not CONNECTED(u, v)
then UNION(v, w)
and we want to support connectivity queries:
CONNECTED(u, v): FIND-SET(u) = FIND-SET(v)
Are u and v in the same connected component?
For example, we want to maintain a spanning forest,
so we check whether each new edge connects a
previously disconnected pair of vertices.
2001 by Erik D. Demaine

Introduction to Algorithms

Day 33

L20.25

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 21
Prof. Charles E. Leiserson

Take-home quiz

No notes (except this one).

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 35

L21.2

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 22
Prof. Charles E. Leiserson

Flow networks
Definition. A flow network is a directed graph
G = (V, E) with two distinguished vertices: a
source s and a sink t. Each edge (u, v) E has
a nonnegative capacity c(u, v). If (u, v) E,
then c(u, v) = 0.
Example:

2
3

3
1

ss
2
2001 by Charles E. Leiserson

3
Introduction to Algorithms

tt

2
2

Day 38

L22.2

Flow networks
Definition. A positive flow on G is a function
p : V V R satisfying the following:
Capacity constraint: For all u, v V,
0 p(u, v) c(u, v).
Flow conservation: For all u V {s, t},

p(u, v) p(v, u ) = 0 .

vV

vV

The value of a flow is the net flow out of the

source:
p ( s , v ) p (v, s ) .
vV

2001 by Charles E. Leiserson

vV

Introduction to Algorithms

Day 38

L22.3

A flow on a network
positive
flow

capacity
2:2
2:3

1:3
0:1

ss
2:2

1:3 1:1 2:3

2:3

tt

1:2
1:2

Flow conservation (like Kirchoffs current law):

Flow into u is 2 + 1 = 3.
Flow out of u is 0 + 1 + 2 = 3.
The value of this flow is 1 0 + 2 = 3.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 38

L22.4

The maximum-flow problem

Maximum-flow problem: Given a flow network
G, find a flow of maximum value on G.
2:2
2:3

2:3
0:1

ss
2:2

0:3 1:1 2:3

3:3

tt

1:2
2:2

The value of the maximum flow is 4.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 38

L22.5

Flow cancellation
Without loss of generality, positive flow goes
either from u to v, or from v to u, but not both.
vv
2:3

vv
1:2

uu

1:3

0:2

uu

Net flow from

u to v in both
cases is 1.

The capacity constraint and flow conservation

are preserved by this transformation.
INTUITION: View flow as a rate, not a quantity.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 38

L22.6

A notational simplification
IDEA: Work with the net flow between two
vertices, rather than with the positive flow.
Definition. A (net) flow on G is a function
f : V V R satisfying the following:
Capacity constraint: For all u, v V,
f (u, v) c(u, v).
Flow conservation: For all u V {s, t},
summation
f (u, v) = 0. One
instead of two.
vV
Skew symmetry: For all u, v V,
f (u, v) = f (v, u).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 38

L22.7

Equivalence of definitions
Theorem. The two definitions are equivalent.
Proof. () Let f (u, v) = p(u, v) p(v, u).
Capacity constraint: Since p(u, v) c(u, v) and
p(v, u) 0, we have f (u, v) c(u, v).
Flow conservation:
( p(u, v) p(v, u ) )
f (u , v) =

vV

= p (u , v) p (v, u )
vV
vV

vV

Skew symmetry:
f (u, v) = p(u, v) p(v, u)
= (p(v, u) p(u, v))
= f (v, u).
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 38

L22.8

Proof (continued)
() Let
p(u, v) =

f (u, v) if f(u, v) > 0,

0
if f(u, v) 0.

Capacity constraint: By definition, p(u, v) 0. Since f

(u, v) c(u, v), it follows that p(u, v) c(u, v).
Flow conservation: If f (u, v) > 0, then p(u, v) p(v, u)
= f (u, v). If f (u, v) 0, then p(u, v) p(v, u) = f (v, u)
= f (u, v) by skew symmetry. Therefore,

p(u, v) p(v, u) = f (u, v) .

vV

2001 by Charles E. Leiserson

vV

Introduction to Algorithms

vV

Day 38

L22.9

Notation
Definition. The value of a flow f, denoted by | f |,
is given by
f = f ( s, v )
vV

= f ( s, V ) .
Implicit summation notation: A set used in
an arithmetic formula represents a sum over
the elements of the set.
Example flow conservation:
f (u, V) = 0 for all u V {s, t}.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 38

L22.10

Simple properties of flow

Lemma.
f (X, X) = 0,
f (X, Y) = f (Y, X),
f (XY, Z) = f (X, Z) + f (Y, Z) if XY = .
Theorem. | f | = f (V, t).
Proof.

|f| =
=
=
=
=

2001 by Charles E. Leiserson

f (s, V)
f (V, V) f (Vs, V)
Omit braces.
f (V, Vs)
f (V, t) + f (V, Vst)
f (V, t).
Introduction to Algorithms

Day 38

L22.11

Flow into the sink

2:2
2:3

2:3
0:1

ss

0:3 1:1 1:3

2:2

3:3

| f | = f (s, V) = 4

2001 by Charles E. Leiserson

Introduction to Algorithms

tt

0:2
2:2

f (V, t) = 4

Day 38

L22.12

Cuts
Definition. A cut (S, T) of a flow network G =
(V, E) is a partition of V such that s S and t T.
If f is a flow on G, then the flow across the cut is
f (S, T).
2:2
2:3

2:3
0:1

ss
2:2

0:3 1:1 1:3

3:3

tt

0:2
2:2

f (S, T) = (2 + 2) + ( 2 + 1 1 + 2)
=4

2001 by Charles E. Leiserson

S
T

Introduction to Algorithms

Day 38

L22.13

Another characterization of
flow value
Lemma. For any flow f and any cut (S, T), we
have | f | = f (S, T).
Proof.

2001 by Charles E. Leiserson

f (S, T) = f (S, V) f (S, S)

= f (S, V)
= f (s, V) + f (Ss, V)
= f (s, V)
= | f |.

Introduction to Algorithms

Day 38

L22.14

Capacity of a cut
Definition. The capacity of a cut (S, T) is c(S, T).
2:2
2:3

2:3
0:1

ss
2:2

0:3 1:1 1:3

tt

0:2

3:3

S
T

2:2

c(S, T) = (3 + 2) + (1 + 2 + 3)
= 11
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 38

L22.15

Upper bound on the maximum

flow value
Theorem. The value of any flow is bounded
above by the capacity of any cut.
Proof.

f = f (S ,T )
= f (u , v)
uS vT

c(u , v)
uS vT

= c( S , T ) .

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 38

L22.16

Residual network
Definition. Let f be a flow on G = (V, E). The
residual network Gf (V, Ef ) is the graph with
strictly positive residual capacities
cf (u, v) = c(u, v) f (u, v) > 0.
Edges in Ef admit more flow.
Example:

G:

0:1

uu

vv

Gf :

3:5

uu

vv
2

Lemma. |Ef | 2|E|.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 38

L22.17

Augmenting paths
Definition. Any path from s to t in Gf is an augmenting path in G with respect to f. The flow
value can be increased along an augmenting
path p by c f ( p ) = min {c f (u , v)}.
(u ,v ) p

Ex.:

3:5

G:

2:5

tt
2

5:5
7

2:3
2

ss

tt
3

2001 by Charles E. Leiserson

0:2

ss

cf (p) = 2
Gf :

2:6

2
Introduction to Algorithms

2
Day 38

L22.18

Max-flow, min-cut theorem

Theorem. The following are equivalent:
1. f is a maximum flow.
2. f admits no augmenting paths.
3. | f | = c(S, T) for some cut (S, T).
Proof (and algorithms). Next time.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 38

L22.19

Introduction to Algorithms
6.046J/18.401J/SMA5503

Lecture 23
Prof. Charles E. Leiserson

Recall from Lecture 22

Flow value: | f | = f (s, V).
Cut: Any partition (S, T) of V such that s S
and t T.
Lemma. | f | = f (S, T) for any cut (S, T).
Corollary. | f | c(S, T) for any cut (S, T).
Residual graph: The graph Gf = (V, Ef ) with
strictly positive residual capacities cf (u, v) =
c(u, v) f (u, v) > 0.
Augmenting path: Any path from s to t in Gf .
Residual capacity of an augmenting path:
c f ( p ) = min {c f (u , v)} .
(u ,v ) p

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.2

Max-flow, min-cut theorem

Theorem. The following are equivalent:
1. | f | = c(S, T) for some cut (S, T).
2. f is a maximum flow.
3. f admits no augmenting paths.
Proof.
(1) (2): Since | f | c(S, T) for any cut (S, T) (by
the corollary from Lecture 22), the assumption that
| f | = c(S, T) implies that f is a maximum flow.
(2) (3): If there were an augmenting path, the
flow value could be increased, contradicting the
maximality of f.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.3

Proof (continued)
(3) (1): Suppose that f admits no augmenting paths.
Define S = {v V : there exists a path in Gf from s to v},
and let T = V S. Observe that s S and t T, and thus
(S, T) is a cut. Consider any vertices u S and v T.

ss

path in Gf

uu

vv
S

We must have cf (u, v) = 0, since if cf (u, v) > 0, then v S,

not v T as assumed. Thus, f (u, v) = c(u, v), since cf (u, v)
= c(u, v) f (u, v). Summing over all u S and v T
yields f (S, T) = c(S, T), and since | f | = f (S, T), the theorem
follows.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.4

Ford-Fulkerson max-flow
algorithm
Algorithm:

f [u, v] 0 for all u, v V

while an augmenting path p in G wrt f exists
do augment f by cf (p)

Can be slow:
109

G:

ss
109

2001 by Charles E. Leiserson

109

tt
109

Introduction to Algorithms

Day 40

L23.5

Ford-Fulkerson max-flow
algorithm
Algorithm:

f [u, v] 0 for all u, v V

while an augmenting path p in G wrt f exists
do augment f by cf (p)

Can be slow:
0:109

G:

0:1

ss
0:109

2001 by Charles E. Leiserson

0:109

tt
0:109

Introduction to Algorithms

Day 40

L23.6

Ford-Fulkerson max-flow
algorithm
Algorithm:

f [u, v] 0 for all u, v V

while an augmenting path p in G wrt f exists
do augment f by cf (p)

Can be slow:
0:109

G:

0:1

ss
0:109

2001 by Charles E. Leiserson

0:109

tt
0:109

Introduction to Algorithms

Day 40

L23.7

Ford-Fulkerson max-flow
algorithm
Algorithm:

f [u, v] 0 for all u, v V

while an augmenting path p in G wrt f exists
do augment f by cf (p)

Can be slow:
1:109

G:

1:1

ss
0:109

2001 by Charles E. Leiserson

0:109

tt
1:109

Introduction to Algorithms

Day 40

L23.8

Ford-Fulkerson max-flow
algorithm
Algorithm:

f [u, v] 0 for all u, v V

while an augmenting path p in G wrt f exists
do augment f by cf (p)

Can be slow:
1:109

G:

1:1

ss
0:109

2001 by Charles E. Leiserson

0:109

tt
1:109

Introduction to Algorithms

Day 40

L23.9

Ford-Fulkerson max-flow
algorithm
Algorithm:

f [u, v] 0 for all u, v V

while an augmenting path p in G wrt f exists
do augment f by cf (p)

Can be slow:
1:109

G:

0:1

ss
1:109

2001 by Charles E. Leiserson

1:109

tt
1:109

Introduction to Algorithms

Day 40

L23.10

Ford-Fulkerson max-flow
algorithm
Algorithm:

f [u, v] 0 for all u, v V

while an augmenting path p in G wrt f exists
do augment f by cf (p)

Can be slow:
1:109

G:

0:1

ss
1:109

2001 by Charles E. Leiserson

1:109

tt
1:109

Introduction to Algorithms

Day 40

L23.11

Ford-Fulkerson max-flow
algorithm
Algorithm:

f [u, v] 0 for all u, v V

while an augmenting path p in G wrt f exists
do augment f by cf (p)

Can be slow:
2:109

G:

1:109
1:1

ss
1:109

tt
2:109

2 billion iterations on a graph with 4 vertices!

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.12

Edmonds-Karp algorithm
Edmonds and Karp noticed that many peoples
implementations of Ford-Fulkerson augment along
a breadth-first augmenting path: a shortest path in
Gf from s to t where each edge has weight 1. These
implementations would always run relatively fast.
Since a breadth-first augmenting path can be found
in O(E) time, their analysis, which provided the first
polynomial-time bound on maximum flow, focuses
on bounding the number of flow augmentations.
(In independent work, Dinic also gave polynomialtime bounds.)
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.13

Monotonicity lemma
Lemma. Let (v) = f (s, v) be the breadth-first
distance from s to v in Gf . During the EdmondsKarp algorithm, (v) increases monotonically.
Proof. Suppose that f is a flow on G, and augmentation
produces a new flow f . Let (v) = f (s, v). Well
show that (v) (v) by induction on (v). For the base
case, (s) = (s) = 0.
For the inductive case, consider a breadth-first path s
L u v in Gf . We must have (v) = (u) + 1, since
subpaths of shortest paths are shortest paths. Certainly,
(u, v) Ef , and now consider two cases depending on
whether (u, v) Ef .
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.14

Case 1
Case: (u, v) Ef .
We have
(v) (u) + 1
(u) + 1
= (v)

(triangle inequality)
(induction)
(breadth-first path),

and thus monotonicity of (v) is established.

2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.15

Case 2
Case: (u, v) Ef .
Since (u, v) Ef , the augmenting path p that produced
f from f must have included (v, u). Moreover, p is a
breadth-first path in Gf :
p=sLvuLt.
Thus, we have
(v) = (u) 1
(breadth-first path)
(u) 1
(induction)
(v) 2
(breadth-first path)
< (v) ,
thereby establishing monotonicity for this case, too.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.16

Counting flow augmentations

Theorem. The number of flow augmentations
in the Edmonds-Karp algorithm (Ford-Fulkerson
with breadth-first augmenting paths) is O(V E).
Proof. Let p be an augmenting path, and suppose that
we have cf (u, v) = cf (p) for edge (u, v) p. Then, we
say that (u, v) is critical, and it disappears from the
residual graph after flow augmentation.
cf (p) = 2
Example:
2

Gf :

ss

tt
3

2001 by Charles E. Leiserson

1
Introduction to Algorithms

2
Day 40

L23.17

Counting flow augmentations

Theorem. The number of flow augmentations
in the Edmonds-Karp algorithm (Ford-Fulkerson
with breadth-first augmenting paths) is O(V E).
Proof. Let p be an augmenting path, and suppose that
the residual capacity of edge (u, v) p is cf (u, v) = cf (p).
Then, we say (u, v) is critical, and it disappears from the
residual graph after flow augmentation.

Example:

Gf :

ss

tt
5

2001 by Charles E. Leiserson

Introduction to Algorithms

4
Day 40

L23.18

Counting flow augmentations

(continued)

The first time an edge (u, v) is critical, we have (v) =

(u) + 1, since p is a breadth-first path. We must wait
until (v, u) is on an augmenting path before (u, v) can
be critical again. Let be the distance function when
(v, u) is on an augmenting path. Then, we have
(u) = (v) + 1
(breadth-first path)
(v) + 1
(monotonicity)
= (u) + 2
(breadth-first path).

Example:

uu

ss

tt

vv
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.19

Counting flow augmentations

(continued)

The first time an edge (u, v) is critical, we have (v) =

(u) + 1, since p is a breadth-first path. We must wait
until (v, u) is on an augmenting path before (u, v) can
be critical again. Let be the distance function when
(v, u) is on an augmenting path. Then, we have
(u) = (v) + 1
(breadth-first path)
(v) + 1
(monotonicity)
= (u) + 2
(breadth-first path).

Example:

(u) = 5

uu

ss

tt

vv
(v) = 6
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.20

Counting flow augmentations

(continued)

The first time an edge (u, v) is critical, we have (v) =

(u) + 1, since p is a breadth-first path. We must wait
until (v, u) is on an augmenting path before (u, v) can
be critical again. Let be the distance function when
(v, u) is on an augmenting path. Then, we have
(u) = (v) + 1
(breadth-first path)
(v) + 1
(monotonicity)
= (u) + 2
(breadth-first path).

Example:

(u) = 5

uu

ss

tt

vv
(v) = 6
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.21

Counting flow augmentations

(continued)

The first time an edge (u, v) is critical, we have (v) =

(u) + 1, since p is a breadth-first path. We must wait
until (v, u) is on an augmenting path before (u, v) can
be critical again. Let be the distance function when
(v, u) is on an augmenting path. Then, we have
(u) = (v) + 1
(breadth-first path)
(v) + 1
(monotonicity)
= (u) + 2
(breadth-first path).

Example:

(u) 7

uu

ss

tt

vv
(v) 6
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.22

Counting flow augmentations

(continued)

The first time an edge (u, v) is critical, we have (v) =

(u) + 1, since p is a breadth-first path. We must wait
until (v, u) is on an augmenting path before (u, v) can
be critical again. Let be the distance function when
(v, u) is on an augmenting path. Then, we have
(u) = (v) + 1
(breadth-first path)
(v) + 1
(monotonicity)
= (u) + 2
(breadth-first path).

Example:

(u) 7

uu

ss

tt

vv
(v) 6
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.23

Counting flow augmentations

(continued)

The first time an edge (u, v) is critical, we have (v) =

(u) + 1, since p is a breadth-first path. We must wait
until (v, u) is on an augmenting path before (u, v) can
be critical again. Let be the distance function when
(v, u) is on an augmenting path. Then, we have
(u) = (v) + 1
(breadth-first path)
(v) + 1
(monotonicity)
= (u) + 2
(breadth-first path).

Example:

(u) 7

uu

ss

tt

vv
(v) 8
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.24

Running time of EdmondsKarp

Distances start out nonnegative, never decrease, and are
at most |V| 1 until the vertex becomes unreachable.
Thus, (u, v) occurs as a critical edge O(V) times, because
(v) increases by at least 2 between occurrences. Since
the residual graph contains O(E) edges, the number of
flow augmentations is O(V E).

Corollary. The Edmonds-Karp maximum-flow

algorithm runs in O(V E 2) time.
Proof. Breadth-first search runs in O(E) time, and all
other bookkeeping is O(V) per augmentation.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.25

Best to date
The asymptotically fastest algorithm to date for
maximum flow, due to King, Rao, and Tarjan,
runs in O(V E logE/(V lg V)V) time.
If we allow running times as a function of edge
weights, the fastest algorithm for maximum
flow, due to Goldberg and Rao, runs in time
O(min{V 2/3, E 1/2} E lg (V 2/E + 2) lg C),
where C is the maximum capacity of any edge
in the graph.
2001 by Charles E. Leiserson

Introduction to Algorithms

Day 40

L23.26