Introduction to Algorithms

6.046J/18.401J/SMA5503

Lecture 6
Prof. Erik Demaine
 Order statistics
Select the ith smallest of n elements (the
element with rank i).
• i = 1: minimum;
• i = n: maximum;
• i = (n+1)/2 or (n+1)/2: median.
Naive algorithm: Sort and index ith element.
Worst-case running time = Θ(n lg n) + Θ(1)
= Θ(n lg n),
using merge sort or heapsort (not quicksort).
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.2
 Randomized divide-and-
conquer algorithm
RAND-SELECT(A, p, q, i) ⊳ ith smallest of A[ p . . q]
if p = q then return A[ p]
r ← RAND-PARTITION(A, p, q)
k←r–p+1 ⊳ k = rank(A[r])
if i = k then return A[ r]
if i < k
then return RAND-SELECT(A, p, r – 1, i )
else return RAND-SELECT(A, r + 1, q, i – k )
k
≤≤ A[r]
A[r] ≥≥ A[r]
A[r]
p r q
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.3
 Example
Select the i = 7th smallest:
66 10
10 13
13 55 88 33 22 11
11 i=7
pivot
Partition:
22 55 33 66 88 13
13 10
10 11
11 k=4

Select the 7 – 4 = 3rd smallest recursively.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.4
 Intuition for analysis
(All our analyses today assume that all elements
are distinct.)
Lucky:
T(n) = T(9n/10) + Θ(n) n log10 / 9 1 = n 0 = 1
= Θ(n) CASE 3
Unlucky:
T(n) = T(n – 1) + Θ(n) arithmetic series
= Θ(n2)
Worse than sorting!
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.5
 Analysis of expected time
The analysis follows that of randomized
quicksort, but it’s a little different.
Let T(n) = the random variable for the running
time of RAND-SELECT on an input of size n,
assuming random numbers are independent.
For k = 0, 1, …, n–1, define the indicator
random variable
1 if PARTITION generates a k : n–k–1 split,
Xk =
0 otherwise.
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.6
 Analysis (continued)
To obtain an upper bound, assume that the ith
element always falls in the larger side of the
partition:
T(max{0, n–1}) + Θ(n) if 0 : n–1 split,
T(max{1, n–2}) + Θ(n) if 1 : n–2 split,
T(n) =
M
T(max{n–1, 0}) + Θ(n) if n–1 : 0 split,
n −1
= ∑ X k (T (max{k , n − k − 1}) + Θ(n)) .
k =0
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.7
 Calculating expectation
 n −1 
E[T (n)] = E  ∑ X k (T (max{k , n − k − 1}) + Θ(n) )
k =0 

Take expectations of both sides.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.8

 Calculating expectation
 n −1 
E[T (n)] = E  ∑ X k (T (max{k , n − k − 1}) + Θ(n) )
k =0 
n −1
= ∑ E[ X k (T (max{k , n − k − 1}) + Θ(n) )]
k =0

Linearity of expectation.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.9

 Calculating expectation
 n −1 
E[T (n)] = E  ∑ X k (T (max{k , n − k − 1}) + Θ(n) )
k =0 
n −1
= ∑ E[ X k (T (max{k , n − k − 1}) + Θ(n) )]
k =0
n −1
= ∑ E[ X k ] ⋅ E[T (max{k , n − k − 1}) + Θ(n)]
k =0

Independence of Xk from other random

choices.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.10

 Calculating expectation
 n −1 
E[T (n)] = E  ∑ X k (T (max{k , n − k − 1}) + Θ(n) )
k =0 
n −1
= ∑ E[ X k (T (max{k , n − k − 1}) + Θ(n) )]
k =0
n −1
= ∑ E[ X k ] ⋅ E[T (max{k , n − k − 1}) + Θ(n)]
k =0
n −1 n −1
= 1 ∑ E [T (max{k , n − k − 1})] + 1 ∑ Θ(n)
n k =0 n k =0

Linearity of expectation; E[Xk] = 1/n .

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.11
 Calculating expectation
 n −1 
E[T (n)] = E  ∑ X k (T (max{k , n − k − 1}) + Θ(n) )
k =0 
n −1
= ∑ E[ X k (T (max{k , n − k − 1}) + Θ(n) )]
k =0
n −1
= ∑ E[ X k ] ⋅ E[T (max{k , n − k − 1}) + Θ(n)]
k =0
n −1 n −1
= 1 ∑ E [T (max{k , n − k − 1})] + 1 ∑ Θ(n)
n k =0 n k =0
n −1
≤ 2 ∑ E [T (k )] + Θ(n) Upper terms
n k = n / 2 
appear twice.
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.12
 Hairy recurrence
(But not quite as hairy as the quicksort one.)
n −1
E[T (n)] = 2 ∑ E [T (k )] + Θ(n)
n k= n/2
 
Prove: E[T(n)] ≤ cn for constant c > 0 .
• The constant c can be chosen large enough
so that E[T(n)] ≤ cn for the base cases.
n −1
Use fact: ∑ 8 (exercise).
k ≤ 3n 2
k = n / 2 
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.13
 Substitution method
n −1
E [T (n)] ≤ 2 ∑ ck + Θ(n)
n k= n/2
 
Substitute inductive hypothesis.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.14

 Substitution method
n −1
E [T (n)] ≤ 2 ∑ ck + Θ(n)
n k= n/2
 
≤ 2c  3 n 2  + Θ(n)
n 8 
Use fact.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.15

 Substitution method
n −1
E [T (n)] ≤ 2 ∑ ck + Θ(n)
n k= n/2
 
≤ 2c  3 n 2  + Θ(n)
n 8 
= cn −  cn − Θ(n) 
4 
Express as desired – residual.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.16

 Substitution method
n −1
E [T (n)] ≤ 2 ∑ ck + Θ(n)
n k= n/2
 
≤ 2c  3 n 2  + Θ(n)
n 8 
= cn −  cn − Θ(n) 
4 
≤ cn ,
if c is chosen large enough so
that cn/4 dominates the Θ(n).
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.17
 Summary of randomized
order-statistic selection
• Works fast: linear expected time.
• Excellent algorithm in practice.
• But, the worst case is very bad: Θ(n2).
Q. Is there an algorithm that runs in linear
time in the worst case?
A. Yes, due to Blum, Floyd, Pratt, Rivest,
and Tarjan [1973].
IDEA: Generate a good pivot recursively.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.18

 Worst-case linear-time order
statistics
SELECT(i, n)
1. Divide the n elements into groups of 5. Find
the median of each 5-element group by rote.
2. Recursively SELECT the median x of the n/5
group medians to be the pivot.
3. Partition around the pivot x. Let k = rank(x).
4. if i = k then return x
elseif i < k Same as
then recursively SELECT the ith RAND-
smallest element in the lower part SELECT
else recursively SELECT the (i–k)th
smallest element in the upper part
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.19
 Choosing the pivot

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.20

 Choosing the pivot

1. Divide the n elements into groups of 5.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.21

 Choosing the pivot

1. Divide the n elements into groups of 5. Find lesser

the median of each 5-element group by rote.

greater
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.22
 Choosing the pivot

1. Divide the n elements into groups of 5. Find lesser

the median of each 5-element group by rote.
2. Recursively SELECT the median x of the n/5
group medians to be the pivot. greater
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.23
 Analysis

At least half the group medians are ≤ x, which lesser

is at least  n/5 /2 = n/10 group medians.

greater
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.24
 Analysis (Assume all elements are distinct.)

At least half the group medians are ≤ x, which lesser

is at least  n/5 /2 = n/10 group medians.
• Therefore, at least 3 n/10 elements are ≤ x.
greater
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.25
 Analysis (Assume all elements are distinct.)

At least half the group medians are ≤ x, which lesser

is at least  n/5 /2 = n/10 group medians.
• Therefore, at least 3 n/10 elements are ≤ x.
• Similarly, at least 3 n/10 elements are ≥ x. greater
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.26
 Minor simplification
• For n ≥ 50, we have 3 n/10 ≥ n/4.
• Therefore, for n ≥ 50 the recursive call to
SELECT in Step 4 is executed recursively
on ≤ 3n/4 elements.
• Thus, the recurrence for running time
can assume that Step 4 takes time
T(3n/4) in the worst case.
• For n < 50, we know that the worst-case
time is T(n) = Θ(1).

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.27

 Developing the recurrence
T(n) SELECT(i, n)
1. Divide the n elements into groups of 5. Find
Θ(n) the median of each 5-element group by rote.
2. Recursively SELECT the median x of the n/5
T(n/5) group medians to be the pivot.
Θ(n) 3. Partition around the pivot x. Let k = rank(x).
4. if i = k then return x
elseif i < k
T(3n/4) then recursively SELECT the ith
smallest element in the lower part
else recursively SELECT the (i–k)th
smallest element in the upper part
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.28
 Solving the recurrence
T (n) = T  1 n  + T  3 n  + Θ(n)
5  4 

Substitution: T (n) ≤ 1 cn + 3 cn + Θ(n)

T(n) ≤ cn 5 4
= 19 cn + Θ(n)
20
= cn −  1 cn − Θ(n) 
 20 
≤ cn ,
if c is chosen large enough to handle both the
Θ(n) and the initial conditions.
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.29
 Conclusions
• Since the work at each level of recursion
is a constant fraction (19/20) smaller, the
work per level is a geometric series
dominated by the linear work at the root.
• In practice, this algorithm runs slowly,
because the constant in front of n is large.
• The randomized algorithm is far more
practical.
Exercise: Why not divide into groups of 3?
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.30