Introduction to Algorithms

6.046J/18.401J/SMA5503

Lecture 4
Prof. Charles E. Leiserson
 Quicksort
• Proposed by C.A.R. Hoare in 1962.
• Divide-and-conquer algorithm.
• Sorts “in place” (like insertion sort, but not
like merge sort).
• Very practical (with tuning).

Day 6 Introduction to Algorithms L4.2

 Divide and conquer
Quicksort an n-element array:
1. Divide: Partition the array into two subarrays
around a pivot x such that elements in lower
subarray ≤ x ≤ elements in upper subarray.
≤≤ xx xx ≥≥ xx
2. Conquer: Recursively sort the two subarrays.
3. Combine: Trivial.
Key: Linear-time partitioning subroutine.
Day 6 Introduction to Algorithms L4.3
 Partitioning subroutine
PARTITION(A, p, q) ⊳ A[ p . . q]
x ← A[ p] ⊳ pivot = A[ p] Running
Running time
time
i←p == O(n)
O(n) for
for nn
for j ← p + 1 to q elements.
do if A[ j] ≤ x elements.
then i ← i + 1
exchange A[i] ↔ A[ j]
exchange A[ p] ↔ A[i]
return i
Invariant: xx ≤≤ xx ≥≥ xx ??
p i j q
Day 6 Introduction to Algorithms L4.4
 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
i j

Day 6 Introduction to Algorithms L4.5

 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
i j

Day 6 Introduction to Algorithms L4.6

 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
i j

Day 6 Introduction to Algorithms L4.7

 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
i j

Day 6 Introduction to Algorithms L4.8

 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
i j

Day 6 Introduction to Algorithms L4.9

 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
i j

Day 6 Introduction to Algorithms L4.10

 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
i j

Day 6 Introduction to Algorithms L4.11

 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
i j

Day 6 Introduction to Algorithms L4.12

 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
66 55 33 22 88 13
13 10
10 11
11
i j

Day 6 Introduction to Algorithms L4.13

 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
66 55 33 22 88 13
13 10
10 11
11
i j

Day 6 Introduction to Algorithms L4.14

 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
66 55 33 22 88 13
13 10
10 11
11
i j

Day 6 Introduction to Algorithms L4.15

 Example of partitioning

66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
66 55 33 22 88 13
13 10
10 11
11
22 55 33 66 88 13
13 10
10 11
11
i
Day 6 Introduction to Algorithms L4.16
 Pseudocode for quicksort
QUICKSORT(A, p, r)
if p < r
then q ← PARTITION(A, p, r)
QUICKSORT(A, p, q–1)
QUICKSORT(A, q+1, r)

Initial call: QUICKSORT(A, 1, n)

Day 6 Introduction to Algorithms L4.17

 Analysis of quicksort

• Assume all input elements are distinct.

• In practice, there are better partitioning
algorithms for when duplicate input
elements may exist.
• Let T(n) = worst-case running time on
an array of n elements.

Day 6 Introduction to Algorithms L4.18

 Worst-case of quicksort
• Input sorted or reverse sorted.
• Partition around min or max element.
• One side of partition always has no elements.
T (n) = T (0) + T (n − 1) + Θ(n)
= Θ(1) + T (n − 1) + Θ(n)
= T (n − 1) + Θ(n)
= Θ( n 2 ) (arithmetic series)

Day 6 Introduction to Algorithms L4.19

 Worst-case recursion tree
T(n) = T(0) + T(n–1) + cn

Day 6 Introduction to Algorithms L4.20

 Worst-case recursion tree
T(n) = T(0) + T(n–1) + cn
T(n)

Day 6 Introduction to Algorithms L4.21

 Worst-case recursion tree
T(n) = T(0) + T(n–1) + cn
cn
T(0) T(n–1)

Day 6 Introduction to Algorithms L4.22

 Worst-case recursion tree
T(n) = T(0) + T(n–1) + cn
cn
T(0) c(n–1)
T(0) T(n–2)

Day 6 Introduction to Algorithms L4.23

 Worst-case recursion tree
T(n) = T(0) + T(n–1) + cn
cn
T(0) c(n–1)
T(0) c(n–2)
T(0) O

Θ(1)

Day 6 Introduction to Algorithms L4.24

 Worst-case recursion tree
T(n) = T(0) + T(n–1) + cn
cn  n 
T(0) c(n–1) Θ ∑ k  = Θ(n 2 )
 k =1 
T(0) c(n–2)
T(0) O

Θ(1)

Day 6 Introduction to Algorithms L4.25

 Worst-case recursion tree
T(n) = T(0) + T(n–1) + cn
cn  n 
Θ(1) c(n–1) Θ ∑ k  = Θ(n 2 )
 k =1 
Θ(1) c(n–2)
h=n T(n) = Θ(n) + Θ(n2)
Θ(1) O = Θ(n2)

Θ(1)

Day 6 Introduction to Algorithms L4.26

 Best-case analysis
(For intuition only!)
If we’re lucky, PARTITION splits the array evenly:
T(n) = 2T(n/2) + Θ(n)
= Θ(n lg n) (same as merge sort)

1 9
What if the split is always 10 : 10 ?
T (n) = T (101 n ) + T (109 n ) + Θ(n)
What is the solution to this recurrence?

Day 6 Introduction to Algorithms L4.27

 Analysis of “almost-best” case
T (n)

Day 6 Introduction to Algorithms L4.28

 Analysis of “almost-best” case
cn
T (101 n ) T (109 n )

Day 6 Introduction to Algorithms L4.29

 Analysis of “almost-best” case
cn
1
10
cn 9
10
cn

T (100
1
n ) T (100
9
n ) T (100
9
n )T (100
81
n)

Day 6 Introduction to Algorithms L4.30

 Analysis of “almost-best” case
cn cn
1
10
cn 9
10
cn cn
log10/9n
1
100
cn 9
100
cn 9
100
cn 81
100
cn cn
…

…
Θ(1) O(n)
O(n) leaves
leaves
Θ(1)

Day 6 Introduction to Algorithms L4.31

 Analysis of “almost-best” case
cn cn
1
10
cn 9
cn cn
log10n 10
log10/9n
1
100
cn 9
100
cn 9
100
cn 81
100
cn cn
…

…
Θ(1) O(n)
O(n) leaves
leaves
Θ(n lg n) Θ(1)
Lucky! cn log10n ≤ T(n) ≤ cn log10/9n + Ο(n)
Day 6 Introduction to Algorithms L4.32
 More intuition
Suppose we alternate lucky, unlucky,
lucky, unlucky, lucky, ….
L(n) = 2U(n/2) + Θ(n) lucky
U(n) = L(n – 1) + Θ(n) unlucky
Solving:
L(n) = 2(L(n/2 – 1) + Θ(n/2)) + Θ(n)
= 2L(n/2 – 1) + Θ(n)
= Θ(n lg n) Lucky!
How can we make sure we are usually lucky?
Day 6 Introduction to Algorithms L4.33
 Randomized quicksort
IDEA: Partition around a random element.
• Running time is independent of the input
order.
• No assumptions need to be made about
the input distribution.
• No specific input elicits the worst-case
behavior.
• The worst case is determined only by the
output of a random-number generator.
Day 6 Introduction to Algorithms L4.34
 Randomized quicksort
analysis
Let T(n) = the random variable for the running
time of randomized quicksort on an input of size
n, assuming random numbers are independent.
For k = 0, 1, …, n–1, define the indicator
random variable
1 if PARTITION generates a k : n–k–1 split,
Xk =
0 otherwise.
E[Xk] = Pr{Xk = 1} = 1/n, since all splits are
equally likely, assuming elements are distinct.
Day 6 Introduction to Algorithms L4.35
 Analysis (continued)
T(0) + T(n–1) + Θ(n) if 0 : n–1 split,
T(1) + T(n–2) + Θ(n) if 1 : n–2 split,
T(n) =
M
T(n–1) + T(0) + Θ(n) if n–1 : 0 split,
n −1
= ∑ X k (T (k ) + T (n − k − 1) + Θ(n)) .
k =0

Day 6 Introduction to Algorithms L4.36

 Calculating expectation
 n −1 
E[T (n)] = E  ∑ X k (T (k ) + T (n − k − 1) + Θ(n) )
k =0 

Take expectations of both sides.

Day 6 Introduction to Algorithms L4.37

 Calculating expectation
 n −1 
E[T (n)] = E  ∑ X k (T (k ) + T (n − k − 1) + Θ(n) )
k =0 
n −1
= ∑ E[ X k (T (k ) + T (n − k − 1) + Θ(n) )]
k =0

Linearity of expectation.

Day 6 Introduction to Algorithms L4.38

 Calculating expectation
 n −1 
E[T (n)] = E  ∑ X k (T (k ) + T (n − k − 1) + Θ(n) )
k =0 
n −1
= ∑ E[ X k (T (k ) + T (n − k − 1) + Θ(n) )]
k =0
n −1
= ∑ E[ X k ] ⋅ E[T (k ) + T (n − k − 1) + Θ(n)]
k =0

Independence of Xk from other random

choices.

Day 6 Introduction to Algorithms L4.39

 Calculating expectation
 n −1 
E[T (n)] = E  ∑ X k (T (k ) + T (n − k − 1) + Θ(n) )
k =0 
n −1
= ∑ E[ X k (T (k ) + T (n − k − 1) + Θ(n) )]
k =0
n −1
= ∑ E[ X k ] ⋅ E[T (k ) + T (n − k − 1) + Θ(n)]
k =0
n −1 n −1 n −1
= 1 ∑ E [T (k )] + 1 ∑ E [T (n − k − 1)] + 1 ∑ Θ(n)
n k =0 n k =0 n k =0

Linearity of expectation; E[Xk] = 1/n .

Day 6 Introduction to Algorithms L4.40

 Calculating expectation
 n −1 
E[T (n)] = E  ∑ X k (T (k ) + T ( n − k − 1) + Θ(n) )
k =0 
n −1
= ∑ E[ X k (T (k ) + T (n − k − 1) + Θ(n) )]
k =0
n −1
= ∑ E[ X k ] ⋅ E[T (k ) + T (n − k − 1) + Θ(n)]
k =0
n −1 n −1 n −1
= 1 ∑ E [T (k )] + 1 ∑ E [T (n − k − 1)] + 1 ∑ Θ(n)
n k =0 n k =0 n k =0
n −1
= 2 ∑ E [T (k )] + Θ(n) Summations have
n k =1
identical terms.
Day 6 Introduction to Algorithms L4.41
 Hairy recurrence
n −1
E[T (n)] = 2 ∑ E [T (k )] + Θ(n)
n k =2
(The k = 0, 1 terms can be absorbed in the Θ(n).)
Prove: E[T(n)] ≤ an lg n for constant a > 0 .
• Choose a large enough so that an lg n
dominates E[T(n)] for sufficiently small n ≥ 2.
n −1
Use fact: ∑ k lg k ≤ 1 n 2 lg n − 1n 2
2 8 (exercise).
k =2
Day 6 Introduction to Algorithms L4.42
 Substitution method
n −1
E [T (n)] ≤ 2 ∑ ak lg k + Θ(n)
n k =2
Substitute inductive hypothesis.

Day 6 Introduction to Algorithms L4.43

 Substitution method
n −1
E [T (n)] ≤ 2 ∑ ak lg k + Θ(n)
n k =2

≤ 2a  1 n 2 lg n − 1 n 2  + Θ(n)
n 2 8 
Use fact.

Day 6 Introduction to Algorithms L4.44

 Substitution method
n −1
E [T (n)] ≤ 2 ∑ ak lg k + Θ(n)
n k =2

≤ 2a  1 n 2 lg n − 1 n 2  + Θ(n)
n 2 8 
= an lg n −  an − Θ(n) 
 4 
Express as desired – residual.

Day 6 Introduction to Algorithms L4.45

 Substitution method
n −1
E [T (n)] ≤ 2 ∑ ak lg k + Θ(n)
n k =2

= 2a  1 n 2 lg n − 1 n 2  + Θ(n)
n 2 8 
= an lg n −  an − Θ(n) 
 4 
≤ an lg n ,
if a is chosen large enough so that
an/4 dominates the Θ(n).
Day 6 Introduction to Algorithms L4.46
 Quicksort in practice

• Quicksort is a great general-purpose

sorting algorithm.
• Quicksort is typically over twice as fast
as merge sort.
• Quicksort can benefit substantially from
code tuning.
• Quicksort behaves well even with
caching and virtual memory.

Day 6 Introduction to Algorithms L4.47