Design and Analysis of Algorithms

Lecture 3
Prof. Piotr Indyk
 Menu

• Randomized algorithms: algorithms that flip coins

• Two examples:
– Matrix product checker: is AB=C ?
– Quicksort:
• Example of divide and conquer
• Fast and practical sorting algorithm
 Menu

• Randomized algorithms: algorithms that flip coins

• Two examples:
– Matrix product checker: is AB=C ?
– Quicksort:
• Example of divide and conquer
• Fast and practical sorting algorithm
 Randomized Algorithms

• Algorithms whose execution is not deterministic given the

input, but depends also on random choices made in various
steps of the algorithm’s execution
• More precisely:
– algorithm has access to a random number generator
• on every call to the random number generator, it outputs out a
fresh sample x from some range {1…R}
– algorithm’s decisions depend not only on the input but also
the random numbers that the generator sampled
• Why is randomization useful?
– short answer: randomization helps us do things we
would not otherwise be able to do
 Where’s the coin?
• You play against an adversary for T rounds. On each round,
you collect the coin if you guess where it is.

• Tricky part: suppose you need to announce to the adversary the algorithm
you use to choose a cup on each round
• If your algorithm always chooses left, the adversary will always put the
coin on the right, so your total payoff = $0
• Similarly, if you use any other deterministic algorithm
• If you choose a uniformly random cup on every round, then no matter
what the adversary does you are guaranteed expected payoff = $T/2 !
 Randomized Algorithms

• Two basic types:

– Las Vegas: Always correct output and typically good runtime
(but sometimes slow)
– Monte Carlo: Always good runtime and typically correct
output (but sometimes output is garbage)
• Typically: measured with respect to the random
numbers sampled by the algorithm during its execution
– in particular, typical runtime (for Las Vegas) and typical
correctness (for Monte Carlo) hold for any input
• Today we’ll see one Las Vegas and one Monte Carlo
algorithm
 Menu

• Randomized algorithms: algorithms that flip coins

• Two examples:
– Matrix product checker: is AB=C ?
– Quicksort:
• Example of divide and conquer
• Fast and practical sorting algorithm
 Menu

• Randomized algorithms: algorithms that flip coins

• Two examples:
– Matrix product checker: is AB=C ?
– Quicksort:
• Example of divide and conquer
• Fast and practical sorting algorithm
 Matrix Product
• Compute C=A×B, where A, B are n x n
– Simple algorithm: O(n3) time
– Multiply two 2×2 matrices using 7 mult.
→O(n2.81…) time [Strassen’69]
–…
– O(n2.376…) [Coppersmith-Winograd’90]
–…
– O(n2.372…) [Alman-Vassilevska Williams’20]
– still open: is it possible to do it in O(n2)
 Matrix Product Checking

• Given n×n matrices A,B,C, is A×B=C ?

• We will give an O(n2) algorithm that:
– If answer=YES, then Pr[output=YES]=1
– If answer=NO, then Pr[output=YES] ≤ 0.001
• Is this Monte Carlo or Las Vegas?
– always fast and typically correct so Monte Carlo!
• Nothing special about 0.001
– We’ll start with an algorithm that has error probability ≤½
if correct answer is NO
– We’ll then boost the algorithm to drop this down to 0.001
(or any other desired constant)
 The algorithm

• Algorithm:
– Choose a random binary vector x=(x1,…,xn),
such that Pr[xi=1]=Pr[xi=0]=½ , i=1…n
– Output YES if ABx=Cx and NO otherwise
• Does it run in O(n2) time ?
– YES, because ABx = A(Bx)
 Correctness Analysis

• Let D=AB, need to check if D=C

• Case 1: D=C
– Then Dx=Cx for any x, so the output is YES
• Case 2: D≠C
– Want to show that Prx[Dx=Cx] ≤ ½
• there certainly exists x such that Dx≠Cx
• we need to show that at least half of all possible
binary vectors satisfy Dx≠Cx
 D≠C

d ? c
x ≠ x
D C

Pr[Dx ≠ Cx] ≥ Pr[dx ≠ cx] ≥ ???

Question: if d, c differ in at least one coordinate (say,
di≠ci) and x is a random binary vector can we lower
bound Pr[dx ≠ cx] ?
 Vector product
• Question: if d, c differ in at least one coordinate (say,
di≠ci) and x is a random binary vector can we lower
bound Pr[dx ≠ cx] ?
– we have dx ≠ cx iff (d-c)x ≠ 0

S1
– we have: (d-c)x = Σj≠i(dj-cj)xj + (di-ci)xi

– If xi=0, then (d-c)x=S1 S2

– If xi=1, then (d-c)x=S2≠S1

– So, at least one of xi=0 or xi=1 gives (d-c)x≠0
⇒ Pr[dx ≠ cx] ≥ ½
 D≠C

d ? c
x ≠ x
D C

Pr[Dx ≠ Cx] ≥ Pr[dx ≠ cx] ≥ 1/2

!
⇒Pr[Dx = Cx]≤
"
 Correctness Analysis

• Let D=AB, need to check if D=C

• What if D=C ?
– Then Pr[Dx=Cx]=1 so output is Pr[output=YES] =1
• What if D≠C ?
– Then Pr[Dx=Cx] ≤ ½ so Pr[output=YES] ≤ ½
 Matrix Product Checker
• Is A×B=C ?
• We have an algorithm that:
– If answer=YES, then Pr[output=YES]=1
– If answer=NO, then Pr[output=YES] ≤ ½
• What if we want to reduce ½ to ¼ ?
– run the algorithm twice, using independent random numbers x
– Output YES only if both runs say YES
• Analysis:
– If answer=YES, then Pr[output1=YES, output2=YES ]=1
– If answer=NO, then
Pr[output=YES] = Pr[output1=YES, output2=YES]
= Pr[output1=YES]*Pr[output2=YES] ≤ ¼
• How to reduce probability of error down to 1/8 ? 1/16 ? etc?
 Menu

• Randomized algorithms: algorithms that flip coins

• Two examples:
– Matrix product checker: is AB=C ?
– Quicksort:
• Example of divide and conquer
• Fast and practical sorting algorithm
 Menu

• Randomized algorithms: algorithms that flip coins

• Two examples:
– Matrix product checker: is AB=C ?
– Quicksort:
• Example of divide and conquer
• Fast and practical sorting algorithm
 Quicksort
• Proposed by C.A.R. Hoare in 1962.
• Divide-and-conquer algorithm.
• Sorts “in place” (like insertion sort, and unlike
merge sort).
• Very practical (with tuning).
• Is a randomized Las Vegas algorithm
 Divide and conquer
Quicksort an n-element array:
1. Choose pivot: Pick some element x from the array
x

2. Divide: Partition the array into two subarrays such

that elements in lower subarray £ x £ elements in
upper subarray.
x £ x£ x x ³x
3. Conquer: Recursively sort the two subarrays.
4. Combine: Trivial.
 Pseudocode for quicksort
QUICKSORT(A, p, r)
if p < r then
i ¬ CHOOSEPIVOT (p, r)
q ¬ PARTITION(A, p, r, i)
QUICKSORT(A, p, q–1)
QUICKSORT(A, q+1, r)
Initial call: QUICKSORT(A, 1, n)
CHOOSEPIVOT (p, r): returns index 𝑝 ≤ 𝑖 ≤ 𝑟
 Analysis of quicksort

• Assume all input elements are distinct.

• In practice, there are better partitioning
algorithms for when duplicate input
elements may exist.
• What is the worst-case running time of
Quicksort ?
 Worst-case execution?
• For every subarray that the algorithm processes
recursively, it chooses as pivot the min or max element
of that subarray
• Then one side of partition always has no elements
𝑇 𝑛 =𝑇 0 +𝑇 𝑛−1 +𝑐⋅𝑛
=2⋅𝑇 0 +𝑇 𝑛−2 +𝑐⋅ 𝑛−1+𝑛
=⋯
= (𝑛 − 1) ⋅ 𝑇 0 + 𝑇 1 + 𝑐 ⋅ 2 + 3 + ⋯ 𝑛
= Θ 𝑛 + Θ 1 + 𝑐 ⋅ Θ 𝑛!
=Θ(𝑛!)
Worst-case recursion (in pictures)
T(n) = T(0) + T(n–1) + cn
Worst-case recursion (in pictures)
T(n) = T(0) + T(n–1) + cn
T(n)
 Worst-case recursion (in pictures)
T(n) = T(0) + T(n–1) + cn
cn
T(0) T(n–1)
 Worst-case recursion (in pictures)
T(n) = T(0) + T(n–1) + cn
cn
T(0) c(n–1)
T(0) T(n–2)
 Worst-case recursion (in pictures)
T(n) = T(0) + T(n–1) + cn
cn
T(0) c(n–1)
T(0) c(n–2)
T(0) !
c⋅2
T(0) T(1)
 Worst-case recursion (in pictures)
T(n) = T(0) + T(n–1) + cn
cn æ n ö
T(0) c(n–1) Qçç å k ÷÷ = Q(n 2 )
è k =1 ø
T(0) c(n–2)
T(0) !
c⋅2
T(0) T(1)
 Worst-case recursion (in pictures)
T(n) = T(0) + T(n–1) + cn
cn æ n ö
Q(1) c(n–1) Qçç å k ÷÷ = Q(n 2 )
è k =1 ø
Q(1) c(n–2)
h=n
Q(1) !
c⋅2
Q(1) Q(1)
Total: T(n) = Θ 𝑛"
 Super-Lucky-Case Execution

If we’re super-lucky, PARTITION always splits the array

evenly:
T(n) = 2T(n/2) + Q(n)
= Q(n lg n) (same as merge sort)

How to be super-lucky?
 Lucky-Case Execution

What if the split is always 1 : 9 ?

10 10
1 9
𝑇 𝑛 ≤𝑇 𝑛 +𝑇 𝑛 + Θ(𝑛)
10 10
Lucky-Case Analysis
T (n)
Lucky-Case Analysis
cn
T (101 n ) T (109 n )
 Lucky-Case Analysis
cn
1
10
cn 9
10
cn

T (100
1
n ) T (100
9
n ) T (100
9
n )T (100
81
n)
 Lucky-Case Analysis
cn cn
1
10
cn 9
10
cn cn
log10/9n
1
100
cn 9
100
cn 9
100
cn 81
100
cn cn
…

…
Q(1)
Q(1)
 Lucky-Case Analysis
cn cn
1
10
cn 9
cn cn
log10n 10
log10/9n
1
100
cn 9
100
cn 9
100
cn 81
100
cn cn
…

…
Q(1)
Q(1)
cn log10n £ T(n) £ cn log10/9n + O(n)
 Lucky-Case Execution

What if the split is no worse than 1 : 9 ?

10 10
𝑇 𝑛 ≤ max 𝑇 𝑖 +𝑇 𝑛−𝑖 + Θ(𝑛)
! &!
$%$ why? because recursive tree has ≤ log!"/$ 𝑛
"# "#
depth no matter what the splits are and there
≤ Θ(𝑛 log 𝑛) is Θ(𝑛) total work for splits at every level

Same is true is we could guarantee no worse split than

any other constant ratio.
Question: Can we avoid running median finding or
similar algorithm, yet guarantee that most our splits
have constant ratio?
 Randomized quicksort
• Partition around a random element. I.e.,
around A[i] , where i chosen uniformly
at random from {p…r}
• Randomized quicksort has expected
running time O(n log n)
• Won’t show the proof, but will analyze an
easier to analyze variant of randomized
quicksort: paranoid quiksort
 “Paranoid” quicksort

PARANOIDPARTITION
• Repeat:
• choose the pivot to be a random element of the array
• perform PARTITION
until resulting split is “lucky,” ie no worse than 1/10: 9/10

PARANOIDQUICKSORT
Same as QUICKSORT but uses PARANOIDPARTITION
 Analysis
• Let T(n) be an upper bound on the expected
running time on any array of n elements
• Consider any input of size n
• The time needed to sort the input is bounded
from the above by a sum of
• The time needed to sort the left subarray
• The time needed to sort the right subarray
• The number of tried partitions until we
get a lucky split, times cn
 Expectations

• Therefore:
𝑇 𝑛 ≤ max 𝑇 𝑖 +𝑇 𝑛−𝑖 + 𝐸 #𝑡𝑟𝑖𝑎𝑙𝑠 ⋅ 𝑐𝑛
! &!
$%$
"# "#
where 𝐸 #𝑡𝑟𝑖𝑎𝑙𝑠 is the expected number of random
partitions we need to do to get a lucky split
• We will show that E[#trials] is £ 10/8
• Therefore:
𝑇 𝑛 ≤ max 𝑇 𝑖 +𝑇 𝑛−𝑖 + 10/8 ⋅ 𝑐𝑛
! &!
$%$
"# "#
≤ Θ(𝑛 log 𝑛)
 Time until Lucky partition

• The probability that a random pivot induces

lucky partition is at least 8/10
– Indeed, we are not lucky if the pivot happens to
be among the smallest/largest n/10 elements
• If we flip a coin, with heads prob. p=8/10 , the
expected waiting time for the first head is 1/p
= 10/8
 Concluding the bound

• Therefore:
𝑇 𝑛 ≤ max 𝑇 𝑖 +𝑇 𝑛−𝑖 + 𝐸 #𝑡𝑟𝑖𝑎𝑙𝑠 ⋅ 𝑐𝑛
! &!
$%$
"# "#
where 𝐸 #𝑡𝑟𝑖𝑎𝑙𝑠 is the expected number of random
partitions we need to do to get a lucky split
• We will show that E[#trials] is £ 10/8
• Therefore:
𝑇 𝑛 ≤ max 𝑇 𝑖 +𝑇 𝑛−𝑖 + 10/8 ⋅ 𝑐𝑛
! &!
$%$
"# "#
≤ Θ(𝑛 log 𝑛)
 Quicksort in practice

• Quicksort is a great general-purpose

sorting algorithm.
• Quicksort is typically over twice as fast
as merge sort.
• Quicksort can benefit substantially from
code tuning.
• Quicksort behaves well even with
caching and virtual memory.
• Quicksort is great!
Optional Material
 Randomized quicksort
analysis
Let T(n) = the random variable for the running
time of randomized quicksort on an input of size
n, assuming random numbers are independent.
For k = 0, 1, …, n–1, define the indicator
random variable
1 if PARTITION generates a k : n–k–1 split,
Xk =
0 otherwise.
E[Xk] = Pr{Xk = 1} = 1/n, since all splits are
equally likely, assuming elements are distinct.
 Analysis (continued)
T(0) + T(n–1) + Q(n) if 0 : n–1 split,
T(1) + T(n–2) + Q(n) if 1 : n–2 split,
T(n) =
!
T(n–1) + T(0) + Q(n) if n–1 : 0 split,
n -1
= å X k (T (k ) + T (n - k - 1) + Q(n)) .
k =0
 Calculating expectation
é n -1 ù
E[T (n)] = E ê å X k (T (k ) + T (n - k - 1) + Q(n) )ú
ëk =0 û

Take expectations of both sides.

 Calculating expectation
é n -1 ù
E[T (n)] = E ê å X k (T (k ) + T (n - k - 1) + Q(n) )ú
ëk =0 û
n -1
= å E[ X k (T (k ) + T (n - k - 1) + Q(n) )]
k =0

Linearity of expectation.
 Calculating expectation
é n -1 ù
E[T (n)] = E ê å X k (T (k ) + T (n - k - 1) + Q(n) )ú
ëk =0 û
n -1
= å E[ X k (T (k ) + T (n - k - 1) + Q(n) )]
k =0
n -1
= å E[ X k ] × E[T (k ) + T (n - k - 1) + Q(n)]
k =0

Independence of Xk from other random

choices.
 Calculating expectation
é n -1 ù
E[T (n)] = E ê å X k (T (k ) + T (n - k - 1) + Q(n) )ú
ëk =0 û
n -1
= å E[ X k (T (k ) + T (n - k - 1) + Q(n) )]
k =0
n -1
= å E[ X k ] × E[T (k ) + T (n - k - 1) + Q(n)]
k =0
n -1 n -1 n -1
= 1 å E [T (k )] + 1 å E [T (n - k - 1)] + 1 å Q(n)
n k =0 n k =0 n k =0

Linearity of expectation; E[Xk] = 1/n .

 Calculating expectation
é n -1 ù
E[T (n)] = E ê å X k (T (k ) + T (n - k - 1) + Q(n) )ú
ëk =0 û
n -1
= å E[ X k (T (k ) + T (n - k - 1) + Q(n) )]
k =0
n -1
= å E[ X k ] × E[T (k ) + T (n - k - 1) + Q(n)]
k =0
n -1 n -1 n -1
= 1 å E [T (k )] + 1 å E [T (n - k - 1)] + 1 å Q(n)
n k =0 n k =0 n k =0
n -1
= 2 å E [T (k )] + Q(n) Summations have
n k =1
identical terms.
 Hairy recurrence
n -1
E[T (n)] = 2 å E [T (k )] + Q(n)
n k =2
(The k = 0, 1 terms can be absorbed in the Q(n).)
Prove: E[T(n)] £ an lg n for constant a > 0 .
• Choose a large enough so that an lg n
dominates E[T(n)] for sufficiently small n ³ 2.
n -1
Use fact: å k lg k £ 1 n 2 lg n - 1n 2
2 8 (exercise).
k =2
Substitution method
n -1
E [T (n)] £ 2 å ak lg k + Q(n)
n k =2
Substitute inductive hypothesis.
Substitution method
n -1
E [T (n)] £ 2 å ak lg k + Q(n)
n k =2

£ 2a æç 1 n 2 lg n - 1 n 2 ö÷ + Q(n)
n è2 8 ø
Use fact.
Substitution method
n -1
E [T (n)] £ 2 å ak lg k + Q(n)
n k =2

£ 2a æç 1 n 2 lg n - 1 n 2 ö÷ + Q(n)
n è2 8 ø
= an lg n - æç an - Q(n) ö÷
è 4 ø
Express as desired – residual.
Substitution method
n -1
E [T (n)] £ 2 å ak lg k + Q(n)
n k =2

= 2a æç 1 n 2 lg n - 1 n 2 ö÷ + Q(n)
n è2 8 ø
= an lg n - æç an - Q(n) ö÷
è 4 ø
£ an lg n ,
if a is chosen large enough so that
an/4 dominates the Q(n).
 Acks
• Previous version of these slides © Charles Leiserson, Piotr
Indyk
 Partitioning subroutine
PARTITION(A, p, r, i)
x ¬ A[ i]
exchange A[ p] « A[i]
i¬p
for j ¬ p + 1 to r
if A[ j] £ x then
i¬i+1
exchange A[i] « A[ j]
exchange A[ p] « A[i]
return i

Loop Invariant: x £x ³x ?
p i j r