A D VA N C E D A L G O R I T H M S

notes for cmu 15-850 (fall 2020)

lecturer: anupam gupta

About this document

This document contains the course notes for 15-850: Advanced

Algorithms, a graduate-level course taught by Anupam Gupta at
Carnegie Mellon University in Fall 2020. Parts of these notes were
written by the students of previous versions of the class (based on the
lectures) and then edited by the professor. The names of the student
scribes will appear soon, as will missing details and bug fixes, more
chapters, exercises, and (better) figures.
The notes have not been thoroughly checked for accuracy, espe-
cially attributions of results. They are intended to serve as study
resources and not as a substitute for professionally prepared publica-
tions. We apologize for any inadvertent inaccuracies or misrepresen-
tations.
More information about the course, including problem sets and
references, can be found on the course website:

https://www.cs.cmu.edu/~15850/

The style files (as well as the text on this page!) are mildly adapted
from the ones developed by Yufei Zhao (MIT), for his notes on Graph
Theory and Additive Combinatorics. As some of you may guess, the
LATEX template used for these notes is called tufte-book.
Contents

I Discrete Algorithms 9

1 Minimum Spanning Trees 11

1.1 Minimum Spanning Trees: History . . . . . . . . . . . . . 11
1.2 The Classical Algorithms . . . . . . . . . . . . . . . . . . 13
1.3 Fredman and Tarjan’s O(m log∗ n)-time Algorithm . . . 15
1.4 A Linear-Time Randomized Algorithm . . . . . . . . . . 18
1.5 Optional: MST Verification . . . . . . . . . . . . . . . . . 21
1.6 The Ackermann Function . . . . . . . . . . . . . . . . . . 25
1.7 Matroids . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2 Arborescences: Directed Spanning Trees 27

2.1 Arborescences . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.2 The Chu-Liu/Edmonds/Bock Algorithm . . . . . . . . . 28
2.3 Linear Programming Methods . . . . . . . . . . . . . . . 30
2.4 Matroid Intersection . . . . . . . . . . . . . . . . . . . . . 35

3 Dynamic Algorithms for Graph Connectivity 37

3.1 Dynamic Connectivity . . . . . . . . . . . . . . . . . . . . 37
3.2 Frederickson’s algorithm . . . . . . . . . . . . . . . . . . . 39
3.3 An Amortized, Deterministic Algorithm . . . . . . . . . . 43
3.4 A Randomized Algorithm for Dynamic Graph Connectiv-
ity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4 Shortest Paths in Graphs 49

4.1 Single-Source Shortest Path Algorithms . . . . . . . . . . 49
4.2 The All-Pairs Shortest Paths Problem (APSP) . . . . . . . 52
4.3 Min-Sum Products and APSPs . . . . . . . . . . . . . . . 55
4.4 Undirected APSP Using Fast Matrix Multiplication . . . 58
4.5 Optional: Fredman’s Decision-Tree Complexity Bound . 61

5 Low-Stretch Spanning Trees 63

5.1 Towards a Definition . . . . . . . . . . . . . . . . . . . . . 63
5.2 Low-Stretch Spanning Tree Construction . . . . . . . . . 66
4

5.3 Bartal’s Construction . . . . . . . . . . . . . . . . . . . . . 67

5.4 Metric Embeddings: a.k.a. Simplifying Metrics . . . . . . 72

6 Graph Matchings I: Combinatorial Algorithms 75

6.1 Notation and Definitions . . . . . . . . . . . . . . . . . . . 75
6.2 Bipartite Graphs . . . . . . . . . . . . . . . . . . . . . . . . 77
6.3 General Graphs: The Tutte-Berge Theorem . . . . . . . . 80
6.4 The Blossom Algorithm . . . . . . . . . . . . . . . . . . . 81
6.5 Subsequent Work . . . . . . . . . . . . . . . . . . . . . . . 86

7 Graph Matchings II: Weighted Matchings 87

7.1 Linear Programming . . . . . . . . . . . . . . . . . . . . . 87
7.2 Weighted Matchings in Bipartite Graphs . . . . . . . . . 90
7.3 Another Perspective: Buyers and sellers . . . . . . . . . . 94
7.4 A Third Algorithm: Shortest Augmenting Paths . . . . . 99
7.5 Perfect Matchings in General Graphs . . . . . . . . . . . 100
7.6 Integrality of Polyhedra . . . . . . . . . . . . . . . . . . . 101

8 Graph Matchings III: Algebraic Algorithms 105

8.1 Preliminaries: roots of low degree polynomials . . . . . . 105
8.2 Detecting Perfect Matchings by Computing a Determinant107
8.3 From Detecting to Finding Perfect Matchings . . . . . . . 110
8.4 Red-Blue Perfect Matchings . . . . . . . . . . . . . . . . . 113
8.5 Matchings in Parallel, and the Isolation Lemma . . . . . 114
8.6 A Matrix Scaling Approach . . . . . . . . . . . . . . . . . 115

II The Curse of Dimensionality, and Dimension Reduction

117

9 Concentration of Measure 119

9.1 Asymptotic Analysis . . . . . . . . . . . . . . . . . . . . . 120
9.2 Non-Asymptotic Convergence Bounds . . . . . . . . . . . 121
9.3 Chernoff bounds, and Hoeffding’s inequality . . . . . . . 124
9.4 Other concentration bounds . . . . . . . . . . . . . . . . . 127
9.5 Application: Oblivious Routing on the Hypercube . . . . 130

10 Dimension Reduction and the JL Lemma 135

10.1 The Johnson Lindenstrauss lemma . . . . . . . . . . . . . 135
10.2 The Construction . . . . . . . . . . . . . . . . . . . . . . . 136
10.3 Intuition for the Distributional JL Lemma . . . . . . . . . 137
10.4 The Direct Proof of Lemma 10.2 . . . . . . . . . . . . . . 138
10.5 Subgaussian Random Variables . . . . . . . . . . . . . . . 140
10.6 JL Matrices using Rademachers and Subgaussian-ness . 142
10.7 Optional: Compressive Sensing . . . . . . . . . . . . . . . 143
10.8 Some Facts about Balls in High-Dimensional Spaces . . . 145
 5

11 Streaming Algorithms 147

11.1 Streams as Vectors, and Additions/Deletions . . . . . . . 148
11.2 Computing Moments . . . . . . . . . . . . . . . . . . . . . 149
11.3 A Matrix View of our Estimator . . . . . . . . . . . . . . . 152
11.4 Application: Approximate Matrix Multiplication . . . . . 153
11.5 Optional: Computing the Number of Distinct Elements . 154

12 Dimension Reduction: Singular Value Decompositions 159

12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 159
12.2 Best fit subspaces of dimension k and the SVD . . . . . . 160
12.3 Useful facts, and rank-k-approximation . . . . . . . . . . 163
12.4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 164
12.5 Symmetric Matrices . . . . . . . . . . . . . . . . . . . . . . 166

III From Discrete to Continuous Algorithms 167

13 Online Learning: Experts and Bandits 169

13.1 The Mistake-Bound Model . . . . . . . . . . . . . . . . . . 169
13.2 The Weighted Majority Algorithm . . . . . . . . . . . . . 170
13.3 Randomized Weighted Majority . . . . . . . . . . . . . . 172
13.4 The Hedge Algorithm, and a Change in Perspective . . . 174
13.5 Optional: The Bandit Setting . . . . . . . . . . . . . . . . 176

14 Solving Linear Programs using Experts 179

14.1 (Two-Player) Zero-Sum Games . . . . . . . . . . . . . . . 179
14.2 Solving LPs Approximately . . . . . . . . . . . . . . . . . 182

15 Approximate Max-Flows using Experts 187

15.1 The Maximum Flow Problem . . . . . . . . . . . . . . . . 187
15.2 A First Algorithm using the MW Framework . . . . . . . 188
15.3 Finding Max-Flows using Electrical Flows . . . . . . . . 190
15.4 An Oe (m3/2 )-time Algorithm . . . . . . . . . . . . . . . . . 195
15.5 Optional: An O e (m4/3 )-time Algorithm . . . . . . . . . . . 196

16 The Gradient Descent Framework 203

16.1 Convex Sets and Functions . . . . . . . . . . . . . . . . . 203
16.2 Unconstrained Convex Minimization . . . . . . . . . . . 205
16.3 Constrained Convex Minimization . . . . . . . . . . . . . 209
16.4 Online Gradient Descent, and Relationship with MW . . 211
16.5 Stronger Assumptions . . . . . . . . . . . . . . . . . . . . 212
16.6 Extensions and Loose Ends . . . . . . . . . . . . . . . . . 215

17 Mirror Descent 217

17.1 Mirror Descent: the Proximal Point View . . . . . . . . . 217
17.2 Mirror Descent: The Mirror Map View . . . . . . . . . . . 220
6

17.3 The Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 223

17.4 Alternative Views of Mirror Descent . . . . . . . . . . . . 225

18 The Centroid and Ellipsoid Algorithms 227

18.1 The Centroid Algorithm . . . . . . . . . . . . . . . . . . . 227
18.2 The Ellipsoid Algorithm . . . . . . . . . . . . . . . . . . . 229
18.3 Ellipsoid for LP Feasibility . . . . . . . . . . . . . . . . . . 230
18.4 Ellipsoid for Convex Optimization . . . . . . . . . . . . . 232
18.5 Getting the New Ellipsoid . . . . . . . . . . . . . . . . . . 233
18.6 Algorithms for Solving LPs . . . . . . . . . . . . . . . . . 236

19 Interior-Point Methods 237

19.1 Barrier Functions . . . . . . . . . . . . . . . . . . . . . . . 238
19.2 The Update Step . . . . . . . . . . . . . . . . . . . . . . . 240
19.3 The Newton-Raphson Method . . . . . . . . . . . . . . . 244
19.4 Self-Concordance . . . . . . . . . . . . . . . . . . . . . . . 246

IV Combating Intractability 247

20 Approximation Algorithms 249

20.1 A Rough Classification into Hardness Classes . . . . . . 249
20.2 The Surrogate . . . . . . . . . . . . . . . . . . . . . . . . . 251
20.3 The Set Cover Problem . . . . . . . . . . . . . . . . . . . . 251
20.4 A Relax-and-Round Algorithm for Set Cover . . . . . . . 252
20.5 The Bin Packing Problem . . . . . . . . . . . . . . . . . . 254
20.6 The Linear Grouping Algorithm for Bin Packing . . . . . 255
20.7 Subsequent Results and Open Problems . . . . . . . . . . 257

21 Approximation Algorithms via SDPs 259

21.1 Positive Semidefinite Matrices . . . . . . . . . . . . . . . . 259
21.2 Semidefinite Programs . . . . . . . . . . . . . . . . . . . . 260
21.3 SDPs in Approximation Algorithms . . . . . . . . . . . . 262
21.4 The MaxCut Problem and Hyperplane Rounding . . . 262
21.5 Coloring 3-Colorable Graphs . . . . . . . . . . . . . . . . 265

22 Online Algorithms 271

22.1 The Competitive Analysis Framework . . . . . . . . . . . 271
22.2 The Ski Rental Problem: Rent or Buy? . . . . . . . . . . . 273
22.3 The Paging Problem . . . . . . . . . . . . . . . . . . . . . 277
22.4 Generalizing Paging: The k-Server Problem . . . . . . . . 279

V Additional Topics 281

23 Smoothed Analysis of Algorithms 283

 7

23.1 The Traveling Salesman Problem . . . . . . . . . . . . . . 284

23.2 The Simplex Algorithm . . . . . . . . . . . . . . . . . . . 286
23.3 The Knapsack Problem . . . . . . . . . . . . . . . . . . . . 289

24 Prophets and Secretaries 295

24.1 The Prophet Problem . . . . . . . . . . . . . . . . . . . . . 295
24.2 Secretary Problems . . . . . . . . . . . . . . . . . . . . . . 301
 Part I

Discrete Algorithms
1
Minimum Spanning Trees

1.1 Minimum Spanning Trees: History

In minimum spanning tree problem, the input is an undirected con-

nected graph G = (V, E) with n nodes and m edges, where the
edges have weights w(e) ∈ R. The goal is to find a spanning tree
of the graph with the minimum total edge-weight. If the graph G A spanning tree/forest is defined to be
is disconnected, we get a spanning forest As a classic (and important) an acyclic subgraph T that is inclusion-
wise maximal, i.e., adding any edge in
problem, it’s been tackled many times. Here’s a brief, not-quite- G \ T would create a cycle.
comprehensive history of its optimization, all without making any
assumptions on the edge weights other that they can be compared in
constant time:

• Otakar Borůvka 1 gave the first known MST algorithm in 1926; 1

it was independently discovered by Gustave Choquet, Georges

Sollin, and others. Vojtečh Jarník 2 gave his algorithm in 1930, and 2

it was independently discovered by Robert Prim (’57) and Edsger

Dijkstra (’59), among others. Joseph Kruskal gave his algorithm J.B. Kruskal, Jr. (1956)
in ’56; this was rediscovered by Loberman and Weinberger in 57. Loberman and Weinberger (1957)
All these can easily be implemented in O(m log n) time; we will
discuss these in this lecture. Both Prim and Kruskal refer to
Borůvka’s paper, but say it is “un-
• In 1975, Andy Yao 3 achieved a runtime of O(m log log n). His necesarily elaborate”. However, while
Borůvka’s paper is written in a compli-
algorithm builds on Borůvka’s algorithm (which he attributes cated fashion, but his essential ideas are
to Sollin), and uses as a subroutine the linear-time algorithm for very clean.
median-finding, which had only recently been invented in 1974. 3

We will work through Yao’s algorithm in HW#1.

• In 1984, Michael Fredman and Bob Tarjan gave an O(m log∗ n) Fredman and Tarjan (1987)
time algorithm, based on their Fibonacci heaps data structure.
Here log∗ is the iterated logarithm function, and denotes the num-
ber of times we must take logarithms before the argument be-
comes smaller than 1. The actual runtime is a bit more nuanced,
which we will not bother with today.
12 minimum spanning trees: history

This result was soon improved by Gabow, Galil, Spencer, and

Tarjan (’86) to get an O(m log log∗ n) runtime—note the logarithm Gabow, Galil, Spencer, and Tarjan (1986)
applied to the iterated logarithm.

• In 1995, David Karger, Phil Klein and Bob Tarjan finally got the Karger, Klein, and Tarjan (1995)
holy grail of O(m) time! . . . but it was a randomized algorithm, so
the search for a deterministic linear-time algorithm continued.

• In 1997, Bernard Chazelle gave an O(mα(n))-time deterministic Chazelle (1997)

algorithm. Here α(n) is the inverse Ackermann function (defined
in §1.6). This function grows extremely slowly, even slower than
the iterated logarithm function. However, it still goes to infinity
as n → ∞, so we still don’t have a deterministic linear-time MST
algorithm.

• In 1998, Seth Pettie and Vijaya Ramachandran gave an optimal Pettie and Ramachandran (1998)
algorithm for computing minimum spanning trees—however,
we don’t know its runtime! More formally, they show that if This was part of Seth’s Ph.D. thesis, and
there exists an algorithm which uses MST ∗ (m, n) comparisons Vijaya was his advisor.

to find MSTs on all graphs with m edges and n nodes, the Pettie-
Ramachandran algorithm will run in time O( MST ∗ (m, n)).)

In this chapter, we’ll go through the three classics (Jarnik/Prim’s,

Kruskal’s, and Borůvka’s). Then we will discuss Fredman and Tar-
jan’s algorithm, and finally present Karger, Klein, and Tarjan’s ran-
domized algorithm. This will lead us to discuss another intriguing
question: how do we verify whether a given tree is an MST?
For the rest of this chapter, assume that the edge weights are dis-
tinct. This does not change things in any essential way, but it ensures
that the MST is unique (Exercise: prove this!), and hence simpli-
fies some statements. Also assume the graph is simple, and hence
m = O(n2 ); you can delete all self-loops and remove all-but-the-
lightest from any collection of parallel edges, all by preprocessing the
graph in linear time.

1.1.1 The Cut and Cycle Rules

Most of these algorithms rely on two rules: the cut rule (known in
Tarjan’s book as the blue rule) and the cycle rule (or the red rule). Tarjan (1983)
Recall that a cut in the graph is a partition of the vertices into two
non-empty sets (S, S̄ = V \ S), and an edge crosses this cut if its two
endpoints lie in different sets.

Theorem 1.1 (Cut Rule). For any cut of the graph, the minimum-weight
edge that crosses the cut must be in the MST. This rule helps us determine
what to add to our MST.
 minimum spanning trees 13

Proof. Let S ⊊ V be any nonempty proper subset of vertices, let

e = {u, v} be the minimum-weight edge that crosses the cut defined
by (S, S̄) (W.l.o.g., u ∈ S, v ∈/ S), and let T be a spanning tree not
containing e. Then T ∪ {e} contains a unique cycle C. Since C crosses
the cut (S, S̄) once (namely at e), it must also cross at another edge
e′ . But w(e′ ) > w(e), so T ′ = ( T − {e′ }) ∪ {e} is a lower-weight tree
than T, so T is not the MST. Since T was an arbitrary spanning tree
not containing e, the MST must contain e.

Theorem 1.2 (Cycle Rule). For any cycle in G, the heaviest edge on that
cycle cannot be in the MST. This helps us determine what we can remove in
constructing the MST.

Proof. Let C be any cycle, let e be the heaviest edge in C. For a con-
tradiction, let T be an MST that contains e. Dropping e from T gives
two components. Now there must be some edge e′ in C \ {e} that
crosses between these two components, and hence T ′ := ( T − {e′ }) ∪
{e} is a spanning tree. (Make sure you see why.) By the choice of e
we have w(e′ ) < w(e), so T ′ is a lower-weight spanning tree than T, a
contradiction.

To find a minimum spanning tree, we repeated apply whichever

of these rules we like. E.g., we choose some cut, use the cut rule to
designate the lightest edge in it as belonging to the MST by coloring
it blue (hence the name). 4 Or we choose a cycle which contains no 4
This edge e cannot have previously
red edge, use the cycle rule to mark the heaviest edge as not being in been colored red—this follows from the
above lemmas. Or more directly, any
the MST, and color it red. (Again, this edge cannot already be blue cycle crosses any cut an even number
for similar reasons.) And if either of the rules is not applicable, we of times, so a cycle containing e also
contains another edge f in the cut,
are done. Indeed, if we cannot apply the blue rule, the blue edges which is heavier.
cross every cut, and hence form a spanning tree, which must be the
MST. Similarly, once the non-red edges do not contain a cycle, they
form a spanning tree, which must be the MST. All known algorithms
differ only in their choice of cut/cycle, and how they find these fast.
Indeed, all the deterministic algorithms we discuss today will just
use the cut rule, whereas the randomized algorithm will use the cycle
rule as well.

1.2 The Classical Algorithms

1.2.1 Kruskal’s Algorithm

For Kruskal’s Algorithm, first sort all the edges such that w(e1 ) <
w(e2 ) < · · · < w(em ). This takes O(m log m) = O(m log n) time.
Start with all edges being uncolored, and iterate through the edges
in the sorted order, coloring an edge blue if and only if it connects
14 the classical algorithms

two vertices which are not currently in the same blue component.
Figure 1.1 gives an example of how edges are added. 1 3

To keep track of which vertex is in which component, use a dis- 10

5
joint set union-find data structure. This data structure has three
4
operations: 2

• makeset(elem), which takes an element elem and creates a new

singleton set for it, Figure 1.1: Dashed lines are not yet in
the MST. Note that 5 will be analyzed
next, but will not be added. 10 will
• find(elem), which finds the canonical representative for the set be added. Colors designate connected
containing the element elem, and components.

• union(elem1 , elem2 ), which merges the two sets that elem1 and
elem2 are in.

There is an implementation of this data structure which allows us to

do m operations in O(m α(m)) amortized time, where α(·) is the in-
verse Ackermann function mentioned above. Note that the naïve im-
plementation of Kruskal’s algorithm spends O(m log m) = O(m log n)
time to sort the edges, and then performs n makesets, m finds, and
n − 1 union operations, the total runtime is O(m log n + m α(m)),
which is dominated by the O(m log n) term.

1.2.2 The Jarnik/Prim Algorithm

For the Jarnik/Prim algorithm, first take an arbitrary root vertex r to
3
start our MST T. At each iteration, take the cheapest edge connecting 1

of our current tree T of blue edges to some vertex not yet in T, and 10
5
color it blue—thereby adding this edge to T and increasing its size by
4
one. Figure 1.2 below shows an example of how we edges are added. 2
We’ll use a priority queue data structure which keeps track of the
lightest edge connecting T to each vertex not yet in T. A priority Figure 1.2: Dashed lines are not yet in
queue data structure is equipped with (at least) three operations: the MST. We started at the red node,
and the blue nodes are also part of T
• insert(elem, key) inserts the given (element, key) pair into the right now.

queue,

• decreasekey(elem, newkey) changes the key of the element elem

from its current key to min(originalkey, newkey), and

• extractmin() removes the element with the minimum key from

the priority queue, and returns the (elem, key) pair.

Note that by using the standard binary heap data structure we can
get O(log n) worst-case time for each priority queue operation above.
To implement the Jarnik/Prim algorithm, we initially insert
each vertex in V \ {r } into the priority queue with key ∞, and
the root r with key 0. The key of an node v denotes the weight of
 minimum spanning trees 15

the least-weight edge from a node in T to v; it is zero if v ∈ T,

and ∞ if there are no edges yet from nodes in T to v. At each step,
use extractmin to find the vertex u with smallest key, and add u
to the tree using this edge. Then for each neighbor of u, say v, do
decreasekey(v, w({u, v})). Overall we do m decreasekey operations, We can optimize slightly by inserting
n inserts, and n extractmins, with the decreasekeys supplying the a vertex into the priority queue only
when it has an edge to the current
dominating O(m log n) term. tree T. This does not seem particularly
useful right now, but will be crucial in
the Fredman-Tarjan proof.
1.2.3 Borůvka’s Algorithm
Unlike Kruskal’s and Jarnik/Prim’s algorithms, Borůvka’s algorithm
adds many edges in parallel, and can be implemented without any
non-trivial data structures. In a “round”, simply take the lightest
edge out of each vertex and color it blue; these edges are guaranteed
to form a forest if edge-weights are distinct. (Exercise: why?)
Now contract the blue edges and recurse on the resulting graph.
At the end, when the resulting graph is a single vertex, uncontract all
the edges to get the MST. Each round can be implemented in O(m)
work: we will work out the details of this in HW #1. Moreover, we’re
1 3
guaranteed to shrink away at least half of the nodes (as each node at
10 10
least pairs up with one other node), and maybe many more if we are 5
lucky. So we have at most ⌈log2 n⌉ rounds of computation, leaving us
4
with O(m log n) total work. 2

1.2.4 A Slight Improvement on Jarnik/Prim Figure 1.3: The red edges will be
chosen and contracted in a single
step, yielding the graph on the right,
We can actually easily improve the performance of Jarnik/Prim’s
which we recurse on. Colors designate
algorithm by using a more sophisticated data structure, namely by components.
using Fibonacci heaps instead of binary heaps to implement the
priority queue. Fibonacci heaps (invented by Fredman and Tarjan)
implement the insert and decreasekey operations in constant amor-
tized time, and extractmin in amortized O(log H ) time, where H is
the maximum number of elements in the heap during the execution.
Since we do n extractmins, and O(m + n) of the other two opera-
tions, and the maximum size of the heap is at most n, this gives us a
total cost of O(m + n log n).
Note that this is linear time on graphs with m = Ω(n log n) edges;
however, we’d like to get linear-time on all graphs. So the remaining
cases are the graphs with m = o (n log n) edges.

1.3 Fredman and Tarjan’s O(m log∗ n)-time Algorithm

Fredman and Tarjan’s algorithm builds on Jarnik/Prim’s algorithm:

the crucial observation uses the following crucial facts.
16 fredman and tarjan’s o ( m log ∗ n ) -time algorithm

The amortized cost of extractmin operations in Fibonacci heaps is

O(log H ), where H is the maximum size of the heap. Moreover, in
Jarnik/Prim’s algorithm, the size of the heap is just the number of
nodes that are adjacent to the current tree T. So if the current tree
always has a “small boundary”, the extractmin cost will be low.

How can we maintain the boundary to be smaller than some

threshold K? Simple: Once the boundary exceeds K, stop growing
the Prim tree, and begin Jarnik/Prim’s algorithm anew from a dif-
ferent vertex. Do this until we have a forest where all vertices lie in
some tree; then contract these trees (much like Borůvka), and recurse
on the smaller graph. Before we formally define the algorithm, here’s
an example.

A H

61 13
1 6 R
2 55 7

56
14 16
8
3

57 62
9 15
4
58 63
11 10

17
52
53 12

51

G 60
5 59 18
54
D
Figure 1.4: We begin at vertices A, H,
R, and D (in that order) with K = 6.
Formally, in each round of the algorithm, all vertices start as un-
Although D begins as its own compo-
marked. nent, it stops when it joins with tree
A. Dashed edges are not chosen in this
1. Pick an arbitrary unmarked vertex and start Jarnik/Prim’s algo- step (though they may be chosen in the
rithm from it, creating a tree T. Keep track of the lightest edge next recursive call), and colors denote
trees.
from T to each vertex in the neighborhood N ( T ) of T, where
N ( T ) := {v ∈ V − T | ∃u ∈ T s.t. {u, v} ∈ E}. Note that
N ( T ) may contain vertices that are marked.
 minimum spanning trees 17

2. If at any time | N ( T )| ≥ K, or if T has just added an edge to some

vertex that was previously marked, stop and mark all vertices in
the current T, and go to step 1.

3. Terminate when each node belongs to some tree.

Let’s first note that the runtime of one round of the algorithm is
O ( m + n log K ) . Each edge is considered at most twice, once from
each endpoint, giving us the O ( m ) term. Each time we grow the
current tree in step 1, the number of connected components decreases
by 1, so there are at most n such steps. Each step calls findmin on a
heap of size at most K, which takes O ( log K ) times. Hence, at the
end of this round, we’ve successfully identified a forest, each edge of
which is part of the final MST, in O ( m + n log K ) time.
Let d v be the degree of the vertex v in the graph we consider in
this round. We claim that every marked vertex u belongs to a com-
ponent C such that ∑ v ∈ C d v ≥ K. Indeed, if u became marked be-
cause the neighborhood of its component had size at least K, then
this is true. Otherwise, u became marked because it entered a com-
ponent C of marked vertices. Since the vertices of C were marked,
∑ v ∈ C d v ≥ K before u joined, and this sum only increased when u
(and other vertices) joined. Thus, if C 1 , . . . , C l are the components at
the end of this routine, we have
l l
2m = ∑ dv = ∑ ∑ dv ≥ ∑ K ≥ Kl
v i = 1 v ∈ Ci i =1

Thus l ≤ 2m 2m
K , i.e. this routine produced at most K trees.
The choice of K will change over the course of the algorithm. How
should we set the thresholds K i ? Say we start round i with n i nodes
and m i ≤ m edges. One clean way is to set
2m
K i : = 2 ni

which ensures that

 
2m
O ( m i + n i log K i ) = O mi + ni · = O(m).
ni

In turn, this means the number of trees, and hence the number of
nodes n i + 1 in the next round, is at most 2m 2m
K i ≤ K i . The number of
i

edges is m i + 1 ≤ m i ≤ m. Rewriting, this gives

2m
Ki ≤ = lg K i + 1 =⇒ K i + 1 ≥ 2 K i .
ni +1

Hence the threshold value exponentiates in each step. Hence after The threshold increases “tetrationally”.
log ∗ n rounds, the value of K would be at least n, and we would
18 a linear-time randomized algorithm

just run Jarnik/Prim’s algorithm to completion, ending with a sin-

gle tree. This means we have at most log ∗ n rounds, and a total of
O ( m log ∗ n ) work.
In retrospect, I don’t know whether to consider the Fredman-
Tarjan algorithm as being trivial (once we have Fibonacci heaps) or
being devilishly clever. I think it is the latter (and that is the beauty
of the best algorithms). Indeed, there’s a lovely idea—of keeping the
neighborhoods small at the beginning when there’s a lot of work to
do, but allow them to grow quickly, as the graph collapses. It is quite
non-obvious at the start, and obvious in hindsight. And once you see
it, you cannot un-see it!

1.4 A Linear-Time Randomized Algorithm

Another algorithm that is extremely clever but almost obvious in

hindsight is the the Karger-Klein-Tarjan randomized MST algorithm, Karger, Klein, and Tarjan (1995)
which runs in O ( m + n ) expected time. The new idea here is to A version of this algorithm was pro-
compute a “rough approximation” to the MST, use that to throw posed by Karger in 1992, but he only
obtained an O(m + n log n) runtime.
away many edges using the cycle rule, and then recurse on the rest of The enhancement to linear time was
the graph. given by Klein and Tarjan at the STOC
1994 conference; the combined paper is
cited above.
1.4.1 Heavy & light edges
The crucial definition is that of edges being heavy and light with
respect to some forest F.

Definition 1.3. Let F be a forest that is a subgraph of G. An edge e ∈

E( G ) is F-heavy if e creates a cycle when added to F, and moreover it
is the heaviest edge in this cycle. Otherwise, we say edge e is F-light.

The next facts follow from the definition: Figure 1.5: Fix this figure, make it
Fact 1.4. Edge e is F-light ⇐⇒ e ∈ MST( F ∪ {e}). interesting. Every edge in F is F-light,
as are the edges on the left, and also
Fact 1.5 (Completeness). If T is an MST of G then edge e ∈ E( G ) is those going between the components.
The edge on the right is F-heavy.
T-light if and only if e ∈ T.
Fact 1.6 (Soundness). For any forest F, the F-light edges contain the
MST of the underlying graph G. In other words, any F-heavy edge is
also heavy with respect to the MST of the entire graph.
This suggests a clear strategy: pick a forest F from the current
edges, and discard all the F-heavy edges. Hopefully the number of
edges remaining is small. By Fact 1.6 these edges contain the MST of
G, so repeat the process on them. To make this idea work, we want
a forest F with many F-heavy edges. The catch is that a forest has
many heavy edges if it has small weight, if there are many off-forest
edges forming cycles where they are the heaviest edges. Indeed, one
 minimum spanning trees 19

such forest in the MST T ∗ of G: Fact 1.5 shows there are m − (n − 1)

many T ∗ -heavy edges, the maximum possible. How do we find some
similarly good tree/forest, but in linear time?
A second issue is to classify edges as light/heavy, given a forest F.
It is easy to classify a single edge e in linear time, but the following
remarkable theorem is also true:

Theorem 1.7 (MST Verification). Given a forest F ⊆ G, we can output

the set of all F-light edges in G in time O(m + n).

This MST verification algorithm itself uses several interesting

ideas; we discuss some of them in Section 1.5. But for now, let us use
it to give the randomized linear-time MST algorithm.

1.4.2 The Randomized MST Algorithm

The idea is simple and elegant: randomly choose half of the edges
and find the minimum-weight spanning forest F on this “half-of-a-
graph”. This forest F should have many F-heavy edges; we discard The random subgraph may not be
these and recursively find the MST on the remaining graph. Since connected, so the maximum spanning
forest is obtained by finding the MST
both the recursive calls are on smaller graphs, hopefully the runtime for each of its connected components.
will be linear.
The actual algorithm below has just one extra step: we first run a
few rounds of Borůvka’s algorithm to force a reduction in the num-
ber of vertices, and then do the steps above.
Algorithm 1: KKT(G)
1.1 Run 3 rounds of Borůvka’s Algorithm on G, contracting the
chosen edges to get a graph G ′ = (V ′ , E′ ) with n′ ≤ n/8
vertices and m′ ≤ m edges.
1.2 If G ′ has a single vertex, return any chosen edges.
1.3 E1 ← random sample of E′ , each edge picked indep. w.p. 1/2.
1.4 F1 ← KKT(G1 = (V ′ , E1 )).
1.5 E2 ← all the F1 -light edges in E′ .
1.6 F2 ← KKT(G2 = (V ′ , E2 )).
1.7 return F2 (combined with Borůvka edges chosen in Step 1).

Theorem 1.8. The KKT algorithm returns MST(G).

Proof. This follows from Fact 1.6, that discarding heavy edges of any
forest F in a graph does not change the MST. Indeed, the MST on
G2 is the same as the MST on G ′ , since the discarded F1 -heavy edges
cannot be in MST ( G ′ ) because of Fact 1.6. Adding back the edges
picked by Borůvka’s algorithm in Step 1 gives the MST on G, by the
cut rule.

Now we need to bound the running time. The following two

20 a linear-time randomized algorithm

claims formalize the intuition that we recurse on “smaller” sub-

graphs:
Claim 1.9. E[#E1 ] = 21 m′ .
Claim 1.10. E[#E2 ] ≤ 2n′ .
The first claim is easy to prove, using linearity of expectations, and
that each edge is picked with probability 1/2. The proof of Claim 1.10
is also short, but before we prove it, let us complete the proof of the
linear running time.

Theorem 1.11. The KKT algorithm, run on a graph with m edges and n
vertices, terminates in expected time O(m + n).

Proof. Let TG be the expected running time on graph G, and

Tm,n := max { TG }.
G =(V,E),|V |=n,| E|=m

In the KKT algorithm, Step 1, 2, 4 and 6 can each be done in linear

time: indeed, the only non-trivial part is Step 4, for which we use
Theorem 1.7. Let the total time for these steps be at most cm. Steps 3
and 5 requires time TG1 and TG2 respectively. Then we have

TG ≤ cm + E[ TG1 + TG2 ] ≤ cm + E[ Tm1 ,n′ + Tm2 ,n′ ],

where m1 = #E1 and m2 = #E2 are both random variables. Induc-

tively assume that Tm,n ≤ c(2m + n), then

TG ≤ cm + E[c(2m1 + n′ )] + E[c(2m2 + n′ )]
≤ c(m + m′ + 6n′ )
≤ c(2m + n)

The second inequality holds because E[m1 ] ≤ 12 m′ and E[m2 ] ≤ 2n′ .

The last inequality holds because n′ ≤ n/8 and m′ ≤ m. Indeed, we
shrunk the graph using Borůvka’s algorithm in the first step just to
ensure n′ ≤ 8n and hence give us some “breathing room”.

Now we prove Claim 1.10. Recall that we randomly subsample

the edges of G ′ to get G1 , compute its maximum spanning forest F1 ,
and now we want to bound the expected number of edges in G ′ that
are F1 -light. The key to the proof is to do all these steps together, This idea to defer looking at the ran-
deferring the random decisions to when we really need them. This dom choices of the algorithm is often
called the principle of deferred deci-
makes it apparent which edges are light, making them easy to count. sions.

Proof of Claim 1.10. For the sake of the proof, we can use any correct
algorithm to compute F1 , so let us use Kruskal’s algorithm. Moreover,
let’s run a lazy version as follows: first sort all the edges in E′ , and
not just those in E1 ⊆ E′ , and consider then in increasing order
 minimum spanning trees 21

of weights. Now if the currently considered edge ei connects two

different trees in the current blue forest, call ei useful and flip an
independent unbiased coin: if the coin comes up “heads”, color ei
blue and add it to F1 , else color ei red. The crucial observation is
that this process produces a forest from the same distribution as first
choosing G1 and then computing F1 by running Kruskal’s algorithm
on it.
Now, let us consider the lazy process again: which edges are F1 -
light? We claim that these are precisely the useful edges. Indeed,
any non-useful edge e j forms a cycle with the previously chosen
blue edges in F1 , and it is the heaviest edge on that cycle. Hence
e j does not belong to MST ( F1 ∪ {e j }), so it is F1 -heavy by Fact 1.4.
Figure 1.6: Illustration of another order
And a useful edge ei would belong to MST ( F1 ∪ {ei }), since run- of coin tossing
ning Kruskal’s algorithm on F1 ∪ {ei } would see that ei connects two
different blue components and hence would pick it.
Finally, how many useful edges are there, in expectation? Let’s
abstract away the details: we’re running a process that periodically
asks us to flip an independent unbiased coin. Since each time we see
a heads, we add an edge to the forest, so we definitely stop when we
see n′ − 1 heads. (We may stop earlier, in case the process runs out of
edges, but then we can pad the random sequence to flip some more
coins.) Since the coins are independent and unbiased, the expected
number of flips until we see n′ − 1 heads is exactly 2(n′ − 1). This
proves Claim 1.10.

That’s it. The algorithm and proof are both short and slick and
beautiful: this result is a real gem. I think it’s an algorithm from The Paul Erdős claimed that God has “The
Book. The one slight annoyance with the algorithm is the relative Book” which contains the most elegant
proof of each mathematical theorem.
complexity of the MST verification algorithm, which we use to find
the F1 -light edges in linear time. Nonetheless, these verification The current verification algorithms are
algorithms also contain many nice ideas, which we now discuss. deterministic; can we use randomness
to simplify these as well?

1.5 Optional: MST Verification

We now come back to the implementation of the MST verification

procedure. Here we only consider only trees (not forests), since we
can run this algorithm separately on each tree in the forest and incur
only a linear extra cost. Let us refine Theorem 1.7 as follows.
Theorem 1.12 (MST Verification). Given a tree T = (V, E) where
|V | = n, and m pairs of vertices (yi , zi ) in T, we can find the heaviest edge
on the unique yi -to-zi path in T for all i, in O(m + n) time.
Since the edge {yi , zi } is T-heavy precisely if it is heavier than the
heaviest edge on the corresponding tree path, this also proves The-
orem 1.7. Observe that the query pairs are given up-front: there is
22 optional: mst verification

an inverse-Ackermann-type lower bound for the problem where the

queries arrive online. Pettie (2006)
How do we get such a linear-time algorithm? A priori, it is not
easy to even show a query-complexity upper bound: that there exists
a procedure that performs a linear number of edge-weight comparisons
to solve the MST verification problem. This problem was solved
by János Komlós. His result was subsequently made algorithmic Komlos (1985)
(“how do you find (in linear time) which linear number of queries to
make?”) by Brendan Dixon, Monika Rauch (now Monika Henzinger)
and Bob Tarjan. This algorithm was futher simplified by Valerie
King 5 , and by Thomas Hagerup 6 . We will just discuss Komlós’s 5
6
query-complexity bound.

1.5.1 A Simpler Case

To start developing the algortihm, it helps to consider special cases:
e.g., what if the tree is a complete binary tree? Let’s assume some- A node v is an ancestor of u if v lies on
thing slightly less restrictive than a complete binary tree: suppose the unique path from u to the root; then
u is a descendent of v.
tree T is rooted at some node r, all internal nodes have at least 2 chil-
dren, and all its leaves are at the same level. Moreover, all queries
{yi , zi } are for pairs where yi is a leaf and zi its ancestor.
Now for an edge (u, v) of the tree, where v is the parent and u
the child, consider all queries starting within subtree Tu and ending
at vertex v or higher. Say these queries go from some leaves inside
Tv up to w1 , w2 , . . . , wk , where w1 is closest to the root. Define the
“query string”
Q e : = ( w1 , w2 , . . . , w k ) .

We want to calculate the “answer string”

A e : = ( a1 , a2 , · · · , a k ),

where ai is the largest weight among the edges between wi and u.

Now given the answer string A(b,a) , we can get the answer string
for a child edge. In the example, say the query string for edge (c, b)
is Q(c,b) = (w1 , w4 , b). We have lost some queries that were in Q(b,a) ,
(e.g., for w3 ) but we now have a query ending at b. To get A(b,a) we
can drop the lost queries, add in the entry for b, and also take the Figure 1.7: Query string Q(b,a) =
component-wise maximum with the weight of (c, b) itself. E.g., if (w1 , w3 , w4 ) means there are three
queries starting from vertices in Tb and
(c, b) has weight t, then ending at w1 , w3 , w4 . The answer string
is A(b,a) = ( a1 , a3 , a4 ) = (6, 4, 4).
A(c,b) = (max{ a1 , t}, max{ a4 , t}, t) = (max{6, 5}, max{4, 5}, 5).

Naïvely this would require us to compare the weight w(c,b) with

all the entries in the answer string, incurring | Ae′ | comparisons. The
crucial observation is this: since the nodes in the query string are
 minimum spanning trees 23

sorted from top to bottom, the answers must be non-increasing: i.e.,

a1 ≥ a2 ≥ · · · ≥ ak . Therefore we can do binary search to reduce
the number of comparisons between edge-weights. Indeed, given
the answer string for some edge e, we can compute answers Ae′ for a
child edge e′ using at most ⌈log(| Ae′ | + 1)⌉ comparisons. This will be
enough to prove the result.
Claim 1.13. The total number of comparisons for all queries is at most
m+n
∑ log (|Qe | + 1) ≤ O(n + n log n
) = O ( m + n ).
e

Proof. Let the number of edges at height i be ni , where height 1 cor-

responds to edges incident to the leaves.

∑ log2 (1 + | Qe |) = ni avge∈height i (log2 (1 + | Qe |))

e∈height i
 
≤ ni log2 1 + avge∈height i (| Qe |)
 
m
≤ ni log2 1 +
ni
 
m+n 4n
= ni log2 + log2 .
4n ni

The first inequality uses concavity of the function log2 (1 + x ), and

Jensen’s inequality. The second holds because each of the m queries Jensen’s inequality says that for any
can only appear on at most one edge, so the average “load” is at most convex function f and any random
variable X, E[ f ( X )] ≥ f (E[ X ]). Con-
m/ni . Summing the first term over all heights gives n log2 m4n+n =
cavity requires flipping the sign, of
O ( m ). course.
To bound the second term (summed over all heights), recall that
i
each node has at least two children, so the number of edges at least S= ∑ 2i
i ≥0
doubles each time the height decreases. Hence, ni ≤ n/2i−1 , and i i+1
2S = ∑ 2i − 1 = ∑ 2i
4n n 4n O (i ) i ≥0 i ≥0
∑ ni log2 ≤ ∑ i−1 log2 i − 1
= n · ∑ i = O ( n ). ( i + 1) − i 1
i ≥1
ni i ≥1 2 n/2 i ≥1 2 =⇒ 2S − S = ∑ 2i
= ∑ i = 2.
i ≥0 i ≥0 2

The inequality above uses that x log(4n/x ) is increasing for x ≤

n.

Converting this into an algorithm that runs in O(m + n) time re-

quires quite a bit more work. The essential idea is to store each query
string Q(u,v) as a bit vector of length log2 n, indicating which nodes
on the path from v to the root belong to it Q(u,v) . Now the answers
A(u,v) can be stored by encoding the locations of the successive max-
ima. And answers for a child edge can be computed from that of the
parent edge using some tricky bit operations (e.g., by precomputing
solutions on bit-strings of length, say (log2 n)/3, of which there are
only n1/3 × n1/3 = n2/3 ). If you are interested, check out these lecture
slides by Uri Zwick.
24 optional: mst verification

1.5.2 Solving the General Case

Finally, we reduce a general instance of MST verification to the spe-
cial instances considered in §1.5.2. First we reduce to a “branch-
ing” tree with the special properties we asked for, then we alter the
queries to become leaf-ancestor queries.
To achieve this reduction, run Borůvka’s algorithm on the tree T.
After the ith round of edge selection and contraction, let Vi be the
remaining vertices, so that V0 = V is the original set of nodes. Define
a new tree T ′ whose vertex set V ′ is the disjoint union V0 ⊎ V1 ⊎ · · · .
A node u ∈ Vi has an edge in T ′ to v ∈ Vi+1 if the component
containing u was contracted into the new vertex v; the weight of this
edge in T ′ is the weight of the minimum-weight edge chosen by u in
this round. Moreover, if r is the single vertex corresponding to the
entire tree T at the end of the run of Borůvka’s algorithm, then root Figure 1.8: Illustration of balancing a
tree T ′ at r. tree. We have maxwtT (v1 , v7 ) is 7 which
is the weight of edge (v4 , v6 ). We can
Exercise 1.14. Show that each node in T ′ has at least two children, check that maxwtT ′ (v1 , v7 ) is also 7.

and all leaves belong to the same level. There are n leaves (corre-
sponding to the nodes in T), and at most 2n − 1 nodes in T ′ . Also
show how to construct T ′ in linear time.

Exercise 1.15. For nodes u, v in a tree T, let maxwtT (u, v) be the maxi-
mum weight of an edge on the (unique) path between u, v in the tree
T. Show that all u, v ∈ V, maxwtT (u, v) = maxwtT ′ (u, v).

This exercise means arbitrary queries (yi , zi ) in the original tree T

can be reduced to leaf-leaf queries in T ′ . To make these leaf-ancestor
queries, we simply find the least-common ancestor ℓi := lca(yi , zi )
for each pair, and replace the original query by the maximum of two
queries (yi , ℓi ), (zi , ℓi ). To show that we can find the least-common
ancestors in linear time, we defer to a theorem of David Harel and
Bob Tarjan: Harel and Tarjan (1984)

Theorem 1.16. Given a tree T, we can preprocess it in O(n) time, so that

all subsequent least-common ancestor queries for T can be answered in O(1)
time.

Interestingly, this algorithm also proceeds by solving the least-

common ancestor problem for complete balanced binary trees, and
then extending the solution to general trees. For a survey of algo-
rithms for this problem, see the paper of Alstrup et al. Alstrup et al. (2004)
This completes Komlós’ proof that the MST verification problem
can be solved using O(m + n) comparisons. An outstanding open
problem is to get a really simple linear-time algorithm for this prob-
lem. (An algorithm that runs in time O(mα(n)) can be given using
the disjoint set union-find data structure.)
 minimum spanning trees 25

1.6 The Ackermann Function

Wilhelm Ackermann defined a fast-growing function that is totally Ackermann (1928)

computable but not primitive recursive. Today, we use the term A similar function was defined by
Ackermann function A(m, n) to refer to one of many variants that are Gabriel Sudan, a Romanian mathemati-
cian, in 1927.
rapidly-growing and have similar propeties. It seems to arise often in
algorithm analysis, so let’s briefly discuss it here.
For illustrative purposes, it is cleanest to define A(m, n) : N ×
N → N recursively as


 2n : m=1
A(m, n) = 2 : m ≥ 1, xn = 1

 A(m − 1, A(m, n − 1)) : m ≥ 2, n ≥ 2

Here are the values of A(m, n) for m, n ≤ 4:

1 2 3 4 ... n
1 2 4 6 8 ... 2n
2 2 4 8 16 ... 2n
.2
2 22 ..
3 2 4 22 22 ... 22
4 2 4 65536 !!! ... huge!

We can define the inverse Ackermann function α(·) to be a functional

inverse of the diagonal A(n, n); by construction, α(·) grows extremely
.2
..
slowly. For example, α(m) ≤ 4 for all m ≤ 22 where the tower has
height 65536.

1.7 Matroids

To come here. See HW1 as well.

2
Arborescences: Directed Spanning Trees

Greedy algorithms worked vey well for minimum weight spanning

tree problem, as we saw in Chapter 1. In this chapter, we define ar-
borescences which are a notion of spanning trees for rooted directed
graphs. We will see that a naïve greedy approach no longer works,
but it requires just a slightly more sophisticated algorithm to effi-
ciently find them. We give two proofs of correctness for this algo-
rithm. The first is a direct inductive proof, but the second makes use
of linear programming duality, and highlights its use in analyzing
the performance of algorithms. This will be a theme that return to
multiple times in this course.

2.1 Arborescences

Consider a graph G = (V, A, w): here V is a set of vertices, and A a

set of directed edges, also known as arcs. The function w : A → R We will use “arcs” instead of “edges”
gives a weight to every arc. Let |V | = n and | A| = m. Once we root to emphasize the directedness of the
graph.
G at a node r ∈ V, we can define a “directed spanning tree” with r
being the sink/root.
Definition 2.1. An r-arborescence is a subgraph T = (V, A′ ) with A branching is the directed analog of
A′ ⊆ A such that a forest; it drops the first reachability
requirement, and asks only for all
1. Every vertex has a directed path in T to the root r, and non-root vertices to have an outgoing
edge.
2. Each vertex except r has one outgoing arc; r has none.
Remark 2.2. Observe that T forms a spanning tree in the undirected
sense. This property (along with either property 1 or property 2) can
alternatively be used to define an arborescence.
Remark 2.3. It’s easy to check if an r-arborescence exists. We can
reverse the arcs and run a depth-first search from the root. If all
vertices are reached, we have produced an r-arborescence.
The focus of this chapter is to find the minimum-weight r-arborescence.
We can simplify things slightly by assuming that all of the weights
28 the chu-liu/edmonds/bock algorithm

are non-negative. Because no outgoing arcs from r will be part If there are negative arc weights, add
of any arborescence, we can assume no such arcs exist in G either. a large positive constant M to every
weight. This increases the total weight
For brevity, we fix r and simply say arborescence when we mean of each arborescence by M (n − 1), and
r-arborescence. hence the identity of the minimum-
weight one remains unchanged.

2.1.1 The Limitations of Greedy Algorithms

It’s natural to ask if greedy algorithms like those in Chapter 1 for the
directed case. E.g., we can try picking the lightest incoming arc into
the component containing r, as in Prim’s algorithm, but this fails,
for example in Figure 2.1. Or we could emulate Kruskal’s algorithm
and consider arcs in increasing order of weight, adding them if they
don’t close a directed cycle. (Exercise: give an example where it fails.)
The problem is that greedy algorithms (that consider the arcs in
some linear order and irrevocably add them in) don’t see to work.
However, the algorithm we eventually get will feel like Borůvka’s
algorithm, but one where we are allowed to revoke some of our past
decisions.

2.2 The Chu-Liu/Edmonds/Bock Algorithm Figure 2.1: A Prim-like algorithm will

select the arc with weight 2 and 3,
The algorithm we present was discovered independently by Yoeng- whereas the optimal choices are the arcs
with weights 3 and 1.
Jin Chu and Tseng-Hong Liu 1 , Jack Edmonds, and F. Bock 2 . We will 1

follow Karp’s presentation of Edmonds’ algorithm. Edmonds (1967)

2

Definition 2.4. For a vertex v ∈ V or subset of vertices S ⊆ V, let Karp (1971)

∂+ v and ∂+ S denote the set of arcs leaving the node v and the set S,
respectively.

Definition 2.5. For a vertex v ∈ V in graph G, define MG (v) :=

mina∈∂+ v w( a) be the minimum weight among arcs leaving v in G.

The first step is to create a new graph G ′ by subtracting some

weight from each outgoing arc from a vertex, such that there is at
least one arc of weight 0. That is, set w( a′ ) ← w( a) − MG (v) for all
a ∈ ∂+ v and each v ∈ V.
Claim 2.6. T is a min-weight arborescence in G ⇐⇒ T is a min-
weight arborescence in G ′ .

Proof. Each arborescence has exactly one arc leaving each vertex.
Decreasing the weight of every arc exiting v by MG (v) decreases the
weight of every possible arborescence by MG (v) as well. Thus, the set
of min-weight arborescences remains unchanged.

Now each vertex has at least one 0-weight arc leaving it. Now, for
each vertex, pick an arbitrary 0-weight arc out of it. If this choice is
 arborescences: directed spanning trees 29

an arborescence, this must be the minimum-weight arborescence,

since all arc weights are still nonnegative. Otherwise, the graph con-
sist of some connected components, each of which has one directed
cycle along with some acyclic incoming components, as shown in the
figure.
For the second step of the algorithm, consider one such 0-weight
cycle C, and construct a new graph G ′′ := G ′ /C by contracting the
cycle C down to a single new node vC , removing arcs within C, and
replacing parallel arcs by the cheapest of these arcs. Let OPT( G )
denote the weight of the min-weight arborescence on G. Figure 2.2: An example of a possible
component after running the first step
Claim 2.7. OPT( G ′ ) = OPT( G ′′ ). of the algorithm

Proof. To show OPT(G ′ ) ≤ OPT(G ′′ ), we exhibit an arborescence in

G ′ with weight at most OPT(G ′′ ). Indeed, let T ′′ be a min-weight
arborescence in G ′′ . Consider arborescence T ′ in G ′ obtained by ex-
panding vC back to the cycle C, and removing one arc in the cycle.
Since the cycle has weight 0 on all its arcs, T ′ has the same weight as
T ′′ . (See Figure 2.3.)
Now to show OPT(G ′′ ) ≤ OPT(G ′ ), take a min-weight arborescence Figure 2.3: The white node is expanded
into a 4-cycle, and the dashed arrow is
T ′ of G ′ , and identify the nodes in C down to get a vertex vC . The
the arc that is removed after expanding.
resulting graph is clearly connected, with each vertex having a di-
rected path to the root. Now remove some arcs to get an arborescence
of G ′′ , e.g., as in Figure 2.4. Since arc weights are non-negative, we
can only lower the weight by removing arcs. Therefore OPT(G ′′ ) ≤
OPT(G ′ ).

The proof also gives an algorithm for finding the min-weight ar-
borescence on G ′ by contracting the cycle C (in linear time), recursing
on G ′′ , and the “lifting” the solution T ′′ back to a solution T ′ . Since
we recurse on a graph which has at least one fewer nodes, there are Figure 2.4: Contracting the two white
nodes down to a cycle, and removing
at most n recursive calls. Moreover, the weight-reduction, contraction, arc b.
and lifting steps in each recursive call take O(m) time, so the runtime
of the algorithm is O(mn).
Remark 2.8. This is not the best known run-time bound: there are
many optimizations possible. Tarjan 3 presents an implementation 3

of the above algorithm using priority queues in O(min(m log n, n2 ))

time, and Gabow, Galil, Spencer and Tarjan 4 give an algorithm to 4

solve the min-weight arborescence problem in O(n log n + m) time.

The best runtime currently known is O(m log log n) due to Mendel-
son et al. 5 . 5

Open problem 2.9. Is there a linear-time (randomized or determinis-

tic) algorithm to find a min-weight arborescence in a digraph G?
30 linear programming methods

2.3 Linear Programming Methods

Let us now see an alternate proof of correctness of the algorithm

above, this time using linear programming duality. This is how Ed-
monds originally proved his algorithm to be optimal. If you have access to the Chu-Liu or
Bock papers, I would love to see them.

2.3.1 Linear Programming Review

Before we actually represent the arborescence problem as a linear
program, we first review some standard definitions and results from
linear programming.

Definition 2.10. For some number of variables (a.k.a. dimension) n ∈

N, number of constraints m ∈ N, objective vector c ∈ Rn , constraint
matrix A ∈ Rn×m , and right-hand side b ∈ Rm , a (minimization)
linear program (LP) is This form of the LP is called the stan-
dard form. More here.
minimize c⊺ x subject to Ax ≥ b and x ≥ 0

Note that c⊺ x is the inner product ∑in=1 ci xi .

The constraints of a linear program form a polyhedron, which is

the convex body formed by the intersection of a finite number of
half spaces. Here we have m + n half spaces. There are m of them
corresponding to the constraints { a⊺i x ≥ bi }im=1 , where ai ∈ Rn is the
vector corresponding to the ith row of the matrix A. Moreover, we Whenever we write a vector, we imag-
have n non-negativity constraints { x j ≥ 0}nj=1 . If the polyhedron is ine it to be a column vector.

bounded, we call it a polytope.

Definition 2.11. A vector x ∈ Rn is called feasible if it satisfies the

constraints: i.e., Ax ≥ b and x ≥ 0.

Definition 2.12. Given a linear program min{c⊺ x | Ax ≤ b, x ≥ 0},

the dual linear program is

maximize b⊺ y subject to A⊺ y ≤ c and y ≥ 0

The dual linear program has a single variable yi for each constraint
in the original (primal) linear program. This variable can be thought
of as giving an importance weight to the constraint, so that taking a
linear combination of constraints with these weights shows that the
primal cannot possibly surpass a certain value for c⊺ x. This purpose
is exemplified by the following theorem.

Theorem 2.13 (Weak Duality). If x and y are feasible solutions to the

linear program min{c⊺ x | Ax ≤ b, x ≥ 0} and its dual, respectively, then
c⊺ x ≥ b⊺ y.

Proof. c⊺ x ≥ ( A⊺ y)⊺ x = y⊺ Ax ≥ y⊺ b = b⊺ y.
 arborescences: directed spanning trees 31

See the strong duality theorem in add

This principle of weak duality tells us that if we have feasible reference for a converse to this theorem.
For now, weak duality will suffice.
solutions x, y where c⊺ x = b⊺ y, then we know that both x and y are
optimal solutions. Our approach will be to give a linear program that
models min-weight arborescences, use the algorithm above to write a
feasible solution to the primal, and then to exhibit a feasible solution
to the dual such that the primal and dual values are the same—hence
both must be optimal!

2.3.2 Arborescences via Linear Programs

To analyze the algorithm, we first need to come up with a linear

program that “captures” the min-weight arborescence problem. Since
we want to find a set of arcs forming an arborescence T, we have
one variable x a for each arc a ∈ A. Ideally, each variable will be
an indicator for the arc being in the arborescence: i.e., it will binary
values: x a ∈ {0, 1}, with x a = 1 if and only if a ∈ T. This choice
of variables allows us to express our objective to minimize the total
weight: w⊺ x := ∑ a∈ A w( a) x a .
Next, we need to come up with a way to express the constraint
that T is a valid arborescence. Let S ⊆ V − {r } be a set of vertices
not containing the root, and some vertex v ∈ S. Every vertex must
be able to reach the root by a directed path. If ∂+ S ∩ T = ∅, there
is no arc in T leaving the set S, and hence we have no path from v to
r. We conclude that, at a minimum, ∂+ S ∩ T ̸= ∅. We represent this
constraint by ensuring that the number of arcs out of S is non-zero,
i.e.,
∑ x a ≥ 1.
a∈∂+ S

We write an integer linear programming (ILP) formulation for min-

weight arborescences as follows:

minimize ∑ w( a) x a
a∈ A
subject to ∑ xa ≥ 1 ∀ S ⊆ V − {r }
a∈∂+ S (2.1)
∑ xa = 1 ∀v ̸= r
a∈∂+ v
x a ∈ {0, 1} ∀ a ∈ A.

The following lemma is easy to verify:

Lemma 2.14. T is an arborescence of G with x a = 1 a∈T if and only if x is

feasible for the integer LP (2.2). Hence the optimal solution to the ILP (2.1)
is exactly the min-weight arborescence.
32 linear programming methods

Relaxing the Boolean integrality constraints gives us the linear

programming relaxation:

minimize ∑ w( a) x a
a∈ A
subject to ∑ xa ≥ 1 ∀ S ⊆ V − {r }
a∈∂+ S (2.2)
∑ xa = 1 ∀v ̸= r
a∈∂+ v
xa ≥ 0 ∀ a ∈ A.

Since we have relaxed the constraints, the optimal solution to the

(fractional) LP (2.2) can only have less value than the ILP (2.1), and
hence the optimal value of the LP is at most OPT( G ). In the follow-
ing, we show that it is in fact equal to OPT( G )!

Exercise 2.15. Suppose all the arc weights are non-negative. Show
that the optimal solution to the linear program remains unchanged
even if drop the constraints ∑ a∈∂+ v x a = 1.

2.3.3 Showing Optimality

The output T of the Chu-Liu/Edmonds/Bock algorithm is an ar-
borescence, and hence the associated solution x (as defined in Lemma 2.14)
is feasible for ILP (2.1) and hence for LP (2.2). To show that x is op-
timal, we now exhibit a vector y feasible for the dual linear program
with objective equal to w⊺ x. Now weak duality implies that both x
and y must be optimal primal and dual solutions.
The dual linear program for (2.2) is

maximize ∑ yS
S⊆V −{r }

subject to ∑ yS ≤ w( a) ∀a ∈ A (2.3)
S:a∈∂+ S
yS ≥ 0 ∀S ⊆ V − {r }, |S| > 1.

Observe that yS is unconstrained when |S| = 1, i.e., S corresponds to

a singleton non-root vertex.
We think of yS as payments raised by vertices inside set S so that
we can buy an arc leaving S. In order to buy an arc a, we need to
raise w( a) dollars. We’re trying to raise as much money as possible,
while not overpaying for any single arc a.

Lemma 2.16. If arc weights are non-negative, there exists a solution for the
dual LP (2.3) such that w⊺ x = 1⊺ y, where all ye values are non-negative.

Proof. The proof is by induction over the execution of the algorithm.

 arborescences: directed spanning trees 33

• The base case is when the chosen zero-weight arcs out of each
node form an arborescence. In this case we can set yS = 0 for
all S; since all arc weights are non-negative, this is a feasible dual
solution. Moreover, both the primal and dual values are zero.

• Suppose we subtract M := MG (v) from all arcs leaving vertex v in

graph G so that v has at least one zero-weight arc leaving it. Let G ′
be the graph with the new weights, and let T ′ be the optimal so-
lution on G ′ . By induction on G ′ , let y′ be a non-negative solution
such that ∑ a∈T ′ we′ = ∑S y′S . Define yv := y′v + M and yS = y′S for
all other subsets; this is the desired feasible dual solution for the
same tree T = T ′ on the original graph G. Indeed, for one of the
arcs a = (v, u) out of the node v, we have

∑ yS = ∑ yS + ∑ yS
S:a∈∂+ S S:a∈∂+ S,|S|=1 S:a∈∂+ S,|S|≥2
1 r
= (y′{u} + M) + ∑ y′S
S:a∈∂+ S,|S|≥2
2 5
3
≤ M + w ′ ( a ) = M + ( w ( a ) − M ) = w ( a ). 2
1

2 3 2
Moreover, the value of the dual increases by M, the same as the
3 2
increase in the weight of the arborescence.
6 7
• Else, suppose the chosen zero-weight arcs contain a cycle C, which
we contract down to a node vC . Using induction for this new 7
graph G ′ , let y′ be the feasible dual solution. For any subset S′ of
nodes in G ′ that contains the new node vC , let S = (S′ \ {vC }) ∪ C,
Figure 2.5: An optimal dual solution:
and define yS = y′S′ . For all other subsets S in G ′ not containing
vertex sets are labeled with dual values,
vC , define yS = y′S . Moreover, for all nodes v ∈ C, define y{v} = 0. and arcs with costs.
The dual value remains unchanged, as does the weight of the
solution T obtained by lifting T ′ . The dual constraint changes only
arcs of the form a = (v, u), where v ∈ C and u ̸∈ C. But such
an arc is replaced by an arc a′ = (vC , u), whose weight is at most
w( a). Hence

∑ yS = y′{v
C}
+ ∑ y′S ≤ w( a′ ) ≤ w( a).
S:a∈∂+ S S′ :a′ ∈∂+ S′ ,S′ ̸={v C}

This completes the inductive proof

Notice that the sets with non-zero weights correspond to single-
ton nodes, or to the various cycles contracted during the algorithm.
Hence these sets form a laminar family; i.e., any two sets S, S′ with
non-zero value in y are either disjoint, or one is contained within the
other.

By Lemma 2.16 and weak duality, we conclude that the solution x

and the associated arborescence T is optimal. It is easy to extend the
argument to potentially negative arc weights.
34 linear programming methods

Corollary 2.17. There exists a solution for the dual LP (2.3) such that
w⊺ x = 1⊺ y. Hence the algorithm produces an optimal arborescence even for
negative arc weights.

Proof. If some arc weights are negative, add M to all arc weights to
get the new graph G ′ where all arc weights are positive. Let y′ be the
optimal dual for G ′ from Lemma 2.16; define yS = y′S for all sets of
size at least two, and y{v} = y′{v} − M for singletons. Note that the
weight of the optimal solution on G is precisely M(n − 1) smaller
than on G ′ ; the same is true for the total dual value. Moreover, for arc
e = (u, v), we have

∑ yS = ∑ y′S + (y′{u} − M) ≤ (we + M) − M = we .

S:a∈∂+ S S:a∈∂+ S,|S|≥2

The inequality above uses that y′ is a feasible LP solution for the

graph G ′ with inflated arc weights. Finally, since the only non-
negative values in the dual solution are for singleton sets, all con-
straints in (2.2) are satisfied for the dual solution y, this completes the
proof.

2.3.4 Integrality of the Polytope

The result of Corollary 2.17 is quite exciting: it says that no matter
what the objective function of the linear program (i.e., the arc weights
w( a)), there is an optimal integral solution to the linear program,
which our combinatorial algorithm finds. In other words, the optimal
solutions to the LP (2.2) and the ILP (2.1) are the same.
We will formally discuss this later in the course, but let us start
playing with these kinds of ideas. A good start is to visualize this ge-
ometrically: let A ⊆ R| A| be the set of all solutions to the ILP (which
correspond to the characteristic vectors of all valid r-arborescences).
This is a finite set of points, and let Karb be the convex hull of these
points. (It can be shown that Karb is a polytope, though we don’t do
it here.) If we optimize a linear function given by some weight vector
w over this polytope, we get the optimal arborescence for this weight.
This is the solution to ILP (2.1).
Moreover, let K ⊆ R| A| be the polytope defined by the constraints
in the LP relaxation (2.2). Note that each point in A is contained
within K, therefore so is their convex hull K. I.e.,

Karb ⊆ K.

In general, the two polytopes are not equal. But in this case, Corol-
lary 2.17 implies that for this particular setting, the two are indeed
equal. Indeed, a geometric hand-wavy argument is easy to make —
if K were strictly bigger than Karb , there would be some direction
 arborescences: directed spanning trees 35

in which K extends beyond Karb . But each direction corresponds to

a weight-vector, and hence for that weight vector the optimal solu-
tion within K (which is the solution to the LP) would differ from the
optimal solution within Karb (which is the solution to the ILP). This
contradicts Corollary 2.17.

2.4 Matroid Intersection

More to come here, maybe just a forward pointer to a later lecture.

3
Dynamic Algorithms for Graph Connectivity

Dynamic algorithms is the study of algorithmic problems in the

setting where the input changes over time. At each timestep, we
get either an update which tells us how the input has changed, or
a query which demands outputting some aspect of the solution to
the underlying problem. In this chapter focus on some basic graph
questions, but the area is much broader, and we can consider efficient
dynamic algorithms for most algorithmic questions.
For graph problems, the most common model is the edge-update
model. Hence, the update operations are InsertEdge (e) and Dele-
teEdge (e), which add and delete some edge e from the graph re-
spectively. (It is useful to also allow InsertNode (v) and DeleteN-
ode (v), which add and delete isolated nodes in the graph; we can-
not delete a node that has an edge incident to it.) If we can handle
both kinds of edge-update operations, we are said to be in the fully-
dynamic case, else we may be in the insert-only or delete-only case.
How can we maintain solutions to basic graph problems in these
models?

3.1 Dynamic Connectivity

We restrict our attention to the most basic of all graph problems:

graph connectivity. As the graph changes via a sequence of Insert
and Delete operations, we want to support two types of queries:

• Connected (u, v): are the vertices u, v in the same connected

component of the current graph?

• Connected ( G ): is the current graph G connected.

Here are two naïve approaches for this problem:

1. We keep track of the updates, say using an adjacency matrix, and

then run depth-first search each time we get a query. This takes
O(1) time for each update, and O(m + n) time for each query.
38 dynamic connectivity

2. We explicitly maintain the connected components of graph by

running depth-first search after each update. Now each query
takes O(1) time, but each update may take O(m) time. If there are no deletions, we can use
disjoint set union-find data structure to
These two extreme approaches suggest a trade-off between update implement both updates and queries in
O(α(n)) amortized time.
time and query time: how can we balance between them to get small
query and update times?

3.1.1 Some Issues to Consider

The quality of our algorithms will depend on the power we allow
them. For example,

• worst-case versus amortized update and query times: does the

algorithm take at most B time on each step? Or can some of the
steps require more time and others less, as long as any sequence of
T operations requires at most BT time?

• deterministic versus randomized algorithms: does the algorithm

flip coins or not?
Even among randomized algorithms, we may consider questions
like Las Vegas versus Monte Carlo algorithms: is the algorithm
always correct, or is it just correct for each output with high prob-
ability? Or that of adaptive versus oblivious adversaries: can the
request at some step depend on the coin-tosses made by the al-
gorithm in previous steps? We defer these subtler distinctions
for now, and just focus on either deterministic algorithms, or on
Monte Carlo algorithms against oblivious adversaries, where the
input is assumed to be fixed before we flip any coins.

3.1.2 And Some Results

The dynamic graph connectivity problem has been studied along all
of these axes, which makes it exhausting to give a complete list of the
current best results. Let us just list the results we will discuss.

1. For deterministic algorithms with worst-case runtimes, Greg Fred-

√
erickson showed how to get O( m) update time and O(1) query
√
time. This was improved to O( n) update times by Eppstein et
al., but then progress stalled for quite some time. We will discuss Only very recently, Chuzhoy et al.
broke through the logjam, and obtained
Frederickson’s approach, and see Eppstein et al.’s extension in a
no(1) worst-case update time; the team
homework. includes CMU almunus Richard Peng
and current student Jason Li.
2. For deterministic algorithms with amortized runtimes, Holm,
deLichtenberg,
 and Thorup showed O(log2 n) update times and
log n
O log log n query times.
 dynamic algorithms for graph connectivity 39

3. For randomized algorithms with worst-case bounds, Kapron, King,

Mountjoy gave an algorithm with poly-logarithmic update and
query times.

All three of these algorithms illustrate some clever algorithmic ideas;

the running times are not the point here.

3.1.3 Oh, and a Lower Bound

A quick word about lower bounds. Sadly, we cannot achieve constant
update and query times: a lower bound of Mihai Pǎtraşcu and Erik
Demaine says that if tu denotes the update time and tq denotes the
query time (both in a deterministic? amortized sense), then

tq log(tu /tq ) = Ω(log n)

tu log(tq /tu ) = Ω(log n)

This implies, for instance, that max{tu , tq } = Ω(log n). More details
to come.

3.2 Frederickson’s algorithm

We describe a weaker version of Frederickson’s algorithm, which has

an O(m2/3 ) update time, and O(1) query time. As we mentioned,
the algorithm maintains a spanning forest of the graph. To find a
replacement edge fast, it clusters the forest into “clusters” that are
roughly the same size. Now it keeps track of which clusters have
edges going between them. So, when searching for a replacement
edge, it can just look at pairs of clusters, instead of at all nodes.

3.2.1 The Basic Operations

To avoid worrying about lower-level details, assume we have a dy-
namic forest data structure that supports the following operations:
1. CreateNode (v) creates an isolated node,

2. Tree (v) returns the identity of the tree containing v,

3. Link (u, v) adds the edge uv to the trees containing u and v (as-
suming these trees are distinct), and

4. Cut (u, v) removes the edge uv from its tree.

For simplicity, we assume that these can be done in constant time. In
practice, these can be implemented, e.g., using the Euler Tree (ET)
data structure of Monika Henzinger and Valerie King, in O(log n)
worst-case time, or using the link-cut data structure of Danny Sleator
and Bob Tarjan, in amortized logarithmic time.
40 frederickson’s algorithm

3.2.2 A First Attempt

Using this data structure, a naïve algorithm to maintain a spanning
forest would be the following:

• When an edge uv is added, check if Tree (u) = Tree (v). If so,

ignore it, else Link (u, v).

• When an edge uv is deleted and does not belong to the current

spanning forest, do nothing. Else, if it belongs to the spanning
forest, Cut (u, v). This creates two trees Tu and Tv . However, there
may be edges in G crossing between these trees, so we need to find a
replacement edge.

Iterating over the edges incident to one these trees (say Tu ) would
take time ∑ x∈Tu deg( x ). How can we do it faster? All the algorithms
of this section address this replacement edge question in different
ways.

3.2.3 Reduction to the Sub-Cubic Case

We first reduce to the case where the maximum degree of any node is
at most 3. Given updates to a graph G = (V, E), we instead maintain Any constant degree greater than three
a graph H = (V ′ , E′ ) with V ⊆ V ′ , such that the answer to the would suffice.

queries on G and H are the same. The graph H has O(m) nodes and
edges.
The mapping is simple: pick any vertex v in G with degree d ≥
2, create a cycle v = v1 , v2 , . . . , vd in H, and connect each vertex
on this cycle to a unique neighbor of v. A vertex of degree one is
unchanged. Moreover, this mapping can be maintained dynamically: We don’t need the cycle for nodes that
if an edge e = uv is inserted into G, we may have to increase the have degree at most 3 in G, but it is
easier to enforce a uniform rule.
size of the cycles for both endpoints (or create a cycle, in case the
degree has gone from 1 to 2), and then add an edge. This requires a b
a constant number of InsertNode and InsertEdge operations in
H (and maybe one DeleteEdge as well), so the number of updates
is maintained up to a constant. Deleting an edge in G is the exact
inverse of this process. v u
We need to remember this mapping between elements of G and H:
this can be done using hash-tables, and we omit the details here.

c d
3.2.4 Clustering a Tree
The advantage of a sub-cubic graph is that any spanning forest F
a b
also has maximum degree 3. This allows us to use the following
elementary clustering algorithm:
v1 v2

v5 u

v3 v4

c d
 dynamic algorithms for graph connectivity 41

Lemma 3.1 (Tree Clustering). Given a positive integer z, and any tree
T with at least z nodes and maximum degree 3, we can partition its vertex
set into clusters such that each cluster (a) induces a connected subtree, and
(b) contains between z and 3z nodes.

Proof. Root the tree T at some leaf node. Since the maximum degree
is 3, each node has at most two children. Find some node u such
that the subtree Tu rooted at u has at least z nodes, but each of its
children has strictly less than z nodes in its subtrees. Since there
are at most two children, the total number of nodes in Tu is at most
2(z − 1) + 1 ≤ 2z, the last +1 to account for u itself. Put all the
nodes in Tu in a cluster, and remove them from T, and repeat. If
this process leaves T with fewer than z nodes at the end, add these
remaining nodes to one of its adjacent clusters, making its size at
most 3(z − 1) + 1 ≤ 3z.

Important Exercise: if T has n T nodes, a naïve application of this

theorem may take O(n2T ) time. Give an implementation that clusters
T in O(n T ) time.

3.2.5 The Algorithm and Analysis

To answer connectivity queries, we maintain a spanning forest F of
the dynamically changing sub-cubic graph H. Moreover, for each tree
in F, we also maintain a clustering like in Lemma 3.1, for a value of
z to be choose later. (If some tree in F has size less than z, it forms
a trivial cluster by itself.) Note that there are O(m/z) non-trivial
clusters. Moreover, since all vertices have degree at most 3 in H, each
cluster is incident to O(z) edges from H.
Concretely, we use a dynamic tree data structure to maintain a
sub-forest of the spanning forest F induced by just the in-cluster
edges. Moreover, we use the data structure to also maintain the clus-
ter forest, obtained from the spanning forest F by shrinking each
cluster down to a single node. For example, figure to come soon.
Moreover, for each pair of clusters Ci , Cj , we keep track of all edges
not in F that go between Ci and Cj . This list is non-empty only for
clusters in the same tree. Finally, each vertex remembers its (at most
three) neighbors in both F and H.
Now we describe how to process updates and queries, first with-
out worrying about the sizes of the clusters:

1. Insert (uv): We call Tree (u) and Tree (v) to find the clusters Cu
and Cv that contain them. Then we call Tree (Cu ) and Tree (Cv )
on the cluster forest to find their trees Tu and Tv in F. If u, v belong
to the same tree in F, update the crossing-edge information for the
42 frederickson’s algorithm

pair (Cu , Cv ). Else, u, v are in different trees in F, so merge these

trees by calling Link (Cu , Cv ) in the cluster tree.

2. Delete (uv): If edge uv is not in the spanning forest F, then this

deletion does not change F. Just remove uv from the list of cross-
cluster edges for the pair of clusters Cu , Cv if needed. Otherwise
uv ∈ T for some tree T in the spanning forest F, and we must find
a replacement edge. There are two cases:

(a) Edge uv goes between two clusters Cu and Cv . Then perform

Cut (Cu , Cv ) in the cluster forest, which breaks the tree T into
Tu , Tv , say. For all clusters Ci ∈ Tu and Cj ∈ Tv , check the
cross-cluster information to see if there is an edge xy from Ci
to Cj . Upon finding the first such edge, add it to F, Link the
corresponding clusters in the cluster forest, and remove the
connecting edge from the list of non-forest edges between Ci
and Cj . (You can imagine having just deleted this edge xy from
H, and then immediately added it back in.) The total runtime is
O(m2 /z2 ), in order to check all pairs of clusters (Ci , Cj ), plus a
constant number of dynamic tree operations.
(b) Now suppose edge uv lies within some cluster C. Then per-
forming Cut (u, v) breaks the cluster into two parts, say Cu , Cv .
First, check if there is an edge within C from Cu to Cv : this takes
O(z) time, because there are O(z) nodes in C, each of degree at
most 3. (This is where we use the bounded cluster size!) If such
an edge xy is found, add it to F and use Link ( x, y) to add the
in-cluster edge, thereby reconnecting cluster C. There is no need
to change any of the other metadata we are maintaining.
However, if there is no such edge, expand the “search area”:
look at the two parts Tu , Tv of tree T created by removing uv,
and try to find an edge from Tu to Tv as in part ( a), which still
takes O(m2 /z2 ). Hence the total runtime in this case is O(z +
m2 /z2 ).

One remaining issue to address: the changes above may have

caused a constant number of non-trivial clusters to have sizes that
are too big or too small. If there is a cluster smaller than z, merge
it with an adjacent cluster (perhaps making the result be too big).
If there is a cluster that is bigger than 3z, use the clustering from
Lemma 3.1 to recluster with sizes being in the range [z, 3z]. Since
each cluster only has O(z) edges, using a linear-time implemen-
tation uses only O(z) time. Moreover, these changes need to be
reflected in the cluster forest, which may require O(z) of the edges
in that forest to be changed. Hence, the total overhead of this pro-
cess is only O(z).
 dynamic algorithms for graph connectivity 43

To minimize this worst-case runtime bound of O(z + m2 /z2 ), set

z = m2/3 to balance the two terms and get O(m2/3 ) time for both
InsertEdge and DeleteEdge. The InsertNode and DeleteN-
ode operations can be implemented in O(1) time.

3. Query: for queries of type Query(u,v), use Tree(Tree (u)) to get

the tree containing u, do the same for v, and check if they are iden-
tical. This takes a constant number of dynamic tree operations.

3.2.6 Improvements and Extensions

As mentioned above, the O(m2/3 ) worst-case update time was im-
proved by Frederickson by reducing the O(m2 /z2 ) time to search
over all pairs of clusters down to O(m/z), using additional features
√
of the Euler tree data structure. Setting z = m in this case to bal-
ance terms gives the O(m1/2 ) update time.
Moreover, the homeworks ask you to derive a cool and fairly
generic way to sparsify graphs for dynamic algorithms, which im-
proves the algorithm’s dependence on the number of edges m down
to a dependence on the number of vertices n.
Finally, the big open question: can we improve on the determin-
√
istic worst-case update times of O( n)? This problem has only re-
cently seen an improvement: work by Chuzhoy et al. gives no(1) up- Chuzhoy, Gao, Li, Nanongkai, Peng,
date times, using the idea of expander decompositions, which we hope Saranurak (2019)

to cover in a later chapter.

3.3 An Amortized, Deterministic Algorithm

The clustering approach of Frederickson’s algorithm was to focus the

search on pairs of clusters, instead on doing the naïve search. How-
ever, if we allow amortized update times instead of worst-case ones,
we can avoid this clustering approach, and instead use a charging
scheme to keep track of the work we do.
In this charging scheme, we associate a level with each edge of the
dynamic graph G, which is zero when the edge is just inserted. Now
when T breaks into L, R because some edge is deleted, we want to
scan the edges incident to one of the sides (say the side L with fewer
vertices) to check if they remain within L, or if they go to R. (Because
T was a tree in a spanning forest, there can be no other options: do
you see why?) When we scan an edge e and it fails to connect L, R,
we increase its level by 1—this “pays” for the failed work we did in
scanning e.
Of course, such a charging scheme is completely useless if the
edge levels can increase unboundedly. So we maintain some invari-
ants that ensure the level of any edge stays at most log2 n.
44 an amortized, deterministic algorithm

3.3.1 A Collection of Layered Spanning Forests

Here is how we do it. We imagine there being many graphs

G = G0 ⊇ G1 ⊇ · · · ⊇ Gi · · · ,

where graph Gi consists of the edges with level i or higher. We main-

tain a spanning forest F0 of G0 , with the additional property that
Fi := F0 ∩ Gi is also a spanning forest of Gi . An equivalent way to
think of this is as follows:

(⋆) F0 is a max-weight spanning forest of G0 , w.r.t. the edge-levels.

Adding an edge is easy: set its level to zero, and add it to F0 if it does
not create a cycle in F0 . Deleting an edge not in F0 is also easy: just
delete it. Finally, if an edge e ∈ F0 (say with level ℓ) is deleted, we
need to search for a replacement. Such a replacement edge can only
be at level ℓ or lower—this follows from property (⋆), and the Cut
Rule from ??. Moreover, to maintain property (⋆), we should also
add a replacement edge of the highest level possible. So we consider
off-tree edges from level ℓ downwards.
Remember, when we scan an edge, we want to raise its level, to
charge to it. That could mess with probability (⋆), because this may
cause off-tree edges to have higher level/weight than tree edges. To
avoid this problem, we first raise the levels of some of the tree edges.
Specifically, we do the following steps:

1. Say deleting e breaks tree T ∈ Fℓ into Lℓ , Rℓ . Let | Lℓ | ≤ | Rℓ |; then

scan the edges incident to Lℓ . (This choice of the smaller side will
help us bound the number of levels by log2 n.) Raise all level-ℓ
edges in Lℓ to have level ℓ + 1.

2. Scan level-ℓ edges incident to Lℓ . If some edge e connects to Rℓ ,

this is a replacement edge. Add e to F0 (and hence to F1 , . . . , Fℓ ).
Do not raise its level, though. Stop: you are done.
Else, if the edge stays within Lℓ , raise its level, and try another.

If we fail to find a replacement edge at level ℓ, we lower ℓ by one,

and try the steps again. Finally if we don’t find a replacement edge
between L0 and R0 , we stop—there is no replacement edge, and
the deletion has increased the number of components in the graph.
(Convince yourself that (⋆) remains satisfied when we do all this.)

3.3.2 Bounding the Maximum Level

It just remains to show that no edge can have level more than log2 n—
this is what all the steps above have been leading up to.
 dynamic algorithms for graph connectivity 45

Lemma 3.2. Each tree in forest Fℓ has at most ⌊n/2ℓ ⌋ nodes. Hence, the
level ℓ of any edge is at most log2 n.

Proof. This is clearly true for F0 . Now a tree in Fℓ is formed by

raising the levels of some “left” piece Lℓ−1 . It was the smaller half
of some tree in level ℓ − 1, which by induction has size at most
⌊n/2ℓ−1 ⌋. Half of that is at most ⌊n/2ℓ ⌋. Finally, since each tree must
have size at least 2, the level ℓ of any edge can be at most log2 n.

Observe: a linear number of edges can have levels ≈ log2 n, but

they must lie in small components. I find this particular charg-
ing quite clever: however, variants of this idea (of “charging to the
smaller side”) arise all the time.
Let us summarize the ideas in this algorithm: we raise levels of
edges to pay for the work we do in finding a replacement edge.
Each edge has level O(log n), so the amortized number of scans is
O(log n). We did not talk about the details of the implementation
here, but each of these operations of raising levels and maintaining
spanning forests in amortized O(log n) time, using a dynamic tree
data structure. This gives an amortized update time of O(log2 n). The
current best amortized update-time bound for deterministic algo-
log2 n
rithms is only slightly better, at O( log log n )1 . 1

3.4 A Randomized Algorithm for Dynamic Graph Connectivity

Finally, let’s sketch a randomized approach to finding replacement

edges, which leads to a Monte-Carlo randomized (albeit worst-case)
update time algorithm due to Bruce Kapron, Valerie King, and Alan
Mountjoy. To simplify matters, we consider the case where a single
edge-deletion occurs: this will be enough to highlight two interesting
techniques of their algorithm.

3.4.1 Giving Nodes a Bit Representation, and the Power of XOR

Suppose we delete edge e = uv, causing the tree T to split into L and
R. As a thought experiment, suppose there is a single replacement
edge f in G that goes between L and R: all other edges incident to L
and R do not cross. How would we find f ?
We begin by assigning to each node v a (log2 n)-bit label ℓ(v).
Assume some total ordering on the nodes, so the edge uv (where
u ≺ v in this ordering) has a 2 log2 n-bit label that concatenates the
names of its two vertices in that order:

ℓ(e) := [ℓ(u), ℓ(v)].

46 a randomized algorithm for dynamic graph connectivity

Definition 3.3 (Node Fingerprint). The fingerprint F (v) of node v is

defined to be the exclusive-or of labels of all edges incident to v in G
M
cF (v) := ℓ (e) .
e∈∂G v

Note: we consider all neighbors in the graph G, not just in the span-
ning forest.

It is easy and fast to maintain the fingerprint F (v) for each vertex
v as edges change. Now when we delete an edge uv (and if there is a
unique replacement edge f ), consider the following:
Fact 3.4. The label of the unique replacement edge is given by the
exclusive-or of the fingerprints of all nodes in L. That is,
M
F (v) = ℓ ( f )
v∈ L

Proof. Indeed, each edge with both endpoints in L will be present

twice in the exclusive-or, and hence will be zeroed out. The only
contribution remaining will be ℓ( f ).

3.4.2 Multiple Edges? Subsample!

What if there are multiple possible replacement edges going from L
to R? The result of the calculation in Fact 3.4 will be the XOR of the
labels of all these edges. If this is non-zero, it would signal that there
is a replacement edge, but not tell us its name. But the XOR may
even be zero if there are crossing edges. However, all is not lost—our
secret weapon will be randomness, which we have not used yet.
Let us make an weaker assumption this time: suppose there are
some k replacement edges, and we know this value k. We now keep a
subsampled graph G ′ by picking each edge in G with probability
1/k. Then for any k ≥ 1,
  k −1
′ 1 1 1
Pr(exists unique L-to-R edge in G ) = k · 1− ≥ . (3.1)
k k e

So with constant probability, there is a unique replacement edge in

G ′ . So if we define the fingerprints in terms of edges in G ′ , the same
idea would work. Note that we don’t need to know k precisely: using
any value in [k/2, 2k] would give some constant probability of having
a unique replacement edge. Moreover, by repeating this process
O(log n) times independently—i.e., keeping O(log n) independent
sets of fingerprints—we can ensure that one of these repetitions will
give a unique crossing edge with probability 1 − 1/ poly(n).
Finally, what if we don’t even have a crude estimate of k? We can
try subsampling at rates 1/2, 1/4, . . . , 1/2i , . . . , 1/n, i.e., at all powers
 dynamic algorithms for graph connectivity 47

of 2 between 1/2 and 1/n. There are log2 n such values, and one of
them will be the correct one, up to a factor of 2. Specifically, we keep
O(log n) different data structures, each one using a different one of
these O(log n) sampling rates; at least one of them (the “right” one)
O(log n)
will succeed with probability 1 − poly(n)
, by a trivial union bound.

3.4.3 Wrapping Up
Many of the random experiments would have multiple L-to-R edges:
the answers for those would not make sense: they may give names
of non-edges, or of edges that do not cross between L and R. Hence,
the algorithm above needs a mechanism to check that the answer is
indeed an L-to-R edge. Details of this can be found in the Kapron,
King, and Mountjoy paper.
More worryingly, what about multiple deletions? We might be
tempted to say that if the input sequence is oblivious to the al-
gorithm’s randomness, we can just take a union bound over all
timesteps. However, the algorithm’s behavior—the structure of the
current spanning forest F, and hence which cut is queried during
later edge deletions—depends on the replacement edges found in
previous steps, which are correlated with the randomness. Hence,
we cannot claim independence for the calculations in (3.1). To han-
dle this, Kapron et al. construct a multi-level data structure; see their
paper for details.
Finally, we did not talk about any of the implementation details
here. For instance, how can we compute the XOR of the set of nodes
in L quickly? How do we check if the replacement edge names are
valid? All these can be done using a dynamic tree data structure.
Putting all this toget, the update time becomes O(log5 n), still in the
worst-case. As an aside, if we allow randomization and amortized
bounds, the update times can be improved to within poly(log log n)
factors of the lower bound of O(log n); see this work by 2 . 2

A closely related algorithm is due to Ahn, Guha, and MacGregor.

Discussion of future developments to come soon.
4
Shortest Paths in Graphs

In this chapter, we look at another basic algorithmic construct: given

a graph where edges have weights, find the shortest path between
two specified vertices in it. Here the weight of a path is the sum of
the weights of the edges in it. Or given a source vertex, find shortest
paths to all other vertices. Or find shortest paths between all pairs of
vertices in the graph. Of course, each harder problem can be solved
by multiple calls of the easier ones, but can we do better?
Let us give some notation. The input is a graph G = (V, E), with
each edge e = uv having a weight/length wuv ∈ R. For most of this
chapter, the graphs will be directed: in this case we use the terms
edges and arcs interchangeably, and an edge uv is imagined as being
directed from u to v (i.e., from left to right). Given a source vertex
s, the single-source shortest paths (SSSP) asks for the weights of
shortest paths from s to each other vertex in V. The all-pairs shortest
paths (APSP) problem asks for shortest paths between each pair of
vertices in V. (In the worst-case, algorithms for the s-t-shortest-path
problem also solve the SSSP, so we do not consider these any further.)
We will consider both these variants, and give multiple algorithms
for both. Given the graph and edge-weights, the
There is another source of complexity: whether the edge-weights weight of a shores path between u and
v is often called their distance.
are all non-negative, or if they are allowed to take on negative values.
In the latter case, we disallow cycles of negative weight, else the
shortest-path is not well-defined, since such a cycle allows for ever-
smaller shortest paths as we can just run around the cycle to reduce
the total weight arbitrarily. We could ask for a shortest simple path.
However, this problem is NP-hard in
general, via a reduction from Hamilton
4.1 Single-Source Shortest Path Algorithms path.

The single-source shortest path problem (SSSP) is to find a shortest

path from a single source vertex s to every other vertex in the graph.
The output of this algorithm can either be the n − 1 numbers giving
the weights of the n − 1 shortest paths, or (some compact representa-
50 single-source shortest path algorithms

tion of) these paths. We first consider Dijkstra’s algorithm for the case
of non-negative edge-weights, and give the Bellman-Ford algorithm
that handles negative weights as well.

4.1.1 Dijkstra’s Algorithm for Non-negative Weights

Dijkstra’s algorithm keeps an estimate dist of the distance from s
to every other vertex. Initially the estimate of the distance from s to
itself is set to 0 (which is correct), and is set to ∞ for all other ver-
tices (which is typically an over-estimate). All vertices are unmarked.
Then repeatedly, the algorithm finds an umarked vertex u with the
smallest current estimate, marks this vertex (thereby indicating that
this estimate is correct), and then updates the estimates for all ver- This update step is often said to relax
tices v reachable by arcs uv thus: the edges out of u, which has a nice
physical interpretation. Indeed, any
edge uv for which the dist(v) is strictly
dist(v) ← min{dist(v), dist(u) + wuv } bigger than dist(u) + wuv can be
imagined to be over-stretched, which
We keep all the vertices that are not marked and their estimated dis- this update fixes.
tances in a priority queue, and extract the minimum in each iteration.
Algorithm 2: Dijkstra’s Algorithm
Input: Digraph G = (V, E) with edge-weights we ≥ 0 and
source vertex s ∈ G
Output: The shortest-path distances from each vertex to s
2.1 add s to heap with key 0
2.2 for v ∈ V \ {s} do
2.3 add v to heap with key ∞
2.4 while heap not empty do
2.5 u ← deletemin
2.6 for v a neighbor of u do
2.7 key(v) ← min{ key(v), key(u) + wuv } // relax uv

To prove the correctness of the algorithm, it suffices to show that

each time we extract a vertex u with the minimum estimated distance
from the priority queue, the estimate for that vertex u is indeed the
distance from s to u. This can be proved by induction on the number
of marked vertices, and left as an exercise. Also left as an exercise are
the modifications to return the shortest-path tree from node s.
The time complexity of the algorithm depends on the prior-
ity queue data structure. E.g., if we use binary heap, which in-
curs O(log n) for decrease-key as well as extract-min operations,
we incur a running time of O(m log n). But just like for spanning
trees, we can do better with Fibonacci heaps, which implement the
decrease-key operation in constant amortized time, and extract-min
in O(log n) time. Since Dijkstra’s algorithm uses n inserts, n delete-
mins, and m decrease-keys, this improves the running time to O(m +
n log n).
 shortest paths in graphs 51

There have been many other improvements since Dijkstra’s origi-

nal work. If the edge-weights are integers in {0, . . . , C }, a clever pri- (Dijkstra (1959)
ority queue data structure of Peter van Emde Boas1 can be used in- The paper also gives his version of
the Járnik/Prim MST algorithm.
stead; this implements all operations in time O(log log C ). Carefully
p The two algorithms are not that
using it can give us runtimes of O(m log log C ) and O(m + n log C ) different, since the MST algorithm
(see Ahuja et al. 2 ). Later, 3 showed a faster implementation for merely changes the update rule to
dist(v) ← min{dist(v), wuv }.
the case that the weights are integer, which has the running time 1

of O(m + n log log(n)) time. Currently, latest results to come here. 2

a
4.1.2 The Bellman-Ford Algorithm 3 1
s −3 t
Dijkstra’s algorithm does not work on instances with negative edge 5

weights; see the example on the right. For such instances, we want b
Figure 4.1: Example with negative
that a correct SSSP algorithm to either return the distances from s to
edge-weights: Dijkstra’s algorithm gives
all other vertices, or else find a negative-weight cycle in the graph. a label of 4 for t, whereas the correct
The most well-known algorithm for this case is the Shimbel- answer is 3.

Bellman-Ford algorithm. 4 Just like Dijkstra’s algorithm, this algo- 4

This algorithm also has a complicated
rithm also starts with an overestimate of the shortest path to each history. The algorithm was first stated
by Shimbel in 1954, then Moore in
vertex. However, instead of relaxing the out-arcs from each vertex ’57, Woodbury and Dantzig in ’57,
once (in a careful order), this algorithm relaxes the out-arcs of all the and finally by Bellman in ’58. Since
it used Ford’s idea of relaxing edges,
vertices n − 1 times, in round-robin fashion. Formally, the algorithm the algorithm “naturally” came to be
is the following. (A visualization can be found 5 .) known as Bellman-Ford.
5
Algorithm 3: The Bellman-Ford Algorithm
Input: A digraph G = (V, E) with edge weights we ∈ R, and
source vertex s ∈ V
Output: The shortest-path distances from each vertex to s, or
report that a negative-weight cycle exists
3.1 dist(s) = 0 // the source has distance 0
3.2 for v ∈ V do
3.3 dist(v) ← ∞
3.4 for |V | iterations do
3.5 for edge e = (u, v) ∈ E do
3.6 dist(v) ← min{ dist(v), dist(u) + weight(e)}
3.7 If any distances changed in the last (nth ) iteration, output “G
has a negative weight cycle”.
The proof relies on the following lemma, which is easily proved by
induction on i.

Lemma 4.1. After i iterations of the algorithm, dist(v) equals the weight of
the shortest-path from s to v containing at most i edges. (This is defined to
be ∞ if there are no such paths.)

If there is no negative-weight cycle, then the shortest-paths are

well-defined and simple, so a shortest-path contains at most n − 1
edges. Now the algorithm is guaranteed to be correct after n − 1
52 the all-pairs shortest paths problem (apsp)

iterations by Lemma 4.4; moreover, none of the distances will change

in the nth iteration.
However, suppose the graph contains a negative cycle that is
reachable from the source. Then the labels dist(u) for vertices on
this cycle continue to decrease in each subsequent iteration, because
we may reach to any point on this cycle and by moving in that cy-
cle we can accumulate negative distance; therefore, the distance will
get smaller and smaller in each iteration. Specifically, they will de-
crease in the nth iteration, and this decrease signals the existence of
a negative-weight cycle reachable from s. (Note that if none of the
negative-weight cycles C are reachable from s, the algorithm out-
puts a correct solution despite C’s existence, and it will produce the
distance of ∞ for all the vertices in that cycle.)
The runtime is O(mn), since each iteration of Bellman-Ford looks
at each edge once, and there are n iterations. This is still the fastest
algorithm known for SSSP with general edge-weights, even though
faster algorithms are known for some special cases (e.g., when the
graph is planar, or has some special structure, or when the edge
weights are “well-behaved”). E.g., for the case where all edge weights
are integers in the range [−C, ∞), we can compute SSSP in time
√
O(m n log C ), using an idea we may discuss in Homework #1.

4.2 The All-Pairs Shortest Paths Problem (APSP)

The obvious way to do this is to run an algorithm for SSSP n times,

each time with a different vertex being the source. This gives an
O(mn + n2 log n) runtime for non-negative edge weights (using n
runs of Dijkstra), and O(mn2 ) for general edge weights (using n runs
of Bellman-Ford). Fortunately, there is a clever trick to bypass this ex-
tra loss, and still get a runtime of O(mn + n2 log n) with general edge
weights. This is known as Johnson’s algorithm, which we discuss
next.
d
4
1
s
4.2.1 Johnson’s Algorithm and Feasible Potentials 1 −1 c
1
a b
The idea behind this algorithm is to (a) re-weight the edges so that Figure 4.2: A graph with negative edges
they are nonnegative yet preserve shortest paths, and then (b) run n in which adding positive constant to all
instances of Dijkstra’s algorithm to get all the shortest-path distances. the edges will change the shortest paths

A simple-minded hope (based on our idea for MSTs) would be to add

a positive number to all the weights to make them positive. Although
this preserves MSTs, it doesn’t preserve shortest paths. For instance,
the example on the right has a single negative-weight edge. Adding
1 to all edge weights makes them all have non-negative weights, but
the shortest path from s to d is changed.
 shortest paths in graphs 53

Don Johnson gave a algorithm that does the edge re-weighting in Johnson (1977)
a slightly cleverer way, using the idea of feasible potentials. Loosely,
it runs the Bellman-Ford algorithm once, then uses the information
gathered to do the re-weighting. At first glance, the concept of a
feasible potential does not seem very useful. It is just an assignment of
weights ϕv to each vertex v of the graph, with some conditions:

Definition 4.2. For a weighted digraph G = (V, A), a function

ϕ : V → R is a feasible potential if for all edges e = uv ∈ A,

ϕ(u) + wuv − ϕ(v) ≥ 0.

Given a feasible potential, we can transform the edge-weights of

the graph from wuv to

buv := wuv + ϕ(u) − ϕ(v).

w

Observe the following facts:

1. The new weights w b are all positive. This comes from the definition
of the feasible potential.

2. Let Pab be a path from a to b. Let ℓ( Pab ) be the length of Pab when
we use the weights w, and ℓ̂( Pab ) be its length when we use the
b Then
weights w.

bℓ( Pab ) = ℓ( Pab ) + ϕa − ϕb .

The change in the path length is ϕa − ϕb , which is independent of

b preserve the shortest a-to-b paths,
the path. So the new weights w
only changing the length by ϕa − ϕb .

This means that if we find a feasible potential, we can compute the

new weights w,b and then run Dijkstra’s algorithm on the remaining
graph. But how can we find feasible potentials? Here’s the short
answer: Bellman-Ford. Indeed, suppose there some source vertex
s ∈ V such that every vertex in V is reachable from s. Then, set It is cleaner (and algorithmically sim-
ϕ(v) = dist(s, v). pler) to just add a new vertex s and
add zero-weight edges from it to all the
original vertices. This does not change
Lemma 4.3. Given a digraph G = (V, A) with vertex s such that all any of the original distances, or create
vertices are reachable from s, ϕ(v) = dist(s, v) is a feasible potential for G. any new cycles.

Proof. Since every vertex is reachable from s, dist(s, v) and therefore

ϕ(v) is well-defined. For an edge e = uv ∈ A, taking the shortest
path from s to u, and adding on the arc uv gives a path from s to v,
whose length is ϕ(u) + wuv . This length is at least ϕ(v), the length of
the shortest path from s to v, and the lemma follows.

In summary, the algorithm is the following:

54 the all-pairs shortest paths problem (apsp)

Algorithm 4: Johnson’s Algorithm

Input: A weighted digraph G = (V, A)
Output: A list of the all-pairs shortest paths for G
4.1 V ′ ← V ∪ {s} // add a new source vertex
4.2 A′ ← E ∪ {(s, v, 0) | v ∈ V }
4.3 dist ← BellmanFord((V ′ , A′ ))
// set feasible potentials
4.4 for e = (u, v) ∈ A do
4.5 weight(e)+ = dist(u) − dist(v)
4.6 L = [] // the result
4.7 for v ∈ V do
4.8 L+ = Dijkstra(G, v)
4.9 return L
We now bound the running time. Running Bellman-Ford once
takes O(mn) time, computing the “reduced” weights w b requires
O(m) time, and the n Dijkstra calls take O(n(m + n log n)), if we
use Fibonacci heaps. Therefore, the overall running time is O(mn +
n2 log n)—almost the same as one SSSP computation, except on very
sparse graphs with m = o (n log n).

4.2.2 More on Feasible Potentials

How did we decide to use the shortest-path distances from s as our
feasible potentials? Here’s some more observations, which give us a
better sense of these potentials, and which lead us to the solution.

1. If all edge-weights are non-negative, then ϕ(v) = 0 is a feasible

potential.

2. Adding a constant to a feasible potential gives another feasible

potential.

3. If there is a negative cycle in the graph, there can be no feasible

potential. Indeed, the sum of the new weights along the cycle is
the same as the sum of the original weights, due to the telescop-
ing sum. But since the new weights are non-negative, so the old
weight of the cycle must be, too.

4. If we set ϕ(s) = 0 for some vertex s, then ϕ(v) for any other vertex
v is an underestimate of the s-to-v distance. This is because for all
the paths from s to v we have

0 ≤ bℓ( Psv ) = ℓ( Psv ) − ϕv + ϕs = ℓ( Psv ) − ϕv ,

giving ℓ( Psv ) ≥ ϕv . Now if we try to set ϕ(s) to zero and try

to maximize summation of ϕ(v) for other vertices subject to the
 shortest paths in graphs 55

feasible potential constraints we will get an LP that is the dual of

the shortest path LP.
Maximize ∑ ϕx
x ∈V
Subject to ϕs = 0
wvu + ϕv − ϕu ≥ 0 ∀(v, u) ∈ E

4.2.3 The Floyd-Warshall Algorithm

The Floyd-Warshall algorithm is perhaps best introduced via its strik- The naming of this algorithm does
ingly simple pseudocode. It first puts down estimates dist(u, v) for not disappoint: it was discovered by
Bernard Roy, Stephen Warshall, Bob
the distances thus: Floyd, and others. The name tells only a
 small part of the story.

wij , i, j ∈ E

distij = ∞ i, j ∈
/ E, i ̸= j .



0, i=j
Then it runs the following series of updates.
Algorithm 5: The Floyd-Warshall Algorithm
Input: A weighted digraph D = (V, A)
Output: A list of the all-pairs shortest paths for D
5.1 set d( x, y) ← wxy if ( x, y) ∈ E, else d( x, y) ← ∞
5.2 for z ∈ V do
5.3 for x, y ∈ V do
5.4 d( x, y) ← min{d( x, y), d( x, z) + d(z, y)}
Importantly, we run over the “inner” index z in the outermost
loop. The proof of correctness is similar to, yet not that same as that
of Algorithm 3, and is again left as a simple exercise in induction.
Lemma 4.4. After we have considered vertices Vk = {z1 , . . . , zk } in the
outer loop, dist(u, v) equals the weight of the shortest x-y path that uses
only the vertices from Vk as internal vertices. (This is ∞ if there are no such
paths.)
The running time of Floyd-Warshall is clearly O(n3 )—no bet-
ter than Johnson’s algorithm. But it does have a few advantages: it
is simple, and it is quick to implement with minimal errors. (The
most common error is nesting the for-loops in reverse.) Another ad- Actually, this paper of Hide, Kumabe,
vantage is that Floyd-Warshall is also parellelizable, and very cache and Maehara (2019) shows that even if
you get the loops wrong, but you run
efficient. the algorithm a few more times, it all
works out in the end. But that proof
requires a bit more work.
4.3 Min-Sum Products and APSPs

A conceptually different way to get shortest-path algorithms is via

matrix products. These may not seem relevant, a priori, but they lead
to deep insights about the APSP problem.
56 min-sum products and apsps

Recall the classic definition of matrix multiplication, for two real-

valued matrices A, B ∈ Rn×n
n
( AB)ij = ∑ ( Aik ∗ Bkj ).
k =0

Hence, each entry of the product AB is a sum of products, both be-

ing the familar operations over the field (R, +, ∗). But now, what if
we change the constituent operations, to replace the sum with the
min operation, and the product with a sum? We get the Min-Sum
Product(MSP): given matrices A, B ∈ Rn×n , the new product is

( A ⊚ B)ij = min{ Aik + Bkj }.

k

This is the usual matrix multiplication, but over the semiring (R, min, +).
A semiring has a notion of addition
It turns out that computing Min-Sum Products is precisely the and one of multiplication. However,
neither the addition nor the multipli-
operation needed for the APSP problem. Indeed, initialize a matrix D cation operations are required to have
exactly as in the Floyd-Warshall algorithm: inverses.


wij , i, j ∈ E

Dij = ∞ i, j ∈
/ E, i ̸= j .



0, i=j

Now ( D ⊚ D )ij represents the cheapest i-j path using at most 2 hops!
(It’s as though we made the outer-most loop of Floyd-Warshall into
the inner-most loop.) Similarly, we can compute

D ⊚k : = D ⊚ D ⊚ D · · · ⊚ D ,
| {z }
k −1 MSPs

whose entries give the shortest i-j paths using at most k hops (or at
most k − 1 intermediate nodes). Since the shortest paths would have
at most n − 1 hops, we can compute D⊚n−1 .
How much time would this take? The very definition of MSP
shows how to implement it in O(n3 ) time. But performing it n − 1
times would be O(n) worse than all other approaches! But here’s a
classical trick, which probably goes back to the Babylonians: for any
integer k,
D⊚2k = D⊚k ⊚ D⊚k .
(Here we use that the underlying operations are associative.) Now it
is a simple exercise to compute D⊚n−1 using at most 2 log2 n MSPs.
This a runtime of O( MSP(n) log n), where MSP(n) is the time it
takes to compute the min-sum product of two n × n matrices. In fact, with some more work, we can
Now using the naive implementation of MSP gives a total runtime implement APSP in time O( MSP(n));
you will probably see this in a home-
of O(n3 log n), which is almost in the right ballpark! The natural work.
question is: can we implement MSPs faster?
 shortest paths in graphs 57

4.3.1 Faster Algorithms for Matrix Multiplication

Can we get algorithms for MSP that run in time O(n3−ε ) for some
constant ε > 0? To answer this question, we can first consider
the more common case, that of matrix multiplication over the re-
als (or over some field)? Here, the answer is yes, and this has been
known for now over 50 years. In 1969, Volker Strassen showed
that one could multiply n × n matrices over any field F, using
O(nlog2 7 ) = O(n2.81 additions and multiplications. (One can allow
divisions as well, but Strassen showed that divisions do not help
asymptotically.) Mike Paterson has a beautiful but still
If we define the exponent of matrix multiplication ω > 0 to be mysterious geometric interpretation
of the sub-problems Strassen comes
smallest real such that two n × n matrices over any field F can be up with, and how they relate to Karat-
multiplied in time O(nω ), then Strassen’s result can be phrased as suba’s algorithm to multiply numbers.
saying:

ω ≤ log2 7.

This value, and Strassen’s idea, has been refined over the years, to The big improvements in this line
its current value of 2.3728 due to François Le Gall (2014). (See this of work were due to Arnold Schön-
hage (1981), Don Coppersmith and
survey by Virginia for a discussion of algorithmic progress until Shmuel Winograd (1990), with recent
2013.) There has been a flurry of work on lower bounds as well, e.g., refinements by Andrew Stothers, CMU
alumna Virginia Vassilevska Williams,
by Josh Alman and Virginia Vassilevska Williams showing limitations and François Le Gall (2014).
for all known approaches.
But how about MSP(n)? Sadly, progress on this has been less im-
pressive. Despite much effort, we don’t even know if it can be done
in O(n3−ϵ ) time. In fact, most of the recent work has been on giving
evidence that getting sub-cubic algorithms for MSP and APSP may
not be possible. There is an interesting theory of hardness within P
developed around this problem, and related ones. For instance, it is
now known that several problems are equivalent to APSP, and truly
sub-cubic algorithms for one will lead to sub-cubic algorithms for all
of them.
Yet there is some interesting progress on the positive side, albeit
qualitatively small. As far back as 1976, Fredman had shown an
log log n
algorithm to compute MSP in O(n3 log n ) time. He used the fact
that the decision-tree complexity of APSP is sub-cubic (a result we
will discuss in §4.5) in order to speed up computations over nearly-
xlogarithmic-sized sub-instances; this gives the improvement above.
More recently, another CMU alumnus  Ryan
 Williams improved on
3
this idea quite substantially to O √n 6, using very interesting 6
log n
2
ideas from circuit complexity. We will discuss this result in a later
section, if we get a chance.
58 undirected apsp using fast matrix multiplication

4.4 Undirected APSP Using Fast Matrix Multiplication

One case were we have seen improvements in APSP algorithms is

that of graphs with small integer edge-weights. Our focus will be on
undirected, unweighted graphs: we present a beautiful algorithm of
Raimund Seidel that runs in time O(nω log n), assuming that ω > 2.
To begin, the adjacency matrix A for graph G is the symmetric
matrix 
1 ij ∈ E
Aij = .
0 ij ∈/E

Now consider the graph G2 , the square of G, which has the same
vertex set as G but where an edge in G2 corresponds to being at most
two hops away in G—that is, uv ∈ E( G2 ) ⇐⇒ dG (u, v) ≤ 2. If
we consider A as a matrix over the finite field (F2 , +, ∗), then the In this field over {0, 1}, observe that the
multiplication operation behaves just
adjacency matrix of G2 has a nice formulation:
like the Boolean AND function, and the
addition like the Boolean XOR.
A G2 = A G ∗ A G + A G .

This shows how to get the adjacency matrix of G2 given one for G,
having spent one Boolean matrix multiplication and one matrix
addition. Suppose we recursively compute APSP on G2 : how can
we translate this result back to G? The next lemma shows that the
shortest-path distances in G2 are nicely related to those in G.

Lemma 4.5. If d xy and Dxy are the shortest-path distances between x, y in

G and G2 respectively, then
 
d xy
Dxy = .
2

Proof. Any u-v path in G can be written as

u, a1 , b1 , a2 , b2 , . . . , ak , bk , v

if the path has odd length; an even-length path can be written as

u, a1 , b1 , a2 , b2 , . . . , ak , bk , ak+1 , v.

In either case, G2 has edges ub1 , b1 b2 , . . . , bk−1 bk , bk v, and thus a u-v

d xy d xy
path of length ⌈ 2 ⌉ in G2 . Therefore Dxy ≤ ⌈ 2 ⌉.
d
To show equality, suppose there is a u-v path of length ℓ < ⌈ 2xy ⌉ in
G2 . Each of these ℓ edges corresponds to either an edge or a 2-edge
path in G, so we can find a u-v path of length at most 2ℓ < d xy in G, a
contradiction.

Lemma 4.5 implies that

duv ∈ {2Duv , 2Duv − 1}.

 shortest paths in graphs 59

But which one? The following lemmas give us simple rule to decide.
Let NG (v) denote the set of neighbors of v in G.

Lemma 4.6. If duv = 2Duv , then for all w ∈ NG (v) we have Duw ≥ Duv .

Proof. Assume not, and let w ∈ NG (v) be such that Duw < Duv .
Since both of them are integers, we have 2Duw < 2Duv − 1. Then the
shortest u-w path in G along with the edge wv forms a u-v-path in G
of length at most 2Duw + 1 < 2Duv = duv , which is in contradiction
with the assumption that duv is the shortest path in G.

Lemma 4.7. If duv = 2Duv − 1, then Duw ≤ Duv for all w ∈ NG (v);
moreover, there exists z ∈ NG (v) such that Duz < Duv .

Proof. For any w ∈ NG (v), considering the shortest u-v path in G

along with the edge vw implies that duw ≤ duv + 1 = (2Duv − 1) + 1,
so Lemma 4.5 gives that Duw = ⌈duw /2⌉ = Duv . For the second claim,
consider a vertex z ∈ NG (v) on a shortest path from u to v. Then
duz = duv − 1, and Lemma 4.5 gives Duz < Duv . Where did we use that G was undi-
rected? In Lemma 4.6 we used that
w ∈ NG (v) =⇒ wv ∈ E.
And in Lemma 4.7 we used that
These lemmas can be summarized thus:
w ∈ NG (v) =⇒ vw ∈ E.

∑w∈ N (v) Duw

duv = 2Duv ⇐⇒ ≥ Duv , (4.1)
deg(v)

where deg( j) = | NG ( j)| is the degree of j. Given D, the criterion on

the right can be checked for each uv in time deg(v) by just comput-
ing the average, but that could be too slow—how can we do better?
Define the normalized adjacency matrix of G to be A b with

bwv = 1wv∈E · 1
A .
deg(v)

Now if D is the distance matrix of G2 , then

∑w∈ NG (v) Duw

b)uv =
(D A ∑ bwv =
Duw A
deg(v)
,
w ∈V

which is conveniently the expression in (4.1). Let 1( D Ab< D) be a matrix

b)uv < Duv , and zero otherwise. Then
with the uv-entry being 1 if ( D A
the distance matrix for G is

2D − 1( D Ab< D) .

This completes the algorithm, which we now summarize:

60 undirected apsp using fast matrix multiplication

Algorithm 6: Seidel’s Algorithm

Input: Unweighted undirected graph G = (V, E) with
adjacency matrix A
Output: The distance matrix for G
6.1 if A = J then
6.2 return A // If A is all-ones matrix, done!
6.3 else
6.4 A′ ← A ∗ A + A // Boolean operations
6.5 D ← Seidel(A′ )
6.6 return 2D − 1( D Ab< D)
Each call to the procedure above performs one Boolean matrix
multiplication in step (6.4), one matrix multiplication with ratio-
nal entries in step (6.6), plus O(n2 ) extra work. The diameter of the
graph halves in each recursive call (by Lemma 4.5), and the algorithm
hits the base case when the diameter is 1. Hence, the overall running
time is O(nω log n).
Ideas similar to these can be used to find shortest paths graphs
with small integer weights on the edges: if the weights are integers
in the interval [0, W ], Avi Shoshan and Uri Zwick 7 give an Õ(Wnω )- 7

time algorithm. In fact, Zwick 8 also extends theideas to directed


8

1 2+ 4−1ω
graphs, and gives an algorithm with runtime Õ W 4− ω n .

4.4.1 Finding the Shortest Paths

How do we find the shortest paths themselves, and not just their
lengths? For the previous algorithms, modifying the algorithms to
output the paths is fairly simple. But for Seidel’s algorithm, things
get tricky. Indeed, since the runtime of Seidel’s algorithm is strictly
sub-cubic, how can we write down the shortest paths in nω time,
since the total length of all these paths may be Ω(n3 )? We don’t:
we just write down the successor pointers. Indeed, for each pair
u, v, define Sv (u) to be the second node on a shortest u-v path (the
first node being u, and the last being v). Then to get the entire u-v
shortest path, we just follow these pointers:

u, Sv (u), Sv (Sv (u)), . . . , v.

So there is a representation of all shortest paths that uses at most

O(n2 log n) bits.
The main idea for computing the successor matrix for Seidel’s
algorithm is to solve the Boolean Product Matrix Witness problem:
given n × n Boolean matrices A, B, compute an n × n matrix W such
that Wuv = k if Aik = Bkj = 1, and Wij = 0 if no such k exists. We will
hopefully see (and solve) this problem in a homework.
 shortest paths in graphs 61

4.5 Optional: Fredman’s Decision-Tree Complexity Bound

Given the algorithmic advances, one may wonder about lower bounds
for the APSP problem. There is the obvious Ω(n2 ) lower bound
from the time required to write down the answer. Maybe even the
decision-tree complexity of the problem is Ω(n3 )? Then no algorithm
can do any faster, and we’d have shown the Floyd-Warshall and the
Matrix-Multiplication methods are optimal.
However, thanks to a result of Fredman 9 , we know this is not the 9

case. If we just care about the decision-tree complexity, we can get

much better. Specifically, Fredman shows

Theorem 4.8. The Min-Sum Product of two n × n matrices A, B can be

deduced in O(n2.5 ) additions and comparisons.

Proof. The proof idea is to split A and B into rectangular sub-matrices,

and compute the MSP on the sub-matrices. Since these sub-matrices
are rectangular, we can substantially reduce the number of compar-
isons needed for each one. Once we have these sub-MSPs, we can
simply compute an element-wise minimum for find the final MSP.
Fix a parameter W which we determine later. Then divide A into
n/W n × W matrices A1 , . . . , An/W , and divide B into n/W W ×
n submatrices B1 , . . . , Bn/W . We will compute each Ai ⊚ Bi . Now
T)
consider ( A ⊚ B)ij = mink∈[W ] ( Aik + Bkj ) = mink∈[W ] ( Aik + Bjk
∗
and let k be the minimizer of this expression. Then we have the
following:
T T
Aik∗ − Bjk ∗ ≤ Aik − B jk ∀ k (4.2)
T T
Aik∗ − Aik ≤ −( Bjk ∗ − Bjk ) ∀k (4.3)

Now for every pair of columns, p, q from Ai , BiT , and sort the follow-
ing 2n numbers

A1p − Aiq , A2p − A2q , . . . , Anp − Anq , −( B1p − B1q ), . . . , −( Bnp − Bnq )

We claim that by sorting W 2 lists of numbers we can compute Ai ⊚

Bi . To see this, consider a particular entry ( A ⊚ B)ij and find a k∗
such that for every k ∈ [W ], Aik∗ − Aik precedes every −( Bjk T − BT )
∗ jk
∗
in their sorted list. By (4.3), such a k is a minimizer. Then we can set
( A ⊚ B)ij = Aik∗ + Bk∗ j .
This computes the MSP for Ai , Bi , but it is possible that another
A j ⊚ Bj produces the actual minimum. So, we must take the element-
wise minimum across all the ( Ai ⊚ Bi ). This produces the MSP of
A, B.
Now for the number of comparisons. We have n/W smaller prod-
ucts to compute. Each sub-product has W 2 arrays to sort, each of
62 optional: fredman’s decision-tree complexity bound

which can be sorted in 2n log n comparisons. Finding the minimizer

requires W 2 n comparisons.So, computing the sub-products requires
n/W ∗ 2W 2 n log n = 2n2 W log n comparisons. Then, reconstructing
the final MSP requires n2 element-wise minimums between n/W − 1
elements, which requires n3 /W comparisons. Summing these bounds
gives us n3 /W + 2n2 W log n comparisons. Optimizing over W gives
p
us O(n2 n log n) comparisons.

This result does not give us a fast algorithm, since it just counts
the number of comparisons, and not the actual time to figure out
which comparisons to make. Regardless, many of the algorithms
that achieve n3 / poly log n time for APSP use Fredman’s result on
tiny instances (say of size O(poly log n), so that we can find the best
decision-tree using brute-force) to achieve their results.
5
Low-Stretch Spanning Trees

Given that shortest paths from a single source node s can be repre-
sented by a single shortest-path tree, can we get an analog for all-
pairs shortest paths? Given a graph can we find a tree T that gives us
the shortest-path distances between every pair of nodes? Does such
a tree even exist? Sadly, the answer is negative—and it remains neg-
ative even if we allow this tree to stretch distances by a small factor,
as we will soon see. However, we show that allowing randomiza-
tion will allow us to circumvent the problems, and get low-stretch
spanning trees in general graphs.
In this chapter, we consider undirected graphs G = (V, E), where
each edge e has a non-negative weight/length we . For all u, v in V,
let dG (u, v) be the distance between u, v, i.e., the length of a shortest
path in G from u to v. Observe that the set V along with the distance
function dG forms a metric space. A metric space is a set V with a dis-
tance function d satisfying symme-
try (i.e., d( x, y) = d(y, x ) for all
x, y ∈ V) and the triangle inequality
5.1 Towards a Definition (d( x, y) ≤ d( x, z) + d(z, y) for all
x, y, z ∈ V). Typically, the definition also
The study of low-stretch spanning trees is guided by two high level asks for x = y ⇐⇒ d( x, y) = 0, but we
will merely assume d( x, x ) = 0 for all x.
hopes:

1. Graphs have spanning trees that preserve their distances. That is,
given G there exists a subtree T = (V, ET ) with ET ⊆ E such that We assume that the weights of edges in
ET are the same as those in G.
dG (u, v) ≈ d T (u, v) for all u, v ∈ V.

2. Many NP-hard problems are much easier to solve on trees.

Supposing these are true, we have a natural recipe for designing

algorithms to solve problems that depend only on distances in G:
(1) find a spanning tree T preserving distances in G, (2) solve the
problem on T, and then (3) return the solution (or some close cousin)
with the hope that it is a good solution for the original graph.
64 towards a definition

5.1.1 An All-Pairs Shortest Path Tree?

The boldest hope would be to find an all-pairs shortest path tree T, i.e.,
one that ensures d T (u, v) = dG (u, v) for all u, v in V. However, such a
tree may not exist: consider Kn , the clique of n nodes, with unit edge
lengths. The distance dG satisfies dG ( x, y) = 1 for all x ̸= y, and zero
otherwise. But any subtree T contains only n − 1 edges, so most pairs
of vertices x, y ∈ V lack an edge between them in T. Any such pair
has a shortest-path distance d T ( x, y) ≥ 2, whereas dG ( x, y) = 1.

5.1.2 A First Relaxation: Low-Stretch Spanning Trees

To remedy the snag above, let us not require distances in T be equal
to those in G, but instead be within a small multiplicative factor
α ≥ 1 of those in G.

Definition 5.1. Let T be a spanning tree of G, and let α ≥ 1. We call

T a (deterministic) α-stretch spanning tree of G if Exercise: show that if T is any subtree
of G with the same edge weights, then
dG ( x, y) ≤ d T ( x, y).
dG (u, v) ≤ d T (u, v) ≤ α dG (u, v).

holds for all u, v ∈ V.

Supposing we had such a low-stretch spanning tree, we could

try our meta-algorithm out on the traveling salesperson problem
(TSP): given a graph, find a closed tour that visits all the vertices, and
has the smallest total length. This problem is NP-hard in general,
but let us see how an α-stretch spanning tree of G gives us an an
α-approximate TSP solution for G. The algorithm is simple:

Algorithm 7: TSP via Low-Stretch Spanning Trees

7.1 Find an α-stretch spanning tree T of G.
7.2 Solve TSP on T to get an ordering π T on the vertices.
7.3 return the ordering π T .

Solving the TSP problem on a tree T is trivial: just take an Euler

tour of T, and let π T be the order in which the vertices are visited.
Let us bound the quality of this solution.
Claim 5.2. π T is an α-approximate solution to the TSP problem on G.

Proof. Suppose that the permutation πG minimizes the length of the

TSP tour for G. The length of the resulting tour is

OPTG := ∑ dG (πG (i ), πG (i + 1)).

i ∈[n]
 low-stretch spanning trees 65

Since distances in the tree T are stretched by only a factor of α,

∑ d T (πG (i ), πG (i + 1)) ≤ α · ∑ dG (πG (i ), πG (i + 1)). (5.1)

i ∈[n] i ∈[n]

Now, since π T is the optimal ordering for the tree T, and πG is some
other ordering,

∑ d T (π T (i ), π T (i + 1)) ≤ ∑ d T (πG (i ), πG (i + 1)). (5.2)

i ∈[n] i ∈[n]
| {z }
OPTT

Finally, since distances were only stretched in going from G to T,

∑ dG (π T (i ), π T (i + 1)) ≤ ∑ d T (π T (i ), π T (i + 1)). (5.3)

i ∈[n] i ∈[n]

Putting it all together, the length of the tour given by π T is

∑ dG (π T (i ), π T (i + 1)) ≤ α · ∑ dG (πG (i ), πG (i + 1)),

i ∈[n] i ∈[n]

which is α · OPTG .

Hence, if we had low-stretch spanning trees T with α ≤ 1.49, we

would get the best approximation algorithm for the TSP problem.
(Assuming we can find T, but we defer this for now.) However, you
may have already noticed that the Kn example above shows that
α < 2 is impossible. But can we achieve α = 2? Indeed, is there
any “small” value for α such that for any graph G we can find an
α-stretch spanning tree of G?
Sadly, things are terrible: take the cycle Cn , again with unit edge
weights. Now any subtree T is missing one edge from Cn , say uv. Exercise: show how to find, for any
The endpoints of this edge are at distance 1 in Cn , but d T (u, v) = n − graph G, a spanning tree T with stretch
α ≤ n − 1.
1, since we have to go all the way around the cycle. Hence, getting
α < (n − 1) is impossible in general.

5.1.3 A Second Relaxation: Randomization to the Rescue

Since we cannot get trees with small stretch deterministically, let
us try to get trees with small stretch “on average”. We amend our
definition as follows:

Definition 5.3. A (randomized) low-stretch spanning tree of stretch α

for a graph G = (V, E) is a probability distribution D over spanning
trees of G such that for all u, v ∈ V, we have Henceforth, all references to low-stretch
trees will only refer to this randomized
version, unless otherwise specified.
dG (u, v) ≤ d T (u, v) for all T in the support of D , and
ET ∼D [d T (u, v)] ≤ α dG (u, v) (5.4)
66 low-stretch spanning tree construction

Observe that the first property must hold with probability 1 (i.e.,
it holds for all trees in the support of the distribution), whereas the
second property holds only on average. Is this definition any good
for our TSP example above? If we change the algorithm to sample a
tree T from the distribution and then return the optimal tour for T,
we get a randomized algorithm that is good in expectation. Indeed,
(5.1) becomes

∑ E[d T (πG (i ), πG (i + 1))] ≤ α · ∑ dG (πG (i ), πG (i + 1)), (5.5)

i ∈[n] i ∈[n]

because the stretch guarantees hold in expectation (and linearity of

expectation). The rest of the inequalities hold unchanged, includ-
ing (5.3)—which requires the probability 1 guarantee of Definition 5.6
(Do you see why?). Hence, we get

∑ E[dG (π T (i ), π T (i + 1))] ≤ α · ∑ dG (πG (i ), πG (i + 1)) . (5.6)

i ∈[n] i ∈[n]
| {z } | {z }
expected algorithm’s tour length OPTG

Even a randomized better-than-1.49 approximation for TSP would

still be amazing! And the algorithmic template here works not just
for TSP: any NP-hard problem whose objective is a linear function
of distances (e.g., many other vehicle routing problems, or the k-
median clustering problem) can be solved in this way. Indeed, the
first approximation algorithms for many such problems came via
low-stretch spanning trees.
Moreover, (randomized) low-stretch spanning trees arise in many
different contexts, some of which are not obvious at all. E.g., they can
be used to more efficiently solve “Laplacian” linear systems of the
form A⃗x = ⃗b, where A is the Laplacian matrix of some graph G. To
do this, we let P be the Laplacian matrix of a low-stretch spanning
tree of G, and then we solve the system P−1 A⃗x = P−1⃗x instead. This
is called preconditioning with P. It turns out that this preconditioning
allows certain algorithms for solving linear systems to converge faster
to a solution. Time permitting, we will discuss this application later
in the course.

5.2 Low-Stretch Spanning Tree Construction

But first, given a graph G, how can we find a randomized low-stretch

spanning tree for G with a small value of α (and efficiently)? As a
sanity check, let us check what we can do on the two examples from
before: A natural first attempt (at least for
unweighted graphs) would be to try
1. For the complete graph Kn , choose a star graph centered at a uni- a uniformly random spanning tree.
This does not work very well (which
formly random vertex of G. For any pair of vertices u, v, they are
I think is not that surprising), even
for the complete graph Kn (which I
think is somewhat surprising). A result
of Moon and Moser shows that for
any pair of vertices u, vinV (Kn ), if we
choose T to be one of the nn−2 spanning
trees uniformly at random, √ the expected
distance is d T (u, v) = Θ( n).
 low-stretch spanning trees 67

at distance 1 in this star if either u or v is the center, else they are

at distance 2. Hence the expected distance is n2 · 1 + n− 2 2
n · 2 = 2 − n.

2. For the cycle Cn , choose a tree by dropping a single edge uni-

formly at random. For any edge uv in the cycle, there is only a 1 in
n chance of deleting the edge from u to v. But when it is deleted, u
and v are at distance n − 1 in the tree. So

n−1 1 2
E[d T (u, v)] = · 1 + · ( n − 1) = 2 − .
n n n
And what about an arbitrary pair of nodes u, v in Cn ? We can use Exercise: Given a graph G, suppose
the exercise on the right to show that the stretch on other pairs is the stretch on all edges is at most α.
Show that the stretch on all pairs of
no worse! nodes is at most α. (Hint: linearity of
expectation.)
While we will not manage to get α < 1.49 for general graphs (or
even for the above examples, for which the bounds of 2 − n2 are the
best possible), we show that α ≈ O(log n) can indeed be achieved.
The following theorem is the current best result, due to Ittai Abra-
ham and Ofer Neiman:

Theorem 5.4. For any graph G, there exists a distribution D over span-
ning trees of G with stretch α = O(log n log log n). Moreover, the
construction is efficient: we can sample trees from this distribution D in
O(m log n log log n) time.

Moreover, the stretch bound of this theorem is almost optimal, up

to the O(log log n) factor, as the following lower bound due to Alon,
Peleg, Karp, and West shows.

Theorem 5.5. For infinitely many n, there exist graphs G on n vertices

such that any α-stretch spanning tree distribution D on G must have α =
Ω(log n). In fact, G can be taken to be the n-vertex square grid, the n-
vertex hypercube, or any n-vertex constant-degree expander.

5.3 Bartal’s Construction

The algorithm underlying Theorem 5.4 is quite involved, but we

can give the entire construction of low-stretch trees for finite metric
spaces.

Definition 5.6. A (randomized) low-stretch tree with stretch α for a

metric space M = (V, d) is a probability distribution D over trees
over the vertex set V such that for all u, v ∈ V, we have

d(u, v) ≤ d T (u, v) for all T in the support of D , and

ET ∼D [d T (u, v)] ≤ α d(u, v). (5.7)
68 bartal’s construction

The difference of this definition from Definition 5.6 is slight: we

now have a metric space instead of a graph, and we are allowed to
output any tree on the vertex set V (since the concept of subtrees
doesn’t make sense now). Note that given a graph G, we can com-
pute its shortest-path metric (V, dG ) and then find a distribution over
(non-spanning) trees that approximate the distance in G. So if we
don’t really need the spanning aspect in our low-stretch trees—e.g.,
as in the TSP example—we can use results for this definition.
We need one more piece of notation: for a metric space M =
(V, d), define its aspect ratio ∆ to be
maxu̸=v∈V d(u, v)
∆ M := .
minu̸=v∈V d(u, v)

We will show the following theorem, due to Yair Bartal:

Theorem 5.7. For any metric space M = (V, d), there exists an efficiently
sampleable α B -stretch spanning tree distribution D B , where

α B = O(log n log ∆ M ).

The proof works in two parts: we first show a good low-diameter

decomposition. This will be a procedure that takes a metric space The diameter of a set S is
and a diameter bound D, and randomly partitions the metric space maxu,v∈S d(u, v), i.e., the maximum
distance between any two points in it.
into clusters of diameter ≤ D, in such a way that close-by points are
unlikely to be separated. Then we show how such a low-diameter
decomposition can be used recursively to constuct a low-stretch tree.

5.3.1 Low-Diameter Decompositions

The notion of a low-diameter decomposition has become ubiquitous
in algorithm design, popping up in approximation and online algo-
rithms, and also in distributed and parallel algorithms. It’s something
worth understanding well.

Definition 5.8 (Low-Diameter Decomposition). A low-diameter de-

composition scheme (or LDD scheme) with parameter β for a metric
M = (V, d) is a randomized algorithm that, given a bound D > 0,
partitions the point set V into “clusters” C1 , . . . , Ct such that
(i) for all i ∈ {1, . . . , t}, the diameter of Ci is at most D, and
(ii) for all x, y ∈ V such that x ̸= y, we have

d( x, y)
Pr[ x, y in different clusters] ≤ β · .
D
Let’s see a few examples, to get a better sense for the definition:

1. Consider a set of points on the real line. One way to partition the
line into pieces of diameter D is simple: imagine making notches
 low-stretch spanning trees 69

on the line at distance D from each other, and then randomly

shifting them. Formally, pick a random value R ∈ [0, D ] uniformly
at random, and partition the line into intervals of the form [ Di +
R, D (i + 1) + R), for i ∈ Z. A little thought shows that points x, y
d( x,y)
are separated with probability exactly D .

2. The infinite 2-dimensional square grid with unit edge-lengths.

One way to divide this up is to draw horizontal and vertical lines
which are D/2 apart, and randomly shift as above. A pair x, y is
d( x,y)
separated with probability exactly D/2 in this case. Indeed, this
approach works for k-dimensional hypergrids (and k-dimensional
ℓ1 -space) with probability k · d(D
x,y)
— in this case the β parameter
is at most the dimension of the space.

3. What about lower bounds? One can show that for the k-dimensional
hypergrid, we cannot get β = o (k). Or for a constant-degree n-
vertex expander, we cannot get β = o (log n). Details to come soon.

Since the aspect ratio of the metric space is invariant to scaling all
the edge lengths by the same factor, it will be convenient to assume
that the smallest non-zero distance in d is 1, so the largest distance is
∆. The basic algorithm is then quite simple:

Algorithm 8: LDD( M = (V, d), D )

4 log n
8.1 p ← min(1, D ).
8.2 while there exist unmarked point do
8.3 v ← any unmarked point.
8.4 sample Rv ∼ Geometric( p).
8.5 cluster Cv ← {unmarked u | d(v, u) < Rv }.
8.6 mark points in Cv .
8.7 return the resulting set of clusters.

Lemma 5.9. The algorithm above ensures that

1. the diameter of every cluster is at most D with probability at least 1 −

1/n, and

2. any pair x, y ∈ V is separated with probability at most 2p d( x, y).

Proof. To show the diameter bound, it suffices to show that Rv ≤

D/2 for each cluster Cv , because then the triangle inequality shows
that for any x, y ∈ Cv ,

d( x, y) ≤ d( x, v) + d(v, y) < D/2 + D/2 = D.

Now the probability that Rv > D/2 for one particular cluster is We use that 1 − z ≤ ez for all z ∈ R.
70 bartal’s construction

1
Pr[ Rv > D/2] = (1 − p) D/2 ≤ e− pD/2 ≤ e−2 log n = .
n2
By a union bound, there exists a cluster with diameter > D with
probability
n 1
1 − Pr[∃v ∈ V, Rv > D/2] ≥ 1 − = 1− .
n2 n
To bound the probability of some pair u, v being separated, we
use the fact that sampling from the geometric distribution with pa-
Cv
rameter p means repeatedly flipping a coin with bias p and counting
the number of flips until we see the first heads. Recall this process
is memoryless, meaning that even if we have already performed k d(v, y)
v y
flips without having seen a heads, the time until the first heads is still Rv ≤ D
2
d(v, x ) d( x, y)
geometrically distributed. x
Hence, the steps of drawing Rv and then forming the cluster can
be viewed as starting from v, where the cluster is a unit-radius ball
around v. Each time we flip a coin of bias p: it is comes up heads we Figure 5.1: A cluster forming around v
in the LDD process, separating x and
set the radius Rv to the current value, form the cluster Cv (and mark y. To reduce clutter, only some of the
its vertices) and then pick a new unmarked point v; on seeing tails, distances are shown.
we just increment the radius of v’s cluster by one and flip again. The
process ends when all vertices lie in some cluster.
For x, y, consider the first time when one of these vertices lies
inside the current ball centered at some point, say, v. (This must hap-
pen at some point, since all vertices are eventually marked.) With-
out loss of generality, let the point inside the current ball be x. At
this point, we have performed d(v, x ) flips without having seen a
heads. Now we will separate x, y if we see a heads within the next
⌈d(v, y) − d(v, x )⌉ ≤ ⌈d( x, y)⌉ flips—beyond that, both x, y will have
been contained in v’s cluster and hence cannot be separated. But
the probability of getting a heads among these flips is at most (by a
union bound)
d( x, y)
⌈d( x, y)⌉ p ≤ 2d( x, y) p ≤ 8 log n .
D
(Here we used that the minimum distance is 1, so rounding up dis-
tances at most doubles things.) This proves the claimed probability of
separation.

Recall that we wanted the diameter bound with probability 1,

whereas Lemma 5.9 only ensures it with high probability. Here’s a
quick fix to this problem: repeat the above process until the returned
partition has clusters of diameter at most D. The probability of any
pair u, v being separated by this last run of Algorithm 8 is at most
the probability of u, v being separated by any of the runs, which is at
most p d(u, v) times the expected number of runs,
d(u, v)
p d(u, v) · (1/(1 − 1/n)) ≤ 2p d(u, v) = O(log n) .
D
 low-stretch spanning trees 71

Lemma 5.10. The low-diameter decomposition scheme above achieves

parameter β = O(log n) for any metric M on n points.

5.3.2 Low-Stretch Trees Using LDDs

Now we can use the low-diameter decomposition scheme to get a

low-stretch tree (LST). Here’s the high-level idea: given a metric with
diameter ∆, use an LDD to decompose it into clusters with diameter
D ≤ ∆/2. Build a tree recursively for each of these clusters, and then
combine these trees into one tree for the entire metric.
Recall we assumed that the metric had minimum distance 1 and
maximum distance ∆. Formally, we invoke the procedure LST below
with the parameters LST(metric M, ⌈log2 ∆⌉).
Algorithm 9: LST(metric M = (V,d), D = 2δ )
Input: Invariant: diameter( M) ≤ 2δ
9.1 if |V | = 1 then
9.2 return tree containing the single point in V.
9.3 C1 , . . . , Ct ← LDD( M, D = 2δ−1 ).
9.4 for j in {1, . . . , t} do
9.5 M j ← metric M restricted to the points in Cj .
9.6 Tj ← LST( M j , δ − 1).
9.7 Add edges of length 2δ from root r1 for tree T1 to the roots of
T2 , . . . , Tt .
9.8 return resulting tree rooted at r1 .
We are ready to prove Theorem 5.7; we will show that the tree has
expected stretch O( β log ∆), and that it does not shrink any distances.
In fact, we show a slightly stronger guarantee.

Lemma 5.11. If the random tree T returned by some call LDD( M′ , δ) has
root r, then (a) every vertex x in T has distance d( x, r ) ≤ 2δ+1 , and (b) the
expected distance between any x, y ∈ T has E[d T ( x, y)] ≤ 8δβ d( x, y).

Proof. The proof is by induction on δ. For the base case, the tree has
a single vertex, so the claims are trivial. Else, let x lie in cluster Cj , so
inductively the distance to the root of the tree Ti is d( x, ri ) ≤ 2(δ−1)+1 .
Now the distance to the new root r is at most 2δ more, which gives
2δ + 2δ = 2δ+1 as claimed.
Moreover, any pair x, y is separated by the LDD with probability
d( x,y)
β 2δ−1 , in which case their distance is at most

d( x, r ) + d(r, y) ≤ 2δ+1 + 2δ+1 = 4 · 2δ .

Else they lie in the same cluster, and inductively have expected dis-
72 metric embeddings: a.k.a. simplifying metrics

tance at most 8(δ − 1) β d( x, y). Hence the expected distance is

E[d( x, y)] ≤ Pr[ x, y separated] · 4 · 2δ +

Pr[ x, y not separated] · 8(δ − 1) β d( x, y)
d( x, y)
≤β · 4 2δ + 8(δ − 1) β d( x, y)
2δ −1
= 8 δ β d( x, y).

This proves Theorem 5.7 because β = O(log n), and the iniitial
call on the entire metric defines δ = O(log ∆). In fact, if we have a
better LDD (with smaller β), we immediately get a better low-stretch
tree. For example, shortest-path metrics of planar graphs admit an
LDD with parameter β = O(1); this shows that planar metrics admit
(randomized) low-stretch trees with stretch O(log ∆).
It turns out this factor of O(log n log ∆) can be improved to O(log n)—
this was done by Fakcharoenphol, Rao, and Talwar. Moreover, the
bound of O(log n) is tight: the lower bounds of Theorem 5.5 continue
to hold even for low-stretch non-spanning trees.

5.4 Metric Embeddings: a.k.a. Simplifying Metrics

We just how to approximate a finite metric space with a simpler

metric space, defined over a tree. (Loosely, “every metric space is
within O(log n) of some tree metric”.) And since trees are simpler
metrics, both conceptually and algorithmically, such an embedding
can help design algorithms for problems on metric spaces.
This idea of approximating metric spaces by simpler ones has
been extensively studied in various forms. For example, another fa-
mous result of Jean Bourgain (with an extension by Jirka Matoušek)
shows that any finite metric space on n points can be embedded
into ℓ p -space with O((log n)/p) distortion 1 . Moreover, the Johnson- 1

Lindenstrauss Lemma, which we will see in a future chapter, shows

that any n point-submetric of Euclidean space can be embedded
into a (low-dimensional) Euclidean space of dimension at most
O(log n/ϵ2 ), such that distances between points are distorted by a
factor of at most 1 ± ϵ 2 . Since geometric spaces, and particularly, 2

low-dimensional Euclidean spaces, are easier to work with and rea-

son about, these can be used for algorithm design as well.

5.4.1 Historical Notes

To be cleaned up. Elkin et al. 3 gave the first polylog-stretch span- 3

ning trees, which took eight years following Bartal’s√ construction.

(The first low-stretch spanning trees had stretch 2O( log n log log n) by
Alon et al. 4 , which is smaller than nϵ for any ϵ > 0 but larger than 4
 low-stretch spanning trees 73

polylogarithmic, i.e., (log n)C for any C > 0.)

6
Graph Matchings I: Combinatorial Algorithms

Another fundamental graph problem is to find matchings: these

are subsets of edges that do not share endpoints. Matchings arise
in various contexts: matching tasks to workers, or advertisements
to slots, or roommates to each other. Moreover, matchings have a
rich combinatorial structure. The classical results can be found in
Matching Theory by Laci Lovász and Michael Plummer 1 , though Lex 1

Schrijver’s Combinatorial Optimization: Polyhedra and Efficiency might

be easier to find, and contains more recent developments as well.
Several different and interesting algortihmic techniques can be
used to find large matchings in graphs; we will discuss them over the
next few chapters. This chapter discusses the simplest, combinatorial
algorithms.

6.1 Notation and Definitions

Consider an undirected (simple and connected) graph G = (V, E)

with |V | = n and | E| = m as usual. The graph is unweighted; we will
consider weighted versions of matching problems in later chapters.
When considering bipartite graphs, where the vertex set has parts
V = L ⊎ R (the “left” and “right”, and the edges E ⊆ L × R, we may
denote the graph as G = ( L, R, E).

Definition 6.1 (Matching). A matching in graph G is a subset of the

edges M ⊆ E which have no endpoints in common. Equivalently,
the edges in M are disjoint, and hence every vertex in (V, M ) has
maximum degree 1.

Given a matching M in G, a vertex v is open or exposed or free if

no edge in the matching is incident to v, else the vertex is closed or
covered or matched. Observe: the empty set of edges is a matching.
Moreover, any matching can have at most |V |/2 edges, since each
edge covers two vertices, and each vertex can be covered by at most
one edge.
76 notation and definitions

Definition 6.2 (Perfect Matching). A perfect matching M is a match-

ing such that | M| = |V |/2. Equivalently, every vertex is matched in
the matching M.

Definition 6.3 (Maximum Matching). A maximum cardinality

matching (or simply maximum matching) in G is a matching with
largest possible cardinality. The size of the maximum matching in
graph G is denoted MM( G ).

Definition 6.4 (Maximal Matching). A maximal matching on a graph

is a matching that is inclusion-wise maximal; that is, no additional
edges can be added to M while maintaining the matching propert.
Hence, M ∪ {e} is not a matching for all edges e ̸∈ M.

The last definition is given to mention something we will not be

focusing on; our interest is in perfect and maximum matchings. That
being said, it is a useful exercise to show that any maximal matching
in G has at least MM( G )/2 edges.

6.1.1 Augmenting Paths for Matchings

Since we want to find a maximum matching, a question we may
ask is: given a matching M, can we (efficiently) decide if it is a maximum
matching? One answer to this was suggested by Berge, who gave a
characterization of maximum matchings in terms of “augmenting”
paths.

Definition 6.5 (Alternating Path). For matching M, an M-alternating

path is a path in which edges in M alternate with those not in M.
Figure 6.1: An alternating path P
(dashed edges are not in P, solid edges
Definition 6.6 (Augmenting Path). For matching M, an M-augmenting are in P)
path is an M-alternating path with both endpoints open.

Given sets S, T, their symmetric difference is denoted Figure 6.2: An augmenting path

S △ T : = ( S \ T ) ∪ ( T \ S ).

The following theorem explains the name for augmenting paths.

Theorem 6.7 (Berge’s Optimality Criterion). A matching M is a maxi-

mum matching in graph G if and only if there are no M-augmenting paths
in G.
Berge (1957)
Proof. If there is an M-augmenting path P, then M′ := M△ P is a
larger matching than M. (Think of getting M′ by toggling the dashed
edges in the path to solid, and vice versa). Hence if M is maximum
matching, there cannot exist an M-augmenting path.
Conversely, suppose M is not a maximum matching, and matching
M has | M′ | > | M|. Consider their symmetric difference S := M△ M′ .
′
 graph matchings i: combinatorial algorithms 77

Every vertex is incident to at most 2 edges in S (at most one each

from M and M′ ), so S consists of only paths and cycles, all of them
having edges in M alternating with edges in M′ . Any cycle with this
alternating structure must be of even length, and any path has at
most one more edge from one matching than form the other. Since
| M′ | > | M|, there must exists a path in S with one more edge from
M′ than from M. But this is an M-augmenting path.

If we could efficiently find an M-augmenting path (if one exists),

we could repeatedly augment the current matching until we have
a maximum matching. However, Berge’s theorem does not imme-
diately give an efficient algorithm: finding an M-augmenting path
could naively take exponential time. We now give algorithms to effi-
ciently find augmenting paths, first in bipartite graphs, and then in
general graphs.

6.2 Bipartite Graphs

Finding an M-augmenting path (if one exists) in bipartite graphs is

an easier task, though it still requires cleverness. A first step is to
consider a “dual” object, which is called a vertex cover.

Definition 6.8 (Vertex Cover). A vertex cover in G is a set of vertices

C such that every edge in the graph has at least one endpoint in C.

Note that the entire set V is trivially a vertex cover, and the chal-
lenge is to find small vertex covers. We denote the size of the smallest
cardinality vertex cover of graph G as VC ( G ). Our motivation for
calling it a “dual” object comes from the following fundamental theo-
rem from the early 20th century:

Theorem 6.9 (König’s Minimax Theorem). In a bipartite graph, the size Dénes König (1916)
of the largest possible matching equals the cardinality of the smallest vertex
cover:
MM( G ) = VC( G ).

This theorem is a special case of the max-flow min-cut theorem,

which you may have seen before. It is first of many min-max rela-
tionships, many of which lead to efficient algorithms. Indeed, the
algorithm for finding augmenting paths will come out of the proof of
this theorem. Exercise: Use König’s theorem to prove
P. Hall’s theorem: A bipartite graph has
Proof. In many such proofs, there is one easy direction. Here, it is a matching that matches all vertices of L if
and only for every subset S ⊆ L of vertices,
proving that MM( G ) ≤ VC ( G ). Indeed, the edges of any matching | N (S)| ≥ |S|. Here N (S) denotes the
share no endpoints, so covering a matching of size MM( G ) requires “neighborhood” of S, i.e., those vertices
at least as many vertices. The minimum vertex cover size is therefore with a neighbor inside S.

at least MM( G ).
78 bipartite graphs

Next, we prove that MM( G ) ≥ VC ( G ). To do this, we give a

linear-time algorithm that takes as input an arbitrary matching M,
and either returns an M-augmenting path (if such a path exists), or
else returns a vertex cover of size | M|. Since a maximum matching
M admits no M-augmenting path by Berge’s theorem, we would
get back a vertex cover of size MM( G ), thereby showing VC ( G ) ≤
MM( G ).
The proof is an “alternating” breadth-first search: it starts with
all open nodes among the left vertex set L, and places them at level
0. Then it finds all the (new) neighbors of these nodes reachable
using non-matching edges, and then all (new) neighbors of those
nodes using matching edges, and so on. Formally, the algorithm is as
follows, where we use X≤ j to denote X0 ∪ . . . ∪ X j .

9.1 X0 ← all open vertices in L

9.2 for i = 0, 1, 2, . . . do
9.3 X2i+1 ← {v | exists u ∈ X2i s.t. uv ̸∈ M, and v ̸∈ X≤2i }
9.4 X2i+2 ← {v | exists u ∈ X2i+1 s.t. uv ∈ M, and v ̸∈ X≤2i+1 }
Let us make a few observations about the procedure. First, since
the graph is bipartite, Xi is a subset of L for even levels i, and of R for
odd levels i. Next, all vertices in X2 ∪ X4 ∪ . . . are matched vertices,
X D A
since they are reached from the previous level using an edge in the Y B

matching. Moreover, if some odd level X2i+1 contains an open node Z C

v, we have found an M-alternating path from an open node in X0 to Layer 0 1 2 3 4

Matched edge
v, and hence we can stop and return this augmenting path. Unmatched edge

Open vertex
Hence, suppose we do not find an open node in an even level, and
Figure 6.3: Illustration of the process
stop when some X j is empty. Let X = ∪ j X j be all nodes added to any to find augmenting paths in a bipartite
of the sets X j ; we call these marked nodes. Define the set C to be the graph. Mistakes here, to be fixed!
vertices on the left which are not marked, plus the vertices on the right
which are marked. That is,

C := ( L \ X ) ∪ ( R ∩ X )

We claim that C is a vertex cover of size | M |.

Claim 6.10. C is a vertex cover.

Proof. G is a bipartite graph, and C hits all edges that touch R ∩ X

and L \ X. Hence we must show there are no edges between L ∩ X
Figure 6.4: X = set of marked vertices,
and R \ X, i.e., between the top-left and bottom-right of the figure. O = marked open vertices, C = claimed
vertex cover of G. To be changed.
1. There can be no unmatched edge from the open vertices in L ∩ X to
R \ X, else that vertex would be reachable from X0 and so belong
to X1 . Moreover, an open vertex has no unmatched edges, by
definition. Hence, any “offending edges” out of L ∩ X must come
from a covered vertex.
 graph matchings i: combinatorial algorithms 79

2. There can be no non-matching edge from a covered vertex in L ∩ X

to some node u in R \ X, else this node u would have been added
to some level X2i+1 .

3. Finally, there can be no matching edge between a covered vertex

in L ∩ X and some vertex in R \ X. Indeed, every covered node in
L ∩ X (i.e., those in X2 , X4 , . . . ) was reached via a matching edge
from some node in R ∩ X. There cannot be another matching edge
from some node in R \ X incident to it.

This shows that C is a vertex cover.

Claim 6.11. |C | ≤ | M|.

We use a simple counting argument:

• Every vertex in R ∩ X has a matching edge incident to it; else it

would be open, giving an augmenting path.

• Every vertex in L \ X has an incident edge in the matching, since

no vertices in L \ X ⊆ L \ X0 are open.

• There are no matching edges between L \ X and R ∩ X, else they

would have been explored and added to X.

Hence, every vertex in C = ( L \ X ) ∪ ( R ∩ X ) corresponds to a unique

edge in the matching, and |C | ≤ | M |.

Observe that the proof of König’s theorem is algorithmic, and

can be implemented to run in O(m) time. Now, starting from some
trivial matching, we can use this linear-time algorithm to repeatedly
augment until we have a maximum matching. This means that maxi-
mum matching on bipartite graphs has an O(mn)-time algorithm.
Observe: this algorithm also gives a “proof of optimality” of the
maximum matching M, in the form of a vertex cover of size | M |. By
the easy direction of König’s theorem, this is a vertex cover of mini-
mum cardinality. Therefore, while finding the smallest vertex cover is
NP-hard for general graphs, we have just solved the minimum vertex
cover problem on bipartite graphs.
One other connection: if you have seen the Ford-Fulkerson al-
gorithm for computing maximum flows, the above algorithm may
seem familiar. Indeed, modeling the maximum matching problem in Figure 6.5: Use Ford-Fulkerson algo-
bipartite graphs as that of finding a maximum integer s-t flow, and rithm to find a matching
running the Ford-Fulkerson “augmenting paths” algorithm results in
the same result. Moreover, the minimum s-t cut corresponds to a ver-
tex cover, and the max-flow min-cut theorem proves König’s theorem.
The figure to the right illustrates this on an example. Figure needs
fixing.
80 general graphs: the tutte-berge theorem

6.2.1 Other algorithms

There are faster algorithms to find maximum matchings in bipartite
graphs. For a long time, the fastest one was an algorithm by John
√
Hopcroft and Dick Karp, which ran in time O(m n). It finds many Hopcroft and Karp (1973)
augmenting paths at once, and then combines them in a clever way.
There is also a related algorithm of Shimon Even and Bob Tarjan, Even and Tarjan (1975)
√
which runs in time O(min(m m, mn2/3 )); in fact, they compute
maximum flows on unit-capacity graphs in this running time.
There was remarkably little progress on the maximum matching
problem until 2016, when Aleksander Madry gave an algorithm Madry (2016)
that runs in time Õ(m10/7 ) time—in fact the algorithm also solves
the unit-capacity maximum-flow problem in that time. It takes an
interior-point algorithm for solving general linear programs, and
specializes it to the case of maximum matchings. We may discuss this
max-flow algorithm in a later chapter. The current best runtime for
the unit-capacity maximum-flow problem is m4/3+o(1) , due to Yang
Liu and Aaron Sidford2 . 2

6.3 General Graphs: The Tutte-Berge Theorem

The matching problem on general (non-bipartite) graphs gets more

involved, since the structure of matchings is richer. For example, the
flow-based approaches do not work any more. And while Berge’s
theorem (Theorem 6.7) still holds in this case, König’s theorem (The-
orem 6.9) is no longer true. Indeed, the 3-cycle C3 has a maximum
matching of size 1, but the smallest vertex cover is of size 2. However,
we can still give a min-max relationship, via the Tutte-Berge theorem.
To state it, let us give a definition: for a subset U ⊆ V, suppose
deleting the nodes of U and their incident edges from G gives con-
nected components {K1 , K2 , . . . , Kt }. The quantity odd( G \ U ) is the
number of such pieces with an odd number of vertices.

Theorem 6.12 (The Tutte-Berge Max-Min Theorem). Given a graph G, Tutte (1947), Berge (1958)
the size of the maximum matching is described by the following equation. Tutte showed that the graph has a
perfect matching precisely if for every
U ⊆ V, odd( G \ U ) ≤ |U |. Berge
n + |U | − odd( G \ U )
MM ( G ) = min . gave the generalization to maximum
U ⊆V 2 matchings.

The expression on the right can seem a bit confusing, so let’s con-
sider some cases.

• If U = ∅, we get that if |V | is even then MM ( G ) ≤ n/2, and if |V |

is odd, the maximum matching cannot be bigger than (n − 1)/2.
(Or if G is disconnected with k odd-sized components, this gives
n/2 − k/2.)
 graph matchings i: combinatorial algorithms 81

• Another special case is when U is any vertex cover with size c.

Then the Ki ’s must be isolated vertices, so odd( G \ U ) = n − c.
c+n−(n−c)
This gives us MM ≤ 2 = c, i.e., the size of the maximum
matching is at most the size of any vertex cover.

• Give example where G is even, connected, but MM < VC.

Trying special cases is a good way to understand the

Proof of the ≤ direction of Theorem 6.12. The easy direction is to show

that MM ( G ) is at most the quantity on the right. Indeed, consider
a maximum matching M. At most |U | of the edges in M can be hit
by nodes in U; the other edges must lie completely within some
connected component of G \ U. The maximum size of a matching
within Ki is ⌊Ki /2⌋, and it are these losses from the odd components
that gives the expression on the right. Indeed, we get

t  
| Ki |
| M | ≤ |U | + ∑
i =1
2
n − |U | odd( G \ U )
= |U | + −
2 2
|U | + n − odd( G \ U )
= .
2
We can prove the “hard” direction using induction (see the webpage
for several such proofs). However, we defer it for now, and derive it
later from the proof of the Blossom algorithm.

6.4 The Blossom Algorithm

The Blossom algorithm for finding the maximum matching in a gen-

eral graph is by Jack Edmonds. Recall: the algorithm for minimum- Edmonds (1965)
weight arborescences in §?? was also due to him, and you may see
some similarities in these two algorithms.

Theorem 6.13. Given a graph G, the Blossom algorithm finds a maximum

matching M in time O(mn2 ).

The rest of this section defines the algorithm, and proves this
theorem. The essential idea of the algorithm is simple, and similar
to the one for the bipartite case: if we have a matching M, Berge’s
characterization from Theorem 6.7 says that if M is not optimal, there
exists an M-augmenting path. So the natural idea would be to find
such an augmenting path. However, it is not clear how to do this
directly. The clever idea in the Blossom algorithm is to either find
an M-augmenting path, or else find a structure called a “blossom”.
The good thing about blossoms is that we can use them to contract
82 the blossom algorithm
Stem Blossom
(a)

the graph in a certain way, and make progress. Let us now give some
definitions, and details. A
(b)
A flower is a subgraph of G that looks like the the object to the
Matched edge
right: it has a open vertex at the base, then a stem with an even num- Unmatched edge
ber of edges (alternating between matched and unmatched edges ), Open vertex

and then a cycle with an odd number of edges (again alternating,

though naturally having two unmatched edges adjacent to the stem). Figure 6.6: An example of blossom and
The cycle itself is called the blossom. the toggling of the stem.

6.4.1 The Main Procedure

The algorithm depends on a subroutine called FindAugPath, which
has the following guarantee.

Lemma 6.14. Given graph G and matching M, the subroutine FindAugPath,

runs in O(m) time. If G has an M-augmenting path, then it returns either
(a) a flower F, or (b) an M-augmenting path.

Note that we have not said what happens if there is no M-augmenting

path. Indeed, we cannot find an augmenting path, but we show that
the FindAugPath returns either a flower, or says “no M-augmenting
path, and returns a Tutte-Berge set U achieving equality in The-
orem 6.12 with respect to M. We can now use this FindAugPath
subroutine within our algorithm as follows.

1. Says “no M-augmenting path” and a set U of nodes. In this case, M is

the maximum matching.

2. Finds augmenting path P. We can now augment along P, by setting

M ← M△ P.

3. Finds a flower F. In this case, we don’t yet know if M is a maxi-

mum matching or not. But we can shrink the blossom down to
get a smaller graph G ′ (and a matching M′ in it), and recurse.
Either we will find a proof of maximality of M′ in G ′ , or an M′ -
augmenting path. This we can extend to the matching M in G.
That’s the whole algorithm!

Let’s give some more details for the last step. Suppose we find
a flower F, with stem S and blossom B. First, toggle the stem (by
setting M ← M△S): this moves the open node to the blossom,
Figure 6.7: The shrinking of a blossom.
without changing the size of the matching M. (It makes the following Image found at http://en.wikipedia.
arguments easier, with one less case to consider.) (Change figure.) org/wiki/Blossom_algorithm.

Next, contract the blossom down into a single vertex v B , which is

now open. Denote the new graph G ′ ← G/B, and M′ ← M/B. Given a graph and a subset C ⊆ V,
Since all the nodes in blossom B, apart from perhaps the base, were recall that G/C denotes the contraction
of C in G.
matched by edges within the blossom, M′ is also a matching in G ′ .
 graph matchings i: combinatorial algorithms 83

Next, we recurse on this smaller graph G ′ with matching M′ .

Finally, if we get back an M′ -augmenting path, we “lift” it to get an
M-augmenting path (as we see soon). Else if we find that M′ is a
maximum matching in G ′ , we declare that M is maximum in G. To
show correctness, it suffices to prove the following theorem.

Lemma 6.15. Given graph G and matching M, suppose we shrink a blos-

som to get G ′ and M′ . Then there exists an M-augmenting path in G if and
only if there exists an M′ -augmenting path in G ′ .
Moreover, given an M′ -augmenting path in G ′ , we can lift it back to an
M-augmenting path P in G in O(m) time.

Proof. Since we toggled the stem, the vertex v at the base of the blos-
som B is open, and so is the vertex v B created in G ′ by contracting
B. Moreover, all other nodes in the blossom are matched by edges
within itself, so all edges leaving B are non-matching edges. The
picture essentially gives the proof, and can be used to follow along.

(⇒) Consider an M-augmenting path in G, denoted by P. If P does

not go through the blossom B, the path still exists in G ′ . Else if
P goes through the blossom, we can assume that one of its end-
points is the base of the blossom (which is the only open node on
Figure 6.8: The translation of augment-
the blossom)—indeed, any other M-augmenting path P can be ing paths from G \ B to G and back.
rerouted to the base. (Figure!) So suppose this path P starts at the
base and ends at some v′ not in B. Because v B is open in G ′ , the
path from v B to v′ is an M′ -augmenting path in G ′ .

(⇐) Again, an M′ -augmenting path P′ in G ′ that does not go through

v B still exists in G. Else, the M′ -augmenting path P′ passes through
v B , and because v B is open in G ′ , the path starts at v B and ends at
some node t. Let the first edge on P′ be e′ = v B y for some node
y, and let it correspond to edge e = xy in G, where x ∈ B. Now,
if v is the open vertex at the base of the blossom, following one
of the two paths (either clockwise or counter-clockwise) along the
blossom from v to x, using the edge xy and then following the rest
of the path P′ from y to t gives an M-augmenting path in G. (This
is where we use the fact that the cycle is odd, and is alternating
except for the two edges incident to v.)

The process to get from P′ in G ′ to the M-augmenting path in G be

done algorithmically in O(m) time, completing the proof.

We can now analyze the runtime, and prove Theorem 6.13:

Proof of Theorem 6.13. We first call FindAugPath, which takes O(m)

time. We are either done (because M is a maximum matching, or else
we have an augmenting path), or else we contract down in another
84 the blossom algorithm

O(m) time to get a graph G ′ with at most n − 3 vertices and at most

m edges. Inductively, the time taken in the recursive call on G ′ is
O(m(n − 3)). Now lifting an augmenting path takes O(m) time more.
So the total runtime to find an augmenting path in G (if one exists) is
O(mn).
Finally, we start with an empty matching, so its size can be aug-
mented at most n/2 times, giving us a total runtime of O(mn2 ).

6.4.2 The FindAugPath Subroutine

The subroutine FindAugPath is very similar to the analogous pro-
cedure in the bipartite case, but since there is no notion of left and
right vertices, we start with level X0 containing all vertices that are
unmatched in M0 , and try to grow M-alternating paths from them, in
the hope of finding an M-augmenting path. As before, let X≤ j denote X0 ∪ . . . ∪ X j ,
and let nodes added to some level X j be
called marked.
9.1 X0 ← all open vertices in V
9.2 for i = 0, 1, 2, . . . do
9.3 X2i+1 ← {v | exists u ∈ X2i s.t. uv ̸∈ M, and v ̸∈ X≤2i }
9.4 X2i+2 ← {v | exists u ∈ X2i+1 s.t. uv ∈ M, and v ̸∈ X≤2i+1 }
9.5 if exists a “cross” edge between nodes of same level then
9.6 return augmenting path or flower
9.7 else
9.8 say “no M-augmenting path”
To argue correctness, let us look at the steps above in more detail.
In line 9.2, for each vertex u ∈ X2i , we consider the possible cases for
each non-matching edge uv incident to it:

1. If v is not in X≤2i+1 already (i.e., not marked already) then we add

it to X2i+1 . Note that v ∈ X2i+1 now has an M-alternating path to
some node in X0 , that hits each layer exactly once.

2. If v ∈ X2i , then uv is an unmatched edge linking two vertices

in the same level. This gives an augmenting path or a blossom!
Indeed, by construction, there are M-alternating paths P and
Q from u and v to open vertices in X0 . If P and Q do not inter-
sect, then concatenating path P, edge uv, and path Q gives an
M-augmenting path. If P and Q intersect, they must first intersect
some vertex w ∈ X2j for some j ≤ i, and the cycle containing u, v, w
gives us the blossom, with the stem being a path from w back to
an open vertex in X0 .

3. If v ∈ X2j for j < i, then u would have been added to the odd level
X2j+1 , which is impossible.

4. Finally, v may belong to some previous odd level, which is fine.

 graph matchings i: combinatorial algorithms 85

Observe that this “backward” non-matching edge uv is also an

even-to-odd edge, like the “forward” edge in the first case.

Now for the edges out of the odd layers considered in line 9.3.
Given u ∈ X2i+1 and matching edge uv ∈ M, the cases are:

1. If v is not in X≤2i+1 then add it to X2i+2 . Notice that v cannot be in

X2i+2 already, since nodes in even layers are matched to nodes in
the preceding odd layer, and there cannot be two matching edges
incident to v.
Again, observe inductively that v has a path to some vertex in X0
that hits each intermediate layer once.

2. If v is in X2i+1 , there is an matching edge linking two vertices in

the same odd level. This gives an augmenting path or a blossom,
as in case 2 above. (Success!)

3. The node v cannot be in a previous level, because all those vertices

are either open, or are matched using other edges.

Observe that if the algorithm does not succeed, all the matching
edges we explored are odd-to-even, whereas all the non-matching
edges are even-to-odd. Now we can prove Lemma 6.14.

Proof of Lemma 6.14. Let P be an M-augmenting path in G. For a

contradiction, suppose we do not succeed in finding an augmenting
path or blossom. Starting from one of the endpoints of P (which is in
X0 , an even level), trace the path in the leveled graph created above.
The next vertex should be in an odd level, the next in an even level,
and so forth. Since the path P is alternating, FindAugPath ensures
that all its edges will be explored. (Make sure you see this!) Now P
has an odd number of edges (i.e., even number of vertices), so the last
vertex has an opposite parity from the starting vertex. But the last
vertex is open, and hence in X0 , an even level. This is a contradiction.

6.4.3 Finding a Tutte-Berge Set⋆

If FindAugPath did not succeed, all the edges we explored form a
bipartite graph. This does not mean that the entire graph is bipar-
tite, of course—there can be non-matching edges incident to nodes
in odd levels that lead to nodes that remain unmarked. But these
components have no open vertices (which are all in X0 and marked).
Now define U = Xodd := X1 ∪ X3 ∪ . . . be the vertices in odd lev-
els. Since there are no cross edges, each of these nodes has a distinct
matching edge leading to the next level. Now G \ U has two kinds of
components:
 86 subsequent work

(a) the marked vertices in the even levels, Xeven which are all single-
tons since there are no cross edges, and

(b) the unmarked components, which have no open vertices, and

hence have even size.

Hence
n + |U | − odd( G \ U ) n + | Xodd | − | Xeven |
=
2 2
2| Xodd | + (n − | X |)
=
2
(n − | X |)
= | Xodd | + = | M |.
2
The last equality uses that all nodes in V \ X are perfectly matched
among themselves, and all nodes in Xodd are matched using unique
edges.
The last piece is to show that a Tutte-Berge set U ′ for a contracted
graph G ′ = G/B with respect to M′ = M/B can be lifted to one for G
with respect to M. We leave it as an exercise to show that adding the
entire blossom B to U ′ gives such an U.

6.5 Subsequent Work

The best runtime of combinatorial algorithms for maximum match-

√
ing in general graphs is O(m n) by an algorithm of Silvio Micali
and Vijay Vazirani3 . The algorithm is based on finding augmenting 3

paths much faster; it is quite involved, though a recent paper of Vijay

Vazirani4 giving a more approachable explanation. In a later chap- 4

ter, we will see a very different “algebraic” algorithm based on fast

matrix multiplication. This algorithm due to Marcin Mucha and Pi-
otr Sankowski gives a runtime of O(nω ), where ω ≈ 2.376. Coming Mucha and Sankowski (2006)
up next, however, is a discussion of weighted versions of matching,
where edges have weights and the goal is to find the matching of
maximum weight.
7
Graph Matchings II: Weighted Matchings

In this chapter, we study the matching problem from the perspective

of linear programs, and also learn results about linear programming
using the matching problem as our running example. In fact, we see
how linear programs capture the structure of many problems we
have been studying: MSTs, min-weight arborescences, and graph
matchings.

7.1 Linear Programming

We start with some basic definitions and results in Linear Program-

ming. We will use these results while designing our linear program
solutions for min-cost perfect matchings, min-weight arborescences
and MSTs. This will be a sufficient jumping-off point for the contents
6
of this lecture; a more thorough introduction to the subject can be
found in the introductory text by Matoušek and Gärtner.
4
Definition 7.1. Let ⃗a ∈ Rn be a vector and let b ∈ R a scalar. Then a
x2

half-space in Rn is a region defined by the set {⃗x ∈ Rn | ⃗a · ⃗x ≥ b}.

" # 2

1
As an example, the half space S = {⃗x | · ⃗x ≥ 3} in R2 is shown
1 0
on the right. (Note that we implicitly restrict ourselves to closed half-
0 1 2 3 4 5 6
spaces.) x1

Figure 7.1: The half-space in R2 given

by the set S
7.1.1 Polytopes and Polyhedra
Definition 7.2 (Polyhedron). A polyhedron in Rn is the intersection of
a finite number of half spaces.

A polyhedron is a convex region which is defined by finitely many

linear constraints. A polyhedron in n dimensions with m constraints
is often written compactly as

K = { Ax ≤ b},
88 linear programming

where A is an m × n constraint matrix, x is an n × 1 vector of vari-

ables, and b is an m × 1 vector of constants.

Definition 7.3 (Polytope). A polytope K ∈ Rn is a bounded polyhe-

dron.

In other words, a polytope is polyhedron such that there exists

some radius R > 0 such that K ⊆ B(0, R) = { x | ∥ x ∥2 ≤ R}. A simple
example of a polytope (where the bounded region of the polytope is
highlighted by ) appears on the right. We can now define a linear Figure 7.2: The polytope in R2 given by
the constraints − x1 − x2 ≤ 1, x1 ≤ 0,
program (often abbeviated as LP) in terms of a polyhedron. and x2 ≤ 0.

Definition 7.4 (Linear Program). For some integer n, a polyhedron

K = { x | Ax ≤ b}, and an n by 1 vector c, a linear program in n
dimensions is the linear optimization problem

min{c · x | x ∈ K } = min{c · x | Ax ≤ b}.

The set K is called the feasible region of the linear program.

Although all linear programs can be put into this canonical form,
in practice they may have many different forms. These presenta-
tions can be shown to be equivalent to one another by adding new
variables and constraints, negating the entries of A and c, etc. For
example, the following are all linear programs:

max{c · x : Ax ≤ b} min{c · x : Ax = b}
x x
min{c · x : Ax ≥ b} min{c · x : Ax ≤ b, x ≥ 0}.
x x

The polyhedron K need not be bounded for the linear program to

have a (finite) optimal solution. For example, the following linear
program has a finite optimal solution even though the polyhedron is
unbounded:
min{ x1 + x2 | x1 + x2 ≥ 3}. (7.1)
x

7.1.2 Vertices, Extreme Points, and BFSs

We now introduce three different classifications of some special
points associated with polyhedra. (Several of these definitions ex-
tend to convex bodies.)

Definition 7.5 (Extreme Point). Given a polyhedron K ∈ Rn , a point

x ∈ K is an extreme point of K if there do not exist distinct x1 , x2 ∈ K,
and λ ∈ [0, 1] such that x = λx1 + (1 − λ) x2 .

In other words, x is an extreme point of K if it cannot be written as Figure 7.3: Here y is an extreme point,
but x is not.
the convex combination of two other points in K. See Figure 7.3 for
an example.
Here’s another kind of point in K. In this course, we will use the notation
c · x, c⊺ x, and ⟨c, x ⟩ to denote the inner-
product between vectors c and x.
 graph matchings ii: weighted matchings 89

Definition 7.6 (Vertex). Given a polyhedron K ⊆ Rn , a point x ∈ K is

a vertex of K if there exists an vector c ∈ Rn such that c · x < c · y for
all y ∈ K y ̸= x.

In other words, a vertex is the unique optimizer of some linear ob-

jective function. Equivalently, the hyperplane {y ∈ Rn | c · y = c · x }
intersects K at the single point x. Note that there may be a poly-
hedron that does not have any vertices: e.g., one given by a single
constraint, or two parallel constraints.
Finally, here’s a third kind of special point in K:

Definition 7.7 (Basic Feasible Solution). Given a polyhedron K ∈ Rn ,

a point x ∈ K is a basic feasible solution (bfs) for K if there exist n
linearly independent defining constraints for K which x satisfies at
equality.

In other words, let K := { x ∈ Rn | Ax ≤ b}, where the m

constraints corresponding to the m rows of A are denoted by ai · x ≤
bi . Then x ∗ ∈ Rn is a basic feasible solution if there exist n linearly
independent constraints for which ai · x ∗ = bi , and moreover ai · x ∗ ≤
bi for all other constraints (because x ∗ must belong to K, and hence
satisfy all other constraints as well). Note there are only (m n ) basic
feasible solutions for K, where m is the total number of constraints
and n is the dimension.
As you may have guessed by now, these three definitions are all
related. In fact, they are all equivalent.
Fact 7.8. Given a polyhedron K and a point x ∈ K, the following are
equivalent:
• x is a basic feasible solution,
• x is an extreme point, and
• x is a vertex.
While we do not prove it here, you could try to prove it yourself,
or consult a textbook. For now, we proceed directly to the main fact
we need for this section. Observe that we claimed Fact 7.9 for
LPs whose feasible region is a polytope,
Fact 7.9. For a polytope K and a linear program LP := min{c · x | x ∈ since that suffices for today, but it can
K }, there exists an optimal solution x ∗ ∈ K such that x ∗ is an extreme be proven with weaker conditions.
However it is not true for all LPs: e.g.,
point/vertex/bfs of K. the LP in (7.1) has an infinite number of
optimal solutions, none of which are at
This fact suggests an algorithm for LPs when K is a polytope: vertices.
simply find all of the (at most (m
n ) basic feasible solutions and pick
the one that gives the minimum solution value. Of course, there
are more efficient algorithm to solve linear programs; we will talk
about them in a later chapter. However, let us state a theorem—a
very restricted form of the general result—about LP solving that will
suffice for now:
90 weighted matchings in bipartite graphs

Theorem 7.10. There exist algorithms that take any LP min{c · x | Ax =

b, x ≥ 0, x ∈ Rn }, where both the constraint matrix A and the RHS b have
entries in {0, 1} and poly(n) rows, and output a basic feasible solution to
this LP in poly(n) time.

We will see a sketch of the proof in a later chapter. Discuss the

dependence on the number of bits to represent c? Or make this an
informal theorem?

7.1.3 Convex Hulls and an Alternate Representation

The next definition allows us to give another representation of poly-
topes:

Definition 7.11 (Convex Hull). Given x1 , x2 , . . . , x N ∈ Rn , the convex

hull of x1 , . . . , x N is the set of all convex combinations of these points.
In other words, CH( x1 , . . . , x N ) is defined as
(  )
 N N
n 
x ∈ R  ∃λ1 , . . . , λ N ≥ 0 s.t. ∑ λi = 1 and x = ∑ λi xi . (7.2)
 i =1 i =1

Put yet another way, the convex hull of x1 , . . . , x N is the intersec-

tion of all convex sets that contain x1 , . . . , x N . It follows from the
definition that the convex hull of finitely many points is a polytope.
(Check!) We also know the following fact:
Fact 7.12. Given a polytope K with extreme points ext(K ),

K = CH(ext(K )).

The important insight that polytopes may be represented in terms

of their extreme points, or their bounding half-planes. One represen-
tation may be easier to work with than the other, depending on the
situation. The rest of this chapter will involve moving between these
two methods of representing polytopes.

7.2 Weighted Matchings in Bipartite Graphs

While the previous chapters focused on finding maximum matchings

in graphs, let us now consider the problem of finding a minimum-
weight perfect matching in a graph with edge-weights. As before,
we start with bipartite graphs, and extend our techniques to general
graphs.
We are given a bipartite graph G = ( L, R, E) with edge-weights
we . We want to use linear programs to solve the problem, so it is
natural to have a variable xe for each edge e of the graph. We want
our solution to set xe = 1 if the edge is in the minimum-weight
 graph matchings ii: weighted matchings 91

perfect matching, and xe = 0 otherwise. Compactly, this collection of

variables gives us a | E|-dimensional vector x ∈ R| E| , that happens to
contain only zeros and ones.
A bit of notation: for any subset S ⊆ E, let χS ∈ {0, 1}| E| de-
note the characteristic vector of this subset S, where χS has ones in
coordinates that correspond to elements in S, and zeros in the other
coordinates.

7.2.1 Goal: the Bipartite Perfect Matching Polytope

It is conceptually easy to define an | E|-dimensional polytope whose Figure 7.4: This graph has one perfect
vertices are precisely the perfect matchings of G: we simply define matching M: it contains edges 1, 4,
5, and 6, represented by the vector
χ M = (1, 0, 0, 1, 1, 1).
CPM(G) = CH ({χ M | M is a perfect matching in G }). (7.3)

And now we get a linear program that finds the minimum-weight

perfect matching in a bipartite graph.

min{w · x | x ∈ CPM(G) }.

By Fact 7.9, there is an optimal solution at a vertex of CPM(G) , which

by construction represents a perfect matching in G.
The good part of this linear program is that its feasible region has
(a) only integer extreme points, (b) which are in bijection with the
objects we want to optimize over. So optimizing over this LP will
immediately solve our problem. (We can assume that there are linear
program solvers which always return an optimal vertex solution, if
one exists.) Moreover, the LP solver runs in time polynomial in the
size of the LP.
The catch, of course, is that we have no control over the size of
the LP, as we have written it. Our graph G may have an exponen-
tial number of matchings, and hence the definition of CPM(G) given
in (7.3) is too unwieldly to work with. Of course, the fact that there The unit cube
are an exponential number of vertices does not mean that there can- K = { x ∈ Rn | 0 ≤ x i ≤ 1 ∀ i }
not be a smaller representation using half-spaces. Can we find a is a polytope with 2n constraints but 2n
compact way to describe CPM(G) ? vertices.

7.2.2 A Compact Linear Program

The beauty of the bipartite matching problem is that the “right”

linear program is perhaps the very first one you may write. Here is
the definition of the polytope using linear constraints:
92 weighted matchings in bipartite graphs

  






∑ xlr = 1 ∀l ∈ L 



 
 r ∈ N (l ) 

  
K PM(G) =

x ∈ R| E | s.t.
 ∑ xlr = 1 ∀r ∈ R


 
 l ∈ N (r ) 


 
 

  
xe ≥ 0 ∀e ∈ E

The constraints of this polytope merely enforce that each coordi-

nate is non-negative (which gives us | E| constraints), and that the
xe values of the edges leaving each vertex sum to 1 (which gives us
| L| + | R| more constraints). All these constraints are satisfied by each
χ M corresponding to a matching M, which is promising. But it still
always comes as a surprise to realize that his first attempt is actually
successful:

Theorem 7.13. For any bipartite graph G, K PM(G) = CPM(G) .

Proof. For brevity, let us refer to the polytopes as K and C. The easy
direction is to show that C ⊆ K. Indeed, the characteristic vector χ M
for each perfect matching M satisfies the constraints for K. Moreover
K is convex, so if it contains all the vertices of C, it contains all their
convex combinations, and hence all of C.
For the other direction, we show that an arbitrary vertex x ∗ of K
is contained within C. Using Fact 7.8, we use the fact that x ∗ is also
an extreme point for K. (We can also use the fact that x ∗ is a basic
feasible solution, or that it is a vertex of the polytope, to prove this
theorem; we will add the former proof soon, the latter proof appears
in §7.3.)
Let supp( x ∗ ) = {e | xe∗ > 0} be the support of this solution.
We claim that supp( x ∗ ) is acyclic. Indeed, suppose not, and cycle
C = e1 , e2 , . . . , ek is contained within the support supp( x ∗ ). Since the
graph is bipartite, this is an even-length cycle. Define

ε := min xe∗ .
e∈supp( x ∗ )

Observe that for all ei ∈ C, xe∗i + xe∗i+1 ≤ 1, so xe∗i ≤ 1 − ε. And

of course xe∗i ≥ ε, merely by the definition of ε. Now consider two
solutions x + and x − , where

xe+i = xe∗i + (−1)i ε Figure 7.5: There cannot be a cycle

in supp( x ∗ ), because this violates the
and assumption that x ∗ is an extreme point.
xe+i = xe∗i − (−1)i ε.
I.e., the two solutions add and subtract ε on alternate edges; this
ensures that both the solutions stay within K. But then x ∗ = 12 x + +
1 − ∗
2 x , contradicting our that x is an extreme point.
 graph matchings ii: weighted matchings 93

Therefore there are no cycles in supp( x ∗ ); this means the sup-

port is a forest. Consider a leaf vertex v in the support. Then, by the
equality constraint at v, the single edge e ∈ supp( x ∗ ) leaving v must
have xe∗ = 1. But this edge e = uv goes to another vertex u; because
x ∗ is in K, this vertex u cannot have other edges in supp( x ∗ ) without
violating its equality constraint. So u and v are a matched pair in x ∗ .
Now remove u and v from consideration. We have introduced no
cycles into the remainder of supp( x ∗ ), so we may perform the same
step inductively to show that x ∗ is the indicator of a perfect match-
ing, and hence x ∗ ∈ C. This means all vertices of K are in C, which
proves C ⊆ K, and completes the proof.

This completes the proof that the polytope K PM(G) exactly captures
precisely the perfect matchings in G, despite having such a simple
description. Now, using the fact that the linear program

min{w · x | x ∈ K PM(G) }

can be solved in polynomial time, we get an efficient algorithm for

finding minimum-weight perfect matching in graphs.

7.2.3 A Proof via Basic Feasible Solutions

Here is how to prove Theorem 7.13 using the notion of basic feasible
solutions (bfs). Suppose x ∗ ∈ R| E| is a bfs: we now show that xe∗ ∈
{0, 1} for all edges. By the definition of a bfs, there is a collection
of | E| tight linearly independent constraints that define x ∗ . These
constraints are of two kinds: the degree constraints ∑e∈∂(v) xe = 1 for
some subset S of vertices, and the non-negativity constraints xe ≥ 0
for some subset E′ ⊆ E be edges. (Hence we have | E′ | + |S| = | E|.)
By reordering columns and rows, we can put the degree con-
straints at the top, and put all the edges in E′ at the end, to get that
x ∗ is defined by: " # " #
C C′ 1 s
x∗ =
0 I 0m−s

where C ∈ {0, 1}s×s , C ′ ∈ {0, 1}(m−s)×s , and m = | E| and s = |S|.

The edges in E′ have xe∗ = 0, so consider edges in E \ E′ . By the linear
independence of the constraints, we have C being non-singular, so

x ∗ | E\ E′ = C −1 (1 − C ′ x ∗ | E′ ) = C −1 1.

By Cramer’s rule,
det(C [1]i )
xe∗ = .
det(C )
The numerator is an integer (since the entries of C are integers), so
showing det(C ) ∈ {±1} means that xe∗ is an integer.
94 another perspective: buyers and sellers

Claim 7.14. Any k × k-submatrix of C has determinant in {−1, 0, 1}.

Proof. The proof is by induction on k; the base case is trivial. If the

submatrix D has a column with a single 1, we can expand using that
entry, and use the inductive hypothesis. Else each column of D has
two non-zeros. Recall that the columns of D correspond to some
edges ED in E \ E′ , and the rows correspond to vertices SD in S—two
non-zeros in each column means each edge in ED has both endpoints
in SD . Now if we sum rows for vertices in SD ∩ L would give the all
ones vector, as will summing up rows for vertices in SD ∩ R. (Here is
the only place we’re using bipartiteness.) In this case det( D ) = 0.

Using the claim and using the fact C is non-singular and hence
det(C ) cannot be zero, we get that the entries of xe∗ are integers. By
the structure of the LP, the only integers possible in a feasible solu-
tion are {0, 1} and the vector x ∗ corresponds to a matching.

7.2.4 Minimum-Weight Matchings

How can we we find a minimum-weight (possibly non-perfect)
matching in a bipartite graph G? If the edge weights are all non-
negative, the empty matching would be the solution—but what if
some edge weights are negative? (In fact, that’s how we would find a
maximum-weight matching–by negating all the weights.) As before,
we can define the matching polytope for G as

C Match(G) = CH ({χ M | M is a matching in G }).

To write a compact LP that describes this polytope, we slightly mod-

ify our linear constraints as follows:
  





 ∑ xij ≤ 1 ∀i ∈ L 



 
 j∈ R 

  
K Match =

x ∈ R| E | s.t.
 ∑ x ji ≤ 1 ∀i ∈ R


 
 j∈ L 


 
 

  
xi,j ≥ 0 ∀i, j

We leave it as an exercise to apply the techniques used in Theo-

rem 7.13 to show that the vertices of K Match are matchings of G, and
hence the following theorem:

Theorem 7.15. For any bipartite graph G, K Match = CH Match

7.3 Another Perspective: Buyers and sellers

The results of the previous section show that the bipartite perfect
matching polytope is integral, and hence the max-weight perfect
 graph matchings ii: weighted matchings 95

matching problem on bipartite graphs can be be solved by “sim-

ply” solving the LP and getting a vertex solution. But do we need a
generic linear program solver? Can we solve this problem faster? In
this section, we develop (a variant of) the Hungarian algorithm that
finds an optimal solutions using a “combinatorial” algorithm. This
proof also shows that any vertex of the polytope K PM(G) is integral,
and hence gives another proof of Theorem 7.13.

7.3.1 The Setting: Buyers and Items

Consider the setting with a set B with n buyers and another set I with
n items, where buyer b has value vbi for item i. The goal is to find a
max-value perfect matching, that matches each buyer to a distinct
item and maximizes the sum of the values obtained by this matching.
Our algorithm will maintain a set of prices for items: each item i
will have price pi . Given a price vector p := ( p1 , . . . , pn ), define the
utility of item i to buyer b to be

ubi ( p) := vbi − pi .

Intuitively, the utility measures how favorable it is for buyer b to buy

item i, since it factors in both the value and the price of the item.
We say that buyer b prefers item i if item i gives the highest utility
to buyer b, among all items. Formally, buyer b ∈ B prefers item i at
prices p if i ∈ arg maxi′ ∈ I ubi′ ( p). The utility of buyer b at prices p is
the utility of this preferred item:

ub ( p) := max ubi ( p) = max(vbi − pi ). (7.4)

i∈ I i∈ I

A buyer has at least one preferred item, and can have multiple
preferred items, since there can be ties. Given prices p, we build a
preference graph H = H ( p), where the vertices are buyers B on the
left, items I on the right, and where bi is an edge if buyer b prefers
item i at prices p. The two examples show preference graphs, where
the second graph results from an increase in price of item 1. Flip the
figure.

Theorem 7.16. For any price vector p∗ , if the preference graph H ( p∗ )

contains a perfect matching M, then M is a max-value perfect matching.

Proof. This proof uses weak linear programming duality. Indeed,

recall the linear program we wrote for the bipartite perfect matching
problem: we allow fractional matchings by assigning each edge bi a
96 another perspective: buyers and sellers

fractional value xbi ∈ [0, 1].

maximize ∑ vbi xbi

bi
n
subject to ∑ xbi = 1 ∀i
b =1
n
∑ xbi = 1 ∀b
i =1
xbi ≥ 0 ∀(b, i )

The perfect matching M is clearly feasible for this LP, so it remains

to show that it achieves the optimum. Indeed, we show this by ex-
hibiting a feasible dual solution with value ∑bi∈ M vbi , the value of the
primal solution. Then by weak duality, both these solutions must be
optimal.
The dual linear program is the following:
n n
minimize ∑ pi + ∑ u b
i =1 b =1
subject to pi + ub ≥ vbi ∀bi

(Observe that u and p are unconstrained variables.) In fact, given

any settings of the pi variables, the ub variables that minimize the
objective function, while still satisfying the linear constraints, are
given by ub := maxi∈ I (vbi − pi ), exactly matching (7.4). Hence, the
dual program can then be rewritten as the following (non-linear,
convex) program with no constraints:

min
p=( p1 ,...,pn )
∑ p i + ∑ u b ( p ).
i∈ I b∈ B

Consider the dual solution given by the price vector p∗ . Recall that
M is a perfect matching in the preference graph H ( p∗ ), and let M(i )
be the buyer matched to item i by it. Since u M(i) ( p) = v M(i)i − pi , the
dual objective is

∑ pi∗ + ∑ ub ( p∗ ) = ∑ pi∗ + ∑(v M(i)i − pi ) = ∑ vbi .

i∈ I b∈ B i∈ I i∈ I bi ∈ M

Since the primal and dual values are equal, the primal matching M
must be optimal.

Prices p = ( p1 , . . . , pn ) are said to be market-clearing if each item

can be assigned to some person who has maximum utility for it at
these prices, subject to the constraints of the problem. In our setting,
having such prices are equivalent to having a perfect matching in the
preference graph. Hence, Theorem 7.16 shows that market-clearing
prices give us an optimal matching, so our goal will be to find such
prices.
 graph matchings ii: weighted matchings 97

7.3.2 The Hungarian algorithm

The “Hungarian” algorithm uses the buyers-and-sellers viewpoint
from the previous section. The idea of the algorithm is to iteratively The algorithm was named the Hungar-
ian algorithm by Harold Kuhn, who
change item prices as long as they are not market-clearing, and the
based his ideas on the works of Jenö
key is to show that this procedure terminates. To make our proofs Egervary and Dénes König. Munkres
easier, we assume for now that all the values vbi are integers. subsequently showed that the algorithm
was in fact implementable in O(n3 ).
The price-changing algorithm proceeds as follows: Later, the algorithm was found to have
been proposed even earlier by Carl
1. Initially, all items have price pi = 0. Gustav Jacobi, before 1851.

2. In each iteration, build the current preference graph H ( p). If it

contains a perfect matching M, return it. Theorem 7.16 ensures
that M is an optimal matching.

3. Otherwise, by Hall’s theorem, there exists a set S of buyers such

that if
N (S) := {i ∈ I | ∃b ∈ S, bi ∈ E( H ( p))}
is the set of items preferred by at least one buyer in S, then | N (S)| <
|S|. (N (S) is the neighborhood of S in the preference graph.) Intu-
itively, we have many buyers trying to buy few items, so logically,
the sellers of those items should raise their prices! The algorithm
increases the price of every item in N (S) by 1, and starts a new
iteration by going back to step 2.

That’s it. Running the algorithm on our running example gives the
prices on the right.
The only way the algorithm can stop is to produce an optimal
matching. So we must show it does stop, for which we use a “semi-
invariant” argument. We keep track of the “potential”

Φ( p) := ∑ p i + ∑ u b ( p ),
i b

where pi are the current prices and ub ( p) = maxi (vbi − pi ) as above.

This is just the dual value, and hence is is lower-bounded by the
optimal value of the dual program. (We assume the optimal value of
the LP is finite, e.g., if all the input values are finite.) Then, it suffices
to prove the following:

Lemma 7.17. Every time we increase the prices in N (S) by 1, the value of
∑i pi + ∑b ub decreases by at least 1.

Proof. The value of ∑i pi increases by | N (S)|, because we increase

the price of each item i ∈ N (S) by 1. For each buyer b ∈ S, the
value ub must decrease by 1, since all their preferred items had their
prices increased by 1, and all other items previously had utilities
at least one lower than the original ub ( p). (Here, we used the fact
98 another perspective: buyers and sellers

that all values were integral.) Therefore, the value of the potential
∑i pi + ∑b ub changes by | N ( B)| − | B| ≤ −1.

Hence the algorithm stops in finite time, and produces a maximum-

value perfect matching. By the arguments above ?? we get yet an-
other proof of integrality of the LP ?? for the bipartite pefect match-
ing problem. A few other remarks about the algorithm:

• In fact, one can get rid of the integrality assumption by raising the
prices by the maximum amount possible for the above proof to
still go through, namely


min ub ( p) − max (vib − pi ) .
b∈S i ̸∈ N (S)

It can be shown that this update rule makes the algorithm stop in
only O(n3 ) iterations.

• If all the values are non-negative, and we don’t like the utilities to
be negative, then we can do one of the following things: (a) when
all the prices become non-zero, subtract the same amount from all
of them to make the lowest price hit zero, or (b) choose S to be a
minimal “consticted” set and raise the prices for N (S). This way,
we can ensure that each buyer still has at least one item which
gives it nonngegative utility. (Exercise!)

• Suppose there are n buyers and a single item, with all non-negative
values. (Imagine there are n − 1 dummy items, with buyers hav-
ing zero values for them.) The above algorithm behaves like the
usual ascending-price English or Vickery auction, where prices
are raised until only one bidder remains. Indeed, the final price
for the “real” item will be such that the second-highest bidder is
indifferent between it and a dummy item.
This is a more general phenomenon: indeed, even in the setting
with multiple items, the final prices are those produced by the
Vickery-Clarke-Groves truthful mechanism, at least if we use the
version of the algorithm that raises prices on minimal constricted
sets. The truthfulness of the mechanism means there is no incen-
tive for buyers to unilaterally lie about their values for items. See,
e.g., 1 for the rich connection of matching algorithms to auction 1

theory and (algorithmic) mechanism design.

Check about negative values, they don’t seem to matter at all,

as long as everything is finite. What about max-weight maximum
matching: we can always convert the graph, but does the algorithm
work out of the box?
 graph matchings ii: weighted matchings 99

This proof shows that for any setting of values, there is an optimal
integer solution to the linear program

max{v · x | x ∈ K LP(G) }.

This implies that every vertex x ∗ of the polytope K LP(G) is integral—

indeed, the definition of vertex means x ∗ is the unique solution to the
linear program for some values v, and our proof just produced an
integral matching that is the optimal solution. Hence, we get another
proof of Theorem 7.13, this time using the notion of vertices instead
of extreme points.

7.4 A Third Algorithm: Shortest Augmenting Paths

Let us now see yet another algorithm for solving weighted matching
problems in bipartite graphs. For now, we switch from maximum-
weight matchings to minimum-weight matchings, because they are
conceptually cleaner to explain here. Of course, the two problems are
equivalent, since we can always negate edges.
In fact, we solve a min-cost max-flow problem here: given an flow
network with terminals s and t, edge capacities ue , and also edge
costs/weights we , find an s-t flow with maximum flow value, and
whose total cost/weight is the least among all such flows. (Moreover,
if the capacities are integers, the flow we find will also have integer
flow values on all edges.) Casting the maximum-cardinality bipartite
matching problem as a integer max-flow problem, as in §blah gives
us a minimum-weight bipartite matching.
This algorithm uses an augmenting path subroutine, much like
the algorithm of Ford and Fulkerson. The subroutine, which takes in
a matching M and returns one of size | M | + 1, is presented below.
Then, we can start with the empty matching and call this subroutine
until we get a maximum matching.
Let the original bipartite graph be G. Construct the directed graph
G M as follows: For each edge e ∈ M, insert that edge directed from
right to left, with weight −we . For each edge e ∈ G \ M, insert that
edge directed from left to right, with weight we . Then, compute the
shortest path P that starts from the left and ends on the right, and
return M △ P. It is easy to see that M △ P is a matching of size | M | +
1, and has total weight equal to the sum of the weights of M and P.
Call a matching M an extreme matching if M has minimum
weight among all matchings of size | M |. The main idea is to show
that the above subroutine preserves extremity, so that the final match-
ing must be extreme and therefore optimal.

Theorem 7.18. If M is an extreme matching, then so is M △ P.

100 perfect matchings in general graphs

Proof. Suppose that M is extreme. We will show that there exists

an augmenting path P such that M △ P is extreme. Then, since the
algorithm finds the shortest augmenting path, it will find a path that
is no longer than P, so the returned matching must also be extreme.
Consider an extreme matching M′ of size | M| + 1. Then, the edges
in M △ M′ are composed of disjoint paths and cycles. Since M △ M′
has more edges in M′ than edges in M, there is some path P ⊂ M △
M′ with one more edge in M′ than in M. This path necessarily starts
and ends on opposite sides, so we can direct it to start from the left
and end on the right. We know that | M′ ∩ P| = | M ∩ P| + 1, which
means that M \ P and M′ \ P must have equal size. The total weight of
M\ P and M′ \ P must be the same, since otherwise, we can swap the
two matchings and improve one of M and M′ . Therefore, M △ P =
( M′ ∩ P) ∪ ( M\ P) has the same weight as M′ and is extreme.

Note that the formulation of G M is exactly the graph constructed

if we represent the minimum matching problem as a min-cost flow.
Indeed, the previous theorem can be generalized to a very similar
statement for the augmenting path algorithm for min-cost flows.

7.5 Perfect Matchings in General Graphs

Interestingly, Theorem 7.13 is false for non-bipartite graphs. Indeed,

consider graph K3 which consists of a single 3-cycle: this graph has
no perfect matching, but setting xe = 1/2 for each edge satisfies all the
constraints. This suggests that the linear constraints defining K PM(G)
are not enough, and we need to add more constraints to capture the
convex hull of perfect matchings in general graphs.
In situations like this, it is instructive to look at the counter-
example, to see what constraints must be satisfied by any integer
solution, but are violated by this fractional solution. For a set of ver-
tices S ⊆ V, let ∂(S) denote the edges leaving S. Here is one such set
of constraints:
 
∑ xe ≥ 1 ∀S ⊆ V such that |S| is odd, .
e∈∂(S)

These constraints say: the vertices belonging to a set S ⊆ V of odd

size cannot be perfectly matched within themselves, and at least one
edge from any perfect matching must leave S. Indeed, this constraint
would be violated by S = V (K3 ) in the example above.
Adding these odd-set constraints to the previous degree con-
straints gives us the following polytope, which was originally pro-
posed by Edmonds 2 : 2
; and
 graph matchings ii: weighted matchings 101

  






∑ xvu = 1 ∀v ∈ V 


  ∈ 

  ∂(v)
 u 

KgenPM(G) =

x ∈ R| E | s.t.
 ∑ xe ≥ 1 ∀S s.t. |S| odd


 
 e ∈ ∂(S) 


 
 

  
xe ≥ 0 ∀e ∈ E

Theorem 7.19. For an undirected graph G, we have

K genPM(G) = CgenPM(G) ,

where CgenPM(G) the convex hull of all perfect matchings of G.

One proof is a more sophisticated version of the one in §7.2.3,

where we may now have tight odd-set constraints; we leave it as a
slightly challenging exercise.
This LP potentially contains a exponential number of constraints,
in contrast with the linear number of constraints needed for the bi-
partite case. In fact, a powerful theorem by Thomas Rothvoß (build- Rothvoß (2014,2017)
ing on work by Mihalis Yannakakis) shows that any polytope whose Yannakakis (1991)
vertices are the perfect matchings of the complete graph on n vertices
must contain an exponential number of constraints.
Given this negative result, this LP seems useless: we cannot even
write it down explicitly in polynomial time! It turns out that despite
this large size, it is possible to solve this LP in polynomial time. In
a later lecture, we will see the Ellipsoid method to solve linear pro-
grams. This method can solve such a large LP, provided we give it a
helper procedure called a “separation oracle”, which, given a point
x ∈ R| E| , outputs Yes if x lies is within the desired polytope, and
otherwise it outputs No and returns a violated constraint of this LP.
It is easy to check if x satisfies the degree constraints, so the challenge
here is to find an odd set S with ∑e∈∂(S) xe < 1, if there exists one.
Such an algorithm can be indeed obtained using a sequence of min-
cut computations in the graph, as was shown by. We will see this in a Padberg and Rao (1982)
HW problem later in the course.

7.6 Integrality of Polyhedra

We just saw several proofs that the bipartite perfect matching poly-
tope has a compact linear program. Moreover, we claimed that the
pefect matching polytope on general graphs has an explicit linear
program that, while exponential-sized, can be solved in polynomial
time. Such results allow us to solve the weighted bipartite matching
problems using generic linear programming solvers (as long as they
return vertex solutions).
102 integrality of polyhedra

Having many different ways to view a problem gives us a deeper

insight, and thereby come up with faster and better ways to solve it.
Moreover, these different perspectives give us a handle into solving
extensions of these problems. E.g., if we have a matching problem
with two different kinds weights w1 and w2 on the edges: we want
to find a matching x ∈ K PM(G) minimizing w1 · x, now subject to the
additional constraint w2 · x ≤ B. While the problem is now NP-hard,
this linear constraint can easily be added to the linear program to
get a fractional optimal solution. Then we can reason about how to
“round” this solution to get a near-optimal matching.
We now show how two problems we considered earlier, namely
minimum-cost arborescence and spanning trees, can be exactly mod-
eled using linear programs. We then conclude with a pointer to a
general theory of integral polyhedra.

7.6.1 Arborescences
We already saw a linear program for the min-weight r-arborescence
polytope in §2.3.2: since each node that is not the root r must have a
path in the arborescence to the root, it is natural to say that for any
subset of vertices S ⊆ V that does not contain the root, there must
be an edge leaving it. Specifically, given the digraph G = (V, A), the
polytope can be written as
  

  ∑ x a ≥ 1 ∀S ⊂ Vs.t.r ̸∈ S
 

| A|
K Arb(G) = x ∈ R s.t. a ∈ ∂ + (S)
.

 
x ≥ 0 

a ∀ a ∈ A
Here ∂+ (S) is the set of arcs that leave set S. The proof in §2.3.2 al-
ready showed that for each weight vector w ∈ R| A| , we can find an
optimal solution to the linear program min{w · x | x ∈ K Arb(G) }.

7.6.2 Minimum Spanning Trees

One way to write an LP for minimum spanning trees is to reduce
it to minimum-weight r-arborescences: indeed, replace each edge
by two arcs in opposite directions, each having the same cost. Pick
any node as the root r. Observe the natural bijection between r-
arborescence in this digraph and spanning trees in the original graph,
having the same weight.
But why go via arborescences? Why not directly model the fact
that any tree has at least one undirected edge crossing each cut
(S, V \ S), perhaps as follows:
  

  ∑ xe ≥ 1 ∀S ⊂ V, S ̸= ∅, V
 

| E|
KSTtry = x ∈ R s.t. e ∈ ∂ ( S ) .

 
x ≥ 0 

e ∀e ∈ E
 graph matchings ii: weighted matchings 103

(The first constraint excludes the case where S is either empty or the
entire vertex set.) Sadly, this does not precisely capture the spanning
tree polytope: e.g., for the familiar cycle graph having three vertices,
setting xe = 1/2 for all three edges satisfies all the constraints. If all
edge weights are 1, this solution get a value of ∑e xe = 3/2, whereas
any spanning tree on 3 vertices must have 2 edges.
One can indeed write a different linear program that captures the
spanning tree polytope, but it is a bit non-trivial:
  





 ∑ xij ≤ |S| − 1 ∀S ⊆ V, S ̸= ∅ 



 
 ij∈ E:i,j∈S 

  
KST (G) =

x ∈ R| E | s.t.
 ∑ xij = |V | − 1 

 
 ij∈ E 


 
 

  
xij ≥ 0 ∀ij ∈ E

Define the convex hull of all minimum spanning trees of G to be

CH MST . Then, somewhat predictably, we will again find that CH MST =
K MST .
Both the polytopes for arborescences and spanning trees had ex-
ponentially many constraints. Again, we can solve these LPs if we
are given separation oracles for them, i.e., procedures that take x and
check if it is indeed feasible for the polytope. If it is not feasible, the
oracle should output a violated linear inequality. We leave it as an
exercise to construct separation oracles for the polytopes above.
A different approach is to represent such a polytope K compactly
′
via an extended formulation: i.e., to define a polytope K ′ ∈ Rn+n
using a polynomial number of linear contraints (on the original vari-
′
ables x ∈ Rn and perhaps some new variables y ∈ Rn ) such that
projecting K ′ down onto the original n-dimensions gives us K. I.e.,
we want that
′
K = { x ∈ Rn | ∃y ∈ Rn s.t. ( x, y) ∈ K ′ }.

The homework exercises will ask you to write such a compact ex-
tended formulation for the arborescence problem.

7.6.3 Integrality of Polyhedra

This section still needs work. We have seen that LPs are a powerful
way of formulating problems like min-cost matchings, min-weight
r-aborescences, and MSTs. We reasoned about the structure of the
polytopes that underly the LPs, and we were able to show that these
LPs do indeed solve their combinatorial problems. But notice that
simply forming the LP is not sufficient–significant effort was ex-
pended to show that these polytopes do indeed have integer solu-
tions at the vertices 3 . Without this guarantee, we could get fractional 3
i.e. that the solution vector is integer
valued
104 integrality of polyhedra

solutions to these LPs that do not actually give us solutions to our

problem.
There is a substantial field of study concerned with proving the
integrality of various LPs. We will briefly introduce a matrix property
that implies the integrality of corresponding LPs. Recall that an LP
can be written as
[ A]m×n · ⃗x ≤ ⃗b
where A is a m × n matrix with each row corresponding to a con-
straint, ⃗x is a vector of n variables, and ⃗b ∈ Rm is a vector corre-
sponding to the m scalars bi ∈ R in the constraint A(i) · ⃗x ≤ bi .

Definition 7.20. A matrix [ A]m×n is called totally unimodular if every

square submatrix B of A has the property that det( B) ∈ {0, ±1}

We then have the following neat theorem, due to Hoffman and

Kruskal:

Theorem 7.21 (Hoffman and Kruskal Theorem). If the constraint A.J. Hoffman and J.B. Kruskal (1956)
matrix [ A]m×n is totally unimodular and the vector ⃗b is integral, i.e: ⃗b ∈
Zm , then, the vertices of the polytope induced by the LP are integer valued.

Thus, to show that the vertices are indeed integer valued, one
need not go through producing combinatorial proofs, as we have.
Instead, one could just check that the constraint matrix A is totally
unimodular.
=
8
Graph Matchings III: Algebraic Algorithms

We now introduce some algebraic methods to find perfect match-

ings in general graphs. We use so-called the “polynomial method”, We focus on perfect matchings here; it
based on the elementary fact that low-degree polynomials have few is an exercise to reduce finding maxi-
mum matchings to perfect matchings.
zeroes. This is a powerful and versatile idea, using a combination of Check!
basic algebra and randomness, that can be used to solve many related
problems as well. For instance, we will use it to get parallel (ran-
domized) algorithms for perfect matchings, and also to find red-Blue
perfect matchings, an algorithm for which we know no deterministic
algorithms. But before we digress to these problems, let us discuss
some of the algebraic results for perfect matchings.

• The first result along these lines is that of Laci Lovász, who intro- Lovász (1979)
duced the general idea, and gave a randomized algorithm to detect
the presence of perfect matchings in time O(nω ), and to find it in
time O(mnω ). We will present all the details of this elegant idea
soon.

• Dick Karp, Eli Upfal, and Avi Wigderson, and then Ketan Mulmu- Karp, Upfal, and Wigderson (1986)
ley, Umesh Vazirani, and Vijay Vazirani showed how to find such a Mulmuley, Vazirani, and Vazirani (1987)
matching in parallel. The question of getting a deterministic paral-
lel algorithm remains an outstanding open problem, despite recent
progress (which discuss at the end of the chapter).

• Michael Rabin and Vijay Vazirani sped up the sequential algorithm Rabin and Vazirani (1989)
to run in O(n · nω ). This was substantially improved by the work
of Marcin Mucha and Piotr Sankowski to get a runtime of O(nω ). Mucha and Sankowski (2006)

8.1 Preliminaries: roots of low degree polynomials

For the rest of this lecture, we fix a field F, and consider (univariate
and multivariate) polynomials over this field. We assume that we can
perform basic arithmetic operations in constant time, though some-
times it will be important to look more closely at this assumption. For finite fields Fq (where q is a prime
power), we can perform arithmetic
operations (addition, multiplication,
division) in time poly log q.
106 preliminaries: roots of low degree polynomials

Given p( x ), a root/zero of this polynomial is some value z such

that p(z) evaluates to zero. The critical idea for today’s lecture is
simple: low-degree polynomials have “few” roots. In this section,
we will see this for both univariate and multivariate polynomials, for
the right notion of “few”. The following theorem well-knwon is for
univariate polynomials. (The proof is essentially by induction on the
degree; will add a reference.)

Theorem 8.1 (Univariate Few-Roots Theorem). A univariate polynomial

p( x ) of degree at most d over any field F has at most d roots, unless p( x ) is
zero polynomial.

Now, for multivariate polynomials, the trivial extension of this

theorem is not true. For example, p( x, y) := xy has degree two, and
the solutions to p( x, y) = 0 over the reals are exactly the points in
{( x, y) ∈ R2 : x = 0 or y = 0}, which is infinite. However, the
roots are still “few”, in the sense that the set of roots is very sparse
in R2 . To formalize this observation, let us write a trivial corollary of
Theorem 8.1:

Corollary 8.2. Given a non-zero univariate polynomial p( x ) over a field F,

such that p has degree at most d. Suppose we choose R uniformly at random
from a subset S ⊆ F. Then

d
Pr [ p( R) = 0] ≤ .
|S|
This statement holds for multivariate polynomials as well, as we
see next. The result is called the Schwartz-Zippel lemma, and it ap-
pears in papers by Richard DeMillo and Richard Lipton 1 , by Richard 1

Zippel, and by Jacob Schwartz. Zippel (1979)

Schwartz (1980)
Theorem 8.3. Let p( x1 , . . . , xn ) be a non-zero polynomial over a field F, Like many powerful ideas, the prove-
such that p has degree at most d. Suppose we choose values R1 , . . . , Rn nance of this result gets complicated.
A version of this for finite fields was
independently and uniformly at random from a subset S ⊆ F. Then apparently already proved in 1922 by
Øystein Ore; anyone have a copy of that
d paper?
Pr [ p( R1 , . . . , Rn ) = 0] ≤ .
|S| A monomial is a product a collection of
variables. The degree of a monomial
Hence, the number of roots of p inside Sn is at most d|S|n−1 . is the sum of degrees of the variables
in it. The degree of a polynomial is the
Proof. We argue by induction on n. The base case of n = 1 considers maximum degree of any monomial in
it.
univariate polynomials, so the claim follows from Theorem 8.1. Now
for the inductive step for n variables. Let k be the highest power of
xn that appears in p, and let q be the quotient and r be the remainder
when dividing p by xnk . That is, let q( x1 , . . . , xn−1 ) and r ( x1 , . . . , xn )
be the (unique) polynomials such that

p( x1 , . . . , xn ) = xnk q( x1 , . . . , xn−1 ) + r ( x1 , . . . , xn ),
 graph matchings iii: algebraic algorithms 107

where the highest power of xn in r is less than k. Now letting E be

the event that q( R1 , . . . , Rn−1 ) is zero, we find

Pr [ p( R1 , . . . , Rn ) = 0] = Pr [ p( R1 , . . . , Rn ) = 0 | E] Pr [E ]
   
+ Pr p( R1 , . . . , Rn ) = 0 | E Pr E
 
≤ Pr [E ] + Pr p( R1 , . . . , Rn ) = 0 | E

By the inductive assumption, and noting that q has degree at most

d − k, we know

Pr [E ] = Pr [q( R1 , . . . , Rn−1 ) = 0] ≤ (d − k)/|S|.

Similarly, fixing the values of R1 , . . . , Rn−1 and viewing p as a poly-

nomial only in variable xn (with degree k), we know
 
Pr p( R1 , . . . , Rn ) = 0 | E ≤ k/|S|.

Thus we get

d−k k d
Pr [ p( R1 , . . . , Rn ) = 0] ≤ + = .
|S| |S| |S|

Remark 8.4. Finding the set S ⊆ F such that |S| ≥ dn2 , guarantees that
if p is a non-zero polynomial,

1
Pr [ p( R1 , . . . , Rn ) = 0] ≤ .
n2
Naturally, if p is zero polynomial, then the probability equals 1.

8.2 Detecting Perfect Matchings by Computing a Determinant

Let us solve the easier problem of detecting a perfect matching in

a graph, first for bipartite graphs, and then for general graphs. We
define the Edmonds matrix of a bipartite graph G.

Definition 8.5. For a bipartite graph G = ( L, R, E) with | L| = | R| = n,

its Edmonds matrix E( G ) is the following n × n matrix of indetermi-
nates/variables.

0 if (i, j) ̸∈ E and i ∈ L, j ∈ R 1 1
Ei,j =
 xi,j if (i, j) ∈ E and i ∈ L, j ∈ R.

Example 8.6. The Edmonds matrix of the graph to the right is

2 2
" #
x11 x12
E= , Figure 8.1: Bipartite graph
0 x22

which has determinant x11 x22 .

 108 detecting perfect matchings by computing a determinant

Recall the Leibniz formula to compute the determinant, and apply

it to the Edmonds matrix:
n
det (E( G )) = ∑ (−1)sign(σ) ∏ Ei,σ(i)
σ ∈ Sn i =1

There is a natural correspondence between potential perfect match-

ings in G and permutations σ ∈ Sn , where we match each vertex
i ∈ L to vertex σ (i ) ∈ R. Moreover, the term in the above expansion
corresponding to a permutation σ gives a non-zero monomial (a prod-
uct of xij variables) if and only if all the edges coresponding to that
permutation exist in G. Moreover, all the monomials are distinct, by
construction. This proves the following simple claim.
Proposition 8.7. Let E( G ) denote the Edmonds matrix of a bipartite graph
G. Then det (E( G )) is a non-zero polynomial (over any field F) if and only
if G contains a perfect matching.
However, writing out this determinant of indeterminates could
take exponential time—it would correspond to a brute-forece check
of all possible perfect matchings. Lovász’s idea was to use the ran-
domized algorithm implicit in Theorem 8.3 to test whether G con-
tains a perfect matching.
Algorithm 10: PM-tester(bipartite graph G, S ⊆ F)
10.1 E ← Edmonds matrix for graph G
10.2 For each non-zero entry Eij , sample Rij ∈ S independently and
uniformly at random
10.3 e ← E({ Rij }i,j ) be matrix with sampled values substituted
E
10.4 e ) = 0 then
if det(E
10.5 return G does not have a perfect matching (No)
10.6 else
10.7 return G contains a perfect matching (Yes)

Lemma 8.8. For |S| ≥ n3 , Algorithm 10 always returns No if G has

no perfect matching, while it says Yes with probability at least 1 − n12
otherwise. Moreover, the algorithm can be implemented in time O(nω ),
where ω is the exponent of matrix multiplication.
Proof. The success probability follows from Remark 8.4, and the fact
that the determinant is a polynomial of degree n. Assuming that
arithmetic operations can be done in constant time, we can compute
e in time O(n3 ), using Gaussian elimination.
the determinant of E
2 Hence Algorithm 10 easuly runs in time O (n3 ). In fact, Bunch 2
If we work over finite fields, the size of
the numbers is not an issue. However,
and Hopcroft 3 proved that both computing matrix inverses and
Gaussian Elimination over the rationals
determinants can be done in asymptotically the same time as matrix could cause some of the numbers to get
multiplication. Thus, we can make Algorithm 10 run in time O(nω ). unreasonably large. Ensuring that the
numbers remain polynomially bounded
requires care; see Edmonds’ paper, or a
book on numerical methods.
3
 graph matchings iii: algebraic algorithms 109

8.2.1 Non-bipartite matching

The extension to the non-bipartite case requires a very small change:
instead of using the Edmonds matrix, which is designed for bipartite
graphs, we use the analogous object for general graphs. This object
was introduced by Bill Tutte in his 1947 paper with the Tutte-Berge Tutte (1947)
theorem.

Definition 8.9. For a general graph G = (V, E) with |V | = n, the

Tutte matrix T( G ) of G is the n × n skew-symmetric matrix given by A matrix A is skew-symmetric if
A⊺ = − A.

 if (i, j) ̸∈ E or i = j
0
 1
Ti,j = xi,j if (i, j) ∈ E and i < j



− x j,i if (i, j) ∈ E and i > j.
2 4
Example 8.10. For the graph to the right, the Tutte matrix is
 
0 x1,2 0 x1,4 3
 −x 0 x2,3 x2,4  Figure 8.2: Non-bipartite graph
 1,2 
 
 0 − x2,3 0 x3,4 
− x1,4 − x2,4 − x3,4 0

And its determinant is blah blah.

Observe that now each variable occurs twice, with the variables
below the diagonal being the negations of those above. We claim the
same property for this matrix as we did for the Edmonds matrix:

Theorem 8.11. For any graph G, the determinant of the Tutte matrix T( G )
is a non-zero polynomial over any field F if and only if there exists a perfect
matching in G.

Proof. As before, the determinant is

n
det(T( G )) = ∑(−1)sign(σ) ∏ Ti,σ(i) .
σ i =1

One direction of the theorem is easy: if G has a perfect matching

M, consider the permutation σ mapping each vertex to the other
endpoint of the matching edge containing it. The corresponding
monomial above is ± ∏e∈ M xe2 , which cannot be cancelled by any
other permutation, and makes the determinant non-zero over any
field.
To prove the converse, suppose the determinant is non-zero. In the
monomial corresponding to permutation σ, either each (i, σ (i )) in the
product corresponds to an edge of G, or else the monomial is zero.
This means each non-zero monomial of det(T( G )) chooses an edge
incident to i, for each vertex i ∈ [n], giving us a cycle cover of G.
 110 from detecting to finding perfect matchings

If the cycle cover for σ has an odd-length cycle, take the permuta-
tion σ′ obtained by reversing the order of, say, the first odd cycle in
it. This does not change the sign of the permutation, which depends
on how many odd and even cycles there are, but the skew symmetry
and the odd cycle length means the product of matrix entries flips
sign. Hence the monomials for σ and σ′ cancel each other. Formally,
we get a bijection between permutations with odd-length cycles that
cancel out.
The remaining monomials corresponding to cycle covers with even
cycles. Choosing either the odd edges or even edges on each such
even cycle gives a perfect matching.

Now given Theorem 8.11, the Tutte matrix can simply be substi-
tuted instead of the Edmonds matrix to extend the results to general
graphs.

8.3 From Detecting to Finding Perfect Matchings

We can convert the above perfect matching tester (which solves the
decision version of the perfect matching problem) into an algorithm
for the search version: one that outputs a perfect matching in a graph
(if one exists), using the simple but brilliant idea of self-reducibility. We are reducing the problem to smaller
Suppose that graph G has a perfect matching. Then we can pick any instances of itself, hence the name.

edge e = uv and check if G [ E − e], the subgraph of G obtained by

dropping just the edge e, contains a perfect matching. If not, then
edge e must be part of every perfect matching in G, and hence we can
find a perfect matching on the induced subgraph G [V \ {u, v}]. The
following algorithm is based on this observation.
Algorithm 11: Find-PM(bipartite graph G, S ⊆ F)
11.1 Assume: G has a perfect matching let e = uv be an edge in G if
PM-tester(G [ E − e], S) == Yes then
11.2 return Find-PM(G [ E − e], S)
11.3 else
11.4 M′ ← Find-PM(G [V − {u, v}], S)
11.5 return M′ ∪ {e}

Theorem 8.12. Let |S| ≥ n3 . Given a bipartite graph G that contains some
perfect matching, Algorithm 11 finds a perfect matching with probability at
least 12 , and runs in time O(m · nω ).

Proof. At each step, we call the tester once, and then recurse after
either deleting an edge or two vertices. Thus, the number of total
recursive steps inside Algorithm 11 is at most max{m, n/2} = m, if
the graph is connected. This gives a runtime of O(m · nω ). Moreover,
 graph matchings iii: algebraic algorithms 111

at each step, the probability that the tester returns a wrong answer is
at most n12 , so the PM-tester makes a mistake with probability at most
m
n2
≤ 1/2, by a union bound.

Observe that the algorithm assumes that G contains a perfect

matching. We could simply run the PM-tester once on G at the be-
ginning to check for a perfect matching, and then proceed as above.
Or indeed, we could just run the algorithm regardless; if G has no
perfect matching, there is no danger of this algorithm erronenously
returning one.
Moreover, there are at least two ways reducing the error proba-
bility to 1/nc : we could increase the size of S to ns+3 , or we could
repeat the above algorithm c log2 n times. For the latter approach,
the probability of not getting a perfect matching in all iterations is at
most (1/2)c log2 n = n1c . Hence we get correctness with high probabil-
ity.

Corollary 8.13. Given a bipartite graph G containing a perfect matching,

there is an O(m · nω log n)-time algorithm which finds a perfect matching
with high probability.

Exercise 8.14. Reduce the time-complexity of the basic version of

Algorithm 11 from O(m · nω ) to O(n log n · nω ).

8.3.1 The Algorithm of Rabin and Vazirani

Rewrite this section. How can we speed up the algorithm further?
The improvement we give here is small, it only removes a logarith-
mic term from the algorithm you get from Exercise 8.14, but it has
a nice idea that we want to emphasize. Again, we only focus on the
bipartite case, and leave the general case for the reader. Also, we
identify the nodes of both L and R with [n], so that we can index the
rows and columns of the Edmonds matrix using vertices in L and R
respectively.
We can view Algorithm 11 as searching for a permutation π such
that M := {iπ (i ) | i ∈ [n]} is a perfect matching in G. Hence it picks
a vertex, say 1 ∈ L, and searches for a vertex j ∈ R such that there
is an edge 1j covering vertex 1, and also the remaining graph has a
perfect matching. Interestingly, the remaining graph has an Edmonds
matrix which is simple: it is simply the matrix E−1,− j , which is our
notation from dropping row 1 and column j from E.
Therefore, our task is simple: find a value j such that 1j is an edge
in G, and also det(E−1,− j ) is a non-zero polynomial. Doing this
naively would require n determinant computations, one for each j,
and we’d be back to square one. But the smart observation is to recall
Cramer’s rule for the inverse for any matrix A: Some jargon you may care for, or not:
adjugate( A)
A −1 = ,
det( A)
where adjugate( A) is the transpose of
the cofactor matrix of A, given by
cofactor( A) p,q := (−1) p+q det( A− p,−q ),

where det( A− p,−q ) is also called a minor

of A.
112 from detecting to finding perfect matchings

(−1)i+ j det( A− j,−i )

A −1 i,j
= . (8.1)
det( A)

Take the matrix E e obtained by substituting random values into the

Edmonds matrix E, and assume the set S and field F are of size at
least n10 , say, so that all error probabilities are tiny. Compute its
inverse in time O(nω ), and use (8.1) to get all the values det(E e −1,− j ):

e −1
e −1,− j ) = E
det(E × (−1) j+1 × det(E
e)
j,1

using just n scalar multiplications. In fact, it suffices to find a non-

e −1 , which indicates that det(E
zero E e −1,− j ) is non-zero, and hence
j,1
the corresponding det(E−1,− j ) (without the tilde) is a non-zero poly-
nomial, so that G [V \ {1, j}] has a perfect matching.
In summary, by computing one matrix inverse and n scalar mul-
tiplications, we can figure out one edge of the matching. Hence the
runtime can be made O(n · nω ). Extending this to general graphs
requires a bit more work; we refer to the Rabin and Vazirani paper
for details. Also, Marcin Mucha’s thesis has a very well-written intro-
duction which discusses these details, and also gives the details of his
improvement (with Sankowski) to O(nω ) time.

8.3.2 The Polynomial Identity Testing (PIT) Problem

In Polynomial Identity Testing we are given a polynomial P( x1 , x2 , . . . , xn )

over some field F, and we want to test if it is identically zero or not.
If P were written out explicitly as a list of monomials and their coef-
ficients, this would not be a problem, since we could just check that
all the coefficients are zero. But if P is represented implicitly, say as a
determinant, then things get more tricky. A big question is whether
polynomial identity testing (PIT) can be derandomized.
We don’t know deterministic algorithms that given P can decide
whether P is identically zero or not, in poly-time. How is P given,
for this question? Even if P is given as an arithmetic circuit (a circuit
whose gates are addition, subtraction and multiplication, and inputs
are the variables and constants), it turns out that derandomizing PIT
will result in surprising circuit lower bounds—for example, via a
result of Kabanets and Impagliazzo. Derandomizing special cases
of PIT can be done. For example, just the PIT instances that come
from matchings can, however, be derandomized. This was shown in
work by Jim Geelen and Nick Harvey, among others; however, but
the runtime seems to get much worse.
 graph matchings iii: algebraic algorithms 113

8.4 Red-Blue Perfect Matchings

To illustrate the power of the algebraic approach, let us now consider

the red-blue matching problem. We solve this problem using the al-
gebraic approach and randomization. Interestingly, no determistic
polynomial-time algorithm is currently known for this problem!
Again, we consider bipartite graphs for simplicity.
Given a graph G where each edge is colored either red or blue,
and an integer k, a k-red matching is a perfect matching in G which
has exactly k red edges. The goal of the red-blue matching problem is
to decide whether G has a k-red matching. (We focus on the decision
version of the problem; the self-reducibility ideas used above can
solve the search version.
To begin, let’s solve the case when G has a unique red-blue match-
ing with k red edges. Define the following n × n matrix:


 if (i, j) ̸∈ E,
0

Mi,j = 1 if (i, j) ∈ E and colored blue,



y if (i, j) ∈ E and colored red.

Claim 8.15. Let G have at most one perfect matching with k red edges.
The determinant det(M) has a term of the form ck yk if and only if G
has a k-red matching.

Proof. Consider p(y) := det(M) as a univariate polynomial in the

variable y. Again, using the Leibniz formula, the only way to get a
non-zero term of the form ck yk is if the graph has a k-red matching.
And since we assumed that G has at most one such perfect matching,
such a term cannot be cancelled out by other such matchings.

The polynomial p(y) has degree at most n, and hence we can re-
cover it by Lagrangian interpolation. Indeed, we can choose n + 1
distinct numbers a0 , . . . , an , and evaluate p( a0 ), . . . , p( an ) by comput-
ing the determinant det(M) at y = ai , for each i. These n + 1 values
are enough to determine the polynomial as follows:

n +1  
x − aj
p(y) = ∑ p ( ai ) ∏ ai − a j
.
i =1 j ̸ =i

(E.g., see 451 lecture notes or Ryan’s lecture notes.) Note this is a
completely deterministic algorithm, so far.
114 matchings in parallel, and the isolation lemma

8.4.1 Getting Rid of the Uniqueness Assumption

To extend to the case where G could have many k-red matchings, we
can redefine the matrix as the following:

 if (i, j) ̸∈ E,
0

Mi,j = xij if (i, j) ∈ E and colored blue,



yxij if (i, j) ∈ E and colored red.

The determinant det(M) is now a polynomial in m + 1 variables

and degree at most 2n. Writing
n
P(x, y) = ∑ y i Q i ( x ),
i =0

where Qi is a multilinear degree-n polynomial that corresponds to Multilinear just means that the degree
all the i-red matchings. If we set the x variables randomly (say, to of each variable in each monomial is at
most one.
values xij = aij ) from a large enough set S, we get a polynomial
R(y) = P(a, y) whose only variable is y. The coefficient of yk in this
polynomial is Qk (a), which is non-zero with high probability, by the
Schwartz-Zippel lemma. Now we can again use interpolation to find
out this coefficient, and decide the red-blue matching problem based
on whether it is non-zero.

8.5 Matchings in Parallel, and the Isolation Lemma

One of the “killer applications” of the algebraic method for finding a

perfect matching is that the approach extends to getting a (random-
ized) parallel algorithm as well. The basic idea is simple when there
is a unique perfect matching. Indeed, computing the determinant
can be done in parallel with poly-logarithmic depth and polynomial
work. Hence, for each edge e we can run the PM-tester algorithm
on G, and also on G [ E − e] to see if e belongs to this unique perfect
matching; we output e if it does.
However, this approach fails when the graph has multiple perfect
matchings. A fix for this problem was given by Mulmuley, Vazirani,
and Vazirani 4 by adding further randomness! The approach is first 4

extend the approach from §8.2 to find a minimum-weight perfect

matching using the Tutte matrix and Schwartz-Zippel lemma, as long
as the weights are polynomially bounded. (Exercise: solve this!) The
trickier part is to show that assigning random polynomially-bounded
weights to the edges of G causes it to have a unique minimum-weight
perfect matching with high probability. Then this unique matching
can also be found in parallel, as we outlined in the previous para-
graph.
 graph matchings iii: algebraic algorithms 115

The proof showing that random weights result in a unique minimum-

weight perfect matching is via a beautiful result called the Isolation
Lemma. Let us give its simple elegant proof.

Theorem 8.16. Consider a collection F = { M1 , M2 , . . . , } of sets over a

universe E of size m. Assign a random weight to each elements of E, where
the weights are drawn independently and uniformly from {1, . . . , 2m}.
Then there exists a unique minimum-weight set with probability at least 12 .

Proof. Call an element e ∈ E “confused” if the weight of a minimum-

weight set containing e is the same as the weight of a minimum-
weight set not containing e. We claim that any specific element e is
confused with probability at most 1/2m. Observe is that there exists
a confused element if and only if there are two minimum-weight sets,
so using the claim and taking a union bound over all elements proves
the theorem.
To prove the claim, make the random choices for all elements
except e. Now the identity (and weight) of the minimum-weight We are using the principle of deferred
set not containing e is determined; let its weight be W − . Also, the decisions again.

identity (but not the weight) of the minimum-weight set containing

e is determined. Its weight is not determined because the random
choice for e’s weight has not been made, so denote its weight by
W + + we , where we is the random variable denoting the weight of
e. Now e will be confused precisely if W − = W + + we , i.e., if we =
W − − W + . But since we is chosen uniformly at random from a set of
size 2m, this probability is at most 1/2m, as claimed.

It is remarkable the result does not depend on number of sets in

F , but only on the size of the universe. We also emphasize that the
weights being drawn from a polynomially-sized set is what gives the
claim its power: it is trivial to obtain a unique minimum-weight set if
the weights are allowed to be in {1, . . . , 2m }. (Exercise: how?) Finally,
the proof strategy for the Isolation Lemma is versatile and worth
remembering.

8.5.1 Towards Deterministic Algorithms

The question of finding perfect matchings deterministically in poly-
logarithmic depth and polynomial work still remains open. Some
recent work of Fenner, Gurjar, and XXXX, and of Svensson and Tar-
nawski has shown how to obtain poly-logarithmic depth and quasi-
polynomial work. We will see some ideas in a HW.

8.6 A Matrix Scaling Approach

Talk about Matrix Scaling approach?

 Part II

The Curse of
Dimensionality, and
Dimension Reduction
9
Concentration of Measure

Consider the following questions:

1. You distribute n tasks among n machines, by sending each task

to a machine uniformly and independently at random: while any
machine has unit expected load, what is the maximum load (i.e.,
the maximum number of tasks assigned to any machine)?

2. You want to estimate the bias p of a coin by repeatedly flipping it

and then taking the sample mean. How many samples suffice to
be within ±ε of the correct answer p with confidence 1 − δ?

3. How many unit vectors can you choose in Rn that are almost


orthonormal? I.e., they must satisfy | vi , v j | ≤ ε for all i ̸= j?

4. A n-dimensional hyercube has N = 2n nodes. Each node i ∈ [ N ]

contains a packet pi , which is destined for node πi , where π is a
permutation. The routing happens in rounds. At each round, each
packet traverses at most one edge, and each edge can transmit at
most one packet. Find a routing policy where each packet reaches
its destination in O(n) rounds, regardless of the permutation π.

All these questions can be answered by the same basic tool, which
goes by the name of Chernoff bounds or concentration inequalities
or tail inequalities or concentration of measure, or tens of other
names. The basic question is simple: if we have a real-valued function
f ( X1 , X2 , . . . , Xm ) of several independent random variables Xi , such that it
is “not too sensitive to each coordinate”, how often does it deviate far from
its mean? To make it more concrete, consider this—
Given n independent random variables X1 , . . . , Xn , each bounded in
the interval [0, 1], let Sn = ∑in=1 Xi . What is
h i
Pr Sn ̸∈ (1 ± ε)ESn ?

This question will turn out to have relations to convex geometry,

to online learning, to many other areas. But of greatest interest to
120 asymptotic analysis

us, this question will solve many problems in algorithm analysis,

including the above four. Let us see some basic results, and then give
the answers to the four questions.

9.1 Asymptotic Analysis

We will be concerned with non-asymptotic analysis, i.e., the qualitative

behavior of sums (and other Lipschitz functions) of finite number of
(bounded) independent random variables. Before we begin that, a
few words about the asymptotic analysis, which concerns the conver-
gence of averages of infinite sequences of random variables.
Given a sequence of random variables { Xn } and another random
variable Y, the following two notions of convergence can be defined.
Definition 9.1 (Convergence in Probability). { Xn } converges in prob-
ability to Y if for every ϵ > 0 we have
lim P(| Xn − Y | > ϵ) = 0 (9.1)
n→∞
p
This is denoted by Xn −
→ Y.
Definition 9.2 (Convergence in Distribution). Let FX (.) denote the
CDF of a random variable X. { Xn } converges in distribution to Y if
lim FXn (t) = FY (t) (9.2)
n→∞
for all points t where the distribution function FY is continuous. This
d
is denoted by Xn −
→ Y.
There are many results known here, and we only mention the two
well-known results below. The weak law of large numbers states that
the average of independent and identically distributed (i.i.d.) random
variables converges in probability to their mean.
Theorem 9.3 (Weak law of large numbers). Let Sn denote the sum of n
i.i.d. random variables, each with mean µ and variance σ2 < ∞, then
p
Sn/n −
→ µ. (9.3)
The central limit theorem tells us about the distribution of the
sum of a large collection of i.i.d. random variables. Let N (0, 1) denote
the standard normal variable with mean 0 and variance 1, whose
2
probability density function is f ( x ) = √1 exp(− x2 ).
2π
Theorem 9.4 (Central limit theorem). Let Sn denote the sum of n i.i.d.
random variables, each with mean µ and variance σ2 < ∞, then
Sn − nµ d
√ −
→ N (0, 1). (9.4)
nσ
There are many powerful asymptotic results in the literature; see
need to give references here.
 concentration of measure 121

9.2 Non-Asymptotic Convergence Bounds

Our focus will be on the behavior of finite sequences of random

variables. The central question here will be: what is the chance of
deviating far from the mean? Given an r.v. X with mean µ, and some
deviation λ > 0, the quantity

Pr[ X ≥ µ + λ]

is called the upper tail, and the analogous quantity

Pr[ X ≤ µ − λ]

is the lower tail. We are interested in bounding these tails for various
values of λ.

9.2.1 Markov’s inequality

Most of our results will stem from the most basic of all results:
Markov’s inequality. This inequality qualitatively generalizes that
idea that a random variable cannot always be above its mean, and
gives a bound on the upper tail.

Theorem 9.5 (Markov’s Inequality). Let X be a non negative random

variable and λ > 0, then

E( X )
P( X ≥ λ ) ≤ (9.5)
λ
With this in hand, we can start substituting various non-negative
functions of random variables X to deduce interesting bounds. For
instance, the next inequality looks at both the mean µ := EX and the
variance σ2 := E[( X − µ)2 ] of a random variable, and bounds both
the upper and lower tails.

9.2.2 Chebychev’s Inequality

Theorem 9.6 (Chebychev’s inequality). For any random variable X with
mean µ and variance σ2 , we have

σ2
Pr[| X − µ| ≥ λ] ≤ .
λ2
Proof. Using Markov’s inequality on the non-negative r.v. Y = ( X −
µ)2 , we get

E [Y ]
Pr[Y ≥ λ2 ] ≤ .
λ2
The proof follows from Pr[Y ≥ λ2 ] = Pr[| X − µ| ≥ λ].
122 non-asymptotic convergence bounds

9.2.3 Examples I
Example 1 (Coin Flips): Let X1 , X2 , . . . , Xn be i.i.d. Bernoulli random
variables with Pr[ Xi = 0] = 1 − p and Pr[ Xi = 1] = p. (Im other
words, these are the outcomes of independently flipping n coins,
each with bias p.) Let Sn := ∑in Xi be the number of heads. Then Sn is
distributed as a binomial random variable Bin(n, p), with Recall that linearity of expectations for
r.v.s X, Y means E[ X + Y ] = E[ X ] +
E[Y ]. For independent we have Var[ X +
E[Sn ] = np and Var[Sn ] = np(1 − p). Y ] = Var[ X ] + Var[Y ].

Applying Markov’s inequality for the upper tail gives

pn 1
Pr[Sn − pn ≥ βn] ≤ = .
pn + βn 1 + ( β/p)

So, for p = 1/2, this is 1+12β ≈ 1 − O( β) for small values of β > 0.

However, Chebychev’s inequality gives a much tighter bound:

np(1 − p) p
Pr[|Sn − pn| ≥ βn] ≤ 2 2
< 2 .
β n β n

In particular, this already says that the sample mean Sn /n lies in the
p
interval p ± β with probability at least 1 − β2 n . Equivalently, to get Concretely, to get within an additive
p p 1% error of the correct bias p with
confidence 1 − δ, we just need to set δ ≥ β2 n
, i.e., take n ≥ β2 δ
. (We probability 99.9%, set β = 0.01 and
will see a better bound soon.) δ = 0.001, so taking n ≥ 107 · p samples
suffices.

Example 2 (Balls and Bins): Throw n balls uniformly at random and

independently into n bins. Then for a fixed bin i, let Li denote the
number of balls in it. Observe that Li is distributed as a Bin(n, 1/n)
random variable. Markov’s inequality gives a bound on the probabil-
ity that Li deviates from its mean 1 by λ ≫ 1 as

  1 1
Pr Li ≥ 1 + λ ≤ ≈ .
1+λ λ

However, Chebychev’s inequality gives a much tighter bound as

h i (1 − 1/n) 1
Pr | Li − 1| ≥ λ ≤ 2
≈ 2.
λ λ
√
So setting λ = 2 n says that the probability of any fixed bin having
√ (1−1/n)
more than 2 n + 1 balls is at most 4n . Now a union bound over Doing this argument with Markov’s
(1−1/n) inequality would give a trivial upper
all bins i means that, with probability at least n · 4n ≤ 1/4, the bound of 1 + 2n on the load. This is
√
load on every bin is at most 1 + 2 n. useless, since there are at most n balls,
so the load can never be more than n.

Example 3 (Random Walk): Suppose we start at the origin and at

each step move a unit distance either left or right uniformly ran-
domly and independently. We can then ask about the behaviour of
 concentration of measure 123

the final position after n steps. Each step (Xi ) can be modelled as a
Rademacher random variable with the following distribution.

1 w.p. 12
Xi =
−1 w.p. 1
2

The position after n steps is given by Sn = ∑in=1 Xi , with mean and

variance being µ = 0 and σ2 = n respectively. Applying Chebyshev’s
√
inequality on Sn with deviation λ = tσ = t n, we get
h √ i 1
Pr Sn > t n ≤ 2 . (9.6)
t
We will soon see how to get a tighter tail bound.

9.2.4 Higher-Order Moment Inequalities

All the bounds in the examples above can be improved by using
higher-order moments of the random variables. The idea is to use the
same recipe as in Chebychev’s inequality.

Theorem 9.7 (2kth -Order Moment inequalities). Let k ∈ Z≥0 . For any
random variable X having mean µ, and finite moments upto order 2k, we
have
E(( X − µ)2k )
Pr[| X − µ| ≥ λ] ≤ .
λ2k
Proof. The proof is exactly the same: using Markov’s inequality on
the non-negative r.v. Y := ( X − µ)2k ,

E [Y ]
Pr[| X − µ| ≥ λ] = Pr[Y ≥ λ2k ] ≤ .
λ2k
We can get stronger tail bounds for large values of k, however
it becomes increasingly tedious to compute E(( X − µ)2k ) for the
random variables of interest.

Example 3 (Random Walk, continued): If we consider the fourth

moment of Sn :
  h n i
E ( S n ) 4 = E ∑ Xi
i =1
h i
=E ∑ Xi4 + 4 ∑ Xi3 Xj + 6 ∑ Xi2 X2j + 12 ∑ Xi2 X j Xk + 24 ∑ Xi X j X k X l
i i< j i< j i < j<k i < j<k<l
 
n
= n+6 ,
2

where we crucially used that the r.v.s are independent and mean-
zero, hence terms like Xi3 X j , Xi2 X j Xk , and Xi X j Xk Xl all have mean
124 chernoff bounds, and hoeffding’s inequality

zero. Now substituting this expectation in the fourth-order moment

√
inequality, we get a stronger tail bound for λ = tσ = t n.
 
 √  E ( Sn )4 n + 6(n2 ) Θ (1)
Pr |Sn | ≥ t n ≤ = = 4 . (9.7)
t4 n2 t4 n2 t
Compare this with the bound in (9.6).

9.2.5 Digression: The Right Answer for Random Walks

We can actually explicitly computing Pr(Sn = k) for sums of Rademacher
random variables. Indeed, we just need to choose the positions for +1
steps, which means
n
Pr[Sn = 2λ] (n+ )
2 λ
= n .
Pr[Sn = 0] (n)
2
√
For large n, we can use Stirling’s formula n! ≈ 2πn( ne )n :

Pr[Sn = 2λ] ( n2 )n/2 ( n2 )n/2 1

≈ n = 2λ n2 +λ n −λ
Pr[Sn = 0] ( 2 + λ)(n/2+λ) ( n2 − λ)(n/2−λ) (1 + n ) (1 − 2λ
n )
2

λ
If λ ≪ n, then we can approximate 1 + k λn by ek n :
Pr[Sn = 2λ] −2λ n 2λ n −4λ2
≈ e n ( 2 +λ) e n ( 2 −λ) = e n .
Pr[Sn = 0]
√
Finally, substituting λ = tσ = t n, we get
2
Pr[Sn = 2λ] ≈ Pr[Sn = 0] · e−4t .

This shows that most of the probability mass lies in the region |Sn | ≤
√
O( n), and drops off exponentially as we go further. And indeed,
this is the bound we will derive next—we will get slightly weaker
constants, but we will avoid these tedious approximations.

9.3 Chernoff bounds, and Hoeffding’s inequality

The main bound of this section is a bit of a mouthful, but as Ryan

O’Donnell says in his notes, you should memorize it “like a poem”. I
find it lies in a sweet spot: it is not difficult to remember, and still is
very broadly applicable: The provenance of these bounds is
again quite complicated. There’s Her-
Theorem 9.8 (Hoeffding’s inequality). Let X1 , . . . , Xn be n independent man Chernoff’s paper, which derives
random variables taking values in [0, 1]. Let Sn := ∑in=1 Xi , with mean the corresponding inequality for i.i.d.
Bernoulli random variables. Wassily
µ := E[Sn ] = ∑i E[ Xi ]. Then for any β ≥ 0 we have Hoeffding gives the generalization
  for independent random variables all
  β2 µ taking values in some bounded interval
Upper tail : Pr Sn ≥ µ(1 + β) ≤ exp − . (9.8)
2+β [ a, b]. Though Chernoff attributes his
  result to another Herman, namely Her-
  β2 µ man Rubin. There’s Harald Cramér (of
Lower tail : Pr Sn ≤ µ(1 − β) ≤ exp − . (9.9)
3 the Cramér-Rao fame, not of Cramer’s
rule). And there’s the bound by Sergei
Bernstein, many years earlier, which is
at least as strong. . .
 concentration of measure 125

Proof. We only prove (9.8); the proof for (9.9) is similar. The idea is to
use Markov’s inequality not on the square or the fourth power, but
on a function which is fast-growing enough so that we get tighter
bounds, and “not too fast” so that we can control the errors. So we
consider the Laplace transform, i.e., the function

x 7→ etx

for some value t > 0 to be chosen carefully. Since this map is mono-
tone,

Pr[Sn ≥ µ(1 + β)] = Pr[etSn ≥ etµ(1+ β) ]

E[etSn ]
≤ (using Markov’s inequality)
etµ(1+ β)
∏i E[etXi ]
= tµ (using independence) (9.10)
e (1+ β )

Bernoulli random variables: Assume that all the Xi ∈ {0, 1}; we will
remove this assumption later. Let the mean be µi = E[ Xi ], so the
moment generating function can be explicitly computed as

E[etXi ] = 1 + µi (et − 1) ≤ exp(µi (et − 1)).

Substituting, we get

∏i E[etXi ]
Pr[Sn ≥ µ(1 + β)] ≤ (9.11)
etµ(1+ β)
∏ exp(µi (et − 1))
≤ i (9.12)
etµ(1+ β)
exp(µ(et − 1))
≤
etµ(1+ β)
(since µ = ∑ µi )
i
t
= exp(µ(e − 1) − tµ(1 + β)). (9.13)

Since this calculation holds for all positive t, and we want the tightest
upper bound, we should minimize the expression (9.13). Setting the
derivative w.r.t. t to zero gives t = ln(1 + β) which is non-negative for
β ≥ 0. This bound on the upper tail is also
one to be kept in mind; it often is
 µ
eβ useful when we are interested in large
Pr[Sn ≥ µ(1 + β)] ≤ . (9.14) deviations where β ≫ 1. One such
(1 + β )1+ β example will be the load-balancing
application with jobs and machines.
We’re almost there: a slight simplification is that
β
≤ ln(1 + β) (9.15)
1 + β/2
for all β ≥ 0, so
 
(9.15) − β2 µ
(9.13) = exp(µ( β − (1 + β) ln(1 + β))) ≤ exp ,
2+β
126 chernoff bounds, and hoeffding’s inequality

with the last inequality following from simple algebra. This proves
the upper tail bound (9.8); a similar proof gives us the lower tail as
well.
Removing the assumption that Xi ∈ {0, 1}: If the r.v.s are not Bernoullis,
then we define new Bernoulli r.v.s Yi ∼ Bernoulli(µi ), which take
value 0 with probability 1 − µi , and value 1 with probability µi , so
that E[ Xi ] = E[Yi ]. Note that f ( x ) = etx is convex for every value
of t ≥ 0; hence the function ℓ( x ) = (1 − x ) · f (0) + x · f (1) satisfies
f ( x ) ≤ ℓ( x ) for all x ∈ [0, 1]. Hence E[ f ( Xi )] ≤ E[ℓ( Xi )]; moreover
ℓ( x ) is a linear function so E[ℓ( Xi )] = ℓ(E[ Xi ]) = E[ℓ(Yi )], since
Xi and Yi have the same mean. Finally, ℓ(y) = f (y) for y ∈ {0, 1}.
Putting all this together,

E[etXi ] ≤ E[etYi ] = 1 + µi (et − 1) ≤ exp(µi (et − 1)),

so the step from (9.11) to (9.12) goes through again. This completes
the proof of Theorem 9.8.

Since the proof has a few steps, let’s take stock of what we did:
i. Markov’s inequality on the function etX ,
ii. independence and linearity of expectations to break into etXi ,
iii. reduction to the Bernoulli case Xi ∈ {0, 1},
iv. compute the MGF (moment generating function) E[etXi ],
v. choose t to minimize the resulting bound, and
vi. use convexity to argue that Bernoullis are the “worst case”. Do make sure you see why the bounds
You can get tail bounds for other functions of random variables of Theorem 9.8 are impossible in
general if we do not assume some kind
by varying this template around; e.g., we will see an application for of boundedness and independence.
sums of independent normal (a.k.a. Gaussian) random variables in
the next chapter.

9.3.1 The Examples Again: New and Improved Bounds

Example 1 (Coin Flips): Since each r.v. is a Bernoulli( p), the sum
Sn = ∑i Xi has mean µ = np, and hence
   β2 n   β2 n 
Pr |Sn − np| ≥ βn ≤ exp − ≤ exp − .
2p + β 2
(For the second inequality, we use that the interesting settings have
2 ln(1/δ)
p + β ≤ 1.) Hence, if n ≥ β2
, the empirical average Sn /n is
within an additve β of the bias p with probability at least 1 − δ. This
has an exponentially better dependence on 1/δ than the bound we
obtained from Chebychev’s inequality.
This is asymptotically the correct answer: consider the problem
where we have n coins, n − 1 of them having bias 1/2, and one having
bias 1/2 + 2β. We want to find the higher-bias coin. One way is to es-
1
timate the bias of each coin to within β with confidence 1 − 2n , using
 concentration of measure 127

the procedure above—which takes O(log n/ε2 ) flips per coin—and

Ω(n log n)
then take a union bound. It turns out any algorithm needs ε2
flips, so this the bound we have is tight. .

Example 2 (Load Balancing): Since the load Li on any bin i behaves

like Bin(n, 1/n), the expected load is 1. Now (9.8) says:
 
β2
Pr[ Li ≥ 1 + β] ≤ exp − .
2+β

If we set β = Θ(log n), the probability of the load Li being larger than
1 + β is at most 1/n2 . Now taking a union bound over all bins, the
probability that any bin receives at least 1 + β balls is at most n1 . I.e.,
the maximum load is O(log n) balls with high probability.
In fact, the correct answer is that the maximum load is (1 +
o (1)) lnlnlnnn with high probability. For example, the proofs in cite show
this. Getting this precise bound requires a bit more work, but we can
get an asymptotically correct bound by using (9.14) instead, with a
C ln n
setting of β = ln ln n with a large constant C.
Moreover, this shows that the asymmetry in the bounds (9.8)
and (9.9) is essential. A first reaction would have been to believe The situation where β ≪ 1 is often
our proof to be weak, and to hope for a better proof to get called the Gaussian regime, since the
bound on the upper tail behaves like
exp(− β2 µ). In other cases, the upper
Pr[Sn ≥ (1 + β)µ] ≤ exp(− β2 µ/c) tail bound behaves like exp(− βµ), and
is said to be the Poisson regime.
for some constant c > 0, for all values of β. This is not possible,
p
however, because it would imply a max-load of Θ( log n) with high
probability.

Example 3 (Random Walk): In this case, the variables are [−1, 1]

valued, and hence we cannot apply the bounds from Theorem 9.8
directly. But define Yi = 1+2Xi to get Bernoulli(1/2) variables, and
define Tn = ∑in=1 Yi . Since Tn = Sn /2 + n/2, In general, if Xi takes values in [ a, b],
X −a
we can define Yi := bi−a and then use
 √   √ 
Pr |Sn | ≥ t n = Pr | Tn − n/2| ≥ (t/2) n Theorem 9.8.
 
(t2 /n) · (n/2)
≤ 2 exp − √ using (9.8)
2 + t/n
≤ 2 exp(−t2 /6).

Recall from §9.2.5 that the tail bound of ≈ exp(−t2 /O(1)) is indeed
in the right ballpark.

9.4 Other concentration bounds

Many of the extensions address the various assumptions of Theo-

rem 9.8: that the variables are bounded, that they are independent,
128 other concentration bounds

and that the function Sn is the sum of these r.v.s. Add details and refs
to this section.
But before we move on, let us give the bound that Sergei Bern-
stein gave in the 1920s: it uses knowledge about the variance of the
random variable to get a potentially sharper bound than Theorem 9.8

Theorem 9.9 (Bernstein’s inequality). Consider n independent random

variables X1 , . . . , Xn with | Xi − E[ Xi ]| ≤ 1 for each i. Let Sn := ∑i Xi
have mean µ and variance σ2 . Then for any λ ≥ 0 we have
 
λ2
Pr[|Sn − µ| ≥ λ] ≤ 2 exp − 2 .
2σ + 2λ/3

9.4.1 Mildly Correlated Variables

The only place we used independence in the proof of Theorem 9.8
was in (9.10). So if we have some set of r.v.s where this inequality
holds even without independence, the proof can proceed unchanged.
Indeed, one such case is when the r.v.s are negatively correlated.
Loosely speaking, this means that if some variables are “high” then
it makes more likely for the other variables to be “low”. Formally,
X1 , . . . , Xn are negatively associated if for all disjoint sets A, B and
for all monotone increasing functions f , g, we have

E[ f ( Xi : i ∈ A) · g( X j : j ∈ B)] ≤ E[ f ( Xi : i ∈ A)] · E[ g( X j : j ∈ B)].

We can use this in the step (9.10), since the function etx is monotone
increasing for t > 0.
Negative association arises in many settings: say we want to
choose a subset S of k items out of a universe of size n, and let
Xi = 1i∈S be the indicator for whether the ith item is selected. The
variables X1 , . . . , Xn are clearly not independent, but they are nega-
tively associated.

9.4.2 Martingales
A different and powerful set of results can be obtained when we
stop considering random variables are not independent, but al-
low variables X j to take on values that depend on the past choices
X1 , X2 , . . . , X j−1 but in a controlled way. One powerful formalization
is the notion of a martingale. A martingale difference sequence is a se-
quence of r.v.s Y1 , Y2 , . . . , Yn , such that E[Yi | Y1 , . . . , Yi−1 ] = 0 for each
i. (This is true for mean-zero independent r.v.s, but may be true in
other settings too.)

Theorem 9.10 (Hoeffding-Azuma inequality). Let Y1 , Y2 , . . . , Yn be a

martingale difference sequence with |Yi | ≤ ci for each i, for constants ci .
 concentration of measure 129

Then for any t ≥ 0,

" # !
 n  λ2
Pr  ∑ Yi  ≥ λ ≤ 2 exp − .
i =1 2 ∑in=1 c2i

For instance, applying the Azuma-Hoeffding bounds to the ran-

dom walk in Example 3, where each Yi is a Rademacher r.v. gives
√ 2
Pr[|Sn | ≥ t n] ≤ 2e−t /8 , which is very similar to the bounds we
derived above. But we can also consider, e.g., a “bounded” random
walk that starts at the origin, say, and stops whenever it reaches ei-
ther −ℓ or +r. In this case, the step size Yi = 0 with unit probability
if ∑ij− 1
=1 Yj ∈ {−ℓ, r }, else it is {±1} independently and uniformly at
random.

9.4.3 Going Beyond Sums of Random Variables

The Azuma-Hoeffding inequality can be used to bound functions of
X1 , . . . , Xn other than their sum—and there are many other bounds
for more general classes of functions. In all these cases we want any
single variable to affect the function only in a limited way—i.e., the
function should be Lipschitz. One popular packaging was given by
Colin McDiarmid:

Theorem 9.11 (McDiarmid’s inequality). Consider n independent r.v.s

X1 , . . . , Xn , with Xi taking values in a set Ai for each i, and a function
f : ∏ Ai → R satisfying | f ( x ) − f ( x ′ )| ≤ ci whenever x and x ′ differ only
in the ith coordinate. Let µ := E[ f ( X1 , . . . , Xn )] be the expected value of the
random variable f ( X ). Then for any non-negative β,
!
2µ2 β2
Upper tail : Pr[ f ( X ) ≥ µ(1 + β)] ≤ exp −
∑i c2i
!
2µ2 β2
Lower tail : Pr[ f ( X ) ≤ µ(1 − β)] ≤ exp −
∑i c2i

This inequality does not assume very much about the function,
except it being ci -Lipschitz in the ith coordinate; hence we can also
use this to the truncated random walk example above, or for many
other applications.

9.4.4 Moment Bounds vs. Chernoff-style Bounds

One may ask how moment bounds relate to Chernoff-Hoeffding
bounds: Philips and Nelson 1 showed that bounds obtained using 1

this approach of bounding the moment-generating function are never

stronger than moment bounds:
130 application: oblivious routing on the hypercube

Theorem 9.12. Consider n independent random variables X1 , . . . , Xn , each

with mean 0. Let Sn = ∑ Xi . Then

E[ X k ] E[etX ]
Pr[Sn ≥ λ] ≤ min ≤ inf
k ≥0 λk t≥0 etλ

9.4.5 Matrix-Valued Random Variables

Finally, an important line of research considers concentration for
vector-valued and matrix valued functions of independent (and
mildly dependent) r.v.s. One object that we will see in a homework,
and also in later applications, is the matrix-valued case: here the no-
tation A ⪰ 0 means the matrix is positive-semidefinite (i.e., all its
eigenvalues are non-negative), and A ⪰ B means A − B ⪰ 0. See, e.g.,
the lecture notes by Joel Tropp!

Theorem 9.13 (Matrix Chernoff bounds). Consider n independent

symmetric matrices X1 , . . . , Xn of dimension d. Moreover, I ⪰ Xi ⪰ 0 for
each i, i.e., the eigenvalues of each matrix are between 0 and 1. If µmax :=
λmax (∑ E[ Xi ]) is the largest eigenvalue of their expected sum, then
   

γ2
Pr λmax ∑ Xi ≥ µmax + γ ≤ d exp −
2µmax + γ
.

As an example, if we are throwing n balls into n bins, then we

can let matrix Xi have a single 1 at position ( j, j) if the ith ball falls
into bin j, and zeros elsewhere. Now the sum of these matrices has
the loads of the bins on the diagonal, and the maximum eigenvalue
is precisely the highest load. This bound therefore gives that the
2
probability of a bin with load 1 + γ is at most n · eγ /(2+γ) —again
implying a maximum load of O(log n) with high probability.
But we can use this for a lot more than just diagonal matrices
(which can be reasoned about using the scalar-valued Chernoff
bounds, plus the naïve union bound). Indeed, we can sample edges
of a graph at random, and then talk about the eigenvalues of the
resulting adjacency matrix (or more interestingly, of the resulting
Laplacian matrix) using these bounds. We will discuss this in a later
chapter.

9.5 Application: Oblivious Routing on the Hypercube

Now we return to fourth application mentioned at the beginning of

the chapter. (The first two applications have already been considered
above, the third will be covered as a homework problem.)
The setting is the following: we are given the d-dimensional hyper-
cube Qd , with n = 2d vertices. We have n = 2d vertices, each labeled
 concentration of measure 131

with a d-bit vector. Each vertex i has a single packet (which we also
call packet i), destined for vertex π (i ), where π is a permutation on
the nodes [n].
Packets move in synchronous rounds. Each edge is bi-directed,
and at most one packet can cross each directed edge in each round.
Moreover, each packet can cross at most one edge per round. So if
uv ∈ E( Qd ), one packet can cross from u to v, and one from v to u,
in a round. Each edge e has an associated queue; if several packets
want to cross e in the same round, only one can crosse, and the rest
wait in the queue, and try again the next round. (So each node has
d queues, one for each edge leaving it.) We assume the queues are
allowed to grow to arbitrary size (though one can also show queue
length bounds in the algorithm below). The goal is to get a simple
routing scheme that delivers the packets in O(d) rounds.
One natural proposal is the bit-fixing routing scheme: each packet
i looks at its current position u, finds the first bit position where u
differs from π (i ), and flips the bit (which corresponds to traversing
an edge out of u). For example:

0001010 → 1001010 → 1101010 → 1100010 → 1100011.

However, this proposal can create “congestion hotspots” in the net-

work, and therefore delay some packets by 2Ω(d) : see example on
Piazza. In fact, it turns out any deterministic oblivious strategy (that
does not depend
p on the actual sources and destinations) must have a
delay of Ω( 2d /d) rounds.

9.5.1 A Randomized Algorithm. . .

Here’s a great randomized strategy, due to Les Valiant, and to Valiant Valiant (1982)
and Brebner. It requires no centralized control, and is optimal in
the sense of requiring O(d) rounds (with high probability) on any
permutation.
Each node i picks a randomized midpoint Ri independently and uni-
formly from [n]: it sends its packet to Ri . Then after 5d rounds have
elapsed, the packets proceed to their final destinations π (i ). All routing
is done using bit-fixing.

9.5.2 . . . and its Analysis

Theorem 9.14. The random midpoint algorithm above succeeds in deliver-
ing the packets in at most 10d rounds, with probability at least 1 − n2 .
Proof. We only prove that all packets reach their midpoints by time
5d, with high probability. The argument for the second phase is then
identical. Let Pi be the bit-fixing path from i to the midpoint Ri . The
following claim is left as an exercise:
132 application: oblivious routing on the hypercube

Claim 9.15. Any two paths Pi and Pj intersect in one contiguous

segment.
Since Ri is chosen uniformly at random from {0, 1}d , the labels of i
and Ri differ in d/2 bits in expectation. Hence Pi has expected length
d/2. There are d2d = dn (directed) edges, and all n = 2d paths behave
symmetrically, so the expected number of paths Pj using any edge e
is n·dn
d/2
= 1/2. Now define

S(i ) = { j | path Pj shares an edge with Pi }.

Claim 9.16. Packet i reaches the midpoint by time at most d + |S(i )|.

Proof. This is a clever, cute argument. Let Pi = ⟨e1 , e2 , . . . , eℓ ⟩. Say

that a packet in {i } ∪ S(i ) that wants to cross edge ek at the start of
round t has lag t − k. Hence packet i reaches Ri at time equal to the
length of Pi , plus its lag just before it crosses the last edge eℓ . We now
show that if i’s lag increases from L to L + 1 at some point, then some
packet leaves the path Pi (forever, because of Claim 9.15) with final
lag L at some future point in time. Indeed, if i’s lag increased from
L to L + 1 at edge ek , then some packet crossed ek instead and its lag
was L. Now either this packet leaves path Pi with lag L, or else it is
delayed at some subsequent edge on Pi (and the edge traversing that
edge has lag L).
Hence each increase in i’s lag L → L + 1 can be charged to some
packet in S(i ) that eventually leaves Pi with lag L; this bounds the
maximum delay by | Pi | + |S(i )| ≤ d + |S(i )|.

Claim 9.17. Pr[|S(i )| ≥ 4d] ≤ e−2d .

Proof. If Xij is the indicator of the event that Pi and Pj intersect, then
|S(i )| = ∑ j̸=i Xij , i.e., it is a sum of a collection of independent {0, 1}-
valued random variables. Now conditioned on any choice of Pi
(which is of length at most d), the expected number of paths using
each edge in it is at most 1/2, so the conditional expectation of S(i ) is
at most d/2. Since this holds for any choice of Pi , the unconditional
expectation µ = E[S(i )] is also at most d/2. Now apply the Chernoff
bound to S(i ) with βµ = 4d − µ and µ ≤ d/2 to get
 
(4d − µ)2
Pr[|S(i )| ≥ 4d] ≤ exp − ≤ e−2d .
2µ + (4d − µ)

Note that we could apply the bound even though the variables Xij
were not i.i.d., and moreover we did not need estimates for E[ Xij ],
just an upper bound for their expected sum.

Now applying a union bound over all n = 2d packets i means

that all n packets reach their midpoints within d + 4d steps with
 concentration of measure 133

probability 1 − 2d · e−2d ≥ 1 − e−d ≥ 1 − 1/n. Similarly, the second

phase has a probability at most 1/n of failing to complete in 5d steps,
completing the proof.

9.5.3 Graph Sparsification

10
Dimension Reduction and the JL Lemma

For a set of n points { x1 , x2 , . . . , xn } in RD , can we map them into

some lower dimensional space Rk and still maintain the Euclidean
distances between them? We can always take k ≤ n − 1, since any set
of n points lies on a n − 1-dimensional subspace. And this is (exis-
tentially) tight, e.g., if x2 − x1 , x3 − x1 , . . . , xn − x1 are all orthogonal
vectors.
But what if we were fine with distances being approximately pre-
served? There can only be k orthogonal unit vectors in Rk , but there
are as many as exp(cε2 k) unit vectors which are ε-orthogonal—i.e.,
whose mutual inner products all lie in [−ε, ε]. Near-orthogonality al-
lows us to pack exponentially more vectors! (Indeed, we will see this
in a homework exercise.)
This near-orthogonality of the unit vectors means that distances
are also approximately preserved. Indeed, for any two a, b ∈ Rk ,

∥ a − b∥22 = ⟨ a − b, a − b⟩ = ⟨ a, a⟩ + ⟨b, b⟩ − 2⟨ a, b⟩ = ∥ a∥22 + ∥b∥22 − 2⟨ a, b⟩,

so the squared Euclidean distance between any pair of the points

defined by these ε-orthogonal vectors falls in the range 2(1 ± ε). So,
if we wanted n points at exactly the same (Euclidean) distance from
each other, we would need n − 1 dimensions. (Think of a triangle in
2-dims.) But if we wanted to pack in n points which were at distance Having n ≥ exp(cε2 k ) vectors in d
(1 ± ε) from each other, we could pack them into dimensions means the dimension is
k = O(log n/ε2 ).
 
log n
k = O ε2

dimensions.

10.1 The Johnson Lindenstrauss lemma

The Johnson Lindenstrauss “flattening” lemma says that such a claim

is true not just for equidistant points, but for any set of n points in
Euclidean space:
136 the construction

Lemma 10.1. Let ε ∈ (0, 1/2). Given any set of points

 X=  { x1 , x2 , . . . , x n }
log n
in R , there exists a map A : R → R with k = O ε2
D D k such that

∥ A( xi ) − A( x j )∥22
1−ε ≤ ≤ 1 + ε.
∥ xi − x j ∥22
Moreover, such a map can be computed in expected poly(n, D, 1/ε) time.
Note that the target dimension k is independent of the original
dimension D, and depends only on the number of points n and the
accuracy parameter ε. Given n points with Euclidean distances
It is not difficult to show that we need at least Ω(log n) dimen- in (1 ± ε), the balls of radius 1− 2
ε

around these points must be mutually

sions in such a result, using a packing argument. Noga Alon showed disjoint, by the minimum distance,
log n

a lower bound of Ω ε2 log 1/ε , and then Kasper Green Larson and and they are contained within a ball
of radius (1 + ε) + 1− ε
2 around x0 .
log n

Jelani Nelson showed a tight and matching lower bound of Ω ε2 Since volumes of balls in Rk of radius r
dimensions for any dimensionality reduction scheme from n dimen- behave like ck r k , we have
 1 − ε k  3 + ε k
sions that preserves pairwise distances. n · ck ≤ ck
2 2
The JL Lemma was first considered in the area of metric embed-
or k ≥ Ω(log n) for ε ≤ 1/2.
dings, for applications like fast near-neighbor searching; today we
Alon (2003)
use it to speed up algorithms for problems like spectral sparsification Larson and Nelson (2017)
of graphs, and solving linear programs fast.

10.2 The Construction

The JL lemma is pretty surprising, but the construction of the map

is perhaps even more surprising: it is a super-simple randomized
construction. Let M be a k × D matrix, such that every entry of M is
filled with an i.i.d. draw from a standard normal N (0, 1) distribution
(a.k.a. the “Gaussian” distribution). For x ∈ RD , define
1
A( x ) = √ Mx.
k
That’s it. You hit the vector x with a Gaussian matrix M, and scale it
√
down by k. That’s the map A.
Since A( x ) is a linear map and satisfies αA( x ) + βA(y) = A(αx +
βy), it is enough to show the following lemma:
Lemma 10.2. [Distributional Johnson-Lindenstrauss] Let ε ∈ (0, 1/2).
If A is constructed as above with k = cε−2 log δ−1 , and x ∈ RD is a unit
vector, then
Pr[ ∥ A( x )∥22 ∈ 1 ± ε] ≥ 1 − δ.
To prove Lemma 10.1, set δ = 1/n2 , and hence k = O(ε−2 log n).
Now for each xi , x j ∈ X, use linearity of A(·) to infer

A ( x i ) − A ( x j ) 2 A ( x i − x j ) 2 2
2 = 2 = A(vij ) ∈ (1 ± ε)
xi − x j xi − x j
 dimension reduction and the jl lemma 137

with probability at least 1 − 1/n2 , where vij is the unit vector in the
direction of xi − x j . By a union bound, all (n2 ) pairs of distances in
( X2 ) are maintained with probability at least 1 − (n2 ) n12 ≥ 1/2. A few
comments about this construction:

• The above proof shows not only the existence of a good map, we
also get that a random map as above works with constant prob-
ability! In other words, a Monte-Carlo randomized algorithm
for dimension reduction. (Since we can efficiently check that the
distances are preserved to within the prescribed bounds, we can
convert this into a Las Vegas algorithm.) Or we can also get deter-
ministic algorithms: see here.

• The algorithm (at least the Monte Carlo version) is data-oblivious:

it does not even look at the set of points X: it works for any set X
with high probability. Hence, we can pick this map A before the
points in X arrive.

10.3 Intuition for the Distributional JL Lemma

Let us recall some basic facts about Gaussian distributions. The prob-
ability density function for the Gaussian N (µ, σ2 ) is
( x − µ )2
f (x) = √1 e 2σ2 .
2πσ

We also use the following; the proof just needs some elbow grease. The fact that the means and the vari-
ances take on the claimed values should
Proposition 10.3. If G1 ∼ N (µ1 , σ12 ) and G2 ∼ N (µ2 , σ22 ) are indepen- not be surprising; this is true for all
r.v.s. The surprising part is that the
dent, then for c ∈ R, resulting variables are also Gaussians.

c G1 ∼ N (cµ1 , c2 σ12 ) (10.1)

G1 + G2 ∼ N (µ1 + µ2 , σ12 + σ22 ). (10.2)

Now, here’s the main idea in the proof of Lemma 10.2. Imagine
that the vector x is the elementary unit vector e1 = (1, 0, . . . , 0). Then
M e1 is just the first column of M, which is a vector with independent
and identical Gaussian values.
    
G1,1 G1,2 ···
G1,D 1 G1,1
    
 G2,1 G2,2 ···
G2,D  0  G2,1 
M e1 = 
 .. .. ..
 
..   ..  =  .. 
 
.
 . . ..  .  . 
Gk,1 Gk,2 · · · Gk,D 0 Gk,1
√
A( x ) is a scaling-down of this vector by k: every entry in this
random vector A( x ) = A(e1 ) is distributed as
√
1/ k · N (0, 1) = N (0, 1/k) (by (10.1)).
138 the direct proof of Lemma 10.2

Thus, the expected squared length of A( x ) = A(e1 ) is If G has mean µ and variance σ2 , then
" # E[ G2 ] = Var[ G ] + E[ G ]2 = σ2 + µ2 .
h i k k h i k
1
E ∥ A( x )∥ = E ∑ A( x )2i = ∑ E A( x )2i =
2
∑k = 1.
i =1 i =1 i =1

So the expectation of ∥ A( x )∥2 is 1; the heart is in the right place!

Now to show that ∥ A( x )∥2 does not deviate too much from the
mean—i.e., to show a concentration result. Indeed, ∥ A( x )∥2 is a sum
of independent N (0, 1/k )2 random variables, so if these N (0, 1/k)2
variables were bounded, we would be done by the Chernoff bounds
of the previous chapter. Sadly, they are not. However, their tails are
fairly “thin”, so if we squint hard enough, these random variables
can be viewed as “pretty much bounded”, and the Chernoff bounds
can be used.
Of course this is very vague and imprecise. Indeed, the Laplace
distribution with density function f ( x ) ∝ e−λ| x| for x ∈ R also has
pretty thin tails—“exponential tails”. But using a matrix with Laplace
entries does not work the same, no matter how hard we squint. It
turns out you need the entries of M, the matrix used to define A( x ),
to have “sub-Gaussian tails”. The Gaussian entries have precisely this
property.
We now make all this precise, and also remove the assumption
that the vector x = e1 . In fact, we do this in two ways. First we
give a direct proof: it has several steps, but each step is elementary,
and you are mostly following your nose. The second proof formally
defines the notion of sub-Gaussian random variables, and builds
some general machinery for concentration bounds.

10.4 The Direct Proof of Lemma 10.2

Recall that we want to argue about the squared length of A( x ) ∈ Rk ,

where A( x ) = √1 Mx, and x is a unit vector. To start off, observe that
k
the ith coordinate of the vector Mx is the inner product of a row of M
with the vector x. This is distributed as

Yi ∼ ⟨ G1 , G2 , . . . , GD ⟩ · x = ∑ x j Gj
j

where the Gj ’s are the i.i.d. N (0, 1) r.v.s on the ith row of M. Now
Proposition 10.3 tells us that Yi ∼ N (0, x12 + x22 + . . . + x2D ). Since x is
a unit length vector, we get

Yi ∼ N (0, 1).

So, each of the k coordinates of Mx behaves just like an independent

Gaussian!
 dimension reduction and the jl lemma 139

10.4.1 The Expectation

Given the observation above, the squared length of A( x ) = √1 Mx is
k

k
1
Z := ∥ A(z)∥2 = ∑ k · Yi2
i =1

where each Yi ∼ N (0, 1), independent of the others. And since

E[Yi2 ] = Var(Yi ) + E[Yi ]2 = 1, we get E[ Z ] = 1.

10.4.2 Concentration about the Mean

Now to show that Z does not deviate too much from 1. And Z is the
sum of a bunch of independent and identical random variables. Let’s
start down the usual path for a Chernoff bound, for the upper tail,
say: The easy way out is to observe that
the squares of Gaussians are chi-
Pr[ Z ≥ 1 + ε] ≤ Pr[etkZ ≥ etk(1+ε) ] ≤ E[etkZ ]/etk(1+ε) (10.3) squared r.v.s, the sum of k of them
  is χ2 with k degrees of freedom, and
= ∏ E[etYi ]/et(1+ε)
2
(10.4) the internet conveniently has tail
bounds for these things. But if you
i
don’t recall these facts, and don’t have
2 internet connectivity and cannot check
for every t > 0. Now E[etG ], the moment-generating function for G2 , Wikipedia, things are not that difficult.
where G ∼ N (0, 1) is easy to calculate for t < 1/2:
Z Z
1 2 2 /2 1 2 /2 dz 1
√ etg e− g dg = √ e−z √ = √ .
2π g ∈R 2π z ∈R 1 − 2t 1 − 2t
(10.5)

So our current bound on the upper tail is that for all t ∈ (0, 1/2) we
have
 k
1
Pr[ Z ≥ (1 + ε)] ≤ √ .
et(1+ε) 1 − 2t
Let’s just focus on part of this expression:
   
1 1
√ = exp −t − log(1 − 2t)) (10.6)
et 1 − 2t 2
 
= exp (2t)2 /4 + (2t)3 /6 + · · · (10.7)
 
≤ exp t2 (1 + 2t + 2t2 + · · · ) (10.8)

= exp(t2 /(1 − 2t)).

Plugging this back, we get

 k
1
Pr[ Z ≥ (1 + ε)] ≤ √
et(1+ε) 1 − 2t
2 /8
≤ exp(kt2 /(1 − 2t) − ktε) ≤ e−kε ,
140 subgaussian random variables

if we set t = ε/4 and use the fact that 1 − 2t ≥ 1/2 for ε ≤ 1/2. (Note:
this setting of t also satisfies t ∈ (0, 1/2), which we needed from our
previous calculations.)
Almost done: let’s take stock of the situation. We observed that
∥ A( x )∥22 was distributed like an average of squares of Gaussians, and
by a Chernoff-like calculation we proved that

Pr[∥ A( x )∥22 > 1 + ε] ≤ exp(−kε2 /8) ≤ δ/2

for k = ε82 ln 2δ . A similar calculation bounds the lower tail, and

finishes the proof of Lemma 10.2.
The JL Lemma was first proved by Bill Johnson and Joram Linden-
strauss. There have been several proofs after theirs, usually trying to Johnson and Lindenstrauss (1982)
tighten their results, or simplify the algorithm/proof (see citations in
some of the newer papers): the proof above is some combinations of
those by Piotr Indyk and Rajeev Motwani, and Sanjoy Dasgupta and Indyk and Motwani (1998)
myself. Dasgupta and Gupta (2004)

10.5 Subgaussian Random Variables

While Gaussians have all kinds of nice properties, they are real-
valued and hence require more randomness to generate. What other
classes of r.v.s could give us bounds that are comparable? E.g., what
about setting each Mij ∈ R {−1, +1}? A random sign is also called a
It turns out that Rademacher r.v.s also suffice, and we can prove Rademacher random variable, the
name Bernoulli being already taken for
this with some effort. But instead of giving a proof from first princi- a random bit in {0, 1}.
ples, let us abstract out the process of proving Chernoff-like bounds,
and give a proof using this abstraction.
Recall the basic principle of a Chernoff bound: to bound the upper
tail of an r.v. V with mean µ, we can choose any t ≥ 0 to get

Pr[V − µ ≥ λ] = Pr[et(V −µ) ≥ etλ ] ≤ E[et(V −µ) ] · e−tλ .

Now if we define the (centered) log-MGF of V as

ψ(t) := ln E[et(V −µ) ],

we get that for any t ≥ 0,

Pr[V − µ ≥ λ] ≤ e−(tλ−ψ(t)) .

The best upper bound is obtained when the expression tλ − ψ(t) is

the largest. The Legendre dual of the function ψ(t) is defined as Exercise: if ψ1 (t) ≥ ψ2 (t) for all t ≥ 0,
then ψ1∗ (λ) ≤ ψ2∗ (λ) for all λ.
ψ∗ (λ) := sup t ≥ 0 {tλ − ψ(t)},

so we get the concise statement for a generic Chernoff bound: Bounds for the lower tail follow from
the arguments applied to the r.v. − X.
 dimension reduction and the jl lemma 141

Pr[V − µ ≥ λ] ≤ exp(−ψ∗ (λ)). (10.9)

This abstraction allows us to just focus on bounds on the dual log-
MGF function ψ∗ (λ), making the arguments cleaner.

10.5.1 A Couple of Examples

Let’s do an example: suppose V ∼ N (µ, σ2 ), then
Z x2
1 −
E[ e t (V − µ ) ] = √ etx e 2σ2 dx
2πσ x ∈R
Z 2 )2
1 2 σ2 /2 − ( x−tσ2 2 σ2 /2
= √ et e 2σ dx = et . (10.10)
2πσ x ∈R

Hence, for N (µ, σ2 ) r.v.s, we have

t2 σ 2 λ2
ψ(t) = and ψ∗ (λ) = ,
2 2σ2
the latter by basic calculus. Now the generic Chernoff bound for says
that for normal N (µ, σ2 ) variables,
− λ2
Pr[V − µ ≥ λ] ≤ e 2σ2 . (10.11)
How about a Rademacher {−1, +1}-valued r.v. V? The MGF is
et + e−t t2 t4 2
E[ e t (V − µ ) ] = = cosh t = 1 + + + · · · ≤ et /2 ,
2 2! 4!
so
t2 λ2
ψ(t) = and ψ∗ (λ) = .
2 2
Note that
∗
ψRademacher (t) ≤ ψN (0,1) (t) =⇒ ψRademacher (λ) ≥ ψ∗N (0,1) (λ).
This means the upper tail bound for a single Rademacher is at least
as strong as that for the standard normal.

10.5.2 Defining Subgaussian Random Variables

Definition 10.4. A random variable V with mean µ is subgaussian
2 2
with parameter σ if ψ(t) ≤ σ 2t .
By the generic Chernoff bound (10.9), such an r.v. has tails that are
smaller than those of a normal r.v. with variance σ2 . The following
fact is an analog of Proposition 10.3.
Lemma 10.5. If V1 , V2 , . . . are independent
q and σi -subgaussian, and
x1 , x2 , . . . are reals, then V = ∑i xi Vi is ∑i xi2 σi2 -subgaussian.
Proof.
E[et(V −µ) ] = E[et ∑i xi (Vi −µi ) ] = ∏ E[etxi (Vi −µi ) ] = ∏ etxi (Vi −µi ) .
i i
t2 x 2 σ 2
Now taking logarithms, ψV (t) = ∑i ψVi (txi ) ≤ ∑i 2i i .
142 jl matrices using rademachers and subgaussian-ness

10.6 JL Matrices using Rademachers and Subgaussian-ness

Suppose we choose each Mij ∈ R {−1, +1} and let A( x ) = √1 Mx

k
again? We want to show that
 2
1 k D
Z := ∥ A( x )∥ = ∑ 2
∑ Mij · x j . (10.12)
k i =1 j =1

has mean ∥ x ∥2 , and is concentrated sharply around that value.

10.6.1 The Expectation

To keep subscripts to a minimum, consider the inner sum for index i
in (10.12), which looks like
 
Yi := ∑ M j · x j . (10.13)
j

where M j s are pairwise independent, with mean zero and unit variance.
h i h i
E Yi2 = E (∑ M j x j )(∑ Ml xl )
j l
h i
=E ∑ M2j x2j + ∑ Mj Ml x j xl
j j̸=l
h h i i
= ∑ E M2j x2j + ∑ E M j Ml x j xl = ∑ x2j .
j j̸=l j

Here E[ M2j ] = Var( M j ) + E[ M j ]2 = 1, and moreover E[ M j Ml ] =

E[ M j ]E[ Ml ] = 0 by pairwise independence. Plugging this into (10.12),

1 k 1 k
E[ Z ] = ∑
k i =1
E[Yi2 ] = ∑ ∑ x2j = ∥ x ∥22 .
k i =1 j
(10.14)

So the expectation is what we want!

10.6.2 Concentration
Now to show concentration: the direct proof from §10.4 showed
the Yi s were themselves Gaussian with variance ∥ x ∥2 . Since the
Rademachers are 1-subgaussian, Lemma 10.5 shows that Yi is sub-
gaussian with parameter ∥ x ∥2 . Next, we need to consider Z, which is
the average of squares of k independent Yi s. The following lemma
shows that the MGF of squares of symmetric σ-subgaussians are
bounded above by the corresponding Gaussians with variance σ2 .
An r.v. X is symmetric if it is dis-
tributed the same as R| X |, where R is
Lemma 10.6. If V is symmetric mean-zero σ-subgaussian r.v., and W ∼ an independent Rademacher.
2 2
N (0, σ2 ), then E[etV ] ≤ E[etW ] for t > 0.
 dimension reduction and the jl lemma 143

Proof. Using the calculation in (10.10) in the “backwards” direction

2
√
2t(|V |/σ ) W
EV [etV ] = EV,W [e ].

(Note that we’ve just introduced W into the mix, without any provo-
cation!) Since W is also symmetric, we get |V |W = V |W |. Hence,
rewriting
√ √
2t(|V |/σ ) W 2t|W |/σ )V
EV,W [e ] = EW [EV [e( ]],

we can use the σ-subgaussian behavior of V in the inner expectation

to get an upper bound of
2(
√
2t|W |/2)2 /2 2
EW [eσ ] = EW [etW ].

Excellent. Now the tail bound for sums of squares of symmetric

mean-zero σ-subgaussians follows from that of Gaussians. Hence we
get the same tail bounds as in §10.4.2, and hence that the Rademacher
matrix also has the distributional JL property, while using far fewer
random bits!
In general one can use other σ-subgaussian distributions to fill
the matrix M—using σ different than 1 may require us to rework the
proof from §10.4.2 since the linear terms in (10.6) don’t cancel any
more, see works by Indyk and Naor or Matousek for details. Indyk and Naor (2008)
Matoušek (2008)

10.6.3 The Fast JL Transform

A different direction to consider is getting fast algorithms for the
JL Lemma: Do we really need to plug in non-zero values into every
entry of the matrix A? What if most of A is filled with zeroes? The
first problem is that if x is a very sparse vector, then Ax might be
zero with high probability? Achlioptas showed that having a random
two-thirds of the entries of A being zero still works fine: Nir Ailon
and Bernard Chazelle showed that if you first hit x with a suitable Ailon and Chazelle
matrix P which caused Px to be “well-spread-out” whp, and then
∥ APx ∥ ≈ ∥ x ∥ would still hold for a much sparser A. Moreover, this
P requires much less randomness, and furthermore, the computa-
tions can be done faster too! There has been much work on fast and
sparse versions of JL: see, e.g., this paper from SOSA 2018 by Michael
Cohen, T.S. Jayram, and Jelani Nelson. Jelani Nelson also has some Cohen, Jayram, and Nelson (2018)
notes on the Fast JL Transform.

10.7 Optional: Compressive Sensing

To rewrite. In an attempt to build a better machine to take MRI scans,

we decrease the number of sensors. Then, instead of the signal x we
144 optional: compressive sensing

intended to obtain from the machine, we only have a small num-

ber of measurements of this signal. Can we hope to recover x from
the measurements we made if we make sparsity assumptions on x?
We use the term r-sparse signal for a vector with at most r nonzero
entries.
Formally, x is a n-dimensional vector, and a measurement of x
with respect to a vector a is a real number given by ⟨ x, a⟩. The ques-
tion we want to answer is how to reconstruct x with r nonzero entries
satisfying Ax = b if we are given k × n matrix A and n dimensional
vector b. This is often written as
n o
min ∥ x ∥0 | Ax = b.

Here the ℓ0 “norm” is the total number of non-zeros in the vector x.

Unfortunately, it turns out that the problem as formulated is
NP-hard: but this is only assuming A and b are contrived by an ad-
versary. Our setting is a bit different. x is some r-sparse signal out
there that we want to determine. We have a handle over A and can
choose it to be any matrix we like, and we are provided with appro-
priate b = Ax, from which we attempt to reconstruct x.
Consider the following similar looking problem called the basis
pursuit (BP) problem:
n o
min ∥ x ∥1 | Ax = b.

This problem can be formulated as a linear program as follows,

and hence can be efficiently solved. Introduce n new variables
y1 , y2 , . . . , yn under the constraints
n o
min ∑ yi | Ax = b, −yi ≤ xi ≤ yi .
i

Definition 10.7. We call a matrix A as BP-exact if for all b = Ax such

that x ∗ is an r-sparse solution, x ∗ is also the unique solution to basis
pursuit.

Call a distribution D over k × n matrices a distributional JL family

if Lemma 10.2 is true when A is drawn from D .

Theorem 10.8 (Donoho, Candes-Tao).

 we pick A ∈ Rk× D from a dis-
 If 
tributional JL family with k ≥ Ω r log D
r , then with high probability
A is BP-exact.

r
We note that the r log Dr comes from log ( Dr) ≈ log Dr =
D


r log r . The last ingredient that one would use to show Theo-
rem 10.8 is the Restricted Isometry Property (RIP) of such a matrix A.

Definition 10.9. A matrix A is (t, ε)-RIP if for all unit vectors x with
∥ x ∥0 ≤ t, we have ∥ Ax ∥22 ∈ [1 ± ε].
 dimension reduction and the jl lemma 145

See Chapter 4 of Ankur Moitra’s book for more on compressed

sensing, sparse recovery and basis pursuit. 10.8 comes from this
paper by Emmanuel Candes and Terry Tao.

10.8 Some Facts about Balls in High-Dimensional Spaces

Consider the unit ball Bd := { x ∈ Rd | ∥ x ∥2 ≤ 1}. Here are two

facts, whose proofs we sketch. These sketches can be made formal
(since the approximations are almost the truth), but perhaps the style
of arguments are more illuminating.

Theorem 10.10 (Heavy Shells). At least 1 − ε of the mass of the unit ball
log 1/ε
in Rd lies within a Θ( d )-width shell next to the surface.

Proof. (Sketch) The volume of a radius-r ball in Rd goes as r d , so the

fraction of the volume not in the shell of width w is (1 − w)d ≈ e−wd ,
log 1/ε
which is ε when w ≈ d .

Given any hyperplane H = { x ∈ Rd | a · x = b} where ∥ a∥ = 1, the

width-w slab around it is K = { x ∈ Rd | b − w ≤ a · x ≤ b + w}.

Theorem 10.11 (Heavy Slabs). At least (1 − ε) of the mass of the unit ball
√
in Rd lies within Θ(1/ d) slab around any hyperplane that passes through
the origin.

Proof. (Sketch) By spherical symmetry we can consider the hyper-

plane { x1 = 0}. The volume of the ball within {−w ≤ x1 ≤ w} is
at Z w q Z w
2 d −1
( 1 − y2 )d−1 dy ≈ e−y · 2 dy.
y =0 y =0
4
If we define σ2 = d −1 , this is
Z w y2
−
e 2σ2 dy ≈ Pr[ G ≤ w],
y =0

2 2
where G ∼ N (0, σ2 ). But we know that Pr[ G ≥ w] ≤ e−w /2σ by
our generic Chernoff bound for Gaussians (10.11). So setting that tail
probability to be ε gives
q r
 log(1/ε) 
2
w ≈ 2σ log(1/ε) = O .
d

This may seem quite counter-intuitive: that 99% of the volume

of the sphere is within O(1/d) of the surface, yet 99% is within
√
O(1/ d) of any central slab! This challenges our notion of the ball Figure 10.1: Sea Urchin (from uncom-
moncaribbean.com)
“looking like” the smooth circular object, and more like a very spiky
sea-urchin. Finally, a last observation:
146 some facts about balls in high-dimensional spaces

Corollary 10.12. If we pick two random vectors from the surface of the unit
ball in Rd (i.e., from the sphere), then they are nearly orthogonal
q with high
log(1/ε)
probability. In particular, their dot-product is smaller than O( d )
with probability 1 − ε.

Proof. Fix u. Then the dot-product |u · v| ≤ w if v lies in the slab

of width w around the hyperplane { x · u = 0}. Now using Theo-
rem 10.11 completes the argument.

This means that if we pick n random vectors in Rd , and

q set ε =
log n
1/n2 , a union bound gives that all have dot-product O( d ). Set-
2
ting this dot-product to ε gives us n = exp(ε d) unit vectors with
mutual dot-products at most ε, exactly as in the calculation at the
beginning of the chapter.
11
Streaming Algorithms

We now consider a slightly different computational model called

the data streaming model. In this model we see elements going past
in a “stream”, and we have very little space to store things. For ex-
ample, we might be running a program on an Internet router with
limited space, and the elements might be IP Addresses. We certainly
don’t have space to store all the elements in the stream. The ques-
tion is: which functions of the input stream can we compute with
what amount of time and space? While we focus on space, similar
questions can be asked for update times.
We denote the stream elements by

a1 , a2 , a3 , . . . , a t , . . .

We assume each stream element is from alphabet U, and takes b =

| log2 U | bits to represent. For example, the elements might be 32-bit
integers IP addresses. We imagine we are given some function, and
we want to compute it continually, on every prefix of the stream. Let
us denote a[1:t] = ⟨ a1 , a2 , . . . , at ⟩. For example, if we have seen the
integers:

3, 1, 17, 4, −9, 32, 101, 3, −722, 3, 900, 4, 32, . . . (11.1)

1. Can we compute the sum of all the integers seen so far? I.e.,
F ( a[1:t] ) = ∑it=1 ai . We want the outputs to be

3, 4, 21, 25, 16, 48, 149, 152, −570, −567, 333, 337, 369, . . .

If we have seen T numbers so far, the sum is at most T2b and

hence needs at most O(b + log T ) space. So we can just keep
a counter, and when a new element comes in, we add it to the
counter.

2. How about the maximum of the elements so far? F ( a[1:t] ) =

maxit=1 ai . Even easier. The outputs are:

3, 1, 17, 17, 17, 32, 101, 101, 101, 101, 900, 900, 900
148 streams as vectors, and additions/deletions

We just need to store b bits.

3. The median? The outputs on the various prefixes of (11.1) now are

3, 1, 3, 3, 3, 3, 4, 3, . . .

And doing this will small space is a lot more tricky.

4. (“distinct elements”) Or the number of distinct numbers seen so

far? We’d want to output:

1, 2, 3, 4, 5, 6, 7, 7, 8, 8, 9, 9, 9 . . .

5. (“heavy hitters”) Or the elements that have appeared most often so

far? Hmm...

We can imagine the applications of the data-stream model. An

Internet router might see a lot of packets whiz by, and may want to
figure out which data connections are using the most space? Or how
many different connections have been initiated since midnight? Or
the median (or the 90th percentile) of the file sizes that have been
transferred. Which IP connections are “elephants” (say the ones that
have used more than 0.01% of our bandwidth)? Even if we are not
working at “line speed”, but just looking over the server logs, we Such a router might see tens of millions
may not want to spend too much time to find out the answers, we of packets per second.

may just want to read over the file in one quick pass and come up
with an answer. Such an algorithm might also be cache-friendly. But
how to do this?
Two of the recurring themes will be:

1. Approximate solutions: in several cases, it will be impossible to

compute the function exactly using small space. Hence we’ll ex-
plore the trade-offs between approximation and space.

2. Hashing: this will be a very powerful technique.

11.1 Streams as Vectors, and Additions/Deletions

An important abstraction will be to view the stream as a vector (in

high dimensional space). Since each element in the stream is an el-
ement of the universe U, we can imagine the stream at time t as a
vector xt ∈ Z|U | . Here

xt = ( x1t , x2t , . . . , x|tU | )

and xit is the number of times the ith element in U has been seen until
time t. (Hence, xi0 = 0 for all i ∈ U.) When the next element comes in
and it is element j, we increment x j by 1.
 streaming algorithms 149

This brings us a extension of the model: we could have another

model where each element of the stream is either a new element,
or an old element departing. Formally, each time we get an update In data stream jargon, the addition-only
at , it looks like (add, e) or (del, e). We usually assume that for each model is called the cash-register model,
whereas the model with both additions
element, the number of deletes we see for it is at most the number of and deletions is called the turnstile
adds we see — the running counts of each element is non-negative. model. I will not use this jargon.
As an example, suppose the stream looks like:

(add, A), (add, B), (add, A), (del, B), (del, A), (add, C ), . . .

and if A is the first element of U, then the first coordinate x1 of the

vector s would be 1, 1, 2, 2, 1, 1, . . .. This vector notation allows us to
formulate some of the problems more easily:

1. The total number of elements currently in the system is just

|U |
∥x∥1 := ∑i=1 xi . (This is easy.)

2. We might want to estimate the norms ∥x∥2 , ∥x∥ p of the vector x.

3. The number of distinct elements is the number of non-zero entries

in x is denoted by ∥x∥0 .

Let’s consider the (non-trivial) problems one by one.

11.2 Computing Moments

Recall that xt was the vector of frequencies of elements seen so far.

Several interesting problems can be posed as computing various
norms of xt : in particular the Euclidean or 2-norm
v
u |U |
u
∥x ∥2 = t ∑ ( xit )2 ,
t
i =1

and the 0-norm (which is not really a norm)

∥xt ∥0 := number of non-zeroes in xt .

Henceforth, we use the notation that F0 := ∥xt ∥0 is the number of

non-zero entries in x. For p ≥ 1, we consider the p-moment, that is,
the pth -power of the p-norm:

|U |
Fp := ∑ (xit ) p . (11.2)
i =1

We’ll develop an algorithm to compute F2 , and to compute F0 ; we

may see extensions from F2 to Fp in the homeworks.
150 computing moments

11.2.1 Computing the Second Moment F2

The “second moment” F2 of the stream is often called the “surprise

number” (since it captures how uneven the data is). This is also
the size of the self-join. Clearly we can store the entire vector x and
compute F2 , but that requires storing |U | counts. Here’s an algorithm
that uses much less space: This estimator is often called the “tug-
of-war” estimator: the hash function
Pick a random hash function h : U → {−1, +1} from family H . randomly partitions the elements into
two parties (those mapping to 1, and
Maintain counter C, which starts off at zero.
those to −1), and the counter keeps the
On update ( add, i ) ∈ U , increment the counter C → C + h(i ). difference between the sizes of the two
On update (delete, i ) ∈ U , decrement the counter C → C − h(i ). parties.
On query about the value of F2 , reply with C2 .

This estimator was given by Noga Alon, Yossi Matias, and Mario
Szegedy, in their God̈el-award winning paper on streaming computa- Alon, Matias, Szegedy (2000)
tion.

11.2.2 Properties of the Hash Family

The choice of the hash family will be crucial: we want a small fam-
ily so that we require only a small amount of space to store the hash
function, but we want it to be rich enough for the subsequent analy-
sis to go through.

Definition 11.1 (k-universal hash family). H is k-universal (also

called uniform and k-wise independent) mapping universe U to
some range R if all distinct elements i1 , . . . , ik ∈ U and for values
α1 , . . . , αk ∈ R,
 
^ 1
Pr ( h (i j ) = α j ) = . (11.3)
h← H
j=1..k
| R |k

In our application, we want the hash family to be 4-universal from

U to the two-element range R = {−1, 1}. This means that for any
element i,
1
Pr [h(i ) = 1] = Pr [h(i ) = −1] = .
h← H h← H 2
Moreover, for four distinct elements i, j, k, l, their maps behave inde-
pendently of each other, and hence

E[h(i ) · h( j) · h(k) · h(l )] = E[h(i )] · E[h( j)] · E[h(k)] · E[h(l )].

E[h(i ) · h( j)] = E[h(i )] · E[h( j)].

We will discuss constructions of such hash families soon, but let us

use them to analyze the tug-of-war estimator.
 streaming algorithms 151

11.2.3 A Direct Analysis

Hence, having seen the stream that results in the frequency vector
|U |
x ∈ Z≥0 , the counter will have the value

C := ∑ x i h ( i ).
i ∈U

Remember, the resulting estimate is C2 : so we need to show that

E[C2 ] = F2 , and variance that is small enough that Chebyshev’s
inequality ensures we are correct with reasonable probability.


E[ C 2 ] = E[ ∑ h ( i ) x i · h ( j ) x j ] = ∑ xi x j E[(h(i) · h( j))]
i,j i,j

=∑ xi2 E[h(i ) · h(i )] + ∑ ∑ xi x j E[h(i)] · E[h( j)]

i i ̸= j i,j

= ∑ xi2 = F2 .
i

So in expectation we are correct! Next, recall that the variance is

defined as Var(C2 ) = E[(C2 )2 ] − E[C2 ]2 :

E[(C2 )2 ] = E[ ∑ h ( p ) h ( q ) h (r ) h ( s ) x p x q xr x s ] =
p,q,r,s

= ∑ x4p E[h( p)4 ] + 6 ∑ x2p xq2 E[h( p)2 h(q)2 ] + other terms
p p<q

= ∑ x4p + 6 ∑ x2p xq2 .

p p<q

This is because all the other terms have expectation zero. Why? The
terms like E[h( p)h(q)h(r )h(s)] where p, q, r, s are all distinct, all be-
come zero because of 4-universality. Terms like E[h( p)2 h(r )h(s)]
become zero for the same reason. It is only terms like E[h( p)2 h(q)2 ]
and E[h( p)4 ] that survive, and since h( p) ∈ {−1, 1}, they have expec-
tation 1. So

Var(C2 ) = ∑ x4p + 6 ∑ x2p xq2 − (∑ x2p )2 = 4 ∑ x2p xq2 ≤ 2E[C2 ]2 .

p p<q p p<q

What does Chebyshev say then?

Var(C2 ) 2
Pr[|C2 − E[C2 ]| > εE[C2 ]] ≤ ≤ 2.
(εE[C2 ])2 ε
This is pretty pathetic: since ε is usually less than 1, the RHS usually
more than 1.

11.2.4 Reduce the Variance by Repetition

The idea is the simplest one: if we have an estimator with mean µ
and variance σ2 , then taking the average of k independent copies of
152 a matrix view of our estimator

this estimator has mean µ and variance σ2 /k. (Why? Summing the
independent copies sums the variances and so increases it by k, but
dividing by k reduces it by k2 .)
So if we k such independent counters C1 , C2 , . . . , Ck , and return
their average C = 1k ∑i Ci , we get

2
2 2 2 Var(C ) 2
Pr[|C − E[C ]| > εE[C ]] ≤ 2
≤ .
(εE[C ])2 kε2

Taking k = ε22δ independent counters gives a probability δ of error

on any query. Each counter uses a 4-universal hash function, which
requires O(log U ) random bits to store.

11.2.5 Estimating the p-Moments

To fix, please skip. A bunch of students (Jason, Anshu, Aram) pro-
posed that for the pth -moment calculation we should use 2p-wise in-
dependent hash functions from U to R, where R = {1, ω, ω 2 , . . . , ω p−1 },
the p primitive roots of unity. Again, we set C := ∑i∈U xi h(i ), and
return the real part of C p as our estimate. This approach has been
explored by Ganguly in this paper. Some calculations (and elbow-
grease) show that E[C p ] = Fp , but it seems that naively Var(C p )
p p
tends to grow like F2 instead of Fk ; this leads to pretty bad bounds.
Ganguly’s paper gives some ways of controlling the variance.
BTW, there is a lower bound saying that any algorithm that out-
puts a 2-approximation for Fk requires at least |U |1−2/k bits of stor-
age. Hence, while we just saw that for k = 2, we can get away with
just O(log |U |) bits to get a O(1)-estimate, for k > 2 things are much
worse.

11.3 A Matrix View of our Estimator

Here’s a equivalent way of looking at this estimator, that also relates

it to the previous chapter and the JL Theorem. Recall that the stream
can be viewed as representing a vector x of size |U |, and F2 = ∥x∥2 .
Take a matrix M of dimensions k × D, where D = |U |: again, M is
a “fat and short” matrix, since k = O(ε−2 δ−1 ) is small and D = |U |
is huge. Pick k independent hash functions h1 , h2 , . . . , hk from the
4-universal hash family, and use each one to fill a row of M:

Mij := hi ( j).

The k counters C1 , C2 , . . . , Ck are now nothing other than the entries

of the matrix-vector product
M x.
 streaming algorithms 153

2
 2
The estimate C = 1
k ∑ik=1 Ci is nothing but

1
∥ Mx∥22 .
k
This is completely analogous to the construction for JL: we’ve got
a slightly taller matrix with k = O(ε−2 δ−1 ) rows instead of k =
O(ε−2 log δ−1 ) rows. However, the matrix entries are not fully inde-
pendent (as in JL), just 4-wise independent. I.e., we need to store only
O(k log D ) bits and can generate any entry of M quickly, whereas the
construction for JL stored all kD bits. Henceforth, we use S = √1 M to denote
k
Let us record two properties of this construction: the “sketch” matrix.

Theorem 11.2 (Tug-of-War Sketch). Take a k × D matrix S whose

−1 }k -valued r.v.s. Then for x, y ∈
columns are 4-wise independent { √1 , √
k k
RD ,

1. E[ ⟨Sx, Sy⟩ ] = ⟨x, y⟩.

2
2. Var( ⟨Sx, Sy⟩ ) = k · ∥x∥22 ∥y∥22 .

The proofs is similar to that in §11.2.3; using y = x gives us ex-

actly the results from that section. Moreover, an analogous theorem
can also be given in the JL construction, with fewer rows but with
completely independent entries.

11.4 Application: Approximate Matrix Multiplication

Suppose we want to multiply square matrices A, B ∈ Rn×n , but

want to solve the problem faster, at the expense of getting only an
approximate solution C ≈ AB. How should we measure the error?
Requiring that the answer be close entry-wise to the actual answer
is a hard problem. Let’s aim for something weaker: we want the
“aggregate error” to be small. It’s as though we think of the matrix as
Formally, the Frobenius norm of matrix M is just a vector and look at its Euclidean
length.
s
∥ M∥ F := ∑ Mij2 .
i,j

Our guarantee for approximate matrix multiplication will be

∥C − AB∥2F ≤ small.

Here’s the idea: we want to do the matrix multiplication:

C = AB
A B = C
154 optional: computing the number of distinct elements

This usually takes O(n3 ) time. Indeed, the ijth entry of the product
C is the dot-product of the ith row Ai⋆ of A with the jth column B⋆ j of
B, and the dot-product takes O(n) time. The intuition is that S⊺ S is an almost-
Suppose instead we use a “fat and short” k × n matrix S (for k ≪ identity matrix, it has 1 on the diag-
onals and at most ε everywhere else.
n), and calculate And hence it gives only a small error.
Ce = AS⊺ SB. Of course, we don’t multiply out S⊺ S,
but instead compute AS⊺ and SB, and
By associativity of matrix multiplication, we could first compute then multiply the smaller matrices.
( AS⊺ ) and (SB) in times O(n2 k), and then multiply the results in
time O(nk2 ). Moreover, the matrix S from the previous section works
pretty well, where we set D = n.
S>
d
A S B ≈ C
Indeed, entries of the error matrix Y = C − Ce satisfy

E[Yij ] = 0

and

E[Yij2 ] = Var(Yij ) + E[Yij ]2 = Var(Yij ) ≤ 2k ∥ Ai⋆ ∥22 ∥ B⋆ j ∥22 .

So The squared Frobenius norm of a

matrix is the sum of squared Euclidean
E[∥ AB − AS⊺ SB∥2F ] = E[∑ Yij2 ] = ∑ E[Yij2 ] = 2k ∑ij ∥ Ai⋆ ∥22 ∥ B⋆ j ∥22 lengths of the columns, or of the rows.
ij ij

= 2k ∥ A∥2F ∥ B∥2F .

Finally, setting k = ε22δ and using Markov’s inequality, we can say that
for any fixed ε > 0, we can compute an approximate matrix product
C := AS⊺ SB such that
 
Pr ∥ AB − C ∥ F ≤ ε · ∥ A∥ F ∥ B∥ F ≥ 1 − δ,

2
in time O( εn2 δ ). (If we want to make δ very small, at the expense of
picking more independent random bits in the sketching matrix S, we
can use the JL matrices instead. Details will appear in a homework.)
Finally, if the matrices A, B are sparse and contains only ≪ n2 entries,
the time can be made to depend on nnz( A, B).
The approximate matrix product question has been considered
often, e.g., by Edith Cohen and David Lewis using a random-walks Cohen and Lewis (1999)
approach. The algorithm we present is due to Tamás Sarlós; his pa-
per gives better results, as well as extensions to computing SVDs
faster. Better bounds have subsequently been given by Clarkson and
Woodruff. More recent refs too.

11.5 Optional: Computing the Number of Distinct Elements

Our last example today will be to compute F0 , the number of distinct

elements seen in the data stream, but in the addition-only model,
with no deletions. (We’ll see another approach in a HW.)
 streaming algorithms 155

11.5.1 A Simple Lower Bound

Of course, if we store x explicitly (using |U | space), we can trivially
solve this problem exactly. Or we could store the (at most) t elements
seen so far, again we could give an exact answer. And indeed, we
cannot do much better if we want no errors. Here’s a proof sketch
for deterministic algorithms (one can extend this to randomized
algorithms with some more work).

Lemma 11.3 (A Lower Bound). Suppose a deterministic algorithm cor-

rectly reports the number of distinct elements for each sequence of length at
most N. Suppose N ≤ 2|U |. Then it must use at least Ω( N ) bits of space.

Proof. Consider the situation where first we send in some subset S

of N − 1 elements distinct elements of U. Look at the information
stored by the algorithm. We claim that we should be able to use this
information to identify exactly which of the ( N|U−|1) subsets of U we
have seen so far. This would require
 
|U |

log2 ≥ ( N − 1) log2 |U | − log2 ( N − 1) = Ω( N )
N−1

bits of memory.1 1
We used the approximation that

k
OK, so why should we be able to uniquely identify the set of el- (mk) ≥ mk , and hence log2 (mk) ≥
k (log2 m − log2 k ).
ements until time N − 1? For a contradiction, suppose we could
not tell whether we’d seen S1 or S2 after N − 1 elements had come
in. Pick any element e ∈ S1 \ S2 . Now if we gave the algorithm e
as the N th element, the number of distinct elements seen would be
N if we’d already seen S2 , and N − 1 if we’d seen S1 . But the algo-
rithm could not distinguish between the two cases, and would return
the same answer. It would be incorrect in one of the two cases. This
contradicts the claim that the algorithm always correctly reports the
number of distinct elements on streams of length N.

OK, so we need an approximation if we want to use little space.

Let’s use some hashing magic.

11.5.2 The Intuition

Suppose there are d = ∥x∥0 distinct elements. If we randomly map
d distinct elements onto the line [0, 1], we expect to see the smallest
mapped value at location ≈ d1 . (I am assuming that we map these
elements consistently, so that multiple copies of an element go to
the same place.) So if the smallest value is δ, one estimator for the
number of elements is 1/δ.
This is the essential idea. To make this work (and analyze it), we
change it slightly: The variance of the above estimator is large. By the
156 optional: computing the number of distinct elements

same argument, for any integer s we expect the sth smallest mapped
value at ds . We use a larger value of s to reduce the variance.

11.5.3 The Algorithm

Assume we have a hash family H with hash functions h : U → [ M ].
(We’ll soon figure out the precise properties we’ll want from this
hash family.) We will later fix the value of the parameter s to be some
large constant. Here’s the algorithm:

Pick a hash function h randomly from H .

If query comes in at time t
Consider the hash values h( a1 ), h( a2 ), . . . , h( at ) seen so far.
Let Lt be the sth smallest distinct hash value h( ai ) in this
set.
M·s
Output the estimate Dt = Lt .

The crucial observation is: it does not matter if we see an element

e once or multiple times — the algorithm will behave the same, since
the output depends on what distinct elements we’ve seen so far.
Also, maintaining the sth smallest element can be done by remember-
ing at most s elements. (So we want to make s small.)
How does this help? As a thought experiment, if we had d distinct
darts and threw them in the continuous interval [0, M ], we would
expect the location of the sth smallest dart to be about s·dM . So if the
sth smallest dart was at location ℓ in the interval [0, M ], we would be
tempted to equate ℓ = s·dM and hence guessing d = s·ℓM would be a
good move. Which is precisely why we used the estimate

M·s
Dt = .
Lt

Of course, all this is in expectation—the following theorem argues

that this estimate is good with reasonable probability.

Theorem 11.4. Consider some time t. If H is a uniform 2-universal hash

family mapping U → [ M ], and M is large enough, then both the following
guarantees hold:

3
Pr[ Dt > 2 ∥xt ∥0 ] ≤ , and (11.4)
s
∥ x t ∥0 3
Pr[ Dt < ]≤ . (11.5)
2 s
We will prove this in the next section. First, some observations.
Firstly, we now use the stronger assumption that that the hash family
2-universal; recall the definition from Section 11.2.2. Next, setting
∥xt ∥
s = 8 means that the estimate Dt lies within [ 2 0 , 2∥xt ∥0 ] with
probability at least 1 − (1/4 + 1/4) = 1/2. (And we can boost the
 streaming algorithms 157

success probability by repetitions.) Secondly, we will see that the

estimation error of a factor of 2 can be made (1 + ε) by changing the
parameters s and k.

11.5.4 Proof of Theorem 11.4

Now for the proof of the theorem. We’ll prove bound (11.5), the other
bound (11.4) is proved identically. Some shorter notation may help.
Let d := ∥xt ∥0 . Let these d distinct elements be T = {e1 , e2 , . . . , ed } ⊆
U.
The random variable Lt is the sth smallest distinct hash value seen
until time t. Our estimate is sMLt , and we want this to be at least d/2.
2sM
So we want Lt to be at most d . In other words,

2sM
Pr[ estimate too low ] = Pr[ Dt < d/2] = Pr[ Lt > ].
d
Recall T is the set of all d (= ∥xt ∥0 ) distinct elements in U that
have appeared so far. How many of these elements in T hashed to
values greater than 2sM/d? The event that Lt > 2sM/d (which
is what we want to bound the probability of) is the same as saying
that fewer than s of the elements in T hashed to values smaller than
2sM/d. For each i = 1, 2, . . . , d, define the indicator

1 if h(ei ) ≤ 2sM/d
Xi = (11.6)
0 otherwise

Then X = ∑id=1 Xi is the number of elements seen that hash to values

below 2sM/d. By the discussion above, we get that
 
2sM
Pr Lt < ≤ Pr[ X < s].
d

We will now estimate the RHS.

Next, what is the chance that Xi = 1? The hash h(ei ) takes on each
of the M integer values with equal probability, so

⌊sM/2d⌋ s 1
Pr[ Xi = 1] = ≥ − . (11.7)
M 2d M
By linearity of expectations,
" #    
d d d
s 1 s d
E[ X ] = E ∑ Xi = ∑ E [Xi ] = ∑ Pr [Xi = 1] ≥ d · 2d
−
M
=
2
−
M
.
i =1 i =1 i =1

s
Let’s imagine we set M large enough so that d/M is, say, at most 100 .
Which means s s  49 s
E[ X ] ≥ − = .
2 100 100
158 optional: computing the number of distinct elements

So by Markov’s inequality,
   100  49
Pr X > s = Pr X > E[ X ] ≤ .
49 100
Good? Well, not so good. We wanted a probability of failure to be
smaller than 2/s, we got it to be slightly less than 1/2. Good try, but
no cigar.

11.5.5 Enter Chebyshev

Recall that Var(∑i Zi ) = ∑i Var( Zi ) for pairwise-independent random
variables Zi . (Why?) Also, if Zi is a {0, 1} random variable, Var( Zi ) ≤
E[ Zi ]. (Why?) Applying these to our random variables X = ∑i Xi , we
get
Var( X ) = ∑ Var( Xi ) ≤ ∑ E[ Xi ] = E( X ).
i i
(The first inequality used that the Xi were pairwise independent,
since the hash function was 2-universal.) Is this variance “low”
enough? Plugging into Chebyshev’s inequality, we get:

100 50 σX2 1 3
Pr[ X > s] = Pr[ X > µ X ] ≤ Pr[| X − µ X | > µX ] ≤ ≤ ≤ .
49 49 (50/49)2 µ2X (50/49)2 µ X s

Which is precisely what we want for the bound (11.4). The proof for
the bound (11.5) is similar and left as an exercise. If we want the estimate to be at most
∥ x t ∥0
,
then we would want to bound
(1+ ε )
E[ X ]
Pr[ X < ]. Similar calculations
11.5.6 Final Bookkeeping (1+ ε )
should give this to be at most ε23s , as
Excellent. We have a hashing-based data structure that answers long as M was large enough. In that
case we would set s = O(1/ε2 ) to get
“number of distinct elements seen so far” queries, such that each some non-trivial guarantees.
answer is within a multiplicative factor of 2 of the actual value ∥xt ∥0 ,
with small error probability.
Let’s see how much space we actually used. Recall that for failure
probability 1/2, we could set s = 12, say. And the space to store
the s smallest hash values seen so far is O(s lg M ) bits. For the hash
functions themselves, the standard constructions use O((lg M) +
(lg U )) bits per hash function. So the total space used for the entire
data structure is
O(log M ) + (lg U ) bits.
What is M? Recall we needed to M large enough so that d/M ≤
s/100. Since d ≤ |U |, the total number of elements in the universe,
set M = Θ(U ). Now the total number of bits stored is

O(log U ).

And the probability of our estimate Dt being within a factor of 2 of

the correct answer ∥xt ∥0 is at least 1/2.
12
Dimension Reduction: Singular Value Decompositions

12.1 Introduction

In this lecture, we see a very popular und useful dimension reduction

technique that is based on the singular value decomposition (SVD)
of a matrix. In contrast to the dimension reduction obtained by the
Johnson-Lindenstrauss Lemma, SVD based dimension reductions are
not distance preserving. That means that we allow that the distances
between pairs of points in our input change. Instead, we want to
keep the shape of the point set by fitting it to a subspace according
to a least squares error. This preserves most of the ‘energy’ of the
points.
More precisely, the problem that we want to solve is the follow-
ing. We are given a matrix A ∈ Rn×d . The points are the rows of
A, which we also name a1 , . . . , an ∈ Rd . Let the rank of A be r, so
r ≤ min{n, d}. Given an integer k, we want to find a subspace V of
dimension k that minimizes the sum of the squared distances of all
points in A to V. Thus, for each point in A, we square the distance
between the point and its projection to V and add these squared
errors, and this term should be minimized by our choice of V.
This task can be solved by computing the SVD of A, a decomposi-
tion of A into matrices with nice properties. We will see that we can
write A as
     
a1 | | σ1 0 v1
     
A=
..  =  u1 · · · u n   ..
.  ..  = UDV ⊺
 .    . 
an | | 0 σr vd

where U ∈ Rn×r and V ∈ Rr×d are matrices with orthonormal

columns and D ∈ Rr×r is a diagonal matrix. Notice that the columns
of V are the d-dimensional points v1 , . . . , vd which appear in the rows
of the above matrix since it is V ⊺ .
Notice that the SVD can give us an intuition of how A acts as a
160 best fit subspaces of dimension k and the svd

mapping. We have that

AV = UDV ⊺ V = UD

because V consists of orthonormal columns. Imagine the r-dimensional

sphere that is spanned by v1 , . . . , vr . The linear mapping defined by
A maps this sphere to an ellipsoid with σ1 u1 , . . . , σr ur as the axes, like
shown in Figure 12.1.
The singular value decomposition was developed by different
mathematicians around the beginning of the 19th century. The survey
by Stewart 1 gives an historical overview on its origins. In the fol- 1

lowing, we see how to obtain the SVD and why it solves our best fit
problem. The lecture is partly based on 2 . 2

12.2 Best fit subspaces of dimension k and the SVD

We start with the case that k = 1. Thus, we look for the line through
the origin that minimizes the sum of the squared errors. See Fig-
ure 12.2. It depicts a one-dimensional subspace V in blue. We look
at a point ai , its distance β i to V, and the length of its projection to
V which is named αi in the picture. Notice that the length of ai is
α2i + β2i . Thus, for our fixed ai , minimizing β i is equivalent to maxi-
mizing αi . If we represent V by a unit vector v that spans V (depicted
in orange in the picture), then we can compute the projection of ai to
V by the dot product ⟨ ai , v⟩. We have just argued that we can find the
best fit subspace of dimension one by solving
n n
max
v∈Rd ,∥v∥=1
∑ ⟨ai , v⟩2 = v∈Rmin
d ,∥ v ∥=1
∑ dist(ai , span(v))2
i =1 i =1

where we denote the distance between a point ai and the line spanned
by v by dist( ai , span(v))2 . Now because Av = (⟨ a1 , v⟩, ⟨ a2 , v⟩, . . . , ⟨ an v⟩)⊺ ,
we can rewrite ∑id=1 ⟨ ai , v⟩2 as ∥ Av∥2 . We define the first right singu-
lar vector to be a unit vector that maximizes ∥ Av∥.3 We thus know 3
There may be many vectors that
that the subspaces spanned by it is the best fit subspace of dimension achieve the maximum: indeed, for
every v that achieves the maximum,
one. −v also has the same maximum. Let us
break ties arbitrarily.

Figure 12.1: A visualization of AV =

UD for r = 2.
σ2 u2
v2
A σ1 u1
v1
 dimension reduction: singular value decompositions 161

a3 Figure 12.2: Finding the best fit sub-

space of dimension one.
ai ∗

βi

a1 a4

αi

a2

Now we want to generalize this concept to more than one dimen-

sion. It turns out that to do so, we can iteratively pick orthogonal
unit vectors that span more and more dimensions. Among all unit
vectors that are orthogonal to those chosen so far, we pick a vector
that maximizes ∥ Av∥. This is formalized in the following definition.

Definition 12.1. Let A ∈ R n × d be a matrix. We define

v 1 = arg max ∥ Av ∥ , σ1 ( A ) : = ∥ Av 1 ∥
∥ v ∥= 1

v 2 = arg max ∥ Av ∥ , σ2 ( A ) : = ∥ Av 2 ∥
∥ v ∥= 1, ⟨ v,v 1 ⟩= 0
..
.
v r = arg max ∥ Av ∥ , σr ( A ) : = ∥ Av r ∥
∥ v ∥= 1, ⟨ v,v i ⟩= 0 ∀ i = 1,...,r − 1

and say that v 1 , . . . , v r are right singular vectors of A and that σ1 : =

σ1 ( A ) , . . . , σr : = σr ( A ) are the singular values of A. Then we define
the left singular vectors by setting
Av i
ui := for all i = 1, . . . , r.
∥ Av i ∥
One worry is that this greedy process picked v 2 after fixing v 1 ,
and hence the span of v 1 , v 2 may not be the best two-dimensional
subspace. The following claim says that Definition 12.1 indeed gives
us the the best fit subspaces.
Claim 12.2. For any k, the subspace Vk , which is the span of v 1 , . . . , v k ,
minimizes the sum of the squared distances of all points among all
subspaces of dimension k.

Proof. Let V2 be the subspace spanned by v 1 and v 2 . Let W be any

other 2-dimensional subspace and let w 1 , w 2 be an orthonormal basis
162 best fit subspaces of dimension k and the svd

of W. Recall that the squared length of the projection of a point a i to

V decomposes into the squared lengths of the projections to the lines
spanned by v 1 and v 2 and the same is true for W, w 1 and w 2 .
Since we chose v 1 to maximize ∥ Av ∥ , we know that ∥ Aw 1 ∥ ≤
∥ Av 1 ∥ . Similarly, it holds that ∥ Aw 2 ∥ ≤ ∥ Av 2 ∥ , which means that

∥ Aw 1 ∥ 2 + ∥ Aw 2 ∥ 2 ≤ ∥ Av 1 ∥ 2 + ∥ Av 2 ∥ 2 .

We can extend this argument by induction to show that the space

spanned by v 1 , . . . , v k is the best fit subspace of dimension k.

We review some properties of the singular values and vectors. No-

tice that as long as i < r, there is always a vector in the row space
of A that is linearly independent to v 1 , . . . , v i , which ensures that
max ∥ Av ∥ is nonzero. For i = r, the vectors v 1 , . . . , v r span the row
space of A. Thus, any vector that is orthogonal to them lies in the
kernel of A, meaning that arg max ∥ v ∥= 1, ⟨ v,v i ⟩= 0 ∀ i = 1,...,i − 1 ∥ Av ∥ =
0, so we end the process at this point. By construction, we know that
the singular values are not increasing. We also see that the right sin-
gular vectors form a orthonormal basis of the row space of A. This is
true for the left singular vectors and the column space as well (home-
work). The following fact summarizes the important properties.
Fact 12.3. The sets { u 1 , . . . , u r } and { v 1 , . . . , v r } as defined in 12.1
are both orthonormal sets and span the column and row space, re-
spectively. The singular values satisfy σ1 ≥ σ2 ≥ . . . ≥ σr > 0.
So far, we defined the v i purely based on the goal to find the best
fit subspace. Now we claim that in doing so, we have actually found
the decomposition we wanted, i.e. that
   
| | σ1 0 v1
  ..  .. 
U DV ⊺ : =  u 1 · · · u n    . 
 .
 = A.

| | 0 σr vd
(12.1)

Claim 12.4. For any matrix A ∈ R n × d and U, V , D as in (12.1), it

holds that
A = U DV ⊺ .

Proof. We prove the claim by using the fact that two matrices A, B ∈
R n × d are identical iff for all vectors v, the images are equal, i.e. Av =
Bv. Notice that it is sufficient to check this for a basis, so it is true if
the following subclaim holds (which we do not prove):

Subclaim: Two matrices A, B ∈ R n × d are identical iff Av = Bv for

all v in a basis of R d .
 dimension reduction: singular value decompositions 163

We use the subclaim for B = U DV ⊺ . Notice that we can extend

v 1 , . . . , v r to a basis of R d by adding orthonormal vectors from the
kernel of A. These additional vectors are orthogonal to all vectors in
the rows of V ⊺ , so V ⊺ v is the zero vector for all of them. Since they
−
→ −
→ − →
are in the kernel of A, it holds 0 = Av = Bv = U D 0 = 0 for the
additional basis vectors. For i = 1, . . . , r, we notice that
Av i
( U DV ⊺ ) v i = U De i = u i σi = · ∥ Av i ∥ = Av i
∥ Av i ∥
which completes the proof.

12.3 Useful facts, and rank-k-approximation

Singular values are a generalization of the concept of eigenvalues

for square matrices. Recall that a square symmetric matrix M can
be written as M = ∑ri=1 λi vi v⊺i where λi and vi are eigenvalues and
eigenvectors, respectively. This decomposition can be used to de-
fine the singular vectors in a different way. In fact, the right singular
vectors of A correspond to the eigenvectors of A⊺ A (notice that this
matrix is square and symmetric), and the left singular vectors corre-
spond to the eigenvectors of AA⊺ .
This fact can also be used to compute the SVD. Computing the
SVD or eigenvalues and -vectors in a numerically stable way is the
topic of a large research area, and there are different ways to obtain
algorithms that converge under the assumption of a finite precision.
Fact 12.5. The SVD can be found (up to arbritrary precision) in time
O(min(nd2 , n2 d)) or even in time O(min(ndω −1 , dnω −1 )) where ω
is the matrix multiplication constant. (Here the big-O term hides the
dependence on the precision.)
The SVD is unique in the sense that for any i ∈ [r ], the subspace
spanned by unit vectors v that maximize ∥ Av∥ is unique. Aside from
the different choices of an orthonormal basis of these subspaces, the
singular vectors are uniquely defined. For example, if all singular
values are distinct, then the subspace of unit vectors that maximize
∥ Av∥ is one-dimensional and the singular vector is unique (up to
sign changes, i.e., up to multiplication by −1).
Sometimes, it is helpful to observe that the matrix product UDV ⊺
can also be written as the sum of outer products of the singular vec-
tors. This formulation has the advantage that we can write the pro-
jection of A to the best fit subspaces of dimension k as the sum of the
first k terms.
Remark 12.6. The SVD can equivalently be written as
r
∑ σi ui vi
⊺
A=
i =1
164 applications

where ui v⊺i is the outer product. For k ≤ r, the projection of A to Vk is

k
∑ σi ui vi .
⊺
Ak :=
i =1

Recall that the Frobenius norm of a matrix A is the square

qroot of
the sum of its squared entries, i.e. it is defined by ∥ A∥ F := ∑i,j a2ij .
This means that ∥ A − B∥2F is equal to the sum of the squared dis-
tances between each row in A and the corresponding row in B for
matrices of equal dimensions. Imagine that B is a rank k matrix.
Then its points lie within a k-dimensional subspace, and ∥ A − B∥2F
cannot be smaller than the distance between A and this subspace.
Since Ak is the projection to the best fit subspace of dimension k, Ak
minimizes ∥ A − B∥ F (notice that Ak has rank at most k). It is there-
fore also called the best rank k-approximation of A.

Theorem 12.7. Let A ∈ Rn×d be a matrix of rank r and let k ≤ r be given.

It holds that
∥ A − Ak ∥ F ≤ ∥ A − B∥ F
for any matrix B ∈ Rn×d of rank at most k.

The theorem is also true if the Frobenius norm is replaced by the

spectral norm.4 For a matrix A, the spectral norm is equal to the 4
In fact, this theorem holds for any
maximum singular value, i.e. ∥ A∥2 := maxv∈Rd ,∥v∥=1 ∥ Av∥ = σ1 . unitarily invariant matrix norm; a
matrix norm ∥ · ∥ is unitarily invariant if
∥ A∥ = ∥U AV ∥ for any unitary matrices
U, V. Other examples of unitarily
12.4 Applications invariant norms are the Schatten norms,
and the Ky Fan norms. J. von Neumann
Topic modeling. Replacing A by Ak is a great compression idea. For characterized all unitarily invariant
matrix norms as those obtained by
example, for topic modeling, we imagine A to be a matrix that stores taking a “symmetric” (vector) norm of
the number of times that any of d words appears in any of n docu- the vector of singular values — here
symmetric means ∥ x ∥ = ∥y∥ when y is
ments. Then we assume that the rank rof A corresponds to r topics. obtained by flipping the signs of some
Recall that entries of x and then permuting them
around. See Theorem 7.4.24 in the text
by Horn and Johnson.
     
a1 | |  σ1 0 v1
 ..    ..  .. 
A=
 .
 =  u1
 ··· un  
 . 
 .
.

an | | 0 σr vd

Assume that the entries in U and V are positive. Since the column
vectors are unit vectors, they define a convex combination of the
r topics. We can thus imagine U to contain information on how
much each of the documents consists of each topic. Then, D assigns
a weight to each of the topics. Finally, we V ⊺ gives information on
how much each topic consists of each of the words. The combination
of the three matrices generates the actual documents. By using the
 dimension reduction: singular value decompositions 165

SVD, we can represent a set of documents based on fewer topics, thus

obtaining an easier model of how they are generated.
Notice that this interpretation of the SVD needs that the entries are
non negative, and that obtaining such a decomposition is an NP-hard
problem.

12.4.1 Pseudoinverse and least squares regression

a4 x
b4

b2 a3 x

a2 x
b3

a1 a2 a3 a4
a1 x

b1

For any diagonal matrix M = diag(d1 , . . . , dℓ ), define M+ :=

diag(1/d1 , . . . , 1/dℓ ). We notice that for the matrices from the SVD, it
holds that
VD + U ⊺ UDV = diag(1, . . . , 1, 0, . . . , 0).
| {z }
r times
If A is an n × n-matrix of rank n, then r = n and the result of this
product is I. Thus, A+ := VD + U ⊺ is then the inverse of A. In gen-
eral, A+ is the (Moore Penrose) pseudoinverse of A. It satisfies that

A( A+ b) = b ∀b in the image of A

The pseudoinverse helps to find the solution to another popular

minimization problem, least squares regression. Given an overcon-
strained system of equations Ax = b, least squares regression asks for
a point x that minimizes the squared error ∥ Ax − b∥22 . I.e., we want

x ∗ := arg min ∥ Ax − b∥22 .

Notice that if there is an x ′ with Ax ′ = b, then it also minimizes

∥ Ax ′ − b∥22 , and if A had full rank this x ′ would be obtained by com-
puting A−1 b. If A does not have full rank, an optimal solution is
obtained by using the pseudoinverse:

x ∗ = A+ b
166 symmetric matrices

(This is often used as another definition for the pseudoinverse.)

Here’s a proof: for any choice of x ∈ Rd , Ax is some point in
the column span of A. So x ∗ , the minimizer, must be the projection
of b onto colspan( A). One orthonormal basis for colspan( A) is the
columns of U. Hence the projection Πb of b onto colspan( A) is given
by UU ⊺ b. (Why? Extend U to a basis for all of Rd , write b in this
basis, and consider what it’s projection must be.) Hence we want
Ax ∗ = UU ⊺ b. For this, it suffices to set x ∗ = VD + U ⊺ b = A+ b.

12.5 Symmetric Matrices

For a (square) symmetric matrix A, the (normalized) eigenvectors vi

and the eigenvalues λi satisfy the following properties: the vi s form
an orthonormal basis, and A = VΛV ⊺ , where the columns of V are
the vi vectors, and Λ is a diagonal matrix with λi s on the diagonal.
It is no longer the case that the eigenvalues are all non-negative. (In
fact, we can match up the eigenvalues and singular values such that
they differ only in sign.)
Given a function f : R → R, we can extend this to a function on
symmetric matrices as follows:

f ( A) = V diag( f (λ1 ), . . . , f (λn )) V ⊺ .

For instance, you can check that Ak or e A defined this way indeed
correspond to what you think they might mean. (The other way to
k
define e A would be ∑k≥0 Ak! .)
 Part III

From Discrete to
Continuous Algorithms
13
Online Learning: Experts and Bandits

In this set of chapters, we consider a basic problem in online algo-

rithms and online learning: how to dynamically choose from among
a set of “experts” in a way that compares favorably to any fixed ex-
pert. Both this abstract problem, and the techniques behind the solu-
tion, are important parts of the algorithm designer’s toolkit.

13.1 The Mistake-Bound Model

The term expert just refers to a person
Suppose there are N experts who make predictions about a certain who has an opinion, and does not
event every day—for example, whether it rains today or not, or reflect whether they are good or bad at
the prediction task at hand.
whether the stock market goes up or not. Let U be the set of pos-
sible choices. The process in the experts setting goes as follows:

1. At the beginning of each time step t, each expert makes a predic-

tion. Let E t ∈ U N be the vector of predictions. Note the order of events: the experts
predictions come first, then the algo-
2. The algorithm makes a prediction at , and simultaneously, the rithm chooses an expert at the same
time as the reality being revealed.
actual outcome o t is revealed.

The goal is to minimize the number of mistakes, i.e., the number of

times our prediction at differs from the outcome o t . Suppose we have 8 experts, and E t =
(0, 1, 0, 0, 0, 1, 1, 0). If we follow the third
Fact 13.1. There exists an algorithm that makes at most ⌈log2 N ⌉ expert and predict at = 0, but the actual
mistakes, if there is a perfect expert. outcome is o t = 1, we make a mistake;
if we would have picked the second
Proof. The algorithm just considers all the experts who have made expert, we would have been correct.

no mistakes so far, and predicts what the majority of them predict.

Note that every time we make a mistake, the number of experts who
have not been wrong yet reduces by a factor of 2 or more. (And when
we do not make a mistake, this number does not increase.) Since
there is at least one perfect expert, we can make at most ⌈log2 N ⌉
mistakes.

Show that any algorithm must make at least ⌈log2 N ⌉ mistakes in

the worst case.
170 the weighted majority algorithm

Fact 13.2. There is an algorithm that, on any sequence, makes at most

M ≤ m∗ (⌈log2 N ⌉ + 1) + ⌈log2 N ⌉ mistakes, where m∗ is the number
of mistakes made by the best of these experts on this sequence.

Proof. Think of time as being divided into “epochs”. In each epoch,

we proceed as in the perfect expert scenario as in Fact 13.1: we keep
track of all experts who have not yet made a mistake in that epoch,
and predict the majority opinion. The set of experts halves (at least)
with every mistake the algorithm makes. When the set becomes
empty, we end the epoch, and start a new epoch with all the N ex-
perts.
Note that in each epoch, every expert makes at least one mistake.
Therefore the number of completed epochs is at most m∗ . Moreover,
we make at most ⌈log2 N ⌉ + 1 mistakes in each completed epoch, and
at most ⌈log2 N ⌉ mistakes the last epoch, giving the result.

However, this algorithm is very harsh and very myopic. Firstly, it

penalizes even a single mistake by immediately discarding the expert.
But then, at the end of an epoch, it wipes the slate clean and forgets
the past performance of the experts. Maybe we should be gentler, but
have a better memory?

13.2 The Weighted Majority Algorithm

This algorithm, due to Littlestone and Warmuth, is remarkable for

(t)
its simplicity. We assign a weight wi to each expert i ∈ [ N ]. Let wi
denote the weight of expert i at the beginning of round t. Initially, all
(1)
weights are 1, i.e., wi = 1.

1. In round t, predict according to the weighted majority of experts.

In other words, choose the outcome that maximizes the sum of
weights of experts that predicted it. I.e.,
(t)
at ← arg max
u ∈U
∑ wi .
i:expert i predicts u
We break ties arbitrarily, say, by picking
the first of the options that achieve the
2. Upon seeing the outcome, set maximum.

1 if i was correct
( t +1) (t)
wi = wi · .
1 if i was incorrect
2

Theorem 13.3. For any sequence of predictions, the number of mistakes

made by the weighted majority algorithm (WM) is at most

2.41(mi + log2 N ),

where mi is the number of mistakes made by expert i.

 online learning: experts and bandits 171

Proof. The proof uses a potential-function argument. Let

(t)
Φt := ∑ wi .
i ∈[ N ]

Note that

1. Φ1 = N, since the weights start off at 1,

2. Φt+1 ≤ Φt for all t, and

3. if Algorithm WM makes a mistake in round t, the sum of weights

of the wrong experts is higher than the sum of the weights of the
correct experts, so
( t +1) ( t +1)
Φ t +1 = ∑ wi + ∑ wi
i wrong i correct
1 (t) (t)
= ∑ wi + ∑ wi
2 i wrong i correct
1 (t)
= Φt − ∑ wi
2 i wrong
3 t
≤ Φ
4
If after T rounds, expert i has made mi mistakes and WM has made
M mistakes, then
  mi  M  M
1 ( T +1) T +1 1 3 3
= wi ≤Φ ≤Φ =N .
2 4 4
Now taking logs, and rearranging,
mi + log2 N
M≤ 4
≤ 2.41(mi + log2 N ).
log2 3
We cannot hope to compare ourselves
In other words, if the best of the N experts on this sequence was to the best way of dynamically choosing
wrong m∗ times, we would be wrong at most 2.41(m∗ + log2 n) times. experts to follow. This result says
that at least we do not much worse to
Note that we are much better on the multiplier in front of the m∗ the best static policy of choosing an
term than Fact 13.2 was, at the expense of being slightly worse on the expert—in fact, choosing the best expert
in hindsight—and sticking with them.
multiplier in front of the log2 N term.
We’ll improve our performance soon,
but all our results will still compare to
the best static policy for now.
13.2.1 A Gentler Penalization
Instead of penalizing each wrong expert by a factor of 1/2, we could
penalize the experts by a factor of (1 − ε). This allows us to trade off
the multipliers on the m∗ term and the logarithmic term.

Theorem 13.4. For ε ∈ (0, 1/2), penalizing each incorrect expert by a factor
of (1 − ε) guarantees that the number of mistakes made by MW is at most
 
log N
2(1 + ε ) m i + O .
ε
172 randomized weighted majority

Proof. Using an analysis identical to Theorem 13.3, we get that

Φt+1 ≤ (1 − 2ε )Φt and therefore
 ε M  ε M

(1 − ε ) m i ≤ Φ T +1 ≤ Φ 1 1 − = N 1− ≤ N exp − εM/2 .
2 2
Now taking logs, and simplifying,
−mi log(1 − ε) + ln N
M≤
ε/2
 
m ( ε + ε2 ) log N
≤2 i +O ,
ε ε
ε2 ε3
because − ln(1 − ε) = ε + 2 + 3 + · · · ≤ ε + ε2 for ε ∈ [0, 1].

This shows that we can make our mistakes bound as close to 2m∗
as we want, but this approach seems to have this inherent loss of
a factor of 2. In fact, no deterministic strategy can do better than a
factor of 2, as we show next.
Proposition 13.5. No deterministic algorithm A can do better than a factor
of 2, compared to the best expert.
Proof. Note that if the algorithm is deterministic, its predictions are
completely determined by the sequence seen thus far (and hence can
also be computed by the adversary). Consider a scenario with two
experts A,B, the first always predicts 1 and the second always pre-
dicts 0. Since A is deterministic, an adversary can fix the outcomes
such that A’s predictions are always wrong. Hence at least one of A
and B will have an error rate of ≤ 1/2, while A’s error rate will be
1.

13.3 Randomized Weighted Majority

Consider the proof of Proposition 13.5, but applied to the WM algo-

rithm: the algorithm alternates between predicting 0 and 1, whereas
the actual outcome is the opposite. The weights of the two experts
remain approximately the same, but because we are deterministic, we
choose the wrong one. What if we interpret the weights being equal
as a signal that we should choose one of the two options with equal
probability?
This is the idea behind the Randomized Weighted Majority algorithm
(RMW) of Littlestone and Warmuth: the weights evolve in exactly the
same way as in Theorem 13.4, but now the prediction at each time is
drawn randomly proportional to the current weights of the experts.
I.e., instead of Step 1 in that algorithm, we do the following:
(t)
∑i:expert i predicts u wi
Pr[action u is picked] = (t)
.
∑ i wi
 online learning: experts and bandits 173

Note that the update of the weights proceeds exactly the same as
previously.

Theorem 13.6. Fix ε ≤ 1/2. For any fixed sequence of predictions, the ex-
pected number of mistakes made by randomized weighted majority (RWM)
is at most  
log N
E[ M ] ≤ (1 + ε ) m i + O
ε
log N

The quantity εmi + O ε gap
between the algorithm’s performance
Proof. The proof is an analysis of the weight evolution that is more and that of the best expert is called the
(t)
careful than in Theorem 13.4. Again, the potential is Φt = ∑i wi . regret with respect to expert i.
Define
(t)
∑i incorrect wi
Ft := (t)
∑ i wi
to be the fraction of weight on incorrect experts at time t. Note that

E[ M ] = ∑ Ft .
t∈[ T ]

Indeed, we make a mistake at step t precisely with the probability Ft ,

since the adversary does not see our random choice when deciding
on the actual outcome at . By our re-weighting rules,

Φt+1 = Φt ((1 − Ft ) + Ft (1 − ε)) = Φt (1 − εFt )

Bounding the size of the potential after T steps,

T
(1 − ε)mi ≤ Φ T +1 = Φ1 ∏ (1 − εFt ) ≤ Ne−ε ∑ Ft = Ne−εE[ M]
t =1

Now taking logs, we get mi ln(1 − ε) ≤ ln N − εE[ M], using the

approximation − log(1 − ε) ≤ ε + ε2 gives us

ln N
E[ M ] ≤ m i (1 + ε ) + .
ε

13.3.1 Classifying Adversaries for Randomized Algorithms

In the above analysis, it was important that the actual random out-
come was independent of the prediction of the algorithm. Let us
formalize the power of the adversary:

Oblivious Adversary. Constructs entire sequence E 1 , o1 , E 2 , o2 , · · · up-

front.

Adaptive Adversary. Sees the previous choices of the algorithm, but

must choose o t independently of our actual prediction at in round
t. Hence, o t can be a function of E 1 , o1 , . . . , E t−1 , o t−1 , E t , as well as
of a1 , . . . , at−1 , but not of at .
174 the hedge algorithm, and a change in perspective

The adversaries are equivalent on deterministic algorithms, because

such an algorithm always outputs the same prediction and the obliv-
ious adversary could have calculated at in advance when creating
E t+1 . They may be different for randomized algorithms. However, it
turns out that RWM works in both models, because our predictions
do not affect the weight updates and hence the future.

13.4 The Hedge Algorithm, and a Change in Perspective

Let’s broaden the setting slightly, and consider the following dot-
product game. In each round, Define the probability simplex as

1. The algorithm produces a vector of probabilities ∆ N := x ∈ [0, 1] N | ∑ xi = 1 .
i
t
p = ( p1t , p2t , · · · , ptN ) ∈ ∆N .

2. The adversary produces

ℓt = (ℓ1t , ℓ2t , · · · , ℓtN ) ∈ [−1, 1] N .


3. The loss of the algorithm in this round is ℓt , pt .
This equivalence between randomized
We can move between this “fractional” model where we play a and fractional algorithms is a common
theme in algorithm design, especially in
point in the probability simplex ∆ N , and the randomized model of
approximation and online algorithms.
the previous section (with an adaptive adversary), where we must
play a single expert (which is a vertex of the simplex ∆ N . Indeed,
setting ℓt to be a vector of 0s and 1s can capture whether an expert is
correct or not, and we can set

pit = Pr[algorithm plays expert i at time t]

to deduce that


Pr[mistake at time t] = ℓt , pt .

13.4.1 The Hedge Algorithm

The Hedge algorithm starts with weights wi1 = 1 for all experts i. In
each round t, it defines pt ∈ ∆ N using:
wit
pit ← , (13.1)
∑ j wtj

and updates weights as follows:

wit+1 ← wit · exp(−εℓit ). (13.2)

Theorem 13.7. Consider a fixed ε ≤ 1/2. For any sequences of loss vectors
in [−1, 1] N and for all indices i ∈ [ N ], the Hedge algorithm guarantees:
T T
ln N
∑ ⟨ pt , ℓt ⟩ ≤ ∑ ℓit + εT + ε
i =1 t =1
 online learning: experts and bandits 175

Proof. As in previous proofs, let Φt = ∑ j wtj , so that Φ1 = N, and

∑ wit+1 = ∑ wit e−εℓi

t
Φ t +1 =
i



≤∑ wit 1 − εℓit + ε2 ℓit )2 (using e x ≤ 1 + x + x2 ∀ x ∈ [−1, 1])
i
≤ ∑ wit (1 + ε2 ) − ε ∑ wit ℓit (because |ℓit | ≤ 1)
= (1 + ε2 )Φt − εΦt ⟨ pt , ℓt ⟩ (because wit = pit · Φt )


= Φ t 1 + ε2 − ε ⟨ p t , ℓ t ⟩
2 − ε ⟨ p t ,ℓ t ⟩
≤ Φt eε (using 1 + x ≤ e x )

Again, comparing to the final weight of the ith coordinate,

t ( t +1) 2 T −ε
∑⟨ pt ,ℓit ⟩ ;
e − ε ∑ ℓi = wi ≤ Φ T +1 ≤ Φ 1 e ε

now using Φ1 = N and taking logs proves the claim.

q √
Moreover, choosing ε = lnTN gives εT + lnεN = 2 T ln N, and the
regret term is concave and sublinear in time T. This suggests that the
further we run the algorithm, the quicker the average regret goes to
zero, which suggests the algorithm is in some sense “learning".

13.4.2 Two Useful Corollaries

The following corollary will be useful in many contexts: it just flips
Theorem 13.7 on its head, and shows that the average regret is small
after sufficiently many steps.
4 log N
Corollary 13.8. For T ≥ ε2
, the average loss of the Hedge algorithm is

1 1
T ∑⟨ pt , ℓt ⟩ ≤ min
i T ∑ ℓit + ε
t t
1

= min
p∗ ∈∆ N T
∑ ℓt , p∗ + ε.
t

The viewpoint of the last expression is useful, since it indicates

that the dynamic strategy given by Hedge for the dot-product game
is comparable (in the sense of having tiny regret) against any fixed
strategy p∗ in the probability simplex.
Finally, we state a further corollary that is useful in future lectures.
It can be proved by running Corollary 13.8 with losses ℓt = − gt /ρ.

Corollary 13.9 (Average Gain). Let ρ ≥ 1 and ε ∈ (0, 1/2). For any
4ρ2 ln N
sequence of gain vectors g1 , . . . , g T ∈ [−ρ, ρ] N with T ≥ ε2 , the gains
version of the Hedge algorithm produces probability vectors pt ∈ ∆ N such
that
1 T
t t 1 T
t 
∑
T t =1
g , p ≥ max ∑ g , ei − ε.
i ∈[ N ] T t=1
176 optional: the bandit setting

In passing we mention that if the gains or losses lie in the range

N
[−γ, ρ], then we can get an asymmetric guarantee of T ≥ 4γρεln 2 .

13.5 Optional: The Bandit Setting

The model of experts or the dot-product problem is often called the

full-information model, because the algorithm gets to see the entire
loss vector ℓt at each step. (Recall that we view the entries of the
probability vector pt played by the algorithm as the probability of


playing each of the actions, and hence ℓt , pt is just the expected
loss incurred by the algorithm. Now we consider a different model,
where the algorithm only gets to see the loss of the action it plays.
Specifically, in each round,

1. The algorithm again produces a vector of probabilities

pt = ( p1t , p2t , · · · , ptN ) ∈ ∆ N .

It then chooses an action at ∈ [ N ] with these marginal probabili-

ties.

2. In parallel, the adversary produces

ℓt = (ℓ1t , ℓ2t , · · · , ℓtN ) ∈ [−1, 1] N .

However, now the algorithm only gets to see the loss ℓtat corre-
sponding to the action chosen by the algorithm, and not the entire
loss vector.

This limited-information setting is called the bandit setting. The name comes from the analysis of
slot machines, which are affectionately
known as “one-armed bandits”.
13.5.1 The Exp3 Algorithm
Surprisingly, we can obtain algorithms for the bandit setting from
algorithms for the experts setting, by simply “hallucinating” the cost
vector, using an idea called importance sampling. This causes the
parameters to degrade, however.
Indeed, consider the following algorithm: we run an instance A
of the RWM algorithm, which is in the full information model. So at
each timestep,

1. A produces a probability vector pt ∈ ∆ N .

2. We choose an expert I t ∈ [ N ], where

1
Pr[ I t = i ] = qit := γ · + (1 − γ) · pit .
N
I.e., with probability γ we pick a uniformly random expert, else we
follow the suggestion given by pt .
 online learning: experts and bandits 177

3. We get back the loss value ℓtI t for this chosen expert.

4. We construct an “estimated loss” ℓ̂t ∈ [0, 1] N by setting

 t

 ℓ j if j = I t
t t
ℓ̃ j = q j .

0 if j ̸= I t

We now feed ℓ̃t to the RWM instance A, and go back to Step 1.

We now show this algorithm achieves low regret. The first obser-
vation is that the estimated loss vector is an unbiased estimate of the
actual loss, just because of the way we reweighted the answer by the
inverse of the probability of picking it. Indeed,
ℓit t
E[ℓ̃it ] = · q + 0 · (1 − qit ) = ℓit . (13.3)
qit i
Since each true loss value lies in [−1, 1], and each probability value
is at least γ/N, the absolute value of each entry in the ℓ̃ vectors is at
most N/γ. Now, since we run RWM on these estimated loss vectors
belonging to [0, N/γ] N , we know that
 

t t N log N
∑ p , ℓ̃ ≤ ∑ i γ
ℓ̃ t
+ εT +
ε
.
t t

Taking expectations over both sides, and using (13.3),

 

t t N log N
∑ p , ℓ ≤ ∑ i γ
ℓ t
+ εT +
ε
.
t t

However, the LHS is not our real loss, since we chose I t according to
qt and not pt . This means our expected total loss is really


 γ

∑ qt , ℓt = (1 − γ) ∑ pt , ℓt + N ∑ 1, ℓt
t t t
 
N log N
≤ ∑ ℓit + εT + + γT.
t γ ε
q √  log N 1/4
log N
Now choosing ε = T and γ = N T gives us a regret
of ≈ N 1/2 T 3/4 . The interesting fact here is that the regret is again
sub-linear in T, the number of timesteps: this means that as T → ∞,
the per-step regret tends to zero.
The dependence on N, the number of experts/options, is now
polynomial, instead of being logarithmic as in the full-information
√
case. This is necessary: there is a lower bound of Ω( NT ) in the
bandit setting. And indeed, the Exp3 algorithm itself achieves a near-
p
optimal regret bound of O( NT log N ); we can show this by using a
finer analysis of Hedge that makes more careful approximations. We
defer these improvements for now, and instead give an application of
this bandit setting to a problem in item pricing.
178 optional: the bandit setting

13.5.2 Item Pricing via Bandits

To be added in.
14
Solving Linear Programs using Experts

We can now use the low-regret algorithms for the experts problem to
show how to approximately solve linear programs (LPs). As a warm-
up, we use it to solve two-player zero-sum games, which are a special
case of LPs. In fact, zero-sum games are equivalent
to linear programming, see this work of
Ilan Adler. Is there an earlier reference?
14.1 (Two-Player) Zero-Sum Games

There are two players in such a game, traditionally called the “row
player" and the “column player". Each of them has some set of ac-
tions: the row player with m actions (associated with the set [m]), and
the column player with the n actions in [n]. Finally, we have a payoff
matrix M ∈ Rm×n . In a play of the game, the row player chooses a
row i ∈ [m], and simultaneously, the column player chooses a column
j ∈ [n]. If this happens, the row player gets Mi,j , and the column
player loses Mi,j . The winnings of the two players sum to zero, and
so we imagine that the payoff is from the row player to the column
player. Henceforth, when we talk about pay-
offs, these will always refer to payoffs to
the row player from the column player.
14.1.1 Strategies, and Best-Response This payoff may be negative, which
would capture situations where the
Each player is allowed to have a randomized strategy. Given strate- column player does better.
gies p ∈ ∆m for the row player, and q ∈ ∆n for the column player, the
expected payoff (to the row player) is

E[payoff to row] = p⊺ Mq = ∑ pi q j Mi,j .

i,j

The row player wants to maximize this value, while the column
player wants to minimize it.
Suppose the row player fixes a strategy p ∈ ∆m . Knowing p, the
column player can choose an action to minimize the expected payoff:

C ( p) := min p⊺ Mq = min p⊺ Me j .
q∈∆n j∈[n]
180 (two-player) zero-sum games

The equality holds because the expected payoff is linear, and hence
the column player’s best strategy is to choose a column that mini-
mizes the expected payoff. The column player is said to be playing
their best response. Analogously, if the column player fixes a strategy
q ∈ ∆n , the row player can maximize the expected payoff by playing
their own best response:

R(q) := max p⊺ Mq = max ei⊺ Mq.

p∈∆m i ∈[m]

Now, the row player would love to play the strategy p such that
even if the column player plays best-response, the payoff is as large
as possible: i.e., it wants to achieve

max C ( p).
p∈∆m

Similarly, the column player wants to choose q to minimize the payoff

against a best-response row player, i.e., to achieve

min R(q).
q∈∆n

Lemma 14.1. For any p ∈ ∆m , q ∈ ∆n , we have

C ( p) ≤ R(q) (14.1)

Proof. Intuitively, since the column player commits to a strategy

q, it hence gives more power to the row player. Formally, the row
player could always play strategy p in response to q, and hence could
always get value C ( p). But R(q) is the best response, which could be
even higher.

Interestingly, there always exist strategies p ∈ ∆m , q ∈ ∆n which

achieve equality. This is formalized by the following theorem:

Theorem 14.2 (Von Neumann’s Minimax Theorem). For any finite

zero-sum game M ∈ Rm×n ,

max C ( p) = min R(q).

p∈∆m q∈∆n

This common value V is called the value of the game M.

Proof. We assume for the sake of contradiction that ∃ M ∈ [−1, 1]m×n

such that max p∈∆m C ( p) ≤ minq∈∆n R(q) − δ for some δ > 0. (The
assumption that Mij ∈ [−1, 1] follows by scaling. Now we use the
fact that the average regret of the Hedge algorithm tends to zero to
construct strategies pb and qb that have R(qb) − C ( pb) < δ, thereby giving
us a contradiction.
We consider an instance of the experts problem, with m experts,
one for each row of M. At each time step t, the row player produces
 solving linear programs using experts 181

 
pt ∈ ∆m . Initially p1 = m1 , . . . , m1 , which represents that the row
player chooses each row with equal probability, when they have no
information to work with.
At each time t, the column player plays the best-response to pt , i.e.,
jt := arg max ( pt )⊺ Me j .
j∈[n]

This defines a gain vector for the row player:

gt := Me jt ,
which is the jth column of M. The row player uses this to update the
weights and get pt+1 , etc. Define
T T
1 1
pb :=
T ∑ pt and qb :=
T ∑ e jt
t =1 t =1

to be the average long-term plays of the row player, and of the best
responses of the column player to those plays. We know that
C ( pb) ≤ R(qb)
4 ln m
by (14.1). But by Corollary 13.9, after T ≥ ε2
steps,
1 1

T ∑⟨ pt , gt ⟩ ≥ max
i T∑
ei , g t − ε (by Hedge)
t t
D 1 E
= max ei , ∑ gt − ε
i T t
D 1 E
= max ei , M
i
∑
T t
e jt −ε (by definition of gt )

= max⟨ei , Mb
y⟩ − ε
i
= R(qb) − ε.
Since pt is the row player’s strategy, and C is concave (i.e., the payoff
on the average strategy pb is no more than the average of the payoffs: To see this, recall that

1 1 1  C ( p) := min p⊺ Mq.
T ∑ ⟨ pt , gt ⟩ = ∑ C ( pt ) ≤ C
T T ∑ pt = C ( xb). q

Let q∗be the optimal value of q that

Putting it all together: minimizes C ( p). Then for any a, b ∈ ∆m ,
we have that
R(qb) − ε ≤ C ( pb) ≤ R(qb).
C ( a + b) = ( a + b)⊺ Mq∗ = a⊺ Mq∗ + b⊺ Mq∗
Now for any δ > 0 we can choose ε < δ to get the contradiction. ≥ min a⊺ Mq + min b⊺ Mq = C ( a) + C (b)
q q

Observe that the proof gives us an explicit algorithm to find strate-

gies pb, qb that have a small gap. The minimax theorem is also im-
plied by strong duality of linear programs: indeed, we can write
minq∈∆n R(q) as a linear program, take its dual and observe that it
computes min p∈∆m C ( p). The natural question is: we can solve linear
programs using low-regret algorithms. We now show how to do this.
We should get a clean proof of strong duality this way?
182 solving lps approximately

14.2 Solving LPs Approximately

Consider an LP with constraint matrix A ∈ Rm×n :

max ⟨c, x ⟩ (14.2)

Ax ≤ b
x≥0

Suppose x ∗ is an optimal solution, with OPT := ⟨c, x ∗ ⟩. Let K ⊆ Rn

be the polyhedron defined by the “easy” constraints, i.e.,

K := { x ∈ Rn | ⟨c, x ⟩ = OPT, x ≥ 0},

where OPT is found by binary search over possible objective values.

Binary search over the reals is typically not a good idea, since it may
never reach the answer. (E.g., searching for 1/3 by binary search over
[0, 1].) However, we defer this issue for now, and imagine we know
the value of OPT. We now use low-regret algorithms to find xb ∈ K
such that ⟨ ai , x ⟩ ≤ bi + ε for all i ∈ [m]. The fix to the “binary search over reals”
problem is this: the optimal value of a
linear program in n dimensions where
14.2.1 The Oracle all numbers integers using at most b
bits is a rational p/q, whereboth p, q use
The one assumption we make is that we can solve the feasibility at most poly(nb) bits. So once we the
granularity of the search is fine enough,
problem obtained by intersecting the “easy” constraints K with a
there is a unique rational close the
single linear constraint. Suppose α ∈ Rn , β ∈ R, then we want to query point, and we can snap to it. See,
solve the problem: e.g., the problem on finding negative
cycles in the homeworks.
Oracle: find a point x ∈ K ∩ { x | ⟨α, x ⟩ ≤ β}. (14.3)

Proposition 14.3. There is an O(n)-time algorithm to solve (14.3), when

K = { x ≥ 0 | ⟨c, x ⟩ = OPT }.
Proof. We give the proof only for the case where ci > 0 for all i; the
general case is left as an exercise. Let j∗ := arg min j α j/c j , and define
x = (OPT/c j∗ )e j∗ . Say “infeasible” if x does not satisfy ⟨α, x ⟩ ≤ β, else
return x.

Of course, this problem can be solved in time linear in the num-

ber of variables (as Proposition 14.3 above shows), but the situation
can be more interesting when the number of variables is large. For
instance, when we solve flow LPs, the number of variables will be
exponential in the size of the graph, yet the oracle will be imple-
mentable in time poly(n).

14.2.2 The Algorithm

The key idea to solving general LPs is then similar to that for zero-
sum games. We have m experts, one corresponding to each con-
straint. In each round, we combine the multiple constraints using a
 solving linear programs using experts 183

weighted sum, we call the above oracle on this single-constraint LP

to get a solution, we construct a gain vector from this solution and
feed this to Hedge, which then updates the weights that we use for
the next round. The gain of an expert in a round is based based on
how badly the constraint was violated by the current solution. The
intuition is simple: greater violation means more gain, and hence
more weight in the next iteration, which forces us to not violate the
constraint as much.
An upper bound on the maximum possible violation is the width
ρ of the LP, defined by

ρ := max {| ⟨ ai , x ⟩ − bi |}. (14.4)

x ∈K,i ∈[m]

We assume that ρ ≥ 1.

Algorithm 12: LP-Solver

12.1 p1 ← (1/m, . . . , 1/m). T ← Θ(ρ2 ln m/ε2 )
12.2 for t = 1 to T do
12.3 Define αt := ∑im=1 pit ai ∈ Rn and βt = ∑im=1 pit bi ∈ R.


12.4 Use Oracle to find x ∈ K ∩ { αt , x t ≤ βt }.
12.5 if oracle says infeasible then
12.6 return infeasible
12.7 else
12.8 git ← ⟨ ai , x t ⟩ − bi for all i.
12.9 feed gt to Hedge(ε) to get pt+1 .
12.10 return xb ← ( x1 + · · · + x T )/T.

14.2.3 The Analysis

Theorem 14.4. Fix 0 ≤ ε ≤ 1/4. Then Algorithm 12 calls the oracle
O(ρ2 ln m/ε2 ) times, and either correctly returns “infeasible”, or returns
xb ∈ K such that
Ab x ≤ b − ε1.
Proof. Observe that if x ∗ is feasible for the original LP (14.2) then it is
feasible for any of the calls to the oracle, since it satisfies any positive
linear combination of the constraints. Hence, we are correct if we ever
return “infeasible”. Moreover, x t ∈ K in each iteration, and xb is an
average of x t ’s, so it also lies in K by convexity. So it remains to show
that xb approximately satisfies the other linear constraints.
Recall the guarantee from Corollary 13.9:
1 1

T ∑⟨ pt , gt ⟩ ≥ max
i T∑
ei , gt − ε, (14.5)
t t

for precisely the choice of T in Algorithm 12, since the definition of

width in (14.4) ensures that gt ∈ [−ρ, ρ]m .
184 solving lps approximately

Let i ∈ [m], and recall the definitions of αt = ∑im=1 pit ai , βt =

∑im=1
pit bi , and gt = Ax t − b from the algorithm. Then

⟨ pt , gt ⟩ = ⟨ pt , Ax t − b⟩
= ⟨ pt , Ax t ⟩ − ⟨ pt , b⟩
= ⟨αt , x t ⟩ − βt ≤ 0,

the last inequality because x t satisfies the single linear constraint

t 
α , x ≤ βt . Averaging over all times, the left hand side of (14.5) is

T
1
T ∑ ⟨ pt , gt ⟩ ≤ 0.
t =1

However, the average on the RHS in (14.5) for constraint/expert i is:

1 T
 D 1 T E
T ∑ ei , g t = ei ,
T ∑ gt
t =1 t =1

1 T 
 
=
T ∑ ai , xbt − bi
t =1
= ⟨ ai , xb⟩ − bi .

Substituting into (14.5) we have

T

1
0≥
T ∑ ⟨ pt , gt ⟩ ≥ max
i
⟨ ai , xb⟩ − bi − ε.
t =1

x ≤ b + ε1.
This shows that Ab

14.2.4 A Small Extension: Approximate Oracles

Recall the definition of the problem width from (14.4). A few com-
ments:

• In the above analysis, we do not care about the maximum value

of | a⊺i x − bi | over all points x ∈ K, but only about the largest this
expression gets over points that are potentially returned by the
oracle. This seems a pedantic point, but if there are many solutions
to (14.3), we can return one with small width. But we can do more,
as the next point outlines.

• We can also relax the oracle to satisfy ⟨α, x ⟩ ≤ β + δ for some

small δ > 0 instead. Define the width of the LP with respect such a
relaxed oracle to be

ρrlx := max {| a⊺i x − bi |}. (14.6)

i ∈[m],x returned by relaxed oracle
 solving linear programs using experts 185

Now running the algorithm with a relaxed oracle gives us a

slightly worse guarantee that

x ≤ b + (ε + δ)1,
Ab

but now the number of calls to the relaxed oracle can be even
smaller, namely O(ρ2rlx ln m/ε2 ).

• Of course, if we violations can be bounded in some better way,

e.g., if we can ensure that violations are always positive or nega-
tive, then we can give stronger bounds on the regret, and hence
reduce the number of calls even further. Details to come.

All these improvements will be crucial in the upcoming applications.

 15
Approximate Max-Flows using Experts

We now use low-regret multiplicative-weight algorithms to give

approximate solutions to the s-t-maximum-flow problem. In the
previous chapter, we already saw how to get approximate solutions
to general linear programs. We now show how a closer look at those
algorithms give us improvements in the running time (albeit in the
setting of undirected graphs), which go beyond those known via
usual “combinatorial” techniques. The first set of results we give will
hold for directed graphs as well, but the improved results will only
hold for undirected graphs.

15.1 The Maximum Flow Problem

In the s-t maximum flow problem, we are given a graph G = (V, E), and
distinguished vertices s and t. Each edge has a capacities ue ≥ 0; we
will mostly focus on the unit-capacity case of ue = 1 in this chapter.
The graph may be directed or undirected; an undirected edge can be
modeled by two oppositely directed edges having the same capacity.
Recall that an s-t flow is an assignment f : E → R+ such that

(a) f (e) ∈ [0, ue ], i.e., capacity-respecting on all edges, and

(b) ∑e=(u,v)∈E f (e) = ∑e=(v,w)∈E f (e), i.e., flow-conservation at all

non-{s, t}-nodes.

The value of flow f is ∑e=(s,w)∈E f (e) − ∑e=(u,s)∈E f (e), the net

amount of flow leaving the source node s. The goal is to find an
s-t flow in the network, that satisfies the edge capacities, and has
maximum value.
Algorithms by Edmonds and Karp, by Yefim Dinitz, and many
others can solve the s-t max-flow problem exactly in polynomial
time. For the special case of (directed) graphs with unit capaci-
ties, Shimon Even and Bob Tarjan, and independently, Alexander
Karzanov showed in 1975 that the Ford-Fulkerson algorithm finds
188 a first algorithm using the mw framework

the maximum flow in time O(m · min(m1/2 , n2/3 )). This runtime
was eventually matched for general capacities (up to some poly-
logarithmic factors) by an algorithm of Andrew Goldberg and Satish
Rao in 1998. For the special case of m = O(n), these results gave a
runtime of O(m1.5 ), but nothing better was known even for approx-
imate max-flows, even for unit-capacity undirected graphs—until a
breakthrough in 2010, which we will see at the end of this chapter.

15.1.1 A Linear Program for Maximum Flow

We formulate the max-flow problem as a linear program. There are

many ways to do this, and we choose to write an enormous LP for
it. Let P be the set of all s-t paths in G. Define a variable f P denoting
the amount of flow going on path P ∈ P . We can now write:

max ∑ fP (15.1)
P∈P

∑ f P ≤ ue ∀e ∈ E
P:e∈ P
fP ≥ 0 ∀P ∈ P

The first set of constraints says that for each edge e, the contribution
of all possible flows is no greater than the capacity ue of that edge.
The second set of constraints say that the contribution from each path
must be non-negative. This is a gigantic linear program: there could
be an exponential number of s-t paths. As we see, this will not be a
hurdle.

15.2 A First Algorithm using the MW Framework

To using the framework from the previous section, we just need

to implement the Oracle: i.e., we solve a problem with a single
“average” constraint, as in (14.3). Specifically, suppose we want a
flow value of F, then the “easy” constraints are:

K := { f | ∑ f p = F, f ≥ 0}.
P∈P

Moreover, the constraint ⟨α, f ⟩ ≤ β is not an arbitrary constraint—it

is one obtained by combining the original constraints. Specifically,
given a vector pt ∈ ∆m , the average constraint is obtained by the
convex combination of these constraints:
 
∑ pte ∑ f P ≤ ue , (15.2)
e∈ E P:e∈ P
 approximate max-flows using experts 189

where f e represents the net flow over edge e. By swapping order of

summations, and using the unit capacity assumption, we obtain
 
∑ f P ∑ pte ≤ ∑ pte ue = 1.
P∈P e∈ P e

Now, the inner summation is the path length of P with respect to

edge weights pte , which we denote by lent ( P) := ∑e∈ P pte . The con-
straint now becomes:

∑ f P lent ( P) ≤ 1, (15.3)
P∈P

and we want a point f ∈ K satisfying it. The best way to satisfy it

is to place all F units of flow on the shortest path P, and zero every-
where else; we output “infeasible” if the shortest-path has a length
more than 1. This step can be done by a single call to Dijkstra’s algo-
rithm, which takes O(m + n log n) time.   We already argued in Theorem 14.4
ρ2 log m that if there exists a feasible flow of
Now Theorem 14.4 says that running this algorithm for Θ ε2 value F in the graph, we never output
iterations gives a solution f ∈ K, that violates the constraints by “infeasible”. Here is a direct proof.
If there is a flow of value F, there are
an additive ε. Hence, the scaled-down flow f /(1 + ε) would satisfy F disjoint s-t paths. The vector pt ∈ ∆m ,
all the capacity constraints, and have flow value F/(1 + ε), which so its values sum to 1. Hence, one of
is what we wanted. To complete the runtime analysis, it remains to the F s-t paths P∗ has ∑e∈ P pte ≤ 1/F.
Setting f P = F for that path satisfies the
bound the value of ρ, the maximum amount by which any constraint constraint.
gets violated by a solution from the oracle. Since we send all the
F units of flow on a single edge, the maximum violation is F − 1.
Hence the total runtime is at most
F2 log m
O(m + n log n) · .
ε2
Moreover, the maximum flow F is m, by the unit capacity assump-
tion, which gives us an upper bound of O(m3 poly(log m/ε)).

15.2.1 A Better Bound, via an Asymmetric Guarantee for Hedge

Let us state (without proof, for now) a refined version of the Hedge
algorithm for the case of asymmetric gains, where the gains lie in the
range [−γ, ρ].

Theorem 15.1 (Asymmetric Hedge). Let ε ∈ (0, 1/2), and γ, ρ ≥ 1.

Θ(γρ ln N )
Moreover, let T ≥ ε2
. There exists an algorithm for the experts
problem such that for every sequence g1 , . . . , g T of gains with g ∈ [−γ, ρ] N ,
produces probability vectors { pt ∈ ∆ N }t∈[T ] online such that for each i:
T
 T

1 1
T ∑ gt , pt ≥
T ∑ gt , ei − ε.
t =1 t =1

The proof is a careful (though not difficult) reworking of the

standard proof for Hedge. (We will add it soon; a hand-written
190 finding max-flows using electrical flows

proof is on the webpage.) Moreover, we can use this statement to

Θ(γρ ln m)
prove that the approximate LP solver can stop after ε2
calls
to an oracle, as long as each of the oracle’s answer x guarantee that
( Ax )i − bi ∈ [−γ, ρ].
Since a solution f found by our shortest-path oracle sends all F
flow on a single path, and all capacities are 1, we have γ = 1 and
ρ = F − 1 ≤ F. The runtime now becomes
1 · ( F − 1) log m
O(m + n log n) · .
ε2
Again, using the naïve bound of F ≤ m, we have a runtime of
O(m2 poly(log m/ε)) to find a (1 + ε)-approximate max-flow, even
in directed graphs.

15.2.2 An Intuitive Explanation and an Example

Observe that the algorithm repeats the following natural process:

1. it finds a shortest path in the graph,

2. it pushes F units of flow on it, and then

3. it increases the length of each edge on this path multiplicatively. The factor happens to be (1 + ε/F ), be-
cause of how we rescale the gains, but
This length-increase makes congested edges (those with a lot of flow) that does not matter for this intuition.
be much longer, and hence become very undesirable when search-
ing for short paths. Note that the process is repeated some number
of times, and then we average all the flows we find. So unlike usual
network flow algorithms based on residual networks, these algo-
rithms are truly greedy and cannot “undo” past actions (which is
what pushing flow in residual flow networks does, when we use an
arc backwards). This means these MW-based algorithms must ensure
that very little flow goes on edges that are “wasteful”.
To illustrate this point, consider an example commonly used to
show that the greedy algorithm does not work for max-flow: Change
the figure to make it more instructive.

15.3 Finding Max-Flows using Electrical Flows

The approach of the previous sections suggests a way to get faster

algorithms for max-flow: reduce the width of the oracle. The approach
of the above section was to push all F flow along a single path, which
is why we have a width of Ω( F ). Can we implement the oracle in a
way that spreads the flow over several paths, and hence has smaller
width? Of course, one such solution is to use the max-flow as the
oracle response, but that would defeat the purpose of the MW ap-
proach. Indeed, we want a fast way of implementing the oracle. We use the notation O e ( f (n)) to hide
factors that are poly-logarithmic in
f (n). E.g., O(n log2 n) lies in Oe (n), and
O(log n log log n) lies in O e (log n), etc.
 approximate max-flows using experts 191

For undirected graphs, one good solution turns out to be to use

electral flows: to model the graph as an electrical network, set a volt-
age difference between s and t, and compute how electrical current
would flow between them. We now show how this approach gives us
an O e (m1.5 /εO(1) )-time algorithm quite easily; then with some more
work, we improved this to get a runtime of O e (m4/3 /εO(1) ). While we
focus only on unit-capacity graphs, the algorithm can be extended
to all undirected graphs with a further loss of poly-logarithmic fac-
tors in the maximum capacity, and moreover to get a runtime of
Oe (mn1/3 / poly(ε)).
At the time this result was announced (by Christiano et al.), it was Christiano, Kelner, Madry, Spielman,
the fastest algorithm for the approximate maximum s-t-problem in Teng (2010)

undirected graphs. Since then, works by Jonah Sherman, and Kelner Sherman (2013)
et al. gave O(m1+o(1) /εO(1) )-time algorithms for the problem. The Kelner, Lee, Oracchia, Sidford (2013)
current best runtime is O(m poly log m/εO(1) )-time, due to Richard Peng (2014)
Peng. Interestingly, Shang-Hua Teng, Jonah
Sherman, and Richard Peng are all
CMU graduates.
15.3.1 Electrical Flows
Given a connected undirected graph with general edge-capacities, we
can view it as an electrical circuit, where each edge e of the original 𝜑 𝑠 =1 𝜑 𝑡 =0

graph represents a resistor with resistance re = 1/ue , and we connect s t

(say, a 1-volt) battery between s to t. This causes electrical current to

flow from s (the node with higher potential) to t. Recall the following
laws about electrical flows. + -

Theorem 15.2 (Kirchoff’s Voltage Law). The directed potential changes

Figure 15.1: The currents on the wires
along any cycle sum to 0. would produce an electric flow (where
all the wires within the graph have
This means we can assign each node v a potential ϕv . Now the resistance 1).
actual amount of current on any edge is given by Ohm’s law, and is
related to the potential drop across the edge.

Theorem 15.3 (Ohm’s Law). The electrical flow f uv on the edge e = uv is

the ratio between the difference in potential ϕ (or voltage) between u, v and
the resistance re of the edge:

ϕu − ϕv
f uv = .
ruv

Finally, we have flow conservation, much like in traditional net-

work flows:

Theorem 15.4 (Kirchoff’s Current Law). If we set s and t to some volt-

ages, the electrical current ensures flow-conservation at all nodes except s, t:
the total current entering any non-terminal node equals the current leaving
it.
192 finding max-flows using electrical flows

These laws give us a set of linear constraints that allow us to go

between the voltages and currents. In order to show this, we define
the Laplacian matrix of a graph.

15.3.2 The Laplacian Matrix

The conductance of an edge is the
Given an undirected graph on n nodes and m edges, with non- reciprocal of the resistance of the edge:
negative conductances cuv for each edge e = uv, we define the Lapla- ce = 1/re .

cian matrix to be a n × n matrix LG , with entries


 ∑w:uw∈E cuw
 if u = v
( LG )uv = −cuv if (u, v) ∈ E .

 0 otherwise

For example, if we take the 6-node graph in Figure 15.1 and assume
that all edges have unit conductance, then its Laplacian LG matrix is:

s t u v w x
 
s 2 0 −1 −1 0 0
 
t  0 2 0 0 −1 −1 
 
u −1 0 3 0 −1 −1 
.
LG =
v 
 −1 0 0 2 0 −1 

 
w 0 −1 −1 0 2 0 
x 0 −1 −1 −1 0 3

Note that LG is not a full-rank matrix

Equivalently, we can define the Laplacian matrix Luv for the graph since, e.g., the columns sum to zero.
However, if the graph G is corrected,
consisting of a single edge uv as then the vector 1 is the only vector in
the kernel of LG , so its rank is n − 1.
(proof?)
Luv := cuv (eu − ev )⊺ (eu − ev ).

Now for a general graph G, we define the Laplacian to be: This Laplacian for the single edge uv
has 1s on the diagonal at locations
(u, u), (v, v), and −1s at locations
LG = ∑ Luv . (u, v), (v, u). Draw figure.
uv∈ E

In other words, LG is the sum of little ‘per-edge’ Laplacians Luv . A symmetric matrix A ∈ Rn×n is called
(Since each of those Laplacians is clearly positive semidefinite (PSD), PSD if x⊺ Ax ≥ 0 for all x ∈ Rn , or
equivalently, if all its eigenvalues are
it follows that LG is PSD too.) non-negative.
For yet another definition for the Laplacian, first consider the
edge-vertex incidence matrix B ∈ {−1, 0, 1}m×n , where the rows are
indexed by edges and the columns by vertices. The row correspond-
ing to edge e = uv has zeros in all columns other than u, v, it has
an entry +1 in one of those columns (say u) and an entry −1 in the
 approximate max-flows using experts 193

other (say v).

su sv uw ux vx wt xt
 
s 1 1 0 0 0 0 0
 
t  0 0 0 0 0 −1 −1 
 
u −1 0 1 1 0 0 0 .
B=
v 
 0 −1 0 0 1 0 0 
 
w 0 0 −1 0 0 1 0 
x 0 0 0 −1 −1 0 1

The Laplacian matrix is now defined as LG := B⊺ CB, where C ∈

Rm×m is a diagonal matrix with entry Cuv containing the conductance
for edge uv. E.g., for the example above, here’s the edge-vertex inci-
dence matrix, and since all conductances are 1, we have LG = B⊺ B.

15.3.3 Solving for Electrical Flows: Lx = b

Given the Laplacian matrix for the electrical network, we can figure
out how the current flows by solving a linear system, i.e., a system of
linear equations. Indeed, by Theorem 15.4, all the current flows from
s to t. Suppose k units of current flows from s to t. By Theorem 15.3,
the net current flow into a node v is precisely
ϕu − ϕv
∑ f uv = ∑ ruv
.
u:uv∈ E u:uv∈ E

A little algebra shows this to be the vth entry of the vector Lϕ. Finally,
by 15.4, this net current into v must be zero, unless v is either s or t,
in which case it is either −k or k respectively. Summarizing, if ϕ are
the voltages at the nodes, they satisfy the linear system:

Lϕ = k(es − et ).

(Recall that k is the amount of current flowing from s to t, and es , et

are elementary basis vectors.) It turns out the solutions ϕ to this
linear system are unique up to translation, as long as the graph is
connected: if ϕ is a solution, then {ϕ + a | a ∈ R} is the set of all
solutions.
Great: we have n + 1 unknowns so far: the potentials at all the
nodes, and the current value k. The above discussion gives us poten-
tials at all the nodes in terms of the current value k. Now we can set
unit potential at s, and ground t (i.e., set its potential to zero), and
solve the linear system (with n − 1 linearly independent constraints)
for the remaining n − 1 variables. The resulting value of k gives us
the s-t current flow. Moreover, the potential settings at all the other
nodes can now be read off from the ϕ vector. Then we can use Ohm’s
law to also read off the current on each edge, if we want.
194 finding max-flows using electrical flows

How do we solve the linear system Lϕ = b (subject to these bound-

ary conditions)? We can use Gaussian elimination, of course, but
currently takes nω time in the worst-case. Thankfully, there are faster
(approximate) methods, which we discuss in §15.3.5.

15.3.4 Electrical Flows Minimize Energy Burn

Here’s another useful way of characterizing this current flow of k
units from s and t: the current flow is one minimizing the total energy
dissipated. Indeed, for a flow f , the energy burn on edge e is given by
(ϕu −ϕv )2
( f uv )2 ruv = ruv , and the total energy burn is

(ϕu − ϕv )2
E ( f ) := ∑ f e2 re = ∑ r uv
= ϕ⊺ Lϕ.
e∈ E (u,v)∈ E

The electrical flow f produced happens to be

arg min f is an s-t flow of value k {E ( f )}.

We often use this characterization when arguing about electrical

flows.

15.3.5 Solving Linear Systems

We can solve a linear system Lx = b fast? If L is a Laplacian matrix
and we are fine with approximate solutions, we can do things much
faster than Gaussian elimination. A line of work starting with Dan
Spielman and Shang-Hua Teng, and then refined by Ioannis Koutis, Spielman and Teng (200?)
Gary Miller, and Richard Peng shows how to (approximately) solve Koutis, Miller, and Peng (2010)
a Laplacian linear system in the time essentially near-linear in the
number of non-zeros of the matrix L.

Theorem 15.5 (Laplacian Solver). There exists an algorithm that given a

linear system Lx = b with L being a Laplacian matrix (and having solution
x̄), find a vector x̂ such that the error vector z := L x̂ − b satisfies Given a positive semidefinite matrix
A, the A-norm is defined as ∥ x ∥ A :=
√
z⊺ Lz ≤ ε( x̄⊺ L x̄ ). x⊺ Ax. Hence the guarantee here says

∥ L x̂ − b∥ L ≤ ε ∥ x̄ ∥ L .
The algorithm is randomized? and runs in time O(m log2 n log 1/ε).

Moreover, Theorem 15.5 can be converted to what we need; details

appear in the Christiano et al. paper.

Corollary 15.6 (Laplacian Solver II). There is an algorithm given a linear

system Lx = b corresponding to an electrical system as above, outputs an
electrical flow f that satisfies

E ( f ) ≤ (1 + δ)E ( fe),
 approximate max-flows using experts 195

where fe is the min-energy flow. The algorithm runs in O e ( m log R ) time,

δ
where R is the ratio between the largest and smallest resistances in the
network.

For the rest of this lecture we assume we can compute the corre-
sponding minimum-energy flow exactly in time O e (m). The arguments
can easily be extended to incorporate the errors.

15.4 e (m3/2 )-time Algorithm

An O

Recall the setup from §15.2: given the polytope

K = {f | ∑ f P = F, f ≥ 0},
P∈P

and some edge weights pe , we wanted a vector in K that satisfies

∑ pe f e ≤ 1. (15.4)
e

where f e := ∑ P:e∈ P f P . Previously, we set f P∗ = F for P∗ being the

shortest s-t path according to edge weights pe , but that resulted in the
width—the maximum capacity violation—being too as large as Ω( F ).
So we want to spread the flow over more paths.
Our solution will now be to have the oracle return a flow with
√
width O( m/ε), and which satisfies the following weaker version of
the length bound (15.4) above:

∑ pe f e ≤ (1 + ε) ∑ pe + ε = 1 + 2ε.
e∈ E e∈ E

It is a simple exercise to check that this weaker oracle changes the

analysis of Theorem 14.4 only slightly, still showing that the multiplicative-
weights-based process finds an s-t-flow of value F, but now the edge-
capacities are violated by 1 + O(ε) instead of just 1 + ε.
Indeed, we replace the shortest-path implementation of the ora-
cle by the following electrical-flow implementation: we construct a
weighted electrical network, where the resistance for each edge e is
defined to be This idea of setting the edge length
ε to be pe plus a small constant term is a
ret := pte + .
m general technique useful in controlling
the width in other settings, as we will
We now compute currents f et by solving the linear system Lt ϕ = see in a HW problem.
F (es − et ) and return the resulting flow. It remains to show that this
flow spreads its mass around, and yet achieves a small “length” on
average.

Theorem 15.7. If f ∗ is a flow with value F and f is the minimum-energy

flow returned by the oracle, then

1. (length) ∑e∈E pe f e ≤ (1 + ε) ∑e∈E pe ,

196 o ( m 4/3 ) -time algorithm
optional: an e

√
2. (width) maxe f e ≤ O( m/ε).

Proof. Since the flow f ∗ satisfies all the constraints, it burns energy
ε
E ( f ∗ ) = ∑( f e∗ )2 re ≤ ∑ re = ∑( pe + ) = 1 + ε.
e e e m

Here we use that ∑e pe = 1. But since f is the flow K that minimizes

the energy,
E ( f ) ≤ E ( f ∗ ) ≤ 1 + ε.
Now, using Cauchy-Schwarz,
√ √ r √ √
∑ re f e = ∑ ( re f e · re ) ≤ (∑ re f e2 )(∑ re ) ≤ 1 + ε 1 + ε = 1 + ε.
e e e e

This proves the first part of the theorem. For the second part, we may
use the bound on energy burnt to obtain
ε  ε
∑ f e2 m ≤ ∑ f e2 pe + m = ∑ f e2 re = E ( f ) ≤ 1 + ε.
e e e

Since each term in the leftmost summation is non-negative,

r r
2 ε m (1 + ε ) 2m
fe ≤ 1 + ε =⇒ f e ≤ ≤
m ε ε
for each edge e.

Using this oracle within the MW framework means the width is

√ ρ log m
e (m) time by
ρ = O( m), and each of the O ε2 iterations takes O
Corollary 15.6, giving a runtime of Oe (m3/2 ).
In fact, this bound on the width is tight: consider the example
√
network on the right. The effective resistance of the entire collection Figure 15.2: There are k = Θ( m) black
paths of length k each. All edges have
of black edges is 1, which matches the effective resistance of the red unit capacities.
edge, so half the current goes on the top red edge. If we set F = k + 1
√
(which is the max-flow), this means a current of Θ( m) goes on the
top edge.
Sadly, while the idea of using electrical flows is very cool, the run-
time of O(m3/2 ) is not that impressive. The algorithms of Karzanov,
and of Even and Tarjan, for exact flow on directed unit-capacity
graphs in time O(m min(m1/2 , n2/3 )) were known even back in the
1970s. (Algorithms with similar runtime are known for capacitated
cases too.) Thankfully, this is not the end of the story: we can take
the idea of electrical flows further to get a better algorithm, as we
show in the next section.

15.5 e (m4/3 )-time Algorithm

Optional: An O

The idea to get an improved bound on the width is to use a crude but
effective trick: if we have an edge with electrical flow of more than
 approximate max-flows using experts 197

ρ ≈ m1/3 in some iteration, we delete it for that iteration (and for the
rest of the process), and find a new flow. Clearly, no edge now carries
a flow more than ρ. The main thrust of the proof is to show that we
do not end up butchering the graph, and that the maximum flow
value reduces by only a small amount due to these edge deletions.
Formally, we set:

m 1/3 log m
ρ= . (15.5)
ε
and show that at most εF edges are ever deleted by the process. The
crucial ingredient in this proof is this observation: every time we
delete an edge, the effective resistance between s and t increases by a
lot. We assume that a flow value of F is
Since we need to argue about how many edges are deleted in the feasible; moreover, F ≥ ρ, else Ford-
Fulkerson can be implemented in time
entire algortihm (and not just in one call to the oracle), we explic- O(mF ) ≤ O e (m4/3 ).
itly maintain edge-weights wet , instead of using the results from the
previous sections as a black-box.

15.5.1 The Effective Resistance

Loosely speaking, the effective resistance between nodes u and v is
the resistance offered by the network to electrical flows between u
and v. There are many ways of formalizing this: the most useful one
in this context is the following.

Definition 15.8 (Effective Resistance). The effective resistance be-

tween s and t, denoted by Reff (st), is the energy burned if we send
one unit of electrical current from s to t.

Since we only consider the effective resistance between s and t in

this lecture, we simply write Reff . The following results relate the
effective resistances before and after we change the resistances of
some edges.

Lemma 15.9. Consider an electrical network with edge resistances re .

1. (Rayleigh Monotonicity) If we increase the resistances to re′ ≥ re for all e,

the resulting effective resistance is

′
Reff ≥ Reff .

2. Suppose f is an s-t electrical flow, suppose e is an edge with energy burn

f e2 re ≥ βE ( f ). If we set re′ ← ∞, then the new effective resistance

′ Reff
Reff ≥( ).
1−β
198 o ( m 4/3 ) -time algorithm
optional: an e

Proof. Recall that if we send electrical flow from s to t, the resulting

flow f minimizes the total energy burned E ( f ) = ∑e f e2 re . To prove
the first statement: for each flow, the energy burned with the new
resistances is at least that with the old resistances. Need to add in
second part.

15.5.2 A Modified Algorithm

Let’s give our algorithm that explicitly maintains the edge weights:
We start off with weights w1e = 1 for all e ∈ E. At step t of the
algorithm:

1. Find the min-energy flow f t of value F in the remaining graph

with respect to edge resistances ret := wet + mε W t .

2. If there is an edge e with f et > ρ, delete e (for the rest of the algo-
rithm), and go back to Item 1.

3. Update the edge weights wet+1 ← wet (1 + ρε f et ). This division by ρ

accounts for the edge-capacity violations being as large as ρ.
ρ log m
Stop after T := ε2
iterations, and output fb = 1
T ∑t f t .

15.5.3 The Analysis

Let us first comment on the runtime: each time we find an electrical
flow, we either delete an edge, or we push flow and increment t. The
latter happens for T steps by construction; the next lemma shows that
we only delete edges in a few iterations.

Lemma 15.10. We delete at most m1/3 ≤ εF edges over the run of the
algorithm.

We defer the proof to later, and observe that the total number of
electrical flows computed is therefore O( T ). Each such computation
takes Oe (m/ε) by Corollary 15.6, so the overall runtime of our algo-
rithm is O(m4/3 / poly(ε)).
Next, we show that the flow fb is an (1 + O(ε)-approximate maxi-
mum s-t flow. We start with an analog of Theorem 15.7 that accounts
for edge deletions.

Lemma 15.11. Suppose ε ≤ 1/10. If we delete at most εF edges from G:

1. the flow f t at step t burns energy E ( f t ) ≤ (1 + 3ε)W t ,

2. ∑e wet f et ≤ (1 + 3ε)W t ≤ 2W t , and

3. if fb ∈ K is the flow eventually returned, then fbe ≤ (1 + O(ε)).

 approximate max-flows using experts 199

Proof. We assumed there exists a flow f ∗ of value F that respects

all capacities. Deleting εF edges can only hit εF of these flow paths,
so there exists a capacity-respecting flow of value at least (1 − ε) F.
Scaling up by (1− 1
ε)
, there exists a flow f ′ of value F using each edge
1
to extent (1− ε )
. The energy of this flow according to resistances ret is
at most
1 Wt
E ( f ′) = ∑ r et ( f e′ ) 2 ≤ (1 − ε )2 ∑ r et ≤ (1 − ε )2
≤ ( 1 + 3ε ) W t ,
e e

for ε small enough. Since we find the minimum energy flow, E ( f t ) ≤

E ( f ′ ) ≤ W t ( 1 + 3ε ) . For the second part, we again use the Cauchy-
Schwarz inequality:
r r q
∑ w et f et ≤ ∑ w et ∑ w et ( f et ) 2 ≤ W t · W t ( 1 + 3ε ) ≤ ( 1 + 3ε ) W t ≤ 2W t .
e e e

The last step is very loose, but it will suffice for our purposes.
To calculate the congestion of the final flow, observe that even
though the algorithm above explicitly maintains weights, we can just
wt
appeal directly to the guarantees . Indeed, define p te : = Wet for each
time t; the previous part implies that the flow f t satisfies

∑ p te f et ≤ 1 + 3ε
e

for precisely the p t values that the Hedge-based LP solver would

return if we gave it the flows f 0 , f 1 , . . . , f t − 1 . Using the guarantees
of that LP solver, the average flow bf uses any edge e to at most ( 1 +
3ε ) + ε.

Finally, it remains to prove Lemma 15.10.

Proof of Lemma 15.10. We track two quantities: the total weight W t

and the s-t-effective resistance R eff . First, the weight starts at W 0 =
m, and when we do an update,
 
ε ε
W t + 1 = ∑ w et 1 + f et = Wt + ∑ w et f et
e ρ ρ e
t ε t
≤ W + ( 2W ) (From Claim 15.11)
ρ
ρ ln m
Hence we get that for T = ε2
,
 T    
T 0 2ε 2ε · T 2 ln m
W ≤W · 1+ ≤ m · exp = m · exp .
ρ ρ ε

Therefore, the total weight is at most m 1 + 2/ε . Next, we consider the

s-t-effective resistance R eff .
200 o ( m 4/3 ) -time algorithm
optional: an e

1. At the beginning, all edges have resistance 1 + ε. When we send

F flow, some edge has at least F/m flow on it, so the energy burn
is at least ( F/m ) 2 . This means R eff at the beginning is at least
( F/m ) 2 ≥ 1/m 2 .
2. The weights increase each time we do an update, so R eff does not
decrease. (This is one place it is more convenience to argue about
weights w et explicitly, and not just the probabilities p te .)

3. Each deleted edge e has flow at least ρ, and hence energy burn at
least ( ρ 2 ) w et ≥ ( ρ 2 ) mε W t . Since the total energy burn is at most
2W t from Lemma 15.11, the deleted edge e was burning at least
ρ2 ε
β := 2m fraction of the total energy. Hence
ol d  
new R eff ol d ρ2 ε
R eff ≥ ρ2 ε
≥ R eff · exp
(1 − 2m
2m )
1
if we use 1− x ≥ e x/2 when x ∈ [ 0, 1/4 ] .
4. For the final effective resistance, note that we send F flow with
total energy burn 2W T ; since the energy depends on the square of
f inal T
the flow, we have R eff ≤ 2W F2
≤ 2W T .
(All these calculations hold as long as we have not deleted more than
ε F edges.) Now, to show that this invariant is maintained, suppose D
edges are deleted over the course of the T steps. Then
   
0 ρ2 ε f inal T 2 ln m
R eff exp D · ≤ R eff ≤ 2W ≤ 2m · exp .
2m ε
Taking logs and simplifying, we get that
ερ 2 D 2 ln m
≤ ln ( 2m 3 ) +
2m  ε 
2m ( ln m )( 1 + O ( ε ))
=⇒ D ≤ 2 ≪ m 1/3 ≤ εF.
ερ ε
This bounds the number of deleted edges D as desired.

15.5.4 Tightness of the Analysis

This analysis of the algorithm is tight. Indeed, the algorithm needs
Ω ( m 1/3 ) iterations, and deletes Ω ( m 1/3 ) edges for the example on Figure 15.3: Again, all edges have unit
the right. In this example, m = Θ ( n ) . Each black gadget has a unit capacities.
effective resistance, and if we do the calculations, the effective resis-
tance between s and t tends to the golden ratio. If we set F = n 1/3
(which is almost the max-flow), a constant fraction of the current
(about Θ ( n 1/3 ) ) uses the edge e 1 . Once that edge is deleted, the
next red edge e 2 carries a lot of current, etc., until all red edges get
deleted.
 approximate max-flows using experts 201

15.5.5 Subsequent Work

A couple years after this work, Sherman, and independently, Kel- Sherman (2013)
ner et al. gave O ( m 1 + o ( 1 ) /ε O ( 1 ) ) -time algorithms for approximate Kelner, Lee, Oracchia, Sidford (2013)
max-flow problem on undirected graphs. This was improved, using
some more ideas, to a runtime of O ( m poly log m/ε O ( 1 ) ) -time by Peng (2014)
Richard Peng. These are based on the ideas of oblivious routings,
and non-Euclidean gradient descent, and we hope to cover this in an
upcoming lecture.
There has also been work on faster directed flows: work by Madry,
and thereafter by more refs here, have improved the current best re-
sult for max-flow in unweighted directed graphs to O e ( m 4/3 ) , match-
ing the above result.
16
The Gradient Descent Framework

Consider the problem of finding the minimum-energy s-t electrical

unit flow: we wanted to minimize the total energy burn

E(f) = ∑ f e2 r e
e

for flow values f that represent a unit flow from s to t (these form
a polytope). We alluded to algorithms that solve this problem, but
one can also observe that E ( f ) is a convex function, and we want to
find a minimizer within some polytope K. Equivalently, we wanted
to solve the linear system

Lϕ = ( e s − e t ) ,

which can be cast as finding a minimizer of the convex function

∥ Lϕ − ( e s − e t )∥ 2 .

How can we minimize these functions efficiently? In this lecture, we

will study the gradient descent framework for the general problem of
minimizing functions, and give concrete performance guarantees for
the case of convex optimization.

16.1 Convex Sets and Functions

First, recall the following definitions:

Definition 16.1 (Convex Set). A set K ⊆ R n is called convex if for all
x, y ∈ K,
λx + ( 1 − λ ) y ∈ K, (16.1)
for all values of λ ∈ [ 0, 1 ] . Geometrically, this means that for any two
points in K, the line connecting them is contained in K.
Definition 16.2 (Convex Function). A function f : K → R defined on
a convex set K is called convex if for all x, y ∈ K,

f ( λx + ( 1 − λ ) y ) ≤ λ f ( x ) + ( 1 − λ ) f ( y ) , (16.2)
 y

204 convex sets and functions

f [λx + (1 − λ)y]

λ f ( x ) + (1 − λ ) f ( y )

for all values of λ ∈ [ 0, 1 ] .

f (x)

There are two kinds of problems that we will study. The most
basic question is that of unconstrained convex minimization (UCM):
x
given a convex function f , we want to find x λx + (1 − λ)y y

min f ( x ).
x ∈Rn

In some cases we will be concerned with the constrained convex min-

imization (CCM) problem: given a convex function f and a convex
set K, we want to find
min f ( x ).
x ∈K

Note that setting K = Rn gives us the unconstrained case.

16.1.1 Gradient
For most of the following discussion, we assume that the function f
is differentiable. In that case, we can give an equivalent characteriza-
tion, based on the notion of the gradient ∇ f : Rn → Rn . The directional derivative of f at x (in the
direction y) is defined as
Fact 16.3 (First-order condition). A function f : K → R is convex if
f ( x + εy) − f ( x )
and only if f ′ ( x; y) := lim .
ε →0 ε
f (y) ≥ f ( x ) + ⟨∇ f ( x ), y − x ⟩ , (16.3) If there exists a vector g such that
⟨ g, y⟩ = f ′ ( x; y) for all y, then f is called
for all x, y ∈ K. differentiable at x, and g is called the
gradient. It follows that the gradient
Geometrically, Fact 16.3 states that the function always lies above must be of the form
its tangent plane, for all points in K. If the function f is twice-differentiable,  
∂f ∂f ∂f
∇ f (x) = ( x ), ( x ), · · · , (x) .
and if H f ( x ) is its Hessian matrix, i.e. its matrix of second derivatives ∂x1 ∂x2 ∂xn
at x ∈ K:
∂2 f
( H f )i,j ( x ) := ( x ), (16.4)
∂xi ∂x j
then we get yet another characterization of convex functions.
Fact 16.4 (Second-order condition). A twice-differentiable function f
is convex if and only if H f ( x ) is positive semidefinite for all x ∈ K.

16.1.2 Lipschitz Functions

We will need a notion of “niceness” for functions: Figure 16.1: The blue line denotes
the function and the red line is the
tangent line at x. (Figure from Nisheeth
Definition 16.5 (Lipschitz continuity). For a convex set K ⊆ Rn , a Vishnoi.)
function f : K → R is called G-Lipschitz (or G-Lipschitz continuous)
with respect to the norm ∥ · ∥ if

| f ( x ) − f (y)| ≤ G ∥ x − y∥ ,

for all x, y ∈ K.
 the gradient descent framework 205

In this chapter we focus on the Euclidean or ℓ2 -norm, denoted by

∥ · ∥2 . General norms arise in the next chapter, when we talk about
mirror descent. Again, assuming that the function is differentiable
allows us to give an alternative characterization of Lipschitzness.
Fact 16.6. A differentiable function f : K → Rn is G-Lipschitz with
respect to ∥ · ∥2 if and only if

∥∇ f ( x )∥2 ≤ G, (16.5)

for all x ∈ K.

16.2 Unconstrained Convex Minimization

If the function f is convex, any stationary point (i.e., a point x ∗ where

∇ f ( x ∗ ) = 0) is also a global minimum: just use Fact 16.3 to infer that
f (y) ≥ f ( x ∗ ) for all y. Now given a convex function, we can just
solve the equation
∇ f (x) = 0
to compute the global minima exactly. This is often easier said than
done: for instance, if the function f we want to minimize may not
be given explicitly. Instead we may only have a gradient oracle that
given x, returns ∇ f ( x ).
Even when f is explicit, it may be expensive to solve the equation
∇ f ( x ) = 0, and gradient descent may be a faster way. One example
arises when solving linear systems: given a quadratic function f ( x ) =
1 ⊺
2 x Ax − bx for a symmetric matrix A (say having full rank), a simple
calculation shows that

∇ f ( x ) = 0 ⇐⇒ Ax = b ⇐⇒ x = A−1 b.

This can be solved in O(nω ) (i.e., matrix-multiplication) time using

Gaussian elimination—but for “nice” matrices A we are often able to
approximate a solution much faster using the gradient-based meth-
ods we will soon see.

16.2.1 The Basic Gradient Descent Method

Gradient descent is an iterative algorithm to approximate the opti-
mal solution x ∗ . The main idea is simple: since the gradient tells us
the direction of steepest increase, we’d like to move opposite to the
direction of the gradient to decrease the fastest. So by selecting an
initial position x0 and a step size ηt at each time t, we can repeatedly
perform the update:

x t +1 ← x t − η t · ∇ f ( x t ). (16.6)
 206 unconstrained convex minimization

There are many choices to be made: where should we start? What

are the step sizes? When do we stop? While each of these decisions
depend on the properties of the particular instance at hand, we can
show fairly general results for general convex functions.

16.2.2 An Algorithm for General Convex Functions

The algorithm fixes a step size for all times t, performs the up-
date (16.6) for some number of steps T, and then returns the average
of all the points seen during the process.
Algorithm 13: Gradient Descent
13.1 x1 ← starting point
13.2 for t ← 1 to T do
13.3 x t +1 ← x t − η · ∇ f ( x t )
T
1
13.4 return xb :=
T ∑ xi .
t =1
This is easy to visualize in two dimensions: draw the level sets
Figure 16.2: The yellow lines denote
of the function f , and the gradient at a point is a scaled version of the level sets of the function f and the
normal to the tangent line at that point. Now the algorithm’s path is red walk denotes the steps of gradient
often a zig-zagging walk towards the optimum (see Fig 16.2). descent. (Figure from Wikipedia.)

Interestingly, we can give rigorous bounds on the convergence of

this algorithm to the optimum, based on the distance of the starting
point from the optimum, and bounds on the Lipschitzness of the
function. If both these are assumed to be constant, then our error is
smaller than ε in only O(1/ε2 ) steps.
Proposition 16.7. Let f : Rn → R be convex, differentiable and G-
G 2 ∥ x0 − x ∗ ∥2
Lipschitz. Let x ∗ be any point in Rd . If we define T := ε2
and
∥ x0 − ∗
η := √x ∥ , then the solution xb returned by gradient descent satisfies
G T

f ( xb) ≤ f ( x ∗ ) + ε. (16.7)
In particular, this holds when x ∗ is a minimizer of f .
The core of this proposition lies in the following theorem
Theorem 16.8. Let f : Rn → R be convex, differentiable and G-Lipschitz.
Then the gradient descent algorithm ensures that
T T
1 1
∑ f (xt ) ≤ ∑ f (x∗ ) + 2 ηTG2 + 2η ∥ x0 − x∗ ∥2 . (16.8)
t =1 t =1
We will prove Theorem 16.8 in the next section, but let’s first use it
to prove Proposition 16.7.

Proof of Proposition 16.7. By definition of xb and the convexity of f ,

1 T  1 T

T t∑ ∑ f ( x t ).
f ( xb) = f x t ≤
=1 T t =1
 the gradient descent framework 207

By Theorem 16.8,

T
1 1 1
T ∑ f ( xt ) ≤ f ( x ∗ ) + ηG2 +
2 2ηT
∥ x0 − x ∗ ∥2 .
t =1 | {z }
error

∥ x0 − ∗
The error terms balance when η = √x ∥ , giving
G T

∥ x0 − x ∗ ∥ G
f ( xb) ≤ f ( x ∗ ) + √ .
T

Finally, we set T = 1 2
ε2
G ∥ x0 − x ∗ ∥2 to obtain

f ( xb) ≤ f ( x ∗ ) + ε.

Observe: we do not (and cannot) show that the point xb is close in

distance to x ∗ ; we just show that the function value f ( xb) ≈ f ( x ∗ ).
Indeed, if the function is very flat close to x ∗ and we start off at some
remote point, we make tiny steps as we get close to x ∗ , and we can-
not hope to get close to it.
The 1/ε2 dependence of the number of oracle calls was shown
to be tight for gradient-based methods by Yurii Nesterov, if we al-
low f to be any G-Lipschitz function. However, if we assume that
the function is “well-behaved”, we can indeed improve on the 1/ε2
dependence. Moreover, if the function is strongly convex, we can
show that x ∗ and xb are close to each other as well: see §16.5 for such
results.
The convergence guarantee in Proposition 16.7 is for the time-
averaged point xb. Indeed, using a fixed step size means that our
iterates may get stuck in a situation where xt+2 = xt after some point
and hence we never improve, even though xb is at the minimizer.
One can also show that f ( x T ) ≤ f ( x ∗ ) + ε if we use a time-varying
√
step size ηt = O(1/ t), and increase the time horizon slightly to
O(1/ε2 log 1/ε). We refer to the work of Shamir and Zhang.

16.2.3 Proof of Theorem 16.8

Like in the proof of the multiplicative weights algorithm, we will use
a potential function. Define

∥ x t − x ∗ ∥2
Φt := . (16.9)
2η

We start the proof of Theorem 16.8 by understanding the one-step

change in potential:

Lemma 16.9. f ( xt ) + (Φt+1 − Φt ) ≤ f ( x ∗ ) + 21 ηG2 .

208 unconstrained convex minimization

Proof. Using the identity

∥ a + b∥2 = ∥ a∥2 + 2 ⟨ a, b⟩ + ∥b∥2 ,

with a + b = xt+1 − x ∗ and a = xt − x ∗ , we get

1

Φ t +1 − Φ t = ∥ x t +1 − x ∗ ∥ 2 − ∥ x t − x ∗ ∥ 2 (16.10)
2η
1

= 2 ⟨x − x t , x t − x ∗ ⟩ + ∥ x t +1 − x t ∥ 2 ;
2η | t+1 {z } | {z }
⟨b,a⟩ ∥ b ∥2

now using xt+1 − xt = −η ∇ f ( xt ) from gradient descent,

1

= 2 ⟨−η ∇ f ( xt ), xt − x ∗ ⟩ + ∥η ∇ f ( xt )∥2 .
2η

Since f is G-Lipschitz, ∥∇ f ( x )∥ ≤ G for all x. Thus,

1
f ( xt ) + (Φt+1 − Φt ) ≤ f ( xt ) + ⟨∇ f ( xt ), x ∗ − xt ⟩ + ηG2
2
Since f is convex, we know that f ( xt ) + ⟨∇ f ( xt ), x ∗ − xt ⟩ ≤ f ( x ∗ ).
Thus, we conclude that
1
f ( xt ) + (Φt+1 − Φt ) ≤ f ( x ∗ ) + ηG2 .
2
Now that we understand how our potential changes over time,
proving the theorem is straightforward.

Proof of Theorem 16.8. We start with the inequality we proved above:

1
f ( xt ) + (Φt+1 − Φt ) ≤ f ( x ∗ ) + ηG2 .
2
Summing over t = 1, . . . , T,

T T T
1
∑ f (xt ) + ∑ (Φt+1 − Φt ) ≤ ∑ f (x∗ ) + 2 ηG2 T
t =1 t =1 t =1

The sum of potentials on the left telescopes to give:

T T
1
∑ f (xt ) + ΦT+1 − Φ1 ≤ ∑ f (x∗ ) + 2 ηG2 T
t =1 t =1

Since the potentials are nonnegative, we can drop the Φ T term:

T T
1
∑ f (xt ) − Φ1 ≤ ∑ f (x∗ ) + 2 ηG2 T
t =1 t =1

Substituting in the definition of Φ1 and moving it over to the right

hand side completes the proof.
 the gradient descent framework 209

16.2.4 Some Remarks on the Algorithm

We assume a gradient oracle for the function: given a point x, it
returns the gradient ∇ f ( x ) at that point. If the function f is not
given explicitly, we may have to estimate the gradient using, e.g.,
random sampling. One particularly sample-efficient solution is to
pick a uniformly random point u ∼ Sn−1 from the sphere in Rn , and
return h f ( x + δu) i As δ → 0, the expectation of this
expression tends to ∇ f ( x ), using
d u Stokes’ theorem.
δ
for some tiny δ > 0. It is slightly mysterious, so perhaps it is useful to
consider its expectation in the case of a univariate function:
h f ( x + δu) i f ( x + δ) − f ( x − δ)
Eu∼{−1,+1} u = ≈ f ′ ( x ).
δ 2δ
In general, randomized strategies form the basis of stochastic gra-
dient descent, where we use an unbiased estimator of the gradient,
instead of computing the gradient itself (because it is slow to com-
pute, or because enough information is not available). The challenge
is now to control the variance of this estimator.
Another concern is that the step-size η and the number of steps
T both require knowledge of the distance ∥ x1 − x ∗ ∥ as well as the
bound on the gradient. More here. As an exercise, show that using
∥ x −x∗ ∥
the time-varying step-size ηt := 0 √ also gives a very similar
G t
convergence rate.
Finally, the guarantee is for f ( xb), where xb is the time-average of
the iterates. What about returning the final iterate? It turns out this
has comparable guarantees, but the proof is slightly more involved.
Add references.

16.3 Constrained Convex Minimization

Unlike the unconstrained case, the gradient at the minimizer may not
be zero in the constrained case—it may be at the boundary. In this This is the analog of the minimizer of a
case, the condition for a convex function f : K → R to be minimized single variable function being achieved
either at a point where the derivative is
at x ∗ ∈ K is now zero, or at the boundary.

⟨∇ f ( x ∗ ), y − x ∗ ⟩ ≥ 0 for all y ∈ K. (16.11)

In other words, all vectors y − x ∗ pointing within K are “positively

correlated” with the gradient. When x ∗ is in the interior of K, the
condition (16.11) is equivalent to
∇ f ( x ∗ ) = 0.
16.3.1 Projected Gradient Descent
While the gradient descent algorithm still makes sense: moving in
the direction opposite to the gradient still moves us towards lower
 210 constrained convex minimization

function values. But we must change our algorithm to ensure that the
new point xt+1 lies within K. To ensure this, we simply project the
new iterate xt+1 back onto K. Let projK : Rn → K be defined as

projK (y) = arg minx∈K ∥ x − y∥2 .

The modified algorithm is given below in Algorithm 14, with the

changes highlighted in blue.
Algorithm 14: Projected Gradient Descent For CCM xt

14.1 x1 ← starting point

x t +1 xt′ +1
14.2 for t ← 1 to T do
14.3 xt′ +1 ← xt − η · ∇ f ( xt )
14.4 xt+1 ← projK ( xt′ +1 )
T
14.5 return xb := 1
T ∑ xt
t =1
Figure 16.3: Projection onto a convex
We will show below that a result almost identical to that of Theo- body
rem 16.8, and hence that of Proposition 16.7 holds.

Proposition 16.10. Let K be a closed convex set, and f : K → R be convex,

G 2 ∥ x0 − x ∗ ∥2
differentiable and G-Lipschitz. Let x ∗ ∈ K, and define T := ε2
and
∥ x0 − ∗
η := √x ∥ . Then the solution xb returned by projected gradient descent
G T
satisfies
f ( xb) ≤ f ( x ∗ ) + ε. (16.12)

In particular, this holds when x ∗ is a minimizer of f .

Proof. We can reduce to an analogous constrained version of Theo-

rem 16.8. Let us start the proof as before:

1

Φ t +1 − Φ t = ∥ x t +1 − x ∗ ∥ 2 − ∥ x t − x ∗ ∥ 2 (16.13)
2η

But xt+1 is the projection of xt′ +1 onto K, which is difficult to reason

about. Also, we know that −η ∇ f ( xt ) = xt′ +1 − x ∗ , not xt+1 − x ∗ ,
so we would like to move to the point xt′ +1 . Indeed, we claim that
′
x ∗ ≥ ∥x ∗
t +1 − x t+1 − x ∥, and hence we get

1

Φ t +1 − Φ t = ∥ xt′ +1 − x ∗ ∥2 − ∥ xt − x ∗ ∥2 . (16.14)
2η

Now the rest of the proof of Theorem 16.8 goes through unchanged.

Why is the claim xt′ +1 − x ∗ ≥ ∥ xt+1 − x ∗ ∥ true? Since K is
convex, projecting onto it gets us closer to every point in K, in particular
to x ∗ ∈ K. To formally prove this fact about projections, consider
the angle x ∗ → xt+1 → xt′ +1 . This is a non-acute angle, since the
orthogonal projection means K likes to one side of the hyperplane
defined by the vector xt′ +1 − xt+1 , as in the figure on the right.
 the gradient descent framework 211

Note that restricting the play to K can be helpful in two ways: we

can upper-bound the distance ∥ x ∗ − x1 ∥ by the diameter of K, and
moreover we need only consider the Lipschitzness of f for points
within K.

16.4 Online Gradient Descent, and Relationship with MW

We considered gradient descent for the offline convex minimization

problem, but one can use it even when the function changes over
time. Indeed, consider the online convex optimization (OCO) prob-
lem: at each time step t, the algorithm proposes a point xt ∈ K and
an adversary gives a function f t : K → R with ∥∇ f t ∥ ≤ G. The cost of
each time step is f t ( xt ) and your objective is to minimize

regret = ∑ f t (xt ) − xmin

∗ ∈K ∑
f t ( x ∗ ).
t t

For instance if K = ∆n , and f t ( x ) := ⟨ℓt , x ⟩ for some loss vector

ℓt ∈ [−1, 1]n , then we are back in the experts setting of the previous
chapters. Of course, the OCO problem is far more general, allowing
arbitrary convex functions.
Surprisingly, we can use the almost same algorithm to solve the
OCO problem, with one natural modification: the update rule is now
taken with respect to gradient of the current function f t : This was first observed by Martin
Zinkevich in 2002, when he was a Ph.D.
x t +1 ← x t − η · ∇ f t ( x t ). student here at CMU.

Looking back at the proof in §16.2, the proof of Lemma 16.9 immedi-
ately extends to give us
1
f t ( xt ) + Φt+1 − Φt ≤ f t ( x ∗ ) + ηG2 .
2
Now summing this over all times t gives
T
T
1
∑ f t ( xt ) − f t ( x ∗ ) ≤ ∑ Φt − Φt+1 + ηTG2
2
t =1 t =1
1
≤ Φ1 + ηTG2 ,
2
∥ x1 − x ∗ ∥2 G 2
since Φ T +1 ≥ 0. The proof is now unchanged: setting T ≥ ε2
∥ x1 − ∗
and η = √x ∥ , and doing some elementary algebra as above,
G T

1 T
∥ x − x∗ ∥G
T ∑ f t ( xt ) − f t ( x ∗ ) ≤ 1 √
T
≤ ε.
t =0

16.4.1 Comparison to the MW/Hedge Algorithms

One advantage of the gradient descent approach (and analysis) over
the multiplicative weight-based ones is that the guarantees here hold
212 stronger assumptions

for all convex bodies K and all convex functions, as opposed to being
just for the unit simplex ∆n and linear losses f t ( x ) = ⟨ℓt , x ⟩, say for
ℓt ∈ [−1, 1]n . However, in order to make a fair comparison, suppose
we restrict ourselves to ∆n and linear losses, and consider the number
of rounds T before we get an average regret of ε.

• If we consider ∥ x1 − x ∗ ∥ (which, in the worst case, is the diameter

of K), and G (which is an upper bound on ∥∇ f t ( x )∥ over points in
K) as constants, then the T = Θ( ε12 ) dependence is the same.
√
• For a more quantitative comparison, note that ∥ x1 − x ∗ ∥ ≤ 2 for
√
x1 , x ∗ ∈ ∆n , and ∥∇ f t ( x )∥ = ∥ℓt ∥ √
≤ n for ℓt ∈ [−1, 1]n . Hence,

log n

Proposition 16.10 gives us T = Θ ε2n , as opposed to T = Θ ε2
for multiplicative weights.

The problem, at a high level, is that we are “choosing the wrong

norm”: when dealing with probabilities, the “right” norm is the ℓ1
norm and not the Euclidean ℓ2 norm. In the next lecture we will for-
malize what this means, and how this dependence on n be improved
via the Mirror Descent framework.

16.5 Stronger Assumptions

If the function f is “well-behaved”, we can improve the guarantees

for gradient descent in two ways: we can reduce the dependence on
ε, and we can weaken (or remove) the dependence on the parameters
G and ∥ x1 − x ∗ ∥. There are two standard assumptions to make on
the convex function: that it is “not too flat” (captured by the idea of
strong convexity), and it is not “not too curved” (i.e., it is smooth).
We now use these assumptions to improve the guarantees.

16.5.1 Strongly-Convex Functions

Definition 16.11 (Strong Convexity). A function f : K → R is α-
strongly convex if for all x, y ∈ K, any of the following holds:

1. (Zeroth order) f (λx + (1 − λ)y) ≤ λ f ( x ) + (1 − λ) f (y) − α2 λ(1 −

λ)∥ x − y∥2 for all λ ∈ [0, 1].

2. (First order) If f is differentiable, then

α
f (y) ≥ f ( x ) + ⟨∇ f ( x ), y − x ⟩ + ∥ x − y ∥2 . (16.15)
2

3. (Second order) If f is twice-differentiable, then all eigenvalues of

H f ( x ) are at least α at every point x ∈ K.
 the gradient descent framework 213

We will work with the first-order definition, and show that the
1


gradient descent algorithm with (time-varying) step size ηt = O αt
2
converges to a value at most f ( x ∗ ) + ε in time T = Θ( Gαε ). Note there
is no more dependence on the diameter of the polytope. Before we
give this proof, let us give the other relevant definitions.

16.5.2 Smooth Functions

Definition 16.12 (Lipschitz Smoothness). A function f : K → R is
β-(Lipschitz)-smooth if for all x, y ∈ K, any of the following holds:
β
1. (Zeroth order) f (λx + (1 − λ)y) ≥ λ f ( x ) + (1 − λ) f (y) − 2 λ(1 −
λ)∥ x − y∥2 for all λ ∈ [0, 1].

2. (First order) If f is differentiable, then

β
f (y) ≤ f ( x ) + ⟨∇ f ( x ), y − x ⟩ + ∥ x − y ∥2 . (16.16)
2

3. (Second order) If f is twice-differentiable, then all eigenvalues of

H f ( x ) are at most β at every point x ∈ K.

In this case, the gradient descent algorithm with fixed step size


ηt = η = O β1 yields an xb which satisfies f ( xb) − f ( x ∗ ) ≤ ε when
β ∥ x1 − x ∗ ∥

T = Θ ε . In this case, note we have no dependence on the
Lipschitzness G any more; we only depend on the diameter of the
polytope. Again, we defer the proof for the moment.

16.5.3 Well-conditioned Functions

Functions that are both β-smooth and α-strongly convex are called
well-conditioned functions. From the facts above, the eigenvalues of
their Hessian H f must lie in the interval [α, β] at all points x ∈ K.
In this case, we get a much stronger convergence—we can achieve
ε-closeness in time T = Θ(log 1ε ), where the constant depends on the
condition number κ = β/α.

Theorem 16.13. For a function f which is β-smooth and α-strongly con-

vex, let x ∗ be the solution to the unconstrained convex minimization prob-
lem arg minx∈Rn f ( x ). Then running gradient descent with ηt = 1/β
gives

β  −t 
f ( xt ) − f ( x ∗ ) ≤ exp ∥ x1 − x ∗ ∥2 .
2 κ
Proof. For β-smooth f , we can use Definition 16.12 to get

β
f ( xt+1 ) ≤ f ( xt ) − η ∥∇ f ( xt )∥2 + η 2 ∥∇ f ( xt )∥2 .
2
214 stronger assumptions

The right hand side is minimized by setting η = β1 , when we get

1
f ( x t +1 ) − f ( x t ) ≤ − ∥∇ f ( xt )∥2 . (16.17)
2β

For α-strongly-convex f , we can use Definition 16.11 to get:

α
f ( xt ) − f ( x ∗ ) ≤ ⟨∇ f ( xt ), xt − x ∗ ⟩ − ∥ x t − x ∗ ∥2 ,
2
α
≤ ∥∇ f ( xt )∥ ∥ xt − x ∗ ∥ − ∥ xt − x ∗ ∥2 ,
2
1 2
≤ ∥∇ f ( xt )∥ , (16.18)
2α

where we use that the right hand side is maximized when ∥ xt − x ∗ ∥ =

∥∇ f ( xt )∥ /α. Now combining with (16.17) we have that
 
α ∗
f ( x t +1 ) − f ( x t ) ≤ − f ( xt ) − f ( x ) , (16.19)
β

or setting ∆t = f ( xt ) − f ( x ∗ ) and rearranging, we get

   t  
α 1 t
∆ t +1 ≤ 1− ∆t ≤ 1− ∆1 ≤ exp − · ∆1 .
β κ κ

We can control the value of ∆1 by using (16.16) in x = x ∗ , y = x1 ;

since ∇ f ( x ∗ ) = 0, get ∆1 = f ( x1 ) − f ( x ∗ ) ≤ 2 ∥ x1 − x ∗ ∥2 .
β

Strongly-convex (and hence well-conditioned) functions have

the nice property that if f ( x ) is close to f ( x ∗ ) then x is close to x ∗ :
intuitively, since the function is curving at least quadratically, the
function values at points far from the minimizer must be significant.
Formally, use (16.15) with x = x ∗ , y = xt and the fact that ∇ f ( x ∗ ) = 0
to get
2
∥ x t − x ∗ ∥2 ≤ ( f ( xt ) − f ( x ∗ )).
α
We leave it as an exercise to show the claimed convergence bounds
using just strong convexity, or just smoothness. (Hint: use the state-
ments proved in (16.17) and (16.18).
Before we end, a comment on the strong O(log 1/ε) convergence
result for well-conditioned functions. Suppose the function values
lies in [0, 1]. The Θ(log 1/ε) error bound means that we are correct
up to b bits of precision—i.e., have error smaller than ε = 2−b —after
Θ(b) steps. In other words, the number of bits of precision is linear in
the number of iterations. The optimization literature refers to this as
linear convergence, which can be confusing when you first see it.
 the gradient descent framework 215

16.6 Extensions and Loose Ends

16.6.1 Subgradients
What if the convex function f is not differentiable? Staring at the
proofs above, all we need is the following:

Definition 16.14 (Subgradient). A vector z x is called a subgradient at

point x if

f (y) ≥ f ( x ) + ⟨z x , y − x ⟩ for all y ∈ Rn .

Now we can use subgradients at the point x wherever we used

∇ f ( x ), and the entire proof goes through. In some cases, an approxi-
mate subgradient may also suffice.

16.6.2 Stochastic Gradients, and Coordinate Descent

16.6.3 Acceleration

16.6.4 Reducing to the Well-conditioned Case

 17
Mirror Descent

The gradient descent algorithm of the previous chapter is general

and powerful: it allows us to (approximately) minimize convex func-
tions over convex bodies. Moreover, it also works in the model of
online convex optimization, where the convex function can vary over
time, and we want to find a low-regret strategy—one which performs
well against every fixed point x ∗ .
This power and broad applicability means the algorithm is not
always the best for specific classes of functions and bodies: for in-
stance, for minimizing linear functions over the probability simplex
∆n , we saw in §16.4.1 that the generic gradient descent algorithm
does significantly worse than the specialized Hedge algorithm. This
suggests asking: can we somehow change gradient descent to adapt to the
“geometry” of the problem?
The mirror descent framework of this section allows us to do pre-
cisely this. There are many different (and essentially equivalent) ways
to explain this framework, each with its positives. We present two
of them here: the proximal point view, and the mirror map view,
and only mention the others (the preconditioned or quasi-Newton
gradient flow view, and the follow the regularized leader view) in
passing.

17.1 Mirror Descent: the Proximal Point View

Here is a different way to arrive at the gradient descent algorithm

from the last lecture: Indeed, we can get an expression for xt+1 by

Algorithm 15: Proximal Gradient Descent Algorithm

15.1 x1 ← starting point
15.2 for t ← 1 to T do
1
15.3 xt+1 ← arg minx {η ⟨∇ f t ( xt ), x ⟩ + 2 ∥x − x t ∥2 }
218 mirror descent: the proximal point view

setting the gradient of the function to zero; this gives us the expres-
sion

η · ∇ f t ( x t ) + ( x t +1 − x t ) = 0 =⇒ x t +1 = x t − η · ∇ f t ( x t ),

which matches the normal gradient descent algorithm. Moreover, the

intuition for this algorithm also makes sense: if we want to minimize
the function f t ( x ), we could try to minimize its linear approximation
f t ( xt ) + ⟨∇ f t ( xt ), x − xt ⟩ instead. But we should be careful not to
“over-fit”: this linear approximation is good only close to the point
xt , so we could add in a penalty function (a “regularizer”) to prevent
us from straying too far from the point xt . This means we should
minimize
1
xt+1 ← arg min{ f t ( xt ) + ⟨∇ f t ( xt ), x − xt ⟩ + ∥ x − xt ∥2 }
x 2
or dropping the terms that don’t depend on x,
1
xt+1 ← arg min{⟨∇ f t ( xt ), x ⟩ + ∥ x − x t ∥2 } (17.1)
x 2
If we have a constrained problem, we can change the update step to:
1
xt+1 ← arg min{η ⟨∇ f t ( xt ), x ⟩ + ∥ x − x t ∥2 } (17.2)
x ∈K 2
The optimality conditions are a bit more complicated now, but they
again can show this algorithm is equivalent to projected gradient
descent from the previous chapter.
Given this perspective, we can now replace the squared Euclidean
norm by other distances to get different algorithms. A particularly
useful class of distance functions are Bregman divergences, which we
now define and use.

17.1.1 Bregman Divergences

Given a strictly convex function h, we can define a distance based on
how the function differs from its linear approximation:
Definition 17.1. The Bregman divergence from x to y with respect to
function h is

Dh (y∥ x ) := h(y) − h( x ) − ⟨∇h( x ), y − x ⟩.

The figure on the right illustrates this definition geometrically for a

univariate function h : R → R. Here are a few examples:
1. For the function h( x ) = 21 ∥ x ∥2 from Rn to R, the associated Breg-
Figure 17.1: Dh (y∥ x ) for function
man divergence is h : R → R.

Dh (y∥ x ) = 21 ∥y − x ∥2 ,

the squared Euclidean distance.

 mirror descent 219

2. For the (un-normalized) negative entropy function h( x ) = ∑in=1 ( xi ln xi −

x i ),
y

Dh (y∥ x ) = ∑i yi ln xi − yi + xi .
i

Using that ∑i yi = ∑i xi = 1 for y, x ∈ ∆n gives us Dh (y∥ x ) =

y
∑i yi ln xii for x, y ∈ ∆n : this is the Kullback-Leibler (KL) divergence
between probability distributions.

Many other interesting Bregman divergences can be defined.

17.1.2 Changing the Distance Function

Since the distance function 21 ∥ x − y∥2 in (17.1) is a Bregman diver-
gence, what if we replace it by a generic Bregman divergence: what
algorithm do we get in that case? Again, let us first consider the un-
constrained problem, with the update:

xt+1 ← arg min{η ⟨∇ f t ( xt ), x ⟩ + Dh ( x ∥ xt )}.

x

Again, setting the gradient at xt+1 to zero (i.e., the optimality condi-
tion for xt+1 ) now gives:

η ∇ f t ( xt ) + ∇h( xt+1 ) − ∇h( xt ) = 0,

or, rephrasing

∇ h ( x t +1 ) = ∇ h ( x t ) − η ∇ f t ( x t ) (17.3)


=⇒ xt+1 = ∇h−1 ∇h( xt ) − η ∇ f t ( xt ) (17.4)

Let’s consider this for our two running examples:

1. When h( x ) = 1
2 ∥ x ∥2 , the gradient ∇h( x ) = x. So we get

x t +1 = x t − η ∇ f t ( x t ),

the standard gradient descent update.

2. When h( x ) = ∑i ( xi ln xi − xi ), then ∇h( x ) = (ln x1 , . . . , ln xn ), so

( xt+1 )i = eln(xt )i −η ∇ f t (xt ) = ( xt )i e−η ∇ f t (xt ) .

Now if f t ( x ) = ⟨ℓt , x ⟩, its gradient is just the vector ℓt , and we get

back precisely the weights maintained by the Hedge algorithm!

The same ideas also hold for constrained convex minimization:

we now have to search for the minimizer within the set K. In this
case the algorithm using negative entropy results in the same Hedge-
like update, followed by scaling the point down to get a probability
vector, thereby giving the probability values in Hedge.
To summarize: this algorithm that tries to minimize the linear ap- What would be the “right” choice of h
to minimize the function f ? It would
be h = f , because adding D f ( x ∥ xt )
to the linear approximation of f at xt
gives us back exactly f . Of course, the
update now requires us to minimize
f ( x ), which is the original problem. So
we should choose an h that is “similar”
to f , and yet such that the update step
is tractable.
 220 mirror descent: the mirror map view

Algorithm 16: Proximal Gradient Descent Algorithm

16.1 x1 ← starting point
16.2 for t ← 1 to T do
16.3 xt+1 ← arg minx∈K {η ⟨∇ f t ( xt ), x ⟩ + Dh ( x ∥ xt )}

proximation of the function, regularized by a Bregman distance Dh ,

gives us vanilla gradient descent for one choice of h (which is good
for quadratic-like functions over Euclidean space), and Hedge for an-
other choice of h (which is good for linear functions over the space of
probability distributions). Indeed, depending on how we choose the
function, we can get different properties from this algorithm—this is
the mirror descent framework.

17.2 Mirror Descent: The Mirror Map View

A different view of the mirror descent framework is the one orig-

inally presented by Nemirovski and Yudin. They observe that in
gradient descent, at each step we set xt+1 = xt − η ∇ f t ( xt ). However,
the gradient was actually defined as a linear functional on Rn and A linear functional on vector space X is
hence naturally belongs to the dual space of Rn . The fact that we a linear map from X into its underlying
field F.
represent this functional (i.e., this covector) as a vector is a matter of
convenience, and we should exercise care.
In the vanilla gradient descent method, we were working in Rn en-
dowed with ℓ2 -norm, and this normed space is self-dual, so it is per-
haps reasonable to combine points in the primal space (the iterates xt
of our algorithm) with objects in the dual space (the gradients). But
when working with other normed spaces, adding a covector ∇ f t ( xt )
to a vector xt might not be the right thing to do. Instead, Nemirovski
and Yudin propose the following:

1. we map our current point xt to a point θt in the dual space using a

mirror map.

2. Next, we take the gradient step

θ t +1 ← θ t − η ∇ f t ( x t ). Figure 17.2: The four basic steps in

each iteration of the mirror descent
algorithm
3. We map θt+1 back to a point in the primal space xt′ +1 using the
inverse of the mirror map from Step 1.

4. If we are in the constrained case, this point xt′ +1 might not be in

the convex feasible region K, so we to project xt′ +1 back to a “close-
by” xt+1 in K.
 mirror descent 221

The name of the process comes from thinking of the dual space as be-
ing a mirror image of the primal space. But how do we choose these mir-
ror maps? Again, this comes down to understanding the geometry
of the problem, the kinds of functions and the set K we care about,
and the kinds of guarantees we want. In order to discuss these, let us
discuss the notion of norms in some more detail.

17.2.1 Norms and their Duals

Definition 17.2 (Norm). A function ∥ · ∥ : Rn → R is a norm if
• If ∥ x ∥ = 0 for x ∈ Rn , then x = 0;
• for α ∈ R and x ∈ Rn we have ∥αx ∥ = |α|∥ x ∥; and
• for x, y ∈ Rn we have ∥ x + y∥ ≤ ∥ x ∥ + ∥y∥.
The well-known ℓ p -norms for p ≥ 1 are defined by
n
∥ x ∥ p := ( ∑ | xi | p )1/p
i =1

for x ∈ Rn . The ℓ∞ -norm is given by

n
∥ x ∥∞ := max | xi |
i =1

for x ∈ Rn .

Definition 17.3 (Dual Norm). Let ∥ · ∥ be a norm. The dual norm of

∥ · ∥ is a function ∥ · ∥∗ defined as

∥y∥∗ := sup{⟨ x, y⟩ : ∥ x ∥ ≤ 1}.

Figure 17.3: The unit ball in ℓ1 -norm
The dual norm of the ℓ2 -norm is again the ℓ2 -norm; the Euclidean (Green), ℓ2 -norm (Blue), and ℓ∞ -norm
(Red).
norm is self-dual. The dual for the ℓ p -norm is the ℓq -norm, where
1/p + 1/q = 1.

Corollary 17.4 (Cauchy-Schwarz for General Norms). For x, y ∈ Rn ,

we have ⟨ x, y⟩ ≤ ∥ x ∥ ∥y∥∗ .

Proof. Assume ∥ x ∥ ̸= 0, otherwise both sides are 0. Since ∥ x/∥ x ∥∥ =

1, we have ⟨ x/∥ x ∥, y⟩ ≤ ∥y∥∗ .

Theorem 17.5. For a finite-dimensional space with norm ∥ · ∥, we have

(∥ · ∥∗ )∗ = ∥ · ∥.
Using the notion of dual norms, we can give an alternative charac-
terization of Lipschitz continuity for a norm ∥ · ∥, much like Fact 16.6
for Euclidean norms:
Fact 17.6. For f be a differentiable function. Then f is G-Lipschitz
with respect to norm ∥ · ∥ if and only if for all x ∈ R,

∥∇ f ( x )∥∗ ≤ G.
 222 mirror descent: the mirror map view

17.2.2 Defining the Mirror Maps

To define a mirror map, we first fix a norm ∥ · ∥, and then choose a
differentiable convex function h : Rn → R that is α-strongly-convex
with respect to this norm. Recall from §16.5.1 that such a function
must satisfy
α
h(y) ≥ h( x ) + ⟨∇h( x ), y − x ⟩ + ∥ y − x ∥2 .
2
We use two familiar examples:

1. h( x ) = 12 ∥ x ∥22 is 1-strongly convex with respect to ∥ · ∥2 , and

2. h( x ) := ∑in=1 xi (log xi − 1) is 1-strongly convex with respect to

∥ · ∥1 ; the proof of this is called Pinsker’s inequality. Check out the two proofs pointed to by
Aryeh Kontorovich, or this proof (part
Having fixed ∥ · ∥ and h, the mirror map is 1, part 2) by Madhur Tulsiani.

∇(h) : Rn → Rn .

Since h is differentiable and strongly-convex, we can define the in-

verse map as well. This defines the mappings that we use in the The function h used in this way is often
Nemirovski-Yudin process: we set called a distance-generating function.

θt = ∇ h( xt ) and xt′ +1 = (∇h)−1 (θt+1 ).

For our first running example of h( x ) = 21 ∥ x ∥2 , the gradient (and

hence its inverse) is the identity map. For the (un-normalized) nega-
tive entropy example, (∇h( x ))i = ln xi , and hence (∇h)−1 (θ )i = eθi .

17.2.3 The Algorithm (Again)

Let us formally state the algorithm again, before we state and prove
a theorem about it. Suppose we want to minimize a convex function
f over a convex body K ⊆ Rn . We first fix a norm ∥ · ∥ on Rn and
choose a distance-generating function h : Rn → R, which gives the
mirror map ∇h : Rn → Rn . In each iteration of the algorithm, we do
the following:

(i) Map to the dual space θt ← ∇h( xt ).

(ii) Take a gradient step in the dual space: θt+1 ← θt − ηt · ∇ f t ( xt ).

(iii) Map θt+1 back to the primal space xt′ +1 ← (∇h)−1 (θt+1 ).

(iv) Project xt′ +1 back into the feasible region K by using the Bregman
divergence: xt+1 ← minx∈K Dh ( x ∥ xt′ +1 ). In case xt′ +1 ∈ K, e.g., in
the unconstrained case, we get xt+1 = xt′ +1 .

Note that the choice of h affects almost every step of this algorithm.
 mirror descent 223

17.3 The Analysis

We prove the following guarantee for mirror descent, which captures

the guarantees for both Hedge and gradient descent, and for other
variants that you may use. The theorem is stated for the uncon-
strained version, but extending it to the
Theorem 17.7 (Mirror Descent Regret Bound). Let ∥ · ∥ be a norm on constrained version is an easy exercise.
Rn , and h be an α-strongly convex function with respect to ∥ · ∥. Given
f 1 , . . . , f T be convex, differentiable functions such that ∥∇ f t ∥∗ ≤ G, the
mirror descent algorithm starting with x0 and taking constant step size η in
every iteration produces x1 , . . . , x T such that for any x ∗ ∈ Rn ,
T T
Dh ( x ∗ ∥ x1 ) η ∑tT=1 ∥∇ f t ( xt )∥2∗
∑ f t ( xt ) ≤ ∑ f t ( x ∗ ) + η
+
2α
. (17.5)
t =1 t =1 | {z }
regret

Before proving Theorem 17.7, observe that when ∥ · ∥ is the ℓ2 -

norm and h = 21 ∥ · ∥2 , the regret term is

∥ x ∗ − x1 ∥22 η ∑tT=1 ∥∇ f t ( xt )∥22

+ ,
2η 2

which is what Theorem 16.8 guarantees. Similarly, if ∥ · ∥ is the ℓ1 -

norm and h is the negative entropy, the regret versus any point x ∗ ∈
∆n is
1 n ∗ x∗ η ∑tT=1 ∥∇ f t ( xt )∥2∞
∑
η i =1
xi ln i +
( x1 )i 2/ ln 2
.

For linear functions f t ( x ) = ⟨ℓt , x ⟩ with ℓt ∈ [−1, 1]n , and x1 = 1/n1,

the regret is

KL( x ∗ ∥ x1 ) ηT ln n
+ ≤ + ηT.
η 2/ ln 2 η

The last inequality uses that the KL divergence to the uniform dis-
tribution is at most ln n. (Exercise!) In fact, this suggests a way to
improve the regret bound: if we start with a distribution x1 that is
closer to x ∗ , the first term of the regret gets smaller.

17.3.1 The Proof of Theorem 17.7

The proof here is very similar in spirit to that of Theorem 16.8: we
give a potential function

Dh ( x ∗ ∥ x t )
Φt =
η

and bound the amortized cost at time t as follows:

f t ( xt ) − f t ( x ∗ ) + (Φt+1 − Φt ) ≤ f t ( x ∗ ) + blaht . (17.6)

224 the analysis

Summing over all times,

T T T
∑ f t (xt ) − ∑ f t (x∗ ) ≤ Φ1 − ΦT+1 + ∑ blaht
t =1 t =1 t =1
T
Dh ( x ∗ ∥ x1 ) T
≤ Φ1 + ∑ blaht = η
+ ∑ blaht .
t =1 t =1

The last inequality above uses that the Bregman divergence is always
non-negative for convex functions.
To complete the proof, it remains to show that blaht in inequal-
ity (17.6) can be made 2α ∥∇ f t ( xt )∥2∗ . Let us focus on the uncon-
η

strained case, where xt+1 = xt′ +1 . The calculations below are fairly
routine, and can be skipped at the first reading:

1

Φ t +1 − Φ t = Dh ( x ∗ ∥ x t +1 ) − Dh ( x ∗ ∥ x t )
η
1

= h( x ∗ ) − h( xt+1 ) − ⟨∇h( xt+1 ), x ∗ − xt+1 ⟩ − h( x ∗ ) + h( xt ) + ⟨∇h( xt ), x ∗ − xt ⟩
η | {z } | {z }
θ t +1 θt
1

= h ( x t ) − h ( x t +1 ) − ⟨ θ t − η ∇ f t ( x t ), x ∗ − x t +1 ⟩ + ⟨ θ t , x ∗ − x t ⟩
η | {z }
∇t
1

= h( xt ) − h( xt+1 ) − ⟨θt , xt − xt+1 ⟩ + η ⟨∇ f t ( xt ), x ∗ − xt+1 ⟩
η
1 α 
≤ − ∥ xt+1 − xt ∥2 + η ⟨∇ f t ( xt ), x ∗ − xt+1 ⟩ (By α-strong convexity of h wrt to ∥ · ∥)
η 2

Substituting this back into (17.6):

f t ( x t ) − f t ( x ∗ ) + ( Φ t +1 − Φ t )
α
≤ f t ( xt ) − f t ( x ∗ ) − ∥ xt+1 − xt ∥2 + ⟨∇ f t ( xt ), x ∗ − xt+1 ⟩
2η
α
≤ f t ( xt ) − f t ( x ) + ⟨∇ f t ( xt ), x ∗ − xt ⟩ − ∥ xt+1 − xt ∥2 + ⟨∇ f t ( xt ), xt − xt+1 ⟩
∗
| {z } 2η
≤ 0 by convexity of f t
α
≤ − ∥ xt+1 − xt ∥2 + ∥∇ f t ( xt )∥∗ ∥ xt − xt+1 ∥ (By Corollary 17.4)
2η
α 1η α 
≤ − ∥ x t +1 − x t ∥ 2 + ∥∇ f t ( xt )∥2∗ + ∥ xt − xt+1 ∥2 (By AM-GM)
2η 2 α η
η
= ∥∇ f t ( xt )∥2∗ .
2α
This completes the proof of Theorem 17.7. As you observe, it is
syntactically similar to the original proof of gradient descent, just
using more general language. In order to extend this to the con-
strained case, we will need to show that if xt′ +1 ∈
/ K, and xt+1 =
arg minx∈K Dh ( x ∥ xt′ +1 ), then

Dh ( x ∗ ∥ xt+1 ) ≤ Dh ( x ∗ ∥ xt′ +1 )
 mirror descent 225

for any x ∗ ∈ K. This is a “Generalized Pythagoras Theorem” for

Bregman distance, and is left as an exercise.

17.4 Alternative Views of Mirror Descent

To complete and flesh out. In this lecture, we reviewed mirror de-

scent algorithm as a gradient descent scheme where we do the gradi-
ent step in the dual space. We now provide some alternative views of
mirror descent.

17.4.1 Preconditioned Gradient Descent

For any given space which we use a descent method on, we can lin-
early transform the space with some map Q to make the geometry
more regular. This technique is known as preconditioning, and im-
proves the speed of the descent. Using the linear transformation Q,
our descent rule becomes

x t + 1 = x t − η Hh ( x t ) − 1 ∇ f ( x t ) .

Some of you may have seen Newton’s method for minimizing convex
functions, which has the following update rule:

x t +1 = x t − η H f ( x t ) −1 ∇ f ( x t ).

This means mirror descent replaces the Hessian of the function itself
by the Hessian of a strongly convex function h. Newton’s method has
very strong convergence properties (it gets error ε in O(log log 1/ε)
iterations!) but is not “robust”—it is only guaranteed to converge
when the starting point is “close” to the minimizer. We can view
mirror descent as trading off the convergence time for robustness. Fill
in more on this view.

17.4.2 As Follow the Regularized Leader

18
The Centroid and Ellipsoid Algorithms

In this chapter, we discuss some algorithms for convex programming

that have O(log 1/ε)-type convergence guarantees (under suitable
assumptions). This leads to polynomial-time algorithms for Linear
Programming problems. In particular, we examine the Center-of-
Gravity and Ellipsoid algorithms in depth.

18.1 The Centroid Algorithm

In this section, we discuss the Centroid Algorithm in the context

of constrained convex minimization. Besides being interesting in
its own right, it is a good lead-in to Ellipsoid, since it gives some
intuition about high-dimensional bodies and their volumes.
Given a convex body K ⊆ Rn and a convex function f : Rn → R,
we want to approximately minimize f ( x ) over x ∈ K. First, recall This is the natural analog of the cen-
that the centroid of a set K is the point c ∈ Rn such that troid of n points x1 , x2 , . . . , x N , which is
R R defined as ∑Ni xi
. See this blog post for
a discussion about the centroid of an
x ∈K x dx x dx
c := = Rx∈K , arbitrary measure µ defined over Rn .
vol(K ) x ∈K dx

where vol(K ) is the volume of the set K. The following lemma cap-
tures the crucial fact about the center-of-gravity that we use in our
algorithm.

Lemma 18.1 (Grünbaum’s Lemma). For any convex set K ∈ Rn with

a centroid c ∈ Rn , and any halfspace H = { x | a⊺ ( x − c) ≥ 0} passing
through c,  
1 vol(K ∩ H ) 1
≤ ≤ 1− .
e vol(K ) e
This bound is the best possible: e.g., consider the probability sim-
plex ∆n with centroid n1 1. Finish this argument.

18.1.1 The Algorithm

In 1965, A. Yu. Levin and Donald Newman independently (and on Levin (1965)
Newman (1965)
 228 the centroid algorithm

opposite sides of the iron curtain) proposed the following algorithm.

Algorithm 17: Centroid(K, f, T)
17.1 K1 ← K
17.2 for t = 1, . . . T do
17.3 at step t, let ct ← centroid of Kt
17.4 Kt+1 ← Kt ∩ { x | ⟨∇ f (ct ), x − ct ⟩ ≤ 0}
17.5 return xb ← arg mint∈{1,...,T } f (ct )
The figure to the right shows a sample execution of the algorithm,
where K is initially a ball. (Ignore the body K ε for now.) We find the
centroid c1 and compute the gradient ∇ f (c1 ). Instead of moving in ∇ f ( c2 ) K
the direction opposite to the gradient, we consider the halfspace H1
of vectors negatively correlated with the gradient, restrict our search • Kε
to K ← K ∩ H1 , and continue. We repeat this step some number of c2
∇ f ( c0 )
times, and then return the smallest of the function value at all the
• •
centroids seen by the algorithm. c0
c1

18.1.2 An Analysis of the Centroid Algorithm ∇ f ( c1 )

Theorem 18.2. Let B ≥ 0 such that f : K → [− B, B]. If xb is the result of

the algorithm, and x ∗ = arg minx∈K f ( x ), then
Figure 18.1: Sample execution of first
f ( xb) − f ( x ∗ ) ≤ 4B · exp(− T/3n). three steps of the Centroid Algorithm.

Hence, for any ε ≤ 1, as long as T ≥ 3n ln 4B

ε ,

f ( xb) − f ( x ∗ ) ≤ ε.

Proof. For some δ ≤ 1, define the body

K δ := {(1 − δ) x ∗ + δx | x ∈ K }

as a scaled-down version of K centered at x ∗ . The following facts are

immediate:
1. vol(K δ ) = δn · vol(K ).

2. The value of f on any point y = (1 − δ) x ∗ + δx ∈ K δ is

f (y) = f ((1 − δ) x ∗ + δx ) ≤ (1 − δ) f ( x ∗ ) + δ f ( x ) ≤ (1 − δ) f ( x ∗ ) + δB
≤ f ( x ∗ ) + δ( B − f ( x ∗ )) ≤ f ( x ∗ ) + 2δB.

Using Grünbaum’s lemma, the volume falls by a constant factor in

each iteration, so vol(Kt ) ≤ vol(K ) · (1 − 1e )t . If we define δ :=
2(1 − 1/e) T/n , then after T steps the volume of KT is smaller than
that of K δ , so some point of K δ must have been cut off.
Consider such a step t such that K δ ⊆ Kt but K δ ̸⊆ Kt+1 . Let
y ∈ K δ ∩ (Kt \ Kt+1 ) be a point that is “cut off”. By convexity we have

f (y) ≥ f (ct ) + ⟨∇ f (ct ), y − ct ⟩ ;

 the centroid and ellipsoid algorithms 229

moreover, ⟨∇ f (ct ), y − ct ⟩ > 0 since the cut-off point y ∈ Kt \ Kt+1 .

Hence the corresponding centroid has value f (ct ) < f (y) ≤ f ( x ∗ ) +
2δB. Since xb is the centroid with the smallest function value, we get


f ( xb) − f ( x ∗ ) ≤ 2B · 2 1 − 1/e) T/n ≤ 4B exp(− T/3n).

The second claim follows by substituting T ≥ 3n ln 4B

ε into the first
claim, and simplifying.

18.1.3 Some Comments on the Runtime

The number of iterations T to get an error of ε depends on log(1/ε);
compare this linear convergence to gradient descent requiring O(1/ε2 )
steps. One downside with this approach is that the number of itera-
tions explicitly depends on the number of dimensions n, whereas
gradient descent depends on other factors (a bound on the gradient,
and the diameter of the polytope), but not explicitly on the dimen-
sion.
However, the all-important question is: how do we compute the cen-
troid? This is a difficult problem—it is #P-hard to do exactly, which
means it is at least as hard as counting the number of satisfying as-
signments to a SAT instance. In 2002, Bertsimas and Vempala sug-
gested a way to find approximate centroids by sampling random
points from convex bodies (which in turn is done via random walks).
More details here.

18.2 The Ellipsoid Algorithm

The Ellipsoid algorithm is usually attributed to Naum Shor; the fact N. Z. Šor and N. G. Žurbenko (1971)
that this algorithm gives a polynomial-time algorithm for linear pro-
gramming was a breakthrough result due to Khachiyan, and was Khachiyan (1979)
front page news at the time. A great source of information about
this algorithm is the Grötschel-Lovász-Schrijver book. A historical Grötschel, Lovász, and Schrijver (1988)
perspective appears in this this survey by Bland, Goldfarb, and Todd.
Let us mention some theorem statements about the Ellipsoid algo-
rithm that are most useful in designing algorithms. The second-most
important theorem is the following. Recall the notion of an extreme
point or basic feasible solution (bfs) from §7.1.2. Let ⟨ A⟩, ⟨b⟩, ⟨c⟩ de-
note the number of bits required to represent of A, b, c respectively.

Theorem 18.3 (Linear Programming in Polynomial Time). Given a

linear program min{c⊺ x | Ax ≥ b}, the Ellipsoid algorithm produces an
optimal vertex solution for the LP, in time polynomial in ⟨ A⟩, ⟨b⟩, ⟨c⟩.

One may ask: does the runtime depend on the bit-complexity

of the input because doing basic arithmetic on these numbers may
 230 ellipsoid for lp feasibility

require large amounts of time. Unfortunately, that is not the case.

Even if we count the number of arithmetic operations we need to
perform, the Ellipsoid algorithm performs poly(⟨ A⟩ + ⟨b⟩ + ⟨c⟩)
operations. A stronger guarantee would have been for the number
of arithmetic operations to be poly(m, n), where the matrix A ∈
Qm×n : such an algorithm would be called a strongly polynomial-
time algorithm. Obtaining such an algorithm remains a major open
question.

18.2.1 Separation Implies Optimization

In order to talk about the Ellipsoid algorithm, as well as to state
the next (and most important) theorem about Ellipsoid, we need a
definition.

Definition 18.4 (Strong Separation Oracle). For a convex set K ⊆ Rn ,

a strong separation oracle for K is an algorithm that takes a point z ∈
Rn and correctly outputs one of:
Separating Hyper
(i) Yes (i.e., z ∈ K), or

(ii) No (i.e., z ̸∈ K), as well as a separating hyperplane given by a ∈

Rn , b ∈ R such that K ⊆ { x | ⟨ a, x ⟩ ≤ b} but ⟨ a, x ⟩ > b.
x̂
K •
The example on the right shows a separating hyperplane.

Theorem 18.5 (Separation implies Optimization). Given an LP

min{c⊺ x | x ∈ K }

for a polytope K = { x | Ax ≥ b} ⊆ Rn , and given access to a strong

Figure 18.2: Example of separating
separation oracle for K, the Ellipsoid algorithm produces a vertex solution
hyperplanes
for the LP in time poly(n, maxi ⟨ ai ⟩, maxi ⟨bi ⟩, ⟨c⟩).

There is no dependence on the number of constraints in the LP; we

can get a basic solution to any finite LP as long as each constraint has
a reasonable bit complexity, and we can define a separation oracle for
the polytope. This is often summarized by saying: “separation implies
optimization”. Let us give two examples of exponential-sized LPs,
for which we can give a separation oracles, and hence optimize over
them.

18.3 Ellipsoid for LP Feasibility

Instead of solving a linear program, suppose we are given a descrip-

tion of some polytope K, and want to either find some point x ∈ K, or
to report that K is the empty set. This feasibility problem is no harder
than optimization over the polytope; in fact, the GLS book shows that
 the centroid and ellipsoid algorithms 231

feasibility is not much easier than optimization: under certain condi-

tions, the two problems are essentially equivalent to each other. (Very
loosely, we can do binary search on the objective function.)
Given this, let’s discuss solving a feasibility problem; this will
allow us to illustrate some of the major ideas of the Ellipsoid al-
gorithm. Given some description of a polytope K, and two scalars
R, r > 0, suppose we are guaranteed that
(a) K ⊆ Ball(0, R), and

(b) either K = ∅, or else some ball Ball (c, r ) ⊆ K for some c ∈ Rn .

The feasibility problem is to figure out which of the two cases in
condition (b) holds; moreover, if K ̸= ∅ then we also need to find a
point x ∈ K. We assume that K is given by a strong separation oracle:
Theorem 18.6 (Idealized Feasibility using Ellipsoid). Given K, r, R as
above (and a strong separation oracle for K), the feasibility problem can be
solved using O(n log( R/r )) oracle calls. For this result, and the rest of the
chapter: let us assume that we can
Proof. The basic structure is simple, and reminiscent of the Centroid perform exact arithmetic on real
algorithm. At each iteration t, we have a current ellipsoid Et guar- numbers. This assumption is with
considerable loss in generality, since
anteed to contain the set K (assuming we have not found a point the algorithm takes square-roots when
x ∈ K yet). The initial ellipsoid is E0 = Ball(0, R), so condition (a) computing the new ellipsoid. If were
to round numbers when doing this,
guarantees that K ⊆ E0 . that could create all sorts of numerical
In iteration t, we ask the separation oracle whether the center ct problems, and a large part of the
of ellipsoid Et belongs to K? If the oracle answers Yes, we are done. complication in the actual algorithms
comes from these numerical issues.
Else the oracle returns a separating hyperplane ⟨ a, x ⟩ = b, such that
⟨ a, ct ⟩ > b but K ⊆ Ht := { x : ⟨ a, x ⟩ ≤ b}. Consequently, K ⊆ Et ∩ Ht .
Moreover, the half-space Ht does not contain the center ct of the
ellipsoid, so Et ∩ Ht is less than half the entire ellipsoid. The crucial
idea is to find another (small-volume) ellipsoid Et+1 containing this
piece Et ∩ Ht (and hence also K). This allows us to continue.
To show progress, we need that the volume vol(Et+1 ) is con-
siderably smaller than vol(Et ). In §18.5 we show that the ellipsoid
Et+1 ⊇ Et ∩ Ht has volume
vol(Et+1 ) − 1
≤ e 2( n +1) .
vol(Et )
Therefore, after 2(n + 1) iterations, the ratio of the volumes falls by at
least a factor of 1e . Now we are done, because our assumptions say This volume reduction is much weaker
that by a factor of n compared to that of
the Centroid algorithm, so it is often
vol(K ) ≤ vol(Ball(0, R)), worth considering if applications of the
Ellipsoid algorithm can be replaced by
and that the Centroid algorithm.
K ̸= ∅ ⇐⇒ vol(K ) ≥ vol(Ball(0, r )).
Hence, if after 2(n + 1) ln( R/r ) steps, none of the ellipsoid centers
have been inside K, we know that K must be empty.
232 ellipsoid for convex optimization

18.3.1 Finding Vertex Solutions for LPs

There are several issues that we need to handle when solving LPs
using this approach. For instance, the polytope may not be full-
dimensional, and hence we do not have any non-trivial ball within
K. Our separation oracles may only be approximate. Moreover, all
the numerical calculations may only be approximate.
Even after we take care of these issues, we are working over the
rationals so binary search-type techniques may not be able to get us
to a vertex solution. So finally, when we have a solution xt that is Consider the case where we perform
“close enough” to x ∗ , we need to “round” it and get a vertex solu- binary-search over the interval [0, 1]
and want to find the point 1/3: no
tion. In a single dimension we can do the following (and this idea number of steps will get us exactly to
already appeared in a homework problem): we know that the opti- the answer.
mal solution x ∗ is a rational whose denominator (when written in
reduced terms) uses at most some b bits. So we find a solution within
distance to x ∗ is smaller than some δ. Moreover δ is chosen to be
small enough such that there is a unique rational with denominator
smaller than 2b in the δ-ball around xt . This rational can only be x ∗ ,
so we can “round” xt to it.
In higher dimensions, the analog of this is a technique (due to
Lovász) called simultaneous Diophantine equations.

18.4 Ellipsoid for Convex Optimization

Now we want to solve min{ f ( x ) | x ∈ K }. Again, assume that K is

given by a strong separation oracle, and we have numbers R, r such
that K ⊆ Ball(0, R), and K is either empty or contains a ball of radius
r. The general structure is a one familiar by now, and combines ideas
from both the previous sections.

1. Let the starting point x1 ← 0, the starting ellipsoid be E1 ←

Ball(0, R), and the starting convex set K1 ← K.

2. At time t, ask the separation oracle: “Is the center ct of ellipsoid Et

in the convex body Kt ?”

Yes: Define half-space Ht := { x | ⟨∇ f (ct ), x − ct ⟩ ≤ 0}. Observe

that Kt ∩ Ht contains all points in Kt with value at most f (ct ).
No: In this case the separation oracle also gives us a separating
hyperplane. This defines a half-space Ht such that ct ̸∈ Ht , but
Kt ⊆ Ht .

In both cases, set Kt+1 ← Kt ∩ Ht , and Et+1 to an ellipsoid con-

taining Et ∩ Ht . Since we knew that Kt ⊆ Et , we maintain that
K t +1 ⊆ E t +1 .
 the centroid and ellipsoid algorithms 233

3. Finally, after T = 2n(n + 1) ln( R/r ) rounds either we have not seen
any point in K—in which case we say “K is empty”—or else we
output
xb ← arg min{ f (ct ) | ct ∈ Kt , t ∈ 1 . . . T }.

One subtle issue: we make queries to a separation oracle for Kt ,

but we are promised only a separation oracle for K1 = K. However,
we can build separation oracles for Ht inductively: indeed, given
strong separation oracle for Kt−1 , we build one for Kt = Kt−1 ∩ Ht−1
as follows:
Given z ∈ Rn , query the oracle for Kt−1 at z. If z ̸∈ Kt−1 , the separating
hyperplane for Kt−1 also works for Kt . Else, if z ∈ Kt−1 , check if
z ∈ Ht−1 . If so, z ∈ Kt = Kt−1 ∩ Ht−1 . Otherwise, the defining
hyperplane for halfspace Ht−1 is a separating hyperplane between z
and Kt .

Now adapting the analysis from the previous sections gives us the
following result (assuming exact arithmetic again):
Theorem 18.7 (Idealized Convex Minimization using Ellipsoid).
Given K, r, R as above (and a strong separation oracle K), and a function
f : K → [− B, B], the Ellipsoid algorithm run for T steps either correctly
reports that K = ∅, or else produces a point xb such that
2BR n T o
f ( xb) − f ( x ∗ ) ≤ exp − .
r 2n(n + 1)
Note the similarity to Theorem 18.2, as well as the differences: the
exponential term is slower by a factor of 2(n + 1). This is because
the volume of the successive ellipsoids shrinks much slower than
in Grünbaum’s lemma. Also, we lose a factor of R/r because K is
potentially smaller than the starting body by precisely this factor.
(Again, this presentation ignores precision issues, and assumes we
can do exact real arithmetic.)

18.5 Getting the New Ellipsoid

This brings us to the final missing piece: given a current ellipsoid

E and a half-space H that does not contain its center, we want an
ellipsoid E ′ that contains E ∩ H, and as small as possible. To start off,
let us recall some basic facts about ellipsoids. The simplest ellipses in
R2 are axis aligned, say with principal semi-axes having length a and
b, and written as:
x2 y2
+ ≤ 1.
a2 b2
Or in matrix notation we could also say
" #⊺ " #" #
x 1/a2 0 x
≤1
y 0 1/b 2 y
234 getting the new ellipsoid

More generally, any ellipsoid E is perhaps best thought of as a in-

vertible linear transformation L applied to the unit ball B(0, 1), and
then it being shifted to the correct center c. The linear transformation
yields:

L(Ball(0, 1)) = { Lx : x⊺ x ≤ 1}
= { y : ( L −1 y )⊺ ( L −1 y ) ≤ 1 }
= {y : y⊺ ( LL⊺ )−1 y ≤ 1}
= { y : y ⊺ Q −1 y ≤ 1 } ,

where Q−1 := LL⊺ is a positive semidefinite matrix. For an ellipsoid

centered at c we simply write

{ y + 1 : y ⊺ Q −1 y ≤ 1 } = { y : ( y − c )⊺ Q −1 ( y − c ) ≤ 1 } .

It is helpful to note that for any ball A,

q
vol( L( A)) = vol( A) · | det( L)| = vol( A) det( Q)

In the above problems, we are given an ellipsoid Et and a half-

space Ht that does not contain the center of Et . We want to find a
matrix Qt+1 and a center ct+1 such that the resulting ellipsoid Et+1
contains Et ∩ Ht , and satisfies

vol(Et+1 )
≤ e−1/2(n+1) .
vol(Et )

Given the above discussion, it suffices to do this when Et is a unit

ball: indeed, when Et is a general ellipsoid, we apply the inverse
linear transformation to convert it to a ball, find the smaller ellipsoid
for it, and then apply the transformation to get the final smaller
ellipsoid. (The volume changes due to the two transformations cancel
each other out.)
We give the construction for the unit ball below, but first let us
record the claim for general ellipsoids:

Theorem 18.8. Given an ellipsoid Et given by (ct , Qt ) and a separating

hyperplane a⊺t ( x − ct ) ≤ 0 through its center, the new ellipsoid Et+1 with
center ct+1 and psd matrix Qt+1 ) is found by taking

1
c t +1 : = c t − h
n+1
and  
n2 2
Q t +1 = 2
Qk − hh⊺
n −1 n+1
q
where h = a⊺t Qt at .
 the centroid and ellipsoid algorithms 235

Note that the construction requires us to take square-roots: this

may result in irrational numbers which we then have to either trun-
cate, or represent implicitly. In either case, we face numerical issues;
ensuring that these issues are not real problems lies at the heart of
the formal analysis. We refer to the GLS book, or other textbooks for
details and references.

18.5.1 Halving a Ball

Before we end, we show that the problem of finding a smaller ellip-
soid that contains half a ball is, in fact, completely straight-forward.
By rotational symmetry, we might as well find a small ellipsoid that
contains
K = Ball(0, 1) ∩ { x | x1 ≥ 0}.

By symmetry, it makes sense that the center of this new ellipsoid E

should be of the form
c = (c1 , 0, . . . , 0).

Again by symmetry, the ellipsoid can be axis-aligned, with semi-axes

of length a along e1 , and b > a along all the other coordinate axes.
Moreover, for E to contain the unit ball, it should contain the points
(1, 0) and (0, 1), say. So

(1 − c1 )2 c21 1
≤ 1 and + 2 ≤ 1.
a2 a 2 b
Suppose these two inequalities are tight, then we get
s s
(1 − c1 )2 (1 − c1 )2
a = 1 − c1 , b= = ,
(1 − c1 )2 − c21 (1 − 2c1

and moreover the ratio of volume of the ellipsoid to that of the ball is
 (1 − c )2 (n−1)/2
1
abn−1 = (1 − c1 ) · .
1 − 2c1
1
This is minimized by setting c1 = n +1 gives us

vol(E ) − 1
= · · · ≤ e 2( n +1) .
vol(Ball(0, 1))

For a more detailed description and proof of this process, see these
notes from our LP/SDP course for details.
In fact, we can view the question of finding the minimum-volume
ellipsoid that contains the half-ball K: this is a convex program, and
looking at the optimality conditions for this gives us the same con-
struction above (without having to make the assumptions of symme-
try).
236 algorithms for solving lps

18.6 Algorithms for Solving LPs

While the Centroid and Ellipsoid algorithms for convex program-

ming are powerful, giving us linear convergence, they are not typ-
ically used to solve LPs in practice. There are several other algo-
rithms: let us mention them in passing. Let K := { x | Ax ≥ b} ⊆ Rn ,
and we want to minimize {c⊺ x | x ∈ K }.

Simplex: This is perhaps the first algorithm for solving LPs that most
of us see. It was also the first general-purpose linear program
solver known, having been developed by George Dantzig in 1947.
This is a local-search algorithm: it maintains a vertex of the poly-
hedron K, and at each step it moves to a neighboring vertex with-
out decreasing the objective function value, until it reaches an op-
timal vertex. (The convexity of K ensures that such a sequence of
steps is possible.) The strategy to choose the next vertex is called
the pivot rule. Unfortunately, for most known pivot rules, there
are examples on which the following the pivot rule takes expo-
nential (or at least a super-polynomial) number of steps. Despite
that, it is often used in practice: e.g., the Excel software contains an
implementation of simplex.

Interior Point: A very different approach to get algorithms for LPs

is via interior-point algorithms: these happen to be good both in
theory and in practice. The first polynomial-time interior-point
algorithm was proposed by Karmarkar in 1984. We discuss this in
the next chapter.

Geometric Algorithms for LPs: These approaches are geared towards

solving LPs fast when the number of dimensions n is small. If m
is the number of constraints, these algorithms often allow a poor
runtime in n, at the expense of getting a good dependence on m.
As an example, a randomized algorithm of Raimund Seidel’s has
a runtime of O(m · n!) = O(m · nn/2 ); a different algorithm of Ken
Clarkson (based on the multiplicative weights approach!) has a
runtime of O(n2 m) + nO(n) O(log m)O(log n) . One of the fastest such
algorithm is by Jiri Matoušek, Micha Sharir, and Emo Welzl, and
has a runtime of √
O(n2 m) + eO( n log n) .

For details and references, see this survey by Martin Dyer, Nimrod
Megiddo, and Emo Welzl.

Naturally, there are other approaches to solve linear programs as

well: write more here.
19
Interior-Point Methods

In this chapter, we continue our discussion of polynomial-time algo-

rithms for linear programming, and cover the high-level details of an
interior-point algorithm. The runtime for these linear programs has
recently been improved both qualitatively and quantitatively, so this
is an active area of research that you may be interested in. Moreover,
these algorithms contain sophisticated general ideas (duality and the
method of Lagrange multipliers, and the use of barrier functions) that
are important even beyond this context.
Another advantage of delving into the details of these methods
is that we can work on getting better algorithms for special kinds
of linear programs of interest to us. For instance, the line of work
on faster max-flow algorithms for directed graphs, starting with
the work of Madry, and currently resulting in the O(m4/3+ε )-time
algorithms of Kathuria, and of Liu and Sidford, are based on a better
understanding of interior-point methods.
We will consider the following LP with equality constraints:

Figure 19.1: The feasible region for

min c⊺ x an LP in equational form (from the
Ax = b Matoušek and Gärtner book).

x≥0

where A ∈ Rm×n , b ∈ Rm and c, x ∈ Rn . Let K := { x | Ax = b, x ≥ 0}

be the polyhedron, and x ∗ = arg min{c⊺ x | x ∈ K } an optimal
solution.
To get the main ideas across, we make some simplifying assump-
tions and skip over some portions of the algorithm. For more details,
please refer to the book by Jiri Matoušek and Bernd Gärtner (which
has more details), or the one by Steve Wright (which has most de-
tails).
238 barrier functions

19.1 Barrier Functions

The first step in solving the LP using an interior-point method will be

to introduce a parameter η > 0 and exchange our constrained linear
optimization problem for an unconstrained but nonlinear one:
 n
1
f η ( x ) := c⊺ x + η ∑ log xi .
i =1

Let xη∗ := arg min{ f η ( x ) | Ax = b} be the minimizer of this function

over the subspace given by the equality constraints. Note that we’ve
added in η times a barrier function
n
1
B( x ) := ∑ log xi .
i =1

The intuition is that when x approaches the boundary x ≥ 0 of the

feasible region, the barrier function B( x ) will approach +∞. The If we had inequality constraints Ax ≥ b
parameter η lets us control the influence of this barrier function. If as well, we would have added
∑im=1 log a⊺ x1−b to the barrier function.
η is sufficiently large, the contribution of the barrier function domi- i i

nates in f η ( x ), and the minimizer xη∗ will be close to the “center” of

the feasible region. However, as η gets close to 0, the effect of B( x )
will diminish and the term c⊺ x will now dominate, causing that xη∗ to
approach x ∗ .
Now consider the trajectory of the minimizer xη∗ as we lower η
continuously, starting at some large value and tending to zero: this
path is called the central path. The idea of our path-following al-
gorithm will be to approximately follow this path. In essence, such
algorithms conceptually perform the following steps (although we
will only approximate these steps in practice):

1. Pick a sufficiently large η0 and a starting point x (0) that is the

minimizer of f η0 ( x ). (We will ignore this step in our discussion, for
now.)

2. At step t, move to the corresponding minimizer x (t+1) for f ηt+1 ,

where
ηt+1 := ηt · (1 − ϵ).
Since ηt is close to ηt+1 , we hope that the previous minimizer
x (t) is close enough to the current goal x (t+1) for us to find it effi-
Figure 19.2: A visualization of a path-
ciently. following algorithm.

3. Repeat until η is small enough that xη∗ is very close to an optimal

solution x ∗ . At this point, round it to get a vertex solution, like in
§18.3.1.

We will only sketch the high-level idea behind Step 1 (finding the
starting solution), and will skip Step 2 (the rounding); our focus will
 interior-point methods 239

be on the update step. To understand this step, let us look at the

structure of the minimizers for f η ( x ).

19.1.1 The Primal and Dual LPs, and the Duality Gap
Recall the primal linear program:

( P) min c⊺ x
Ax = b
x ≥ 0,

and its dual:

( D ) max b⊺ y
A⊺ y ≤ c.

We can rewrite the dual using non-negative slack variables s:

( D ′ ) max b⊺ y
A⊺ y + s = c
s ≥ 0.

We assume that both the primal ( P) and dual ( D ) are strictly feasible:
i.e., they have solutions even if we replace the inequalities with strict
ones). Then we can prove the following result, which relates the
optimizer for f η to feasible primal and dual solutions:

Lemma 19.1 (Optimality Conditions). The point x ∈ Rn≥0 is a minimizer

of f η ( x ) if and only if there exist y ∈ Rm and s ∈ Rn≥0 such that:

Ax − b = 0 (19.1)
⊺
A y+s = c (19.2)
∀i ∈ [ n ] : si xi = η (19.3)

The conditions (19.1) and (19.2) show that x and (y, s) are feasible
for the primal ( P) and dual ( D ′ ) respectively. The condition (19.3)
is an analog of the usual complementary slackness result that arises
when η = 0. To prove this lemma, we use the method of Lagrange
multipliers. Observe: we get that if there exists a
maximum x ∗ , then x ∗ satisfies these
Theorem 19.2 (The Method of Lagrange Multipliers). Let functions conditions.
f and g1 , · · · , gm be continuously differentiable, and defined on some open
subset of Rn . If x ∗ is a local optimum of the following optimization problem

min f ( x )
s.t. ∀i ∈ [m] : gi ( x ) = 0

then there exists y∗ ∈ Rm such that ∇ f ( x ∗ ) = ∑im=1 yi∗ · ∇ gi ( x ∗ ).

240 the update step

Proof Sketch of Lemma 19.1. We need to show three things:

1. The function f η ( x ) achieves its maximum x ∗ in the feasible region.

2. The point x ∗ satisfies the conditions (19.1)–(19.3).

3. And that no other x satisfies these conditions.

The first step uses that if there are strictly feasible primal and
dual solutions ( x̂, ŷ, ŝ), then the region { x | Ax = b, f µ ( x ) ≤ f µ x̂ }
is bounded (and clearly closed) and hence the continuous function
f µ ( x ) achieves its minimum at some point x ∗ inside this region, by
the Extreme Value theorem. (See Lemma 7.2.1 of Matoušek and Gärt-
ner, say.)
For the second step, we use the functions f µ ( x ), and gi ( x ) = a⊺i x −
bi in Theorem 19.2 to get the existence of y∗ ∈ Rm such that:
m
⊺ m
∑ yi∗ · ∇(ai x∗ − bi ) ∑ yi∗ ai .
⊺
f η (x∗ ) = ⇐⇒ c − η · 1/x1∗ , · · · , 1/xn∗ =
i =1 i =1

Define a vector with si∗

s∗ = η/xi∗ . The above condition is now
equivalent to setting A⊺ y∗ + s∗ = c and si∗ xi∗ = η for all i.
Finally, for the third step of the proof, the function f η ( x ) is strictly
convex and has a unique local/global optimum. Finish this proof.

By weak duality, the optimal value of the linear program lies be-
tween the values of any feasible primal and dual solution, so the
duality gap c⊺ x − b⊺ y bounds the suboptimality c⊺ x − OPT of our
current solution. Lemma 19.1 allows us to relate the duality gap to η
as follows.

c⊺ x − b⊺ y = c⊺ x − ( Ax )⊺ y = x⊺ c − x⊺ (c − s) = x⊺ s = n · η.

If the representation size of the original LP is L := ⟨ A⟩ + ⟨b⟩ + ⟨c⟩,

then making η ≤ 2− poly( L) means we have primal and dual solutions
whose values are close enough to optimal, and can be rounded (us-
ing the usual simultaneous Diophantine equations approach used for
Ellipsoid).

19.2 The Update Step

Let us now return to the question of obtaining x (t+1) from x (t) at

step t? Recall, we want x (t+1) to satisfy the optimality conditions
(19.1)–(19.3) for f ηt+1 . The hurdles to finding this point directly are:
(a) the non-negativity of the x, s variables means this is not just a
linear system, there are inequalities to contend with. And more wor-
ryingly, (b) it is not a linear system at all: we have non-linearity in the
constraints (19.3) because of multiplying xi with si .
 interior-point methods 241

To get around this, we use a “local-search” method. We start with

the solution x (t) “close to” the optimal solution xη∗t for f ηt , and take
a small step, so that we remain non-negative, and also get “close to”
the optimal solution xη∗t+1 for f ηt+1 . Then we lower η and repeat this
process.
Let us make these precise. First, to avoid all the superscripts, we
use ( x, y, s) and η to denote ( x (t) , y(t) , s(t) ) and ηt . Similarly, ( x ′ , y′ , s′ )
and η ′ denote the corresponding values at time t + 1. Now we as-
sume we have ( x, y, s) with x, s > 0, and also:

Ax = b (19.4)
⊺
A y+s = c (19.5)
n
2
∑ s i x i − ηt ≤ (ηt/4)2 . (19.6)
i =1

The first two are again feasibility conditions for ( P) and ( D ′ ). The
third condition is new, and is an approximate version of (19.3). Sup-
pose that
1

η ′ := η ′ · 1 − √ .
4 n

Our goal is a new solution x ′ = x + ∆x, y′ = y + ∆y, s′ = s + ∆s,

which satisfies non-negativity, and ideally also satisfies the original
optimality conditions (19.1)–(19.3) for the new η ′ . (Of course, we will
fail and only satisfy the weaker condition (19.6) instead of (19.3), but
we should aim high.)
Let us write the goal explicitly, by substituting ( x ′ , y′ , s′ ) into
(19.4)–(19.6) and using the feasibility of ( x, y, s). This means the incre-
ments ∆x, ∆y, ∆s satisfy

A (∆x ) = 0
⊺
A (∆y) + (∆s) = 0
si (∆xi ) + (∆si ) xi + (∆si )(∆xi ) = η ′ − xi si .

Note the quadratic term in blue. Since we are aiming for an approxi-
mation anyways, and these increments are meant to be tiny, we drop
the quadratic term to get a system of linear equations in these incre-
ments:

A (∆x ) = 0
⊺
A (∆y) + (∆s) = 0
si (∆xi ) + (∆si ) xi = η ′ − xi si .

This is often written in the following matrix notation (which I am

242 the update step

putting down just so that you recognize it the next time you see it):
    
A 0 0 ∆x 0
    
 0 A⊺ I   ∆y  =  0 .
diag( x ) 0 diag(s) ∆s η′1 − x ◦ s

Here x ◦ s stands for the component-wise product of the vectors x

and s. The bottom line: this is a linear system and we can solve it, say
using Gaussian elimination. Now we can set x ′ = x + ∆x, etc., to get The goal of many modern algorithms
the new point ( x ′ , y′ , s′ ). It remains to check the non-negativity and is to get faster ways to solve this linear
system. E.g., if it were a Laplacian
also the weakened conditions (19.4)–(19.6) with respect to η ′ . system we could (approximately) solve
it in near-linear time.

19.2.1 Properties of the New Solution

While discarding the quadratic terms means we do not satisfy xi si =
η for each coordinate i, we can show that we satisfy it on average,
allowing us to bound the duality gap.

Lemma 19.3. The new duality gap is ⟨ x ′ , s′ ⟩ = n η ′ .

Proof. The last set of equalities in the linear system ensure that

si xi + si (∆xi ) + (∆si ) xi = η ′ , (19.7)

so we get


x ′ , s′ = ⟨ x + ∆x, s + ∆s⟩


= ∑ si xi + si (∆xi ) + (∆si ) xi + ⟨∆x, ∆s⟩
i
= nη ′ + ⟨∆x, − A⊺ (∆y)⟩
= n · η ′ − ⟨ A(∆x ), ∆y⟩
= n · η′,

using the first two equality constraints of the linear system.

We explicitly maintained the invariants given by (19.4), (19.5), so

it remains to check (19.6). This requires just a bit of algebra (that also
√
causes the n to pop out).

2
Lemma 19.4. ∑in=1 si′ xi′ − η ′ ≤ (η ′/4)2 .

Proof. As in the proof of Lemma 19.3, we get that si′ xi′ − η ′ = (∆si )(∆xi ),
so it suffices to show that
s
n
∑ (∆si )2 (∆xi )2 ≤ η′/4.
i =1

We can use the inequality

r 1 1
∑ a2i bi2 ≤ 4 ∑(ai + bi )2 = 4 ∑(a2i + bi2 + 2ai bi ),
i i i
 interior-point methods 243

xi (∆si )2 si (∆xi )2
where we set a2i = si and bi2 = xi . Hence
s  
n
1 n xi s
∑ (∆si ∆xi )2 ≤ ∑
4 i =1 si
· (∆si )2 + i · (∆xi )2 + 2(∆si )(∆xi )
xi
i =1
1 n ( xi ∆si )2 + (si ∆xi )2
4 i∑
= [since (∆s)⊺ ∆x = 0 by Claim 19.3]
=1
si xi
1 ∑in=1 ( xi ∆si + si ∆xi )2
≤
4 mini∈[n] si xi
1 ∑in=1 (η ′ − si xi )2
= . (19.8)
4 mini∈[n] si xi

We now bound the numerator and denominator separately.

This claim and proof are incorrect. However, we can prove that
mini si xi ≥ 3η/4. This weaker claim suffices for the rest of the proof.
The details will come soon, sorry for the mistake!


Claim 19.5 (Denominator). mini si xi ≥ η 1 − 4√1 n .

Proof. By the inductive hypothesis, ∑i (si xi − η )2 ≤ (η/4)2 . This

η
means that maxi |si xi − η | ≤ 4√n , which proves the claim.

Claim 19.6 (Numerator). ∑in=1 (η ′ − si xi )2 ≤ η 2 /8.

1
Proof. Let δ = √
4 n
. Then,

n n
∑ (η ′ − si xi )2 = ∑ ((1 − δ)η − si xi )2
i =1 i =1
n n n
= ∑ (η − si xi )2 + ∑ (δη )2 + 2δη ∑ (η − si xi ).
i =1 i =1 i =1

The first term is at most (η/4)2 , by the induction hypothesis. On the

other hand, by Claim 19.3 we have
n n
∑ (η − si xi ) = nη − ∑ si xi = 0.
i =1 i =1

Thus
n
1
∑ (η ′ − si xi )2 ≤ (η/4)2 + n (4√n)2 η 2 = η 2 /8.
i =1

Substituting these results into (19.8), we get

s
n
1 η 2 /8 1 η′
∑ i i( ∆s ∆x ) 2 ≤
4 (1 − √1 )η
=
32 (1 − √1 )2
.
i =1 4 n 4 n

This expression is smaller than η ′ /4, which completes the proof.

Lemma 19.7. The new values x ′ , s′ are non-negative.

244 the newton-raphson method

Proof. By induction, we assume the previous point has xi > 0 and

si > 0. (For the base case we need to ensure that the starting solution
( x (0) , s(0) ) also satisfies this property.) Now for a scalar α ∈ [0, 1] we
define x ′′ := x + α∆x, s′′ := s + α∆s, and η ′′ := (1 − α)η + αη ′ , to
linearly interpolate between the old values and the new ones. Then
we can show ⟨ x ′′ , s′′ ⟩ = n η ′′ , and also

∑(si′′ xi′′ − η ′′ )2 ≤ (η ′′ /4)2 , (19.9)

i

which are analogs of Lemmas 19.3 and 19.4 respectively. The latter
inequality means that |si′′ xi′′ − η ′′ | ≤ η ′′ /4 for each coordinate i, else
that coordinate itself would violate inequality (19.9). Specifically,
this means that neither xi′′ nor si′′ ever becomes zero for any value of
α ∈ [0, 1]. Now since ( xi′′ , si′′ ) is a linear interpolation between ( xi , si )
and ( xi′ , si′ ), and the former were strictly positive, the latter cannot be
non-positive.

Theorem 19.8. Given an LP min{c⊺ x | Ax = b, x ≥ 0} with an initial

feasible ( x (0) , η0 ) pair, the interior-point algorithm produces a primal-dual
√ nη
pair with duality gap at most ε in O( n log ε 0 ) iterations, each involving
solving one linear system.

The proof of the above theorem follows immediately from the

fact that the duality gap at the beginning is nη0 , and the value of η
drops by (1 − 4√1 n ) in each iteration. If the LP has representation size
L := ⟨ A⟩ + ⟨b⟩ + ⟨c⟩, we can stop when ε = exp(− poly( L)), and then
round this solution to an vertex solution of the LP.
The one missing piece is finding the initial ( x (0) , η0 ) pair: this is
a somewhat non-trivial step. One possible approach is to run the
interior-point algorithm “in reverse”. The idea is that we can start
with some vertex of the feasible region, and then to successively
increase η through a similar mechanism as the one above, until the
value of η is sufficiently large to begin the algorithm.

19.3 The Newton-Raphson Method

A more “modern” way of viewing interior-point methods is via the

notion of self-concordance. To do this, let us revisit the classic Newton-
Raphson method for finding roots.

19.3.1 Finding Zeros of Functions

The basic Newton-Raphson method for finding a zero of a univariate
function is the following: given a function g, we start with a point x1 ,
 interior-point methods 245

and at each time t, set

g( xt )
x t +1 ← x t − . (19.10)
g′ ( xt )
We now show that if f is “nice enough” and we are “close enough”
to a zero x ∗ , then this process converges very rapidly to x ∗ . Since we take O(log log 1/ε) iterations
to get to error ε, the number of bits
Theorem 19.9. Suppose g has continuous second-derivatives, then if is x∗ of accuracy squares each time. This
∗
a zero of g, then if we start at x1 “close enough” to X , the error goes to ε in is called quadratic convergence in the
optimization literature.
O(log log 1/ε) steps. Make this formal!
Proof. By Taylor’s theorem, the existence of continuous second
derivatives means we can approximate f around xt as:
f ( x ∗ ) = f ( xt ) + f ′ ( xt )( x ∗ − xt ) + 1/2 f ′′ (ξ t )( x ∗ − xt )2 ,
where ξ t is some point in the interval [ x ∗ , xt ]. However, x ∗ is a zero of
f , so f ( x ∗ ) = 0. Moreover, using (19.10) to replace xt f ′ ( xt ) − f ( xt ) by
xt+1 f ′ ( xt ), and rearranging, we get
− f ′′ (ξ t )
x ∗ − x t +1 = · ( x ∗ − x t )2 .
| {z } 2 f ′ ( xt ) | {z }
=:δt+1 =:δt2

Above, we use δt to denote the error x ∗ − xt . Taking absolute values

 f ′′ (ξ ) 
 t 
|δt+1 | =  ′  · δt2 .
2 f ( xt )
f ′′ (ξ )
Hence, if we can ensure that | 2 f ′ ( x) | ≤ M for each x and each ξ
that lies bewteen x ∗ and x, then once we have δ0 small enough, then
each subsequent error drops quadratically. This means the number of
significant bits of accuracy double each step. More careful analysis.

19.3.2 An Example
Given an n-bit integer a ∈ Z, suppose we want to compute its re-
ciprocal 1/a without using divisions. This reciprocal is a zero of the
expression This method for computing recip-
g( x ) = 1/x − a. rocals appears in the classic book of
Aho, Hopcroft, Ullman, without any
Hence, the Newton-Raphson method says, we can start with x1 = 1, elaboration—it always mystified me
until I realized the connection to the
say, and then use (19.10) to get Newton-Raphson method. I guess they
(1/xt − a) expected their readers to be familiar
x t +1 ← x t − = xt + xt (1 − a xt ) = 2xt − a xt2 . with these connections, since computer
(−1/xt2 ) science used to have closer connections
If we define ε t := 1 − a xt , then to numerical analysis in the 1970s.

ε t+1 = 1 − a xt+1 = 1 − (2a xt − a2 xt2 ) = (1 − a xt )2 = ε2t .

Hence, if ε 1 ≤ 1/2, say, the number of bits of accuracy double at each
step. Moreover, if we are careful, we can store xt using integers (by
instead keeping track of 2k xt for suitably chosen values k ≈ 2t ).
246 self-concordance

19.3.3 Minimizing Convex Functions

To find the minimum of a function f (especially a convex function)
we can focus on finding a stationary point, i.e., a point x such that
f ′ ( x ) = 0. Setting g = f ′ , the update rule just changes to

f ′ ( xt )
x t +1 ← x t − . (19.11)
f ′′ ( xt )

19.3.4 On To Higher Dimensions

For general functions f : Rn → R, the rule remains the same, with
the natural changes:

xt+1 ← xt − [ H f ( xt )]−1 · ∇ f ( xt ). (19.12)

19.4 Self-Concordance

Analogy between self-concordance and the convergence conditions

for the 1-d case?
Present the view using the “modern view” of self-concordance.
Mention that the current bound is really O(m)-self-concordant. That
universal barrier is O(n) self-concordant, but not efficient. Vaidya’s
volumetric barrier? The entropic barrier? The Lee-Sidford barrier,
based on leverage scores. What’s the cleanest way, without getting
lost in the algebra?
 Part IV

Combating Intractability
20
Approximation Algorithms

In this chapter, we turn to the problem of combating intractability:

many combinatorial optimization problems are NP-hard, and hence
are unlikely to have polynomial-time algorithms. Hence we consider
approximation algorithms: algorithms that run in polynomial-time,
but output solutions whose quality is close the optimal solution’s
quality. We illustrate some of the basic ideas in the context of two
NP-hard problems: Set Cover and Bin Packing. Both have been
studied since the 1970s.
Let us start with some definitions: having fixed an optimization
problem, let I denote an instance of the problem, and Alg denote an
algorithm. Then Alg( I ) is the output/solution produced by the algo-
rithm, and c(Alg( I )) its cost. Similarly, let Opt( I ) denote the optimal
output for input I, and let c(Opt( I )) denote its cost. For minimiza- Typically, if the instance is clear from
tion problems, the approximation ratio of the algorithm A is defined context, we use the notation

to be the worst-case ratio between the costs of the algorithm’s solu- Alg ≤ r · Opt

tion and the optimum: to denote that

c(Alg( I )) ≤ r · c(Opt( I )).
c(Alg( I ))
ρ = ρA := max .
I c(Opt( I ))

In this case, we say that Alg is an ρ-approximation algorithm. For

maximization problems, we define rho to be

c(Alg( I ))
ρ = ρA := min ,
I c(Opt( I ))

and therefore a number in [0, 1].

20.1 A Rough Classification into Hardness Classes

In the late 1990s, there was an attempt to classify combinatorial op-

timization problems into a small number of hardness classes: while
this ultimately failed, a rough classification of NP-bard problems is
still useful.
250 a rough classification into hardness classes

• Fully Poly-Time Approximation Scheme (FPTAS): For problems in this

category, there exist approximation algorithms that take in a pa-
rameter ε, and output a solution with approximation ratio 1 + ε in
time poly(⟨ I ⟩, 1/ε). E.g., one such problem is Knapsack, where As always, let ⟨ I ⟩ denote the bit com-
given a collection of n items, with each item i having size si ∈ Q+ plexity of the input I.

and value vi ∈ Q+ , find the subset of items with maximum value

that fit into a knapsack of unit size.

• Poly-Time Approximation Scheme (PTAS): For a problem in this cat-

egory, for any ε > 0, there exists an approximation algorithm
with approximation ratio 1 + ε, that runs in time O(n f (ε) ) for some
function f (·). For instance, the Traveling Salesman Problem The runtime has been improved to
in d-dimensional Euclidean space has an algorithm due to San- O(n log n + n exp{(1/ε)d }).

jeev Arora (1996) that computes a (1 + ε)-approximation in time

O(n f (ε) ), where f (ε) = exp{(1/ε)d }. Moreover, it is known that
this dependence on ε, with the doubly-exponential dependence on
d, is unavoidable.

• Constant-Factor Approximation: Examples in this class include the

Traveling Salesman Problem on general metrics. In the late Christofides’ result only ever appeared
1970s, Nicos Christofides and Anatoliy Serdyukov discovered as a CMU GSIA technical report in
1976. Serdyukov’s result only came to
the same 1.5-approximation algorithm for metric TSP, using the be known a couple years back.
Blossom algorithm to connect up the odd-degree vertices of an
MST of the metric space to get an Eulerian spanning subgraph,
and hence a TSP tour. This was improved only in 2020, when
Anna Karlin, Nathan Klein, and Shayan Oveis-Gharan gave an Karlin, Klein, and Oveis Gharan (2020)
(1.5 − ε)-approximation, which we hope to briefly outline in a later
chapter. Meanwhile, it has been shown that metric TSP can’t be
approximated with a ratio better than 123
122 under the assumption of
P ̸= NP, by Karpinski, Lampis, Schmied.

• Logarithmic Approximation: An example of this is Set Cover,

which we will discuss in some detail.

• Polynomial Approximation: One example is the Independent Set

problem, for which any algorithm with an approximation ratio
n1−ε for some constant ε > 0 implies that P = NP. The best
approximation algorithm for Independent Set known has an
approximation ratio of O(n/ log3 n).

However, there are problems that do not fall into any of these
clean categories, such as Asymmetric k-Center, for which there
exists a O ( log ∗ n ) -approximation algorithm, and this is best possible
unless P = NP. Or Group Steiner Tree, where the approximation
ratio is O ( log 2 n ) on trees, and this is also best possible.
 approximation algorithms 251

20.2 The Surrogate

Given that it is difficult to find an optimal solution, how can we ar-

gue that the output of some algorithm has cost comparable to that
of Opt ( I ) . An important idea in proving the approximation guar-
antee involves the use of a surrogate, or a lower bound, as follows:
Given an algorithm Alg and an instance I, if we want to calculate
the approximation ratio of Alg, we first find a surrogate map S from
instances to the reals. To bound the approximation ratio, we typically
do the following:
Figure 20.1: The cost diagram on
instance I (costs increase from left to
1. We show that S( I ) ≤ Opt( I ) for all I, and right).

2. then show that Alg( I ) ≤ αS( I ) for all I.

This shows that Alg( I ) ≤ α Opt( I ). Which leaves us with the ques-
tion of how to construct the surrogate. Sometimes we use the com-
binatorial properties of the problem to get a surrogate, and at other
times we use a linear programming relaxation.

20.3 The Set Cover Problem

In the Set Cover problem, we are given a universe U with n ele-

ments, and a family S = {S1 , . . . , Sm } of m subsets of U, such that
U = ∪S∈S S. We want to find a subset S ′ ⊆ S , such that U = ∪S∈S S
while minimizing the size |S ′ |.
In the weighted version of Set Cover, we have a cost cS for each
set S ∈ S , and want to minimize c(S ′ ) = ∑S∈S ′ cS . We will focus
on the unweighted version for now, and indicate the changes to the
algorithm and analysis to extend the results to the weighted case.
The Set Cover problem is NP-complete, even for the unweighted
version. Several approximation algorithms are known: the greedy
algorithm is a ln n-approximation algorithm, with different analyses
given by Vašek Chvátal, David Johnson, Laci Lovász, Stein, and oth-
ers. Since then, the same approximation guarantee was given based
on the relax-and-round paradigm.
This was complemented by a hardness result in 1998 by Uri
Feige (building on previous work of Carsten Lund and Mihalis Yan-
nakakis), who showed that a (1 − ε) ln n-approximation algorithm for
any constant ε > 0 would imply that NP has algorithms that run in
time O(nlog log n ). This was improved by Irit Dinur and David Steurer,
who tightened the result to show that such an approximation algo-
rithm would in fact imply that NP has polynomial-time algorithm
(i.e., that P = NP).
252 a relax-and-round algorithm for set cover

20.3.1 The Greedy Algorithm for Set Cover

The greedy algorithm is simple: Repeatedly pick the set S ∈ S that
covers the most uncovered elements, until all elements of U are covered.

Theorem 20.1. The greedy algorithm is a ln n-approximation.

The greedy algorithm does not achieve a better ratio than Ω(log n):
one example is given by the figure to the right. The optimal sets are
the two rows, whereas the greedy algorithm may break ties poorly
and pick the set covering the left half, and then half the remainder,
etc. A more sophisticated example can show a matching gap of ln n.

Proof of Theorem 20.1. Suppose Opt picks k sets from S . Let ni be the
number of still uncovered when the algorithm has picked i sets. Then
n0 = n = |U |. Since the k sets in Opt cover all the elements of U,
Figure 20.2:
they also cover the uncovered elements in ni . By averaging, there
must exist a set in S that covers ni /k of the yet-uncovered elements.
Hence,
ni+1 ≤ ni − ni /k = ni (1 − 1/k).

Iterating, we get nt ≤ n0 (1 − 1/k)t < n · e−t/k . So setting T = k ln n, As always, we use 1 + x ≤ e x , and here
we get n T < 1. Since n T must be an integer, it is zero, so we have we can use that the inequality is strict
whenever x ̸= 0.
covered all elements using T = k ln n sets.

If the sets are of size at most B, we can show that the greedy al-
gorithm is an (1 + ln B)-approximation. Moreover, for the weighted
case, the greedy algorithm changes to picking the set S in that maxi-
mizes:
number of yet-uncovered elements in S
.
cS
One can give an analysis like the one above for this weighted case as
well. Not quite, the proof here changes a fair bit, need to rephrase
and give the proof?

20.4 A Relax-and-Round Algorithm for Set Cover

The second algorithm for Set Cover uses the popular relax-and-
round framework. The steps of this process are as follows:

1. Write an integer linear program for the problem. This will also be
NP-hard to solve, naturally.

2. Relax the integrality constraints to get a linear program. Since

this is a minimization problem, relaxing the constraints causes the
optimal LP value to be no larger than the optimal IP value (which
is just Opt). This optimal value LP value is the surrogate.
 approximation algorithms 253

3. Now solve the linear program, and round the fractional variables
to integer values, while ensuring that the cost of this integer solu-
tion is not much higher than the LP value.
Let’s see this in action: here is the integer linear program (ILP) that
precisely models Set Cover:

min ∑ cS xS (ILP-SC)
S∈S
s.t. ∑ xS ≥ 1 ∀e ∈ U
S:e∈S
xS ∈ {0, 1} ∀S ∈ S .

The LP relaxation just drops the integrality constraints:

min ∑ cS xS (LP-SC)
S∈S
s.t. ∑ xS ≥ 1 ∀e ∈ U
S:e∈S
xS ≥ 0 ∀S ∈ S .

If LP( I ) is the optimal value for the linear program, then we get:

LP( I ) ≤ Opt( I ).

Finally, how do we round? Suppose x ∗ is the fractional solution ob-

tained by solving the LP optimally. We do the following two phases:
1. Phase 1: Repeat t = ln n times: for each set S, pick S with probabil-
ity xS∗ independently.

2. Phase 2: For each element e yet uncovered, pick any set covering it.
Clearly the solution produced by the algorithm is feasible; it just
remains to bound the number of sets picked by it.
Theorem 20.2. The expected number of sets picked by this algorithm is
(ln n) LP( I ) + 1.
Proof. Clearly, the expected number of sets covered in each round in
phase 1 is ∑S xS∗ = LP( I ), and hence the expected number of sets in
phase 1 is at most ln n times as much.
For the second phase, the number of sets not picked is precisely
the the expected number of elements not covered in Phase 1. To
calculate this, consider an arbitrary element e.

Pr[e not covered in phase 1] = (ΠS:e∈S (1 − xS∗ ))t

≤ (e− ∑S:e∈S xS )t
≤ ( e −1 ) t
1
= ,
n
254 the bin packing problem

since t = ln n. By linearity of expectations, the expected number of

uncovered elements in Phase 2 should be 1, so in expectation we’ll
pick 1 set in Phase 2. This completes the proof. One can derandomize this algorithm to
get a deterministic algorithm with the
same guarantee. We may see this in an
In a homework problem, we will show that if the sizes of the sets exercise. Also, Neal Young has a way to
are bounded by B, then we can get a (1 + ln B)-approximation as solve this problem without solving the
LP at all!
well. And that the analysis can extend to the weighted case, where
sets have costs.

20.5 The Bin Packing Problem

Bin Packing is another classic NP-hard optimization problem. We

are givenn items, each item i having some size si ∈ [0, 1]. We want
to find the minimum number of bins, each with capacity 1, such that
we can pack all n items into them. Formally, we want to find the
partition of [n] into S1 ∪ S2 ∪ . . . ∪ Sk such that ∑i∈S j si ≤ 1 for each
set S j , and moreover, the number of parts k is minimized.
The Bin Packing is NP-hard, and this can be shown via a reduc-
tion from the Partition problem (where we are given n positive
integers s1 , s2 , . . . , sn and an integer K such that ∑in=1 si = 2K, and
we want to decide whether we can find a partition of these integers
into two disjoint sets A, B such that ∑i∈ A si = ∑ j∈ B s j = K). Since
this partition instance corresponds gives us Bin Packing instances
where the optimum is either 2 or at least 3, the reduction shows that
getting an approximation factor of smaller than 3/2 for Bin Packing
is also NP-hard.
We show two algorithms for this problem. The first algorithm
First-Fit uses at most 2 Opt bins, whereas the second algorithm
uses at most (1 + ε) Opt +O(1/ε2 ) bins. These are not the best re-
sults possible: e.g., a recent result by Rebecca Hoberg and Thomas
Rothvoß gives a solution using at most Opt +O(log Opt) bins, and it
is conceivable that we can get an algorithm that uses Opt +O(1) bins.

20.5.1 A Class of Greedy Algorithms: X-Fit

We can define a collection of greedy algorithms that consider the

items in some arbitrary order: for each item they try to fit it into
some “open” bins; if the item does not fit into any of these bins,
then they open a new bin and put it there. Here are some of these
algorithms:

1. First-Fit: add the item to the earliest opened bin where it fits.

2. Next-Fit: add the item to the single most-recently opened bin.

 approximation algorithms 255

3. Best-Fit: consider the bins in increasing order of free space, and

add the item to the first one that can take it.

4. Worst-Fit: consider the open bins in decreasing(!) order of free

space, and add the item to the first one that can take it. The idea is
to ensure that no bin has small amounts of free space remaining,
which is likely to then get wasted.

All these algorithms are 2-approximations. Let us give a proof for

First-Fit, the others have similar proofs.

Theorem 20.3. AlgFF ( I ) ≤ 2 · Opt( I ).

Proof. The surrogate in this case is the total volume V ( I ) = ∑i si of

items. Clearly, ⌈V ( I )⌉ ≤ OPT ( I ). Now consider the bins in the order
they were opened. For any pair of consecutive bins 2j − 1, 2j, the
first item in bin 2j could not have fit into bin 2j − 1 (else we would
not have opened the new bin). So the total size of items in these two
consecutive bins is strictly more than 1. Another way to say this: at most one
Hence, if we open K bins, the total volume of items in these bins is bin is at-most-half-full, because if there
were two, the later of these bins would
strictly more than ⌊K/2⌋. Hence, not have been opened.

⌊K/2⌋ < V ( I ) =⇒ K ≤ 2 ⌈V ( I )⌉ ≤ 2 Opt( I ).

Exercise: if all the items were of size at most ε, then each bin (ex-
cept the last one) would have at least 1 − ε total size, thereby giving
an approximation of

1
Opt( I ) + 1 ≈ (1 + ε) Opt( I ) + 1.
1−ε

20.6 The Linear Grouping Algorithm for Bin Packing

The next algorithm was given by Wenceslas Fernandez de la Vega

and G.S. Luecker, and it uses a clever linear programming idea to get
an “almost-PTAS” for Bin Packing. Observe that we cannot hope
to get a PTAS, because of the hardness result we showed above. But Recall, a PTAS (polynomial-time ap-
we will show that if we allow ourselves a small additive term, the proximation scheme) is an algorithm
that for any ε > 0 outputs a (1 + ε)-
hardness goes away. The main ideas here will be the following: approximation in time n f (ε) . Hence, we
can get the approximation factor to any
1. We can discretize the item sizes down to a constant number of constant above 1 as we want, and still
get polynomial-time—just the degree
values by losing at most ε Opt (where the constant depends on ε).
of the polynomial in the runtime gets
larger.
2. The problem for a constant number of sizes can be solved almost
exactly (up to an additive constant) in polynomial-time.

3. Items of size at most ε can be added to any instance while main-

taining an approximation factor of (1 + ε).
256 the linear grouping algorithm for bin packing

20.6.1 The Linear Grouping Procedure

Lemma 20.4 (Linear Grouping). Given an instance I = (s1 , s2 , . . . , sn ) of
Bin Packing, and a parameter D ∈ N, we can efficiently produce another
instance I ′ = (s1 ′ , s2 ′ , . . . , sn ′ ) with increased sizes si ′ > si and at most D
distinct item sizes, such that

Opt( I ′ ) ≤ Opt( I ) + ⌈n/D⌉ .

Proof. The instance I ′ is constructed as follows:

• Sort the sizes si in non-increasing order to get s1 ≥ s2 ≥ . . . ≥ sn .

• Group items into D groups of ⌈n/D⌉ consecutive items, with the

last group being potentially slightly smaller.

• Define the new size si ′ for each item i to be the size of the largest
element in i’s group.
There are D distinct item sizes, and all sizes are only increased, so
it remains to show a packing for the items in I ′ that uses at most
Opt( I ) + ⌈n/D ⌉ bins. Indeed, suppose Opt( I ) assigns item i to some
bin b. Then we assign item (i + ⌈n/D⌉) to bin b. Since the sizes of the
items only get smaller, this allocates all the items except items in first
group, without violating the sizes. Now we assign each item in the
first group into a new bin, thereby opening up ⌈n/D⌉ more bins.

20.6.2 An Algorithm for a Constant Number of Item Sizes

Suppose we have an instance with at most D distinct item sizes: let
the sizes be s1 < s2 < . . . < SD , with δ > 0 being the smallest
size. The instance is then defined by the number of items for each
size. Define a configuration to be a collection of items that fits into
a bin: there can be at most 1/s1 items in any bin, and each item has
one of D sizes (or it can be the “null” item), so there are at most
N := ( D + 1)1/s1 different configurations. Note that if D and s1 are
both constants, this is (large) constant. (In the next section, we use
this result for the case where s1 ≥ ε.) Let C be the collection of all
configurations.
We now use an integer LP due to Paul Gilmore and Ralph Gomory
(from the 1950s). It has one variable xC for every configuration C ∈ C
that denotes the number of bins with configuration C in the solution.
The LP is:

min ∑ xC ,
C ∈C
s.t. ∑ ACs xC ≥ ns , ∀ sizes s
C
xC ∈ N.
 approximation algorithms 257

Here ACs is the number of items of type s being placed in the config-
uration C, and ns is the total number items of size s in the instance.
This is an exact formulation, and relaxing the integrality constraint to
xC ≥ 0 gives us an LP that we can solve in time poly( N, n). This is
polynomial time when N is a constant. We use the optimal value of In fact, we show in a homework prob-
this LP as our surrogate. lem that the LP can be solved in time
polynomial in n even when N is not a
How do we round the optimal solution for this LP? There are only constant.
D non-trivial constraints in the LP, and N non-negativity constraints.
So if we pick an optimal vertex solution, it must have some N of
these constraints at equality. This means at least N − D of these tight
constraints come from the latter set, and therefore N − D variables
are set to zero. In other words, at most D of the variables are non-
zero. Rounding these variables up to the closest integer, we get a
solution that uses at most LP( I ) + D ≤ Opt( I ) + D bins. Since D is a
constant, we have approximated the solution up to a constant.

20.6.3 The Final Bin Packing Algorithm

Combining the two ideas, we get a solution that uses

Opt( I ) + ⌈n/D ⌉ + D

bins. Now if we could ensure that n/D were at most ε Opt( I ), when
D was f (ε), we would be done. Indeed, if all the items have size at
least ε, the total volume (and therefore Opt( I )) is at least εn. If we
 
now set D = 1/ε2 , then n/D ≤ ε2 n ≤ ε Opt( I ), and the number of
bins is at most l m
(1 + ε) Opt( I ) + 1/ε2 .
What if some of the items are smaller than ε? We now use the
observation that First-Fit behaves very well when the item sizes
are small. Indeed, we first hold back all the items smaller than ε,
and solve the remaining instance as above. Then we add in the small
items using First-Fit: if it does not open any new bins, we are
fine. And if adding these small items results in opening some new
bin, then each of the existing bins—and all the newly opened bins
(except the last one)—must have at least (1 − ε) total size in them.
The number of bins is then at most
1
Opt( I ) + 1 ≈ (1 + O(ε)) Opt( I ) + 1,
1−ε
as long as ε ≤ 1/2.

20.7 Subsequent Results and Open Problems

21
Approximation Algorithms via SDPs

Just like the use of linear programming was a major advance in the
design of approximation algorithms, specifically in the use of lin-
ear programs in the relax-and-round framework, another significant
advantage was the use of semidefinite programs in the same frame-
work. For instance, the approximation guaranteee for the Max-Cut
problem was improved from 1/2 to 0.878 using this technique. More-
over, subsequent results have shown that any improvements to this
approximation guarantee in polynomial-time would disprove the
Unique Games Conjecture.

21.1 Positive Semidefinite Matrices

The main objects of interest in semidefinite programming, not sur-

prisingly, are positive semidefinite matrices.

Definition 21.1 (Positive Semidefinite Matrices). Let A ∈ Rn×n be a

real-valued symmetric matrix and let r = rank( A). We say that A is
positive semidefinite (PSD) if any of the following equivalent conditions
hold:

a. x⊺ Ax ≥ 0 for all x ∈ Rn .

b. All of A’s eigenvalues are nonnegative (with r of them being

strictly positive), and hence A = ∑ri=1 λi vi v⊺i for λ1 , . . . , λr > 0,
and vi ’s being orthonormal.

c. There exists a matrix B ∈ Rn×r such that A = BB⊺ .


d. There exist vectors v1 , . . . , vn ∈ Rr such that Ai,j = vi , v j for all
i, j.

e. There exist jointly distributed (real-valued) random variables

X1 , . . . , Xn such that Ai,j = E[ Xi X j ].

f. All principal minors have nonnegative determinants. A principal minor is a submatrix of A

obtained by taking the columns and
rows indexed by some subset I ⊆ [n].
260 semidefinite programs

The different definitions may be useful in different contexts. As an

example, we see that the condition in Definition 21.1(f) gives a short
proof of the following claim.

Lemma 21.2. Let A ⪰ 0. If Ai,i = 0 then A j,i = Ai,j = 0 for all j. We will write A ⪰ 0 to denote that A is
PSD; more generally, we write A ⪰ B
if A − B is PSD: this partial order on
Proof. Let j ̸= i. The determinant of the submatrix indexed by {i, j} is
symmetric matrices is called the Löwner
order.
Ai,i A j,j − Ai,j A j,i

is nonnegative, by assumption. Since Ai,j = A j,i by symmetry, and

Ai,i = 0, we get A2i,j = A2j,i ≤ 0 and we conclude Ai,j = A j,i = 0.

Definition 21.3 (Frobenius Product). Let A, B ∈ Rn×n . The Frobenius

inner product A • B, also written as ⟨ A, B⟩ is defined as

⟨ A, B⟩ := A • B := ∑ Ai,j Bi,j = Tr( A⊺ B).

i,j

We can think of this as being the usual vector inner product treat-
ing A and B as vectors of length n × n. Note that by the cyclic prop-
erty of the trace, A • xx⊺ = Tr( Axx⊺ ) = Tr( x⊺ Ax ) = x⊺ Ax; we will
use this fact to derive yet another of PSD matrices.

Lemma 21.4. A is PSD if and only if A • X ≥ 0 for all X ⪰ 0.

Proof. Suppose A ⪰ 0. Consider the spectral decomposition X =

⊺
∑i λi xi xi where λi ≥ 0 by Definition 21.1(b). Then

∑ λi ( A • xi xi ) = ∑ λi xi Axi ≥ 0.
⊺ ⊺
A•X =
i i

On the other hand, if A ⪰̸ 0, there exists v such that v⊺ Av < 0,

by 21.1(a). Let X = vv⊺ ⪰ 0. Then A • X = v⊺ Av < 0.

Finally, let us mention a useful fact:

Fact 21.5 (PSD cone). Given two matrices A, B ⪰ 0, and scalars
α, β > 0 then αA + βB ⪰ 0. Hence the set of PSD matrices forms a
convex cone in Rn(n+1)/2 . Here n(n + 1)/2 is the number of
entries on or above the diagonal in an
n × n matrix, and completely specifies a
symmetric matrix.
21.2 Semidefinite Programs

Loosely, a semidefinite program (SDP) is the problem of optimizing a

linear function over the intersection of a convex polyhedron K (given
by finitely many linear constraints, say Ax ≥ b) with the PSD cone K.
Let us give two useful packagings for semidefinite programs.
 approximation algorithms via sdps 261

21.2.1 As Linear Programs with a PSD Constraint

Consider a linear program where the variables are indexed by pairs
i, j ∈ [n], i.e., a typical variable is xi,j . Let X be the n × n dimensional
matrix whose (i, j)th entry is xi,j . As the objective and constraints are
linear, we can write them as C • X and Ak • X ≤ bk for some (not
necessarily PSD) matrices C, A1 , . . . , Am and scalars b1 , . . . , bm . An
SDP is an LP of this form with the additional constraint X ⪰ 0: Observe that if each of the matrices Ai
and C are diagonal matrices, say with
maximize C•X diagonals ai and c, this SDP becomes
X ∈Rn × n the linear program
subject to A k • X ≤ bk , ∀k ∈ [m] max{c⊺ x | a⊺k x ≤ bk , x ≥ 0},
X ⪰ 0. where x denotes the diagonal of the
PSD matrix X.

21.2.2 As Vector Programs

We can use Definition 21.1(d) to rewrite the above program as a “vec-
tor program”: where the linear objective and the linear constraints
are on inner products of vector variables:


maximize ∑ cij vi , v j
v1 ,...,vn ∈R
n
i,j
(k)

subject to ∑ aij v i , v j ≤ bk , ∀ k ∈ [ m ].
i,j

In particular, we optimize over vectors in n-dimensional space; we

cannot restrict the dimension of these vectors, much like we cannot
restrict the rank of the matrices X in the previous representation.

21.2.3 Examples of SDPs

Let A a symmetric n × n real matrix. Here is an SDP to compute the
maximum eigenvalue of A:

maximize A•X
X ∈Rn × n
subject to I•X =1 (21.1)

X⪰0

Lemma 21.6. SDP (21.1) computes the maximum eigenvalue of A.

Proof. Let X maximize SDP (21.1) (this exists as the objective is con-
tinuous and the feasible set is compact). Consider the spectral de-
composition X = ∑in=1 λi xi xi⊺ where λi ≥ 0 and ∥ xi ∥2 = 1. The
trace constraint I • X = 1 implies ∑i λi = 1. Thus the objective value
A • X = ∑i λi xi⊺ Axi is a convex combination of xi⊺ Axi . Hence without
loss of generality, we can put all the weight into one of these terms,
in which case X = yy⊺ is a rank-one matrix with ∥y∥2 = 1. By the
Courant-Fischer theorem, OPT ≤ max∥y∥2 =1 y⊺ Ay = λmax .
262 sdps in approximation algorithms

On the other hand, letting v be a unit eigenvector of A correspond-

ing to λmax , we have that OPT ≥ A • vv⊺ = v⊺ Av = λmax .

Here is another SDP for the same problem: In fact, it turns out that this SDP is dual
to the one in (21.1). Weak duality still
minimize t holds for this case, but strong duality
t does not hold in general for SDPs.
(21.2)
Indeed, there could be a duality gap for
subject to tI − A ⪰ 0. some cases, where both the primal and
dual are finite, but the optimal solutions
Lemma 21.7. SDP (21.2) computes the maximum eigenvalue of A. are not equal to each other. However,
under some mild regularity conditions
Proof. The matrix tI − A has eigenvalues t − λi . And hence the con- (e.g., the Slater conditions) we can show
strong duality. More about SDP duality
straint tI − A ⪰ 0 is equivalent to the constraint t − λ ≥ 0 for all its
here.
eigenvalues λ. In other words, t ≥ λmax , and thus OPT = λmax .

21.3 SDPs in Approximation Algorithms

We now consider designing approximation algorithms using SDPs.

Recall that given a matrix A, we can check if it is PSD in (strongly)
polynomial time, by performing its eigendecomposition. Moreover, if
A is not PSD, we can return a hyperplane separating A from the PSD
cone. Thus using the ellipsoid method, we can approximate SDPs
when OPT is appropriately bounded. Informally, We know that there is an optimal LP
solution where the numbers are singly
Theorem 21.8 (Informal Theorem). Assuming that the radius of the fea- exponential, and hence can be written
using a polynomial number of bits. But
sible set is at most exp(poly(⟨SDP⟩)), the ellipsoid algorithm can weakly this is not true in SDPs, in fact, OPT in
solve SDP in time poly(⟨SDP⟩, log(1/ε)) up to an additive error of ε. an SDP may be as large (or small) as
doubly exponential in the size of the
For a formal statement, see Theorem 2.6.1 of Matoušek and Gärt- SDP. (See Section 2.6 of the Matoušek
and Gärtner.)
ner. However, we will ignore these technical issues in the remainder
of this chapter, and instead suppose that we can solve our SDPs ex-
actly.

21.4 The MaxCut Problem and Hyperplane Rounding

Given a graph G = (V, E), the MaxCut problem asks us to find a

partition of the vertices (S, V \ S) maximizing the number of edges
crossing the partition. This problem is NP-complete. In fact assuming
P ̸= NP, a result of Johan Håstad shows that we cannot approximate
MaxCut better than 17/16 − ε for any ε > 0.

21.4.1 Greedy and Randomized Algorithms

We begin by considering a greedy algorithm: process the vertices
v1 , . . . , vn in some order, and place each vertex vi in the part of the
bipartition that maximizes the number of edges cut so far (breaking
ties arbitrarily).
 approximation algorithms via sdps 263

Lemma 21.9. The greedy algorithm cuts at least | E|/2-many edges.

Proof. Let δi be the number of edges from vertex i to vertices j < i:
then the greedy algorithm cuts at least ∑i δi /2 = | E|/2 edges.

This result shows two things: (a) every graph has a bipartition that
cuts half the edges of the graph, so Opt ≥ | E|/2. Moreover, (b) that
since Opt ≤ | E| on any graph, this means that Alg ≥ | E|/2 ≥ Opt /2.
We cannot hope to prove a better result
Here’s a simple randomized algorithm: place each vertex in either than Lemma 21.9 in terms of | E|, since
the complete graph Kn has (n2 ) ≈ n2 /2
S or in S̄ independently and uniformly at random. Since each edge is edges and any partition can cut at most
cut with probability 1/2, the expected number of cut edges is | E|/2. n2 /4 of them.
Moreover, by the probabilistic method Opt ≥ | E|/2.

21.4.2 Relax-and-Round using LPs

A natural direction would be to write an ILP formulation for Max-
Cut and to relax it: this approach does not give us anything beyond
a factor of 1/2, say.

21.4.3 A Semidefinite Relaxation

We now see a well-known example of an SDP-based approximation
algorithm due to Michel Goemans and David Williamson. Again, we
will use the relax-and-round framework from the previous chapter.
The difference is that we write a quadratic program to model the
problem exactly, and then relax it to get an SDP.
Indeed, observe that the MaxCut problem can be written as the
following quadratic program.

( x i − x j )2
maximize
x1 ,...,xn ∈R
∑ 4
(i,j)∈ E (21.3)
subject to xi2 =1 ∀i.

Since each xi is real-valued, and xi2 = 1, each variable must be as-

signed one of two labels {−1, +1}. Since each term in the objective
contributes 1 for an edge connecting two vertices in different parti-
tions, and 0 otherwise, this IP precisely captures MaxCut.
We now relax this program by replacing the variables xi with
vector variables vi ∈ Rn , where ∥vi ∥2 = 1.
∥ v i − v j ∥2
maximize
v1 ,...,vn ∈R
n ∑ 4
(i,j)∈ E (21.4)
2
subject to ∥ vi ∥ = 1 ∀i.



Noting that ∥vi − v j ∥2 = ∥vi ∥2 + ∥v j ∥2 − 2 vi , v j = 2 − 2 vi , v j , we
rewrite this vector program as The SDP relaxation for the MaxCut
problem was first introduced by Svata
Poljak and Franz Rendl.
264 the maxcut problem and hyperplane rounding


1 − vi , v j
maximize
v1 ,...,vn ∈R
n ∑ 2
(i,j)∈ E (21.5)
subject to ⟨ vi , vi ⟩ = 1 ∀i.

This is a relaxation of the original quadratic program, because we

can model any {−1, +1}-valued solution using vectors, say by a
corresponding {−e1 , +e1 }-valued solution. Since this is a maximiza-
tion problem, the SDP value is now at least the optimal value of the
quadratic program.

21.4.4 The Hyperplane Rounding Technique

In order to round this vector solution {vi } to the MaxCut SDP into
an integer scalar solution to MaxCut, we use the remarkably simple
method of hyperplane rounding. The idea is this: a term in the SDP
objective incurs a tiny cost close to zero when vi , v j are very close to
each other, and almost unit cost when vi , v j point in nearly opposite
directions. So we would like to map close vectors to the same value.
To do this, we randomly sample a hyperplane through the origin
and partition the vectors according to the side on which they land.
Formally, this corresponds to picking a vector g ∈ Rn according to
the standard n-dimensional Gaussian distribution, and setting
v1
S : = { i | ⟨ v i , g ⟩ ≥ 0}. g
v2

We now argue that this procedure gives us a good cut in expectation;

this procedure can be repeated to get an algorithm that succeeds with
high probability.

Theorem 21.10. The partition produced by the hyperplane rounding algo- v3

rithm cuts at least αGW · SDP edges in expectation, where αGW := 0.87856. v4

Proof. By linearity of expectation, it suffices to bound the probability Figure 21.1: A geometric picture of
Goemans-Williamson randomized
of an edge (i, j) being cut. Let rounding


θij := cos−1 ( vi , v j )

be the angle between the unit vectors vi and v j . Now consider the
2-dimensional plane P containing vi , v j and the origin, and let ge be
the projection of the Gaussian vector g onto this plane. Observe that
the edge (i, j) is cut precisely when the hyperplane defined by g
separates vi , v j . This is precisely when the vector perpendicular to
ge in the plane P lands between vi and v j . As the projection onto a
subspace of the standard Gaussian is again a standard Guassian (by
spherical symmetry),
2θij θij
Pr[(i, j) cut] = = .
2π π
 θij
approximation algorithms via sdps 265
g̃

Since the SDP gets a contribution of


1 − vi , v j 1 − cos(θi,j )
=
2 2
for this edge, it suffices to show that Figure 21.2: Angle between two vectors.
θ 1 − cos θ We cut edge (i, j) when the vector
≥α . perpendicular to ge lands in the grey
π 2 area.
Indeed, we can show (either by plotting, or analytically) that α =
0.87856 . . . suffices for the above inequality, and hence
1 − cos(θij )
E[# edges cut] = ∑ θij /π ≥ α ∑ 2
= α SDP .
(i,j)∈ E (i,j)∈ E

This proves the theorem.

Corollary 21.11. For any ε > 0, repeating the hyperplane rounding

algorithm O(1/ε log 1/δ) times and returning the best solution ensures that
we output a cut of value at least (.87856 − ε) Opt with probability 1 − δ.
We leave this proof as an exerise in using Markov’s inequality:
note that we want to show that the algorithm returns something not
too far below the expecation, which seems to go the wrong way, and
hence requires a moment’s thought.
The above algorithm is randomized and the result only holds in
expectation. However, it is possible to derandomize this result to
obtain a polynomial-time deterministic algorithm with the same
approximation ratio.

21.4.5 Subsequent Work and Connections

Can we get a better approximation factor, perhaps using a more so-
phisticated SDP? An influential result of Subhash Khot, Guy Kindler,
Elchanan Mossel, and Ryan O’Donnell says that a constant-better-
than-αGW -approximation would refute the Unique Games Conjecture.
Also, one can ask if similar rounding procedures exist for an
linear-programming relaxation as opposed to the SDP relaxation
here. Unfortunately the answer is again no: a result of Siu-On Chan,
James Lee, Prasad Raghavendra, and David Steurer shows that no
polynomial-sized LP relaxation of MaxCut can obtain a non-trivial
approximation factor, that is, any polynomial sized LP of MaxCut
has an integrality gap of 1/2.

21.5 Coloring 3-Colorable Graphs

Suppose we are given a graph G = (V, E) and a promise that there

is some 3-coloring of G. What is the minimum k such that we can
266 coloring 3-colorable graphs

find a k-coloring of G in polynomial time? It is well-known that 2-

coloring a graph can be done in linear time, but 3-coloring a graph is
NP-complete. Hence, even given a 3-colorable graph, it is NP-hard
to color it using 3 colors. (In fact, a result of Venkat Guruswami and
Sanjeev Khanna shows that it is NP-hard to color it using even 4 col-
ors.) But what if we ask to color a 3-colorable graph using 5 colors?
O(log n) colors? O(nα ) colors, for some fixed constant α? We will see
√
an easy algorithm to achieve an O( n)-coloring, and then will use
semidefinite programming to improve this to an O e (nlog6 (2) ) color-
ing. Before we describe these, let us recall the easy part of Brooks’
theorem. The harder part is to show that in fact ∆
colors suffice unless the graph is either
Lemma 21.12. Let ∆ be the maximum degree of a graph G, then we can a complete graph, or an odd-length
cycle.
find a (∆ + 1)-coloring of G in linear time.

Proof. Pick any vertex v, recursively color the remaining graph, and
then assign v a color not among the colors of its ∆ neighbors.

We will now describe an algorithm that colors a 3-colorable graph

√
G with O( n) colors, originally due to Avi Wigderson: while there
√
exists a vertex with degree at least n, color it using a fresh color.
Moreover, its neighborhood must be 2-colorable, so use two fresh
√
colors to do so. This takes care of n vertices using 3 colors. Remove
these, and repeat. Finally, use Lemma 21.12 to color the remaining
√
vertices using n colors. This proves the following result.

Lemma 21.13. There is an algorithm to color a 3-colorable graph with

√
O( n) colors.

21.5.1 An Algorithm using SDPs

Let’s consider an algorithm that uses SDPs to color a 3-colorable
graph with maximum degree ∆ using O e (∆log3 2 ) ≈ O
e (∆0.63 ) colors.
In general ∆ could be as large as n, so this could be worse than the
algorithm in Lemma 21.13, but we will be able to combine the ideas
together to get a better result.
For some parameter λ ∈ R, consider the following feasibility SDP
(where we are not optimizing any objective):

find v 1 , . . . , v n ∈ Rn


subject to vi , v j ≤ λ ∀(i, j) ∈ E (21.6)
⟨ vi , vi ⟩ = 1 ∀i ∈ V.

Why is this SDP relevant to our problem? The goal is to have vectors
clustered together in groups, such that each cluster represents a color.
Intuitively, we want to have vectors of adjacent vertices to be far
apart, so we want their inner product to be close to −1 (recall we are
 approximation algorithms via sdps 267

dealing with unit vectors, due to the last constraint) and vectors of
the same color to be close together.

Lemma 21.14. For 3-colorable graphs, SDP (21.6) is feasible with λ =

−1/2. 120◦ 120◦

Proof. Consider the vector placement shown in the figure to the right. 120◦
If the graph is 3-colorable, we can assign all vertices with color 1
the red vector, all vertices with color 2 the blue vector and all vertices
with color 3 the green vector. Now for every edge (i, j) ∈ E, we have
that   Figure 21.3: Optimal distribution of

 2π vectors for 3-coloring graph
vi , v j = cos = −1/2.
3
At first sight, it may seem like we are done: if we solve the above
SDP with λ = −1/2, don’t all three vectors look like the figure above?
No, that would only hold if all of them were to be co-planar. And in
n-dimensions we can have an exponential number of cones of angle
2π
3 , like in the next figure, so we cannot cluster vectors as easily as in
the above example.
To solve this issue, we apply a hyperplane rounding technique
similar to that from the MaxCut algorithm. Indeed, for some pa-
rameter t we will pick later, pick t random hyperplanes. Formally, we
pick gi ∈ Rn from a standard n-dimensional Gaussian distribution, Figure 21.4: Dimensionality problem of
for i ∈ [t]. Each of these defines a normal hyperplane, and these split 2π/3 far vectors
the Rn unit sphere into 2t regions (except if two of them point in the
same direction, which has zero probability). Now, each vectors {vi }
that lie in the same region can be considered “close” to each other,
and we can try to assign them a unique color. Formally, this means
that if vi and v j are such that


sign(⟨vi , gk ⟩) = sign( v j , gk )

for all k ∈ [t], then i and j are given the same color. Each region is
given a different color, of course.
However, this may color some neighbors with the same color, so
we use the method of alterations: while there exists an edge between
vertices of the same color, we uncolor both endpoints. When this
uncoloring stops, we remove the still-colored vertices from the graph,
and then repeat the same procedure on the remaining graph, until we
color every vertex. Note that since we use t hyperplanes, we add at
most 2t new colors per iteration. The goal is to now show that (a) the
number of interations is small, and (b) the value of 2t is also small.

Lemma 21.15. If half of the vertices are colored in a single iteration in

expectation, then the expected number of iterations to color the whole graph
is O(log n).
268 coloring 3-colorable graphs

Proof. Since the expected number of uncolored vertices is at most

half, Markov’s inequality says that more than 3/4 of the vertices are
uncolored in a single iteration, with probability at most 2/3. In other
words, at least 1/4 of the vertices are colored with probability 1/3.
Hence, the number of iterations to color the whole graph is domi-
nated by the number of flips of a coin of bias 1/3 to get log4 n heads.
This is 4 log4 n, which proves the result.

Lemma 21.16. The expected number of vertices that remain uncolored after
a single iteration is at most n∆ (1/3)t .

Proof. Fix an edge ij: for a single random hyperplane, the probability
that vi , v j are not separated by it is

π − θij 1
≤ ,
π 3

using that θij ≥ 2π

3 which follows from the constraint in the SDP.
Now if i is uncolored because of j, then vi , v j have the same color,
which happens when all t hyperplanes fail to separate the two. By
independence, this happens with probability at most (1/3)t . Finally,

E[remaining] = ∑ Pr[i uncolored]

i ∈V
≤ ∑ ∑ Pr[i uncolored because of j]. (21.7)
i ∈V (i,j)∈ E

There are n vertices, and each vertex has degree at most ∆, which
proves the result.

Lemma 21.17. There is an algorithm that colors a 3-colorable graph with

maximum degree ∆ with O(∆log3 2 · log n) colors in expectation.

Proof. Setting t = log3 (2∆) in Lemma 21.16, the expected number of

uncolored vertices in any iteration is

n · ∆ · (1/3)t ≤ n/2. (21.8)

Now Lemma 21.15 says we perform O(log n) iterations in expecta-

tion. Since we use most 2log3 (2∆) = (2∆)log3 2 colors in each iteration,
we get the result.

21.5.2 Improving the Algorithms Further

The expected number of colors used by the above algorithm is
√
e (nlog3 2 ) ≈ O
O e (n0.63 ), which is worse than our initial O( n) algo-
rithm. However we can combine the ideas together to get a better
result:
 approximation algorithms via sdps 269

Theorem 21.18. There is an algorithm that colors a 3-colorable graph with

e (nlog6 (2) ) colors.
O

Proof. For some value σ, repeatedly remove vertices with degree

greater than σ and color them and their neighbors with 3 new col-
ors, as in Lemma 21.13. This requires at most 3n/σ colors overall,
and leaves us with a graph having maximum degree σ. Now use
Lemma 21.17 to color the remaining graph with O(σlog3 2 · log n) col-
ors. Picking σ to be nlog6 3 to balance these terms, we get a procedure
that uses Oe (nlog6 2 ) ≈ O
e (n0.38 ) colors.

21.5.3 Final notes on coloring 3-colorable graphs

This result us due to David Karger, Rajeev Motwani, and Madhu Su-
dan. They gave a better rounding algorithm that uses spherical caps
e (n1/4 ) colors. This result was
instead of hyperplanes to achieve O
then improved over a sequence of papers: the current best result by
Ken-Ichi Kawarabayashi and Mikkel Thorup uses O(n0.199 ) colors.
It remains an outstanding open problem to either get a better algo-
rithm, or to show hardness results, even under stronger complexity-
theoretic hypotheses.
22
Online Algorithms

In this chapter we introduce online algorithms and study two classic

online problems: the rent-or-buy problem, and the paging problem.
While the models we consider are reminiscent of those in regret min-
imization in online learning and online convex optimization, there
are some important differences, which lend a different flavor to the
results that are obtained.

22.1 The Competitive Analysis Framework

In the standard setting of online algorithms there is a sequence of

requests σ = (σ1 , σ2 , . . . , σt , . . .) received online. An online algorithm
does not know the input sequence up-front, but sees these requests
one by one. It must serve request σi before it is allowed to see σi+1 .
Serving this request σi involves some choice of actions, and incurs
some cost. We will measure the performance of an algorithm by
considering the ratio of the total cost incurred on σ to the optimal
cost of serving σ in hindsight. To make all this formal, let us xsee an
example of an online problem.

22.1.1 Example: The Paging Problem

The paging problem arises in computer memory systems. Often, a
memory system consists of a large but slow main memory, as well as
a small but fast memory called a cache. The CPU typically communi-
cates directly with the cache, so in order to access an item that is not
contained in the cache, the memory system has to load the item from
the slow memory into the cache. Moreover, if the cache is full, then
some item contained in the cache has to be evicted to create space for
the requested item.
We say that a cache miss occurs whenever there is a request to an
item that is not currently in the cache. The goal is to come up with
an eviction strategy that minimizes the number of cache misses.
272 the competitive analysis framework

Typically we do not know the future requests that the CPU will make
so it is sensible to model this as an online problem. If the entire sequence of requests
We let U be a universe of n items or pages. The cache is a mem- is known, show that Belády’s rule is
optimal: evict the page in cache that is
ory containing at most k pages. The requests are pages σi ∈ U and next requested furthest in the future.
the online algorithm is an eviction policy. Now we return back to
defining the performance of an online algorithm.

22.1.2 The Competitive Ratio

As we said before, the online algorithm incurs some cost as it serves
each request. If the complete request sequence is σ, then we let
Alg(σ ) be the total cost incurred by the online algorithm in serv-
ing σ. Similarly, we let Opt(σ ) be the optimal cost in hindsight of
Figure 22.1: Illustration of the Paging
serving σ. Note that Opt(σ ) represents the cost of an optimal offline Problem
algorithm that knows the full sequence of requests. Now we define
the competitive ratio of an algorithm to be:

Alg(σ )
max
σ Opt(σ )

In some sense this is an “apples to oranges” comparison, since

the online algorithm does not know the full sequence of requests,
whereas the optimal cost is aware of the full sequence and hence is
an “offline” quantity.
Note two differences from regret minimization: there we made
a prediction xt before (or concurrently with) seeing the function f t ,
whereas we now see the request σt before we produce our response
at time t. In this sense, our problem is easier. However, the bench-
mark is different—we now have to compare with the best dynamic se-
quence of actions for the input sequence σ, whereas regret is typically
measured with respect to a static response, i.e., to the cost of playing
the same fixed action for each of the t steps. In this sense, we are now
solving a harder problem. There are is a smaller, syntactic difference
as well: regret is an additive guarantee whereas the competitive ra-
tio is a multiplicative guarantee—but this is more a reflection on the
kind of results that are possible, rather than fundamental difference
between the two models.

22.1.3 What About Randomized Algorithms?

The above definitions generally hold for deterministic algorithms, so
how should we characterize randomized algorithms. For the deter-
ministic case we generally think about some adversary choosing the
worst possible request sequence for our algorithm. For randomized
algorithms we could consider either oblivious or adaptive adver-
saries. Oblivious adversaries fix the input sequence up front and then
 online algorithms 273

let the randomized algorithm process it. An adaptive adversary is

allowed to see the results of the coin flips the online algorithm makes
and thus adapt its request sequence. We focus on oblivious adver-
saries in these notes.
To define the performance of a randomized online algorithm we
just consider the expected cost of the algorithm. Against an obliv-
ious adversary, we say that a randomized online algorithm is α-
competitive if for all request sequences σ,

E[Alg(σ)] ≤ α · Opt(σ ).

22.2 The Ski Rental Problem: Rent or Buy?

Now that we have a concrete analytical framework, let’s apply it to a

simple problem. Suppose you are on a ski trip with your friends. On
each day you can choose to either rent or buy skis. Renting skis costs
$1, whereas buying skis costs $B for B > 1. However, the benefit of
buying skis is that on subsequent days you do not need to rent or
buy again, just use the skis you already bought. The problem that
arises is that for some mysterious reason we do not know how long
the ski trip will last. On each morning we are simply told whether or
not the trip will continue that day. The goal of the problem is to find
a rent/buy strategy that is competitive with regards to minimizing
the cost of the trip.
In the notation that we developed above, the request for the i’th
day, σi , is either “Y” or “N” indicating whether or not the ski trip
continues that day. We also now that once we see a “N” request that
the request sequence has ended. For example a possible sequence
might be σ = (Y, Y, Y, N ). This allows us to characterize all instances
of the problem as follows. Let Ij be the sequence where the ski trip
ends on day j. Suppose we knew ahead of time what instance we re-
ceived, then we have that Opt( Ij ) = min{ j, B} since we can choose to
either buy skis on day 1 or rent skis every day depending on which is
better.

22.2.1 Deterministic Rent or Buy

We can classify and analyze all possible deterministic algorithms
since an algorithm for this problem is simply a rule deciding when
to buy skis. Let Algi be the algorithm that rents skis until day i, then
buys on day i if the trip lasts that long. The cost on instance Ij is then
Algi ( Ij ) = (i − 1 + B) · 1{i≤ j} + j · 1{i> j} . What is the best determin-
istic algorithm from the point of view of competitive analysis? The
following claims answer this question.
274 the ski rental problem: rent or buy?

Lemma 22.1. The competitive ratio of algorithm AlgB is 2 − 1/B and this is
the best possible ratio for any deterministic algorithm.

Proof. There are two cases to consider j < B and j ≥ B. For the
first case, AlgB ( Ij ) = j and Opt( Ij ) = j, so AlgB ( Ij )/ Opt( Ij ) =
1. In the second case, AlgB ( Ij ) = 2B − 1 and Opt( Ij ) = B, so
AlgB ( Ij )/ Opt( Ij ) = 2 − 1/B. Thus the competitive ratio of AlgB
is
AlgB ( Ij )
max = 2 − 1/B
Ij Opt( Ij )

Now to show that this is the best possible competitive ratio for any
deterministic algorithm. Consider algorithm Algi . We find an in-
stance Ij such that Algi ( Ij )/ Opt( Ij ) ≥ 2 − 1/B. If i ≥ B then we take
j = B so that Algi ( Ij ) = (i − 1 + B) and Opt( Ij ) = B so that

Algi ( Ij ) i−1+B i 1 1
= = +1− ≥ 2−
Opt( Ij ) B B B B

Now if i = 1, we take j = 1 so that

Algi ( Ij ) B
= ≥2
Opt( Ij ) 1

Since B is an integer > 1 by assumption. Now for 1 < i < B, we take

j = ⌊(i − 1 + B)/(2 − 1/B)⌋ ≥ 1 so that

Algi ( Ij ) 1
≥ 2− .
Opt( Ij ) B

22.2.2 Randomized Rent or Buy

Can randomization improve over deterministic algorithms in terms
of expected cost? We will show that this is in fact the case. So how
do we design a randomized algorithm for this problem? We use the
following general insight about randomized algorithms, notably that
a randomized algorithm is a distribution over deterministic algorithms.
To keep things simple let’s consider the case when B = 4. We
construct the following table of payoffs where the rows correspond
to deterministic algorithms Algi and the columns correspond to
instances Ij .

I1 I2 I3 I∞
Alg1 4/1 4/2 4/3 4/4
Alg2 1/1 5/2 5/3 5/4
Alg3 1/1 2/2 6/3 6/4
Alg4 1/1 2/2 3/3 7/4
 online algorithms 275

While the real game is infinite along with coordinates, we do not

need to put columns for I4 , I5 , . . . because these strategies for the ad-
versary are dominated by the column I∞ . Now given that the adver-
sary would rather give us only these inputs, we do not need to put
rows after B = 4 since buying after day B is worse than buying on
day B for these inputs. (Please check these for yourself!) This means
we can think of the above table as a 2-player zero-sum game with 4
strategies each. The row player chooses an algorithm and the column
player chooses an instance, then the number in the corresponding
entry indicates the loss of the row player. Thinking along the lines of
the Von Neumann minimax theorem, we can consider mixed strate-
gies for the row player to construct a randomized algorithm for the
ski rental problem.
Let pi be the probability of our randomized algorithm choosing
row i. What is the expected cost of this algorithm? Suppose that
the competitive ratio was at most c in expectation. The expected
competitive ratio of our algorithm against each instance should be at
most c, so this yields the following linear constraints.

4p1 + p2 + p3 + p4 ≤ c
4p2 + 5p2 + 2p3 + 2p4
≤c
2
4p1 + 5p3 + 6p3 + 3p4
≤c
3
4p1 + 5p2 + 6p3 + 7p4
≤c
4

We would like to minimize c subject to p1 + p2 + p3 + p4 = 1 and

pi ≥ 0. It turns out that one can do this by solving the following
system of equations: Why is it OK to set the inequalities to
equalities? Simply because it works
out: in essence, we are guessing that
p1 + p2 + p3 + p4 = 1 making these four constraints tight
4p1 + p2 + p3 + p4 = c gives a basic feasible solution—and our
guess turns out to right. It does not
4p1 + 5p2 + 2p3 + 2p4 = 2c show optimality, but we can do that by
giving a matching dual solution.
4p1 + 5p2 + 6p3 + 3p4 = 3c
4p1 + 5p2 + 6p3 + 7p4 = 4c

Subtracting each line from the previous one gives us

p1 + p2 + p3 + p4 = 1
3p1 = c − 1
4p2 + p3 + p4 = c
4p3 + p4 = c
4p4 = c.
276 the ski rental problem: rent or buy?

This gives us p4 = c/4, p3 = (3/4)(c/4), etc., and indeed that

1
c= and pi = (3/4)i−1 (c/4)
1 − (1 − 1/4)4

for i = 1, 2, 3, 4. For general B, we get

1 e
c = cB = ≤ .
1 − (1 − 1/B) B e−1

Moreover, this value of c B is indeed the best possible competitive

ratio for anyth randomized algorithm for the ski rental problem.
How might one prove such a result? We instead consider playing
a random instance against a deterministic algorithm. By Von Neu-
mann’s minimax theorem the value of this should be the same as
what we considered above. We leave it as an exercise to verify this for
the case when B = 4.

22.2.3 (Optional) A Continuous Approach

This section is quite informal right now, needs to be made formal.
For simplicity, assume B = 1 by scaling, and that both the algorithm
and the length of the season can be any real in [0, 1]. Now our ran-
domized algorithm chooses a threshold t ∈ [0, 1] from the probability
distribution with density function f (t). Let’s say f is continuous.
Then we get that for any season length ℓ,
Z ℓ Z 1
(1 + t) f (t) dt + ℓ f (t) dt = c · ℓ.
t =0 t=ℓ

(Again, we’re setting an equality there without justification, except

that it works out.) Now we can differentiate w.r.t. ℓ to get
Z 1
(1 + ℓ) f (ℓ) + f (t) dt − ℓ f (ℓ) = c.
t=ℓ

(This differentiation is like taking differences of successive lines that

we did above.) Simplifying,
Z 1
f (ℓ) + f (t) dt = c. (22.1)
t=ℓ

Taking derivatives again:

f ′ (ℓ) − f (ℓ) = 0

But this solves to f (ℓ) = Ceℓ for some constant C. Since f is a prob-
R1
ability density function, ℓ=0 f (ℓ) = 1, we get C = e−1 1 . Substituting
into (22.1), we get that the competitive ratio is c = e−e 1 , as desired.
 online algorithms 277

22.3 The Paging Problem

Now we return to the paging problem that was introduced earlier

and start by presenting a disappointing fact.

Lemma 22.2. No deterministic algorithm can be < k-competitive for the

paging problem.

Proof. Consider a universe with k + 1 pages in all. In each step the

adversary requests a page not in the cache (there is always at least
1 such page). Thus the algorithm’s cost over n requests is n. The
optimal offline algorithm can cut losses by always evicting the item
that will be next requested furthest in the future, thus it suffers a
cache miss every k steps so the optimal cost will be n/k. Thus the
n
competitive ratio of any deterministic algorithm is at least n/k =
k.

It is also known that many popular eviction strategies are k-

competitive such as Least Recently Used (LRU) and First-In First-Out
(FIFO). We will show that a 1-bit variant of LRU is k-competitive
and also show that a randomized version of it achieves an O(log k)-
competitive randomized algorithm for paging.

22.3.1 The 1-bit LRU/Marking Algorithm

The 1-bit LRU/Marking algorithm works in phases. The algorithm

maintains a single bit for each page in the universe. We say that a
page is marked/unmarked if its bit is set to 1/0. At the beginning of
each phase, all pages are unmarked. When a request for a page not
in the cache comes, then we evict an arbitrary unmarked page and
put the requested page in the cache, then mark the requested page. If
there are no unmarked pages to evict, then we unmark all pages and
start a new phase.

Lemma 22.3. The Marking algorithm is k-competitive for the paging

problem.

Proof. Consider the i’th phase of the algorithm. By definition of the

algorithm, Alg incurs a cost of at most k during the phase since we
can mark at most k different pages and hence we will have at most
k cache misses in this time. Now consider the first request after the
i’th phase ends. We claim that Opt has incurred at least 1 cache miss
by the time of this request. This follows since we have now seen
k + 1 different pages. Now summing over all phases we see that
Alg ≤ k Opt
278 the paging problem

Now suppose that instead of evicting an arbitrary unmarked page,

we instead evicted an unmarked page uniformly at random. For this
randomized marking algorithm we can prove a much better result.

Lemma 22.4. The Randomized Marking Algorithm is O(log k)-competitive

Proof. We break up the proof into an upper bound on Alg’s cost and
a lower bound on Opt’s cost. Before doing this we set up some no-
tation. For the ith phase, let Si be the set of pages in the algorithm’s
cache at the beginning of the phase. Now define

∆ i = | Si + 1 \ Si | .

We claim that the expected number of cache misses made by the

algorithm in phase i is at most ∆i ( Hk + 1), where Hk is the kth har-
monic number. By summing over all phases we see that E[Alg] ≤
∑i ∆i ( Hk + 1).
Now let Ri be the set of distinct requests in phase i. For each re-
quest in Ri we say that it is clean if the requested page is not Si , oth-
erwise the request is called stale. Every cache miss in the ith phase is
caused by either a clean request or a stale request.

1. The number of cache misses due to clean requests is at most ∆i

since there can be at most ∆i clean requests in phase i: each clean
request brings in a page not belonging to Si into the cache and
marks it, so it will be in Si+1 .

2. To bound the cache misses due to stale requests, suppose there

have been c clean requests and s stale requests so far, and consider
the s + 1st stale request. The probability this request causes a This is like the Airline seat problem,
cache miss is at most k−c s since we have evicted c random pages where we can imagine that c confused
passengers get on at the beginning.
out of k − s remaining stale requests. Now since c ≤ ∆i , we have
that the expected cost due to stale requests is at most

k −1 k −1
c 1
∑ k−s
≤ ∆i ∑
k − s
= ∆i Hk .
s =0 s =0

Now the expected total cost in phase i is at most

∆i Hk + ∆i = ∆i ( Hk + 1).

Now we claim that Opt ≥ 12 ∑i ∆i . Let Si∗ be the pages in Opt’s cache
at the beginning of phase i. Let ϕi be the number of pages in Si but
not in Opt’s cache at the beginning of phase i, i.e., ϕi = |Si \ Si∗ |.
Now let Opti be the cost that Opt incurs in phase i. We have that
Opti ≥ ∆i − ϕi since this is the number of “clean” requests that Opt
sees. Moreover, consider the end of phase i. Alg has the k most recent
 online algorithms 279

requests in cache, but Opt does not have ϕi+1 of them by definition of
ϕi+1 . Hence Opti ≥ ϕi+1 . Now by averaging,
1
Opti ≥ max{ϕi+1 , ∆i − ϕi } ≥ (ϕ + ∆i − ϕi ).
2 i +1
So summing over all phases we have
1 1
Opt ≥ ∑
2 i
∆i + ϕ f inal − ϕinitial ≥ ∑ ∆i ,
2 i
since ϕ f inal ≥ 0 and ϕinitial = 0. Combining the upper and lower
bound yields
E[Alg] ≤ 2( Hk + 1) Opt = O(log k) Opt .
It can also be shown that no randomized algorithm can do better
than Ω(log k )-competitive for the paging problem. For some intuition
as to why this might be true, consider the coupon collector prob-
lem: if you repeatedly sample a uniformly random number from
{1, . . . , k + 1} with replacement, show that the expected number of
samples to see all k + 1 coupons is Hk+1 .

22.4 Generalizing Paging: The k-Server Problem

Another famous problem in online algorithms is the k-server prob-

lem. Consider a metric space M = (V, d) with point set V and dis-
tance function d : V × V → R+ satisfying the triangle inequality.
In the k-server problem there are k servers that are located at various
points of M. At each timestep t we receive a request σt ∈ V. If there
is a server at point σt already, then we can server that request for free.
Otherwise we move some server from point x to point σt and pay a
cost equal to d( x, σt ). The goal of the problem is to serve the requests
while minimizing the total cost of serving them.
The paging problem can be modeled as a k-server problem as fol-
lows. We let U be the points of the metric space and take d( x, y) = 1
for all pages x, y where x ̸= y. This special case shows that every de-
terministic algorithm is at least k-competitive and every randomized
algorithm is Ω(log k)-competitive by the discussion in the previous
section. It is conjectured that there is a k-competitive deterministic
algorithm: the best known result is a (2k − 1)-competitive algorithm
of Elias Koutsoupias and Christos Papadimitriou.
For randomized algorithms, a poly-logarithmic competitive al-
gorithm was given by Nikhil Bansal, Niv Buchbinder, Aleksander
Madry, and Seffi Naor. This was recently improved via an approach
based on Mirror descent by Sebastien Bubeck, Michael Cohen, and
James Lee, Yin Tat Lee, Aleksander Madry; see this paper of Niv
Buchbinder, Marco Molinaro, Seffi Naor, and myself for a discretiza-
tion.
 Part V

Additional Topics
23
Smoothed Analysis of Algorithms

Smoothed analysis originates from an influential paper of Spielman

and Teng, and provides an alternative to worst-case analysis. Assume
that we want to analyze an algorithm’s cost, e.g., the running time
of an algorithm. Let cost( A( I )) be the cost that the algorithm has for
input instance I. We will usually group the possible input instances
according to their “size” (depending on the problem, this might be
the number of jobs, vertices, variables and so on). Let In be the set of
all inputs of ‘size’ n. For a worst case analysis, we are now interested
in
max cost( A( I )),
I ∈In

the maximum cost for any input of size n. Consider Figure 23.1 and
imagine that all instances of size n are lined up on the x-axis. The
blue line could be the running time of an algorithm that is fast for
most inputs but very slow for a small number of instances. The worst
case is the height of the tallest peak. For the green curve, which
could for example be the running time of a dynamic programming
algorithm, the worst case is a tight bound since all instances induce
the same cost.

Figure 23.1: An example of a cost

function with outliers.

Worst case analysis provides a bound that is true for all instances,
but it might be very loose for some or most instances like it happens
for the blue curve. When running the algorithm on real world data
284 the traveling salesman problem

sets, the worst case instances might not occur, and there are different
approaches to provide a more realistic analysis.
The idea behind smoothed analysis is to analyze how an algorithm
acts on data that is jittered by a bit of random noise (notice that for
real world data, we might have a bit of noise anyway). This is mod-
eled by defining a notion of ‘neighborhood’ for instances and then
analyzing the expected cost of an instance from the neighborhood of
I instead of the running time of I.
The choice of the neighborhood is problem specific. We assume
that the neighborhood of I is given in form of a probability distribu-
tion over all instances in In . Thus, it is allowed that all instances in
In are in the neighborhood but their influence will be different. The
distribution depends on a parameter σ that says how much the input
shall be jittered. We denote the neighborhood of I parameterized on
σ by nbrhdσ ( I ). Then the smoothed complexity is given by

max E I ′ ∼nbrhdσ ( I ) [cost( A( I ′ ))]

I ∈In

In Figure 23.1, we indicate the neighborhood of one of the peak

instances by the red brackets (imagine that the distribution is zero
outside of the brackets – we will see one case of this in Section 23.3).
The cost is smoothed within this area. For small σ, the bad instances
get more isolated so that they dominate the expected value for their
neighborhood, for larger σ, their influence decreases. We want the
neighborhood to be big enough to smooth out the bad instances.
So far, we have mostly talked about the intuition behind smoothed
analysis. The method has a lot of flexibility since the neighborhood
can be defined individually for the analysis of each specific problem.
We will see two examples in the following sections.

23.1 The Traveling Salesman Problem

The traveling salesman problem (TSP) is a classic example of an NP-

complete problems with practical importance. In this problem, we are
given an undirected weighted graph with (symmetric) edge weights
wij = w ji ∈ [0, 1], find the minimum weighted cycle that contains all
vertices.
On metric spaces there exists a polynomial 1.5-approximation al-
gorithm (and now slightly better!) and on a d-dimensional Euclidean
spaces (for fixed d) there are 1 + ε polynomial-time approximation
schemes time (due to Sanjeev Arora, and others). In this chapter we
consider the 2-OPT local-search heuristic.
 smoothed analysis of algorithms 285

23.1.1 The 2-OPT Heuristic

Start with an arbitrary cycle (i.e., a random permutation of vertices).
In each iteration find two pairs of adjacent vertices ( a, b) and (c, d)
(i.e. a → b and c → d are edges in the cycle) and consider a new
candidate cycle obtained by removing edges a → b and c → d as a b
well as inserting edges a → c and b → d. See Figure 23.2. If the new
candidate cycle has smaller weight than the current cycle, replace the
current cycle with the candidate one and repeat the heuristic. If no
quadruplet ( a, b, c, d) can decrease the weight, end the algorithm and
report the final cycle.
There are two questions to be asked here: (a) how long does it take
for us to reach the local optimum, and (b) what is the quality of the d c
Figure 23.2: Main step of the 2-OPT
local optima? heuristic: a → b, c → d are replaced by
It is useful to note that the 2-OPT always terminates since there a → c, b → d. The path b → . . . → c is
also reversed.
at finite number of cycles (exactly (n − 1)! distinct ones) and each
iteration strictly decreases the weight. However, there are examples
on which the heuristic takes Ω(exp(n)) time as well as examples
log n
where it achieves value of Ω( log log n ) · OPT. Despite this poor worst-
case behavior, the heuristic performs well in practice, and hence it
makes sense to see if smoothed analysis can explain its performance.

23.1.2 The Smoothing Model and Result

We set up the probability space by sampling each wij ∈ [0, 1] from
a independent distribution with probability density functions f ij ( x )
(note that the distributions can be different for different edges). The
densities are bounded by σ1 , i.e. f ij ( x ) ≤ σ1 for all i, j, x. The density
can be otherwise completely arbitrary can chosen by the adversary.
We will prove that the expected number of iterations taken by 2-
OPT is poly(n, σ1 ). Let X be the the number of iterations; we write its
expectation in terms of the probability that the algorithm takes more
than t iterations via the formula
∞
E[ X ] = ∑ Pr[X ≥ t]
t =1
( n −1) !
= ∑ Pr [ X ≥ t]. (23.1)
t =1

Note that the (n − 1)! represents the maximum possible number of it-
erations the algorithm can take, since each swap is strictly improving
and hence no permutation can repeat.
To bound this probability, we use the simple fact that a large num-
ber of iterations t means there is an iteration where the decrease in
weight was very small, and this is an event with low probability.
286 the simplex algorithm

Lemma 23.1. The probability of an iteration with improvement (i.e., de-

4
crease in cycle weight) in the range (0, ε] is at most nσ ε .

Proof. Fix a quadruplet ( a, b, c, d). We upper-bound the probability

that the (0, ε] improvement was obtained by replacing a → b, c → d
with a → c, b → d. The decrease in weight resulting from this
replacement is −wbd − wac + wab + wcd ∈ (0, ε]. By conditioning
on wbd , wac , wab , the last unconditioned value wcd must lie in some
interval ( L, L + ε]. By the hypothesis on the distribution density, this
can happen with probability at most σε , leading us to conclude that
for a fixed ( a, b, c, d) the probability of such an event is at most σε .
By union-bounding over all n4 quadruplets ( a, b, c, d) we can
4
bound the probability by nσ ε .

Lemma 23.2. The probability that the algorithm takes at least t iterations is
5
at most ntσ .

Proof. Note that wij ∈ [0, 1] implies that the weight of any cycle is in
[0, n]. This implies by pigeonhole principle that there was an iteration
where the decrease in weight was at most ε := nt . By Lemma 23.1 the
n5
probability of this event is at most tσ , as advertised.

Theorem 23.3. 6 The expected number of iterations that the algorithm takes
n log n
is at most O σ .

Proof. Using Equation 23.1 and Lemma 23.2 we have:

( n −1) !
E[ X ] = ∑ Pr [ X ≥ t]
t =1
( n −1) !
n5
≤ ∑ tσ
t =1
( n −1) !
n5 1
σ t∑
=
=1 t
 6 
n log n
=O
σ

Here we used the fact that ∑tN=1 1

t = O(log N ) and O(log(n − 1)!) =
O(n log n).

An improved bound of blah was given by details.

23.2 The Simplex Algorithm

We now turn our attention to the simplex algorithm for solving gen-
eral linear programs. Given a vector c ∈ Rd and a matrix A ∈ Rn×d ,
 smoothed analysis of algorithms 287

this algorithm finds a solution variables x = ( x1 , . . . , xd )⊺ that opti-

mizes

max c⊺ x
Ax ≤ 1.

This is a local-search algorithm, which iterates through solutions

corresponding to vertices of the feasible solution polyhedron until
it finds an optimal solution, or detects that the linear program is
unbounded. It always goes from a solution to one of at least the same
value, and there are situations where we have to make swaps that
do not improve the value. Give example? In case there are many
possible next moves, a pivot rule decides the next solution for the
algorithm. Many pivot rules have been proposed. For most of them,
we now know inputs where the simplex method iterates through an
super-polynomial or exponential number of vertices, and thus has
poor worst-case performance. (One widely used bad example is the
Klee–Minty cube.) Moreover, there is no pivot rule for which we can
prove a polynomial number of steps. In fact, we don’t know the answer to
Despite this, the simplex method is widely used to solve linear a simpler problem: if we imagine the
polyhedron defining the feasible region
programs. In their seminal paper, Dan Spielman and Shang-Hua as a graph, with the extreme points
Teng show that the simplex algorithm has a polynomial smoothed being vertices, and the edges of the
polyhedron connecting them, is the
complexity for a specific pivot rule, the “shadow vertex pivot rule”. diameter of this polytope polynomially
bounded in the dimension d and the
number of constraints n? As far as
23.2.1 The Smoothing Model and Result we know, the diameter could even be
O ( n + d ).
More precisely, they have shown that the simplex method with this
pivot rule provides a polynomial algorithm for the following prob-
lem: They consider a slightly more general
problem, where the right hand side
could be some b ∈ Rn instead of 1.
Input: vector c ∈ Rd , matrix A ∈ Rn×d

Problem: For a random matrix G ∈ Rn×d and a random vector

g ∈ Rn where all entries are chosen independently from a Gaussian
distribution N (0, maxi || ai ||2 ), solve the following LP:

max c⊺ x
( A + G ) x ≤ 1 + g.

This is one specific neighborhood model. Notice that for any input
( A, c, 1), all inputs ( A + G, c, 1, g) are potential neighbors, but the
probability decreases exponentially when we go ‘further away’ from
the original input. The vector c is not changed, only A and 1 are
jittered. The variance of the Gaussian distributions scales with the
smoothness parameter σ. For σ = 0, the problem reduces to the
288 the simplex algorithm

standard linear programming problem and the analysis becomes a

worst case analysis.
Notice that randomly jittering A and 1 means that the feasibility of
the linear program can be switched (from feasible to infeasible or vice
versa). Thus, jittering the instance and then solving the LP does not
necessarily give any solution for the original LP. However, assuming
that the input comes from an appropriate noisy source, the following
theorem gives a polynomial running time bound.

Theorem 23.4. The ‘smoothed’ number of simplex steps executed by

the simplex algorithm with the shadow vertex pivot rule is bounded by
poly(n, d, 1/σ ) for the smoothed analysis model described above.

The original result bounded the number of steps by O((nd/σ)O(1) ),

but the exponents were somewhat large. Roman Vershynin proved an
improved bound of O(log7 n(d9 + d3 /σ4 )). An improved and sim-
plified analysis due to Daniel Dadush and Sophie Huiberts gives a
bound of ≈ d3.5 σ−2 poly log n. More on this later. See this survey
chapter by Daniel Dadush and Sophie Huiberts for many details.

23.2.2 The Shadow Vertex Pivot Rule

We conclude with an informal description of the shadow vertex pivot
rule. Consider Figure 23.3. The three-dimensional polytope stands
for the polyhedron of all feasible solutions, which is in general d-
dimensional. The vector c points in the direction in which we want
to maximize (it is plotted with an offset to emphasize the optimal optimum
vertex). The shadow pivot rule projects the polyhedron to a two-
dimensional space spanned by c and the starting vertex u. c̄
ū
Assume that the original LP has an optimal solution with finite ob-
jective value. Then the polyhedron must be bounded in the direction
of c. It is also bounded in the direction of u since the start vertex u is start vertex
optimal for the direction u.
After projecting the polyhedron we intuitively follow the vertices
that lie on the convex hull of the projection (moving towards the
direction of c)1 . Notice that the extreme vertex on the c-axis is the
projection of an optimal vertex of the original polyhedron.
Since the polyhedron of all feasible solutions is not known, the
projection cannot be done upfront. Instead, in each step, the algo-
Figure 23.3: Illustration of the
rithm projects the vectors to the neighbor vertices onto span{c, u}
shadow vertex pivot rule.
and identifies the neighbor which is the next on the convex hull. 1
The two-dimensional projection
A first step towards proving the result is to show that the shadow is called the shadow of the original
has a small number of vertices if the polyhedron ( A + G ) x ≤ (1 + g) polyhedron, hence the name of the
pivot rule.
is projected onto two fixed vectors u and c. The real result for sim-
plex, however, is complicated by the fact that the vector u depends
 smoothed analysis of algorithms 289

on the polyhedron and on the starting vertex, and so the polyhedron

is projected to a subspace that is correlated to the polyhedron itself.
Another complication: finding a starting vertex is as hard as solv-
ing an LP. Spielman and Teng handle these and other issues; see the
original publications.

23.3 The Knapsack Problem

Finally, we turn to a problem for which we will give (almost) all the
details of the smoothed analysis. The input to the knapsack problem
is a collection of n items, each with size/weight wi ∈ R≥0 and profit
pi ∈ R≥0 , and the goal is to pick a subset of items that maximizes the
sum of profits, subject to the total weight of the subset being at most
some bound B. As always, we can write the inputs
The knapsack problem is weakly NP-hard—e.g., if the sizes are as vectors p, w ∈ Rn≥0 , and hence the
solution as x ∈ {0, 1}n . For example,
integers, then it can be solved by dynamic programming in time with p = (1, 2, 3), w = (2, 4, 5) and B =
O(nB). Notice that perturbing the input by a real number prohibits 10, the optimal solution is x = (0, 1, 1)
with p⊺ x = 5 and w⊺ x = 9.
the standard dynamic programming approach which assumes in-
tegral input. Therefore we show a smoothed analysis for a dif-
ferent algorithm, one due to George Nemhauser and Jeff Ullman.
The smoothed analysis result is by René Beier, Heiko Röglin, and
Berthold Vöcking.

23.3.1 The Smoothing Model

One natural neighborhood is to smooth each weight wi uniformly

over an interval of width σ centered at wi . Figure 23.4 illustrates this.
σ σ σ
Within the interval, the density function is uniform, outside of the
interval, it is zero. The profits are not perturbed. w1 w2
0 = w0
We choose a slightly more general model already used earlier:
The profits are fixed (chosen by the adversary), while the weights Figure 23.4: Distributions for three
are chosen randomly and independently. The weight wi is sampled weights plotted into the same
from a distribution f i : [0, 1] → [0, 1/σ ], where the distribution diagram.
can be different for different i and is chosen by the adversary. The
important thing here is that the distribution is bounded by σ1 and
our complexity will polynomially depend on this value. Note that
we made a simplifying assumption that weights are in [0, 1] to make
our lives easier although it can be avoided. We also assume that it
never happens that two solutions get exactly the same weight. This
is justified since this is an event with zero probability, so it is almost
surely not the case.
290 the knapsack problem

23.3.2 The Nemhauser–Ullman algorithm

The Nemhauser-Ullman algorithm for the knapsack problem com-
putes the Pareto curve and returns the best solution from the curve.
The Pareto curve for the knapsack problem can defined as follows.

Definition 23.5. Given two knapsack solutions x1 , x2 ∈ {0, 1}n we

say that “x1 is dominated by x2 ” if w⊺ x1 > w⊺ x2 and p⊺ x1 ≤ p⊺ x2 . The
Pareto curve is defined as the set of all non-dominated solutions (from
a particular set of solutions). We also call the points on the curve
Pareto optimal.

The definition is visualized in Figure 23.5. Note that the optimal

solution is always on the Pareto curve. Moreover, if P( j) is the collec-
tion of Pareto optimal solutions among the subinstance consisting of
just the first j items, then

sum of profits p T x

sum of profits p T x
P( j + 1) ⊆ P( j) ∪ {S ∪ { j + 1} | S ∈ P( j)} .
| {z }
=:A j

The above containment implies that P( j + 1) can be computed from

sum of weights w T x sum of weights w T x
P( j). In other words, a dominated solution by P( j) will still be domi-
nated in P( j + 1), so it can be safely forgotten.
Figure 23.5: Left side: A point set
If we keep the elements of P( j) in sorted order with regard to the and its Pareto curve. The blue
weights, then P( j + 1) can be computed in linear time by merging points form the Pareto curve. They
them together. Note that P( j) and A j are naturally constructed in belong to the curve because the
increasing-weight order. Then we merge them in parallel (using the areas indicated by the black lines
technique from merge-sort) and eliminate dominated points on the does not contain points. For the
third point on the curve, this area
fly. The result is P( j + 1) with points stored in increasing-weight
is specifically highlighted. Right
order. This technique leads to the following Lemma. side: Worst Case.
Lemma 23.6. The Nemhauser-Ullman algorithm can be implemented to
have a running time of O(∑in=1 | P( j)|) = O(n · max j∈[n] E| P( j)|) in
expectation.

Note that for this analysis we no longer care about the size of
the knapsack g. The remainder of the section will be focused on
bounding E[| P( j)|].

23.3.3 Bounding Expected Size of Pareto Curve

In the worst-case it is possible that all 2n solutions are Pareto optimal.
(Exercise: put wi = 2i , pi = 2i ) In this case, the Nemhauser-Ullman
algorithm will take exponential time. However, we will see that the
expected size of P( j) is polynomial for all j ∈ [n] if the instance is
σ-smoothed.
Fix any j ∈ [n], and let P := P( j). The proof idea is to partition the
interval [0, ∞] of weights into stripes of decreasing width ε := 1/k
 smoothed analysis of algorithms 291

for an integer k and then to count the number of Pareto optimal

solutions in each stripe. See Figure 23.6. There is always an ε > 0 that
is small enough such that each stripe contains at most one point from
P (since we assumed no two solutions have the same weight). Denote
by P ∩ [ a, b] the set of all Pareto optimal solutions that have weight in ε
the range [ a, b]. Thus,

sum of the profits

∞  
ℓ ℓ + 1
| P| = 1 + lim ∑ 1 P ∩ , ̸= ∅ .
k →∞ ℓ=0 k k

We want to restrict the number of terms in the sum. Since the

knapsack has size g, we can ignore all solutions that weight more
than g. However, g might still be large. By our simplification, we sum of the weights
know that all weights are at most 1. Thus, no solution can weight
Figure 23.6: Dividing into stripes of
more than n and it is thus sufficient to consider the interval [0, n]. width ε = 1/k.
The kn stripes at (0, 1/k ], (1/k, 2/k], . . . , (n(k − 1)/k, nk/k] fit into this
interval. We thus get that

nk  
ℓ ℓ + 1
E[| P|] ≤ 1 + lim ∑
k →∞ ℓ=0
Pr P ∩
k
,
k
̸ = ∅ (23.2)

Now we have to bound the probability that there is a point from P in

the interval ( kℓ , ℓ+k 1 ]. The following Lemma establishes this.

Lemma 23.7. For any t ≥ 0 it holds that Pr ( P ∩ (t, t + ε] ̸= ∅) ≤ nε

σ.

Proof. Define x R to be the leftmost point right of t that is on the

Pareto curve.

arg min ⊺ ⊺
x ∈ P { p x | w x > t } if the set is nonempty
x R :=
⊥ else

We define ∆ to be the distance between t and x R :


 w⊺ x R − t x R ̸ = ⊥
∆ :=
∞ otherwise

(See Figure 23.7 for a visualization.) Clearly:


P ∩ t, t + ε ̸= ∅ ⇐⇒ ∆ ∈ (0, ε]


The rest of the proof shows that Pr ∆ ∈ (0, ε] is small.
It is difficult to directly argue about ∆, so we use a set of auxiliary
random variables which make the proof clean. For any item i ∈ [n]
we define several random variables: let xUL,−i be the most profitable
(highest) point left of t without the item i; and let x ∗,+i be the left-
most (least weight) point that is higher than xUL,−i and contains i.
292 the knapsack problem

Formally: ε

sum of the profits

xUL,−i := arg max { p⊺ x | w⊺ x ≤ t, xi = 0}
x ∈2[ n ]
∗,+ i xR
x := arg min {w⊺ x | p⊺ x ≥ p⊺ xUL,−i , xi = 1} xUL
x ∈2[ n ]

The reason for defining these variables is that (i) it is easy to

bound the probability that x ∗,+i has weight within an interval (t, t +
sum of the weights
ε] and (ii) we will later show that x R = x ∗,+i for some i. With this ε
reason in mind we define ∆i = x ∗,+i − t (∞ if undefined).
∆

sum of the profits

xR
  xUL
Subclaim For any item i ∈ [n] it holds that Pr w ⊺ x ∗,+ i ∈ (t, t + ε] ≤
 i 
σ . In particular, Pr ∆ ∈ (0, ε ] ≤ σ .
ε ε

Proof. Assume we fixed all weights except for i. Then xUL,−i is com-
pletely determined. We remind the reader that x ∗,+i is defined as the sum of the weights
leftmost point that contains item i and is higher than xUL,−i .
Now we turn our attention to the “precursor” of x ∗,+i without the Figure 23.7: Illustration of the definition
of xUL and x R . ∆ is only plotted in the
item i, namely the item x ∗,+i − ei where ei is the ith basis vector. The right figure.
claim is that this point is completely determined when we fixed all
weights except i. Name that point y (formal definition of this point
will be given later). The point y is exactly the one that is leftmost
with the condition that yi = 0 and p⊺ y + pi ≥ p⊺ xUL,−i (by definition
of x ∗,+i ). Note that the order of y’s does not change when adding wi .
In particular, if y1 was left of y2 (had smaller weight), then adding the
(currently undetermined) wi will not change their ordering.
More formally, let y := arg miny∈2[n] {w⊺ y | p⊺ y + pi ≥ p⊺ xUL,−i , yi =
0} (we drop the index i from y for clarity). In other words, it is the
leftmost solution without i higher than xUL,−i when we add the profit
of i to it. It holds that w⊺ x ∗,+i = w⊺ y + wi . Therefore,
h i
Pr w⊺ x ∗,+i ∈ (t, t + ε] = Pr [w⊺ y + wi ∈ (t, t + ε]]

= Pr [wi ∈ (t − w⊺ y, t + ε − w⊺ y]]
ε
≤
σ

Subclaim There exists i ∈ [n] s.t. x R = x ∗,+i . In particular, ∃i s.t.

∆ = ∆i .

Proof. Let xUL be the most profitable (highest) point left of t. For-
mally:

xUL := arg max{ p⊺ x | w⊺ x ≤ t}

x∈P
 smoothed analysis of algorithms 293

Since the zero vector is always Pareto optimal there is always at least
one point left of t, so xUL is well-defined. There must be an item i s.t.
that is contained in x R , but is not in xUL . Formally, pick and fix any
i ∈ [n] s.t. xiR = 1, xiUL = 0. It is clear that such i must exist since
otherwise xUL would have higher weight than x R . Also, the height of
x R higher than xUL since they are both on the Pareto curve.
Clearly (for this i) it holds that xUL = xUL,−i . Assume for the sake
of contradiction that x ∗,+i is not x R . Then:

• x ∗,+i must be left of x R , otherwise we would have x ∗,+i = x R and

be done.

• x ∗,+i must be higher than xUL by definition.

• x ∗,+i cannot be higher than x R , otherwise x R would not be on the

Pareto curve (since it’s left of it). So assume it’s below x R .

• x ∗,+i cannot be left of t, otherwise we would pick that point to be

xUL (since it’s higher than xUL ).

• The only remaining spot for x ∗,+i is right of t and left of x R , but
that contradicts the choice of x R as the leftmost point right of t.

Hence we conclude that x R = x ∗,+i for our choice of i, which con-

cludes the proof of the subclaim.

Combining the above Subclaims, we get

n h i
Pr[∆ ∈ (0, ε]] ≤ ∑ Pr ∆ i
∈ ( 0, ε ]
i =1
ε
=n .
σ
This is equivalent to the statement of the Lemma, hence we are done.

Using the above analysis, we conclude with a smoothness theorem.

Theorem 23.8. For σ-smoothed instances, the expected size of P is bounded

by n2 /σ for all j ∈ [n]. In particular, the Nemhauser-Ullman algorithm
for the knapsack problem has a smoothed complexity of O(n3 /σ ) for the
smoothness model described in Subsection 23.3.1.

Proof. By Inequality (23.2), Lemma 23.6 and Lemma 23.7, we con-

clude that
 

ℓ , ℓ+1 ̸ = ∅
E[| P|] ≤ 1 + limk→∞ ∑nk
ℓ=0 Pr P ∩ k k

n 1k n2
≤ 1 + limk→∞ nk · σ = σ.
294 the knapsack problem

23.3.4 More general result

The result can be extended beyond the knapsack problem. Let Π be a
“combinatorial optimization” problem given as

max p⊺ nx
s.t. Ax ≤ b
x∈S

where S ⊆ {0, 1}n . Observe: the knapsack problem is one such

problem. Beier and Vöcking prove the following theorem.

Theorem 23.9. Problem Π has polynomial smoothed complexity if solving

Π on unitarily encoded instances can be done in polynomial time.

For the knapsack problem, the dynamic programming algorithm

has a running time of O(ng), where g is the size of the knapsack. If
the instance is encoded in binary, then we need O(n log g) bits for the
input and hence this algorithm is not polynomial-time; however if
input is encoded in unary then we use O(ng) to encode the instance,
and the dynamic programming algorithm becomes polynomial-time.
24
Prophets and Secretaries

The prophet and secretary problems are two classes of problems where
online decision-making meets stochasticity: in the first set of prob-
lems the inputs are random variables, whereas in the second one
the inputs are worst-case but revealed to the algorithm (a.k.a. the
decision-maker) in random order. Here we survey some results,
proofs, and techniques, and give some pointers to the rich body of
work developing around them.

24.1 The Prophet Problem

The problem setting: there are n random variables X1 , X2 , . . . , Xn . We

know their distributions up-front, but not their realizations. These
realizations are revealed one-by-one (say in the order 1, 2, . . . , n). We
want to give a strategy (which is a stopping rule) that, upon seeing
the value Xi (and all the values before it) decides either to choose i,
in which case we get reward Xi and the process stops. Or we can
pass, in which case we move on to the next items, and are not allowed
to come back to i ever again. We want to maximize our expected
reward. If
Xmax := max{ X1 , X2 , . . . , Xn },
it is clear that our expected reward cannot exceed E[ Xmax ]. But how
close can we get? If we want to find the best strategy,
In fact, we may be off by a factor of almost two against this yard- and we know the order in which we
are shown these r.v.s, there is dynamic
stick in some cases: suppose X1 = 1 surely, whereas X2 = 1/ε with programming algorithm. (Exercise!)
probability ε, and 0 otherwise. Any strategy either picks 1 or passes
on it, and hence gets expected value 1, whereas E[ Xmax ] = (2 − ε).
Surprisingly, this is the worst case.
Theorem 24.1 (Krengel, Sucheston, and Garling). There is a strategy
with expected reward 1/2E[ Xmax ].
Such a result, that gives a stopping rule whose value is comparable
to the E[ Xmax ] is called a prophet inequality, the idea being that one
296 the prophet problem

can come close to the performance of a prophet who is clairvoyant,

can see the future. The result in Theorem 24.1 was proved by Kren-
gel, Sucheston, and Garling; several proofs have been given since.
Apart from being a useful algorithmic construct, the prophet inequal-
ity naturally fits into work on algorithmic auction design: suppose
you know that n potential are interested in an item with valuations
X1 , . . . , Xn , and you want to sell to one person: how do you make
sure your revenue is close to E[ Xmax ]?
We now give three proofs of this theorem. For the moment, let us
ignore computational considerations, and just talk about the informa-
tion theoretic issues.

24.1.1 The Median Algorithm

This proof is due to Ester Samuel-Cahn.
Let τ be the median of the distribution of Xmax : i.e.,

Pr[ Xmax ≥ τ ] = 1/2.

(For simplicity we assume that there is no point mass at τ, the proof

is easily extended to discrete distributions too.) Now the strategy
is simple: pick the first Xi which exceeds τ. We claim this prove Theo-
rem 24.1.

Proof. Observe that we pick an item with probability exactly 1/2, but
how does the expected reward compare with E[ Xmax ]?

E[ Xmax ] ≤ τ + E[( Xmax − τ )+ ])

 n 
≤ τ + E ∑ ( Xi − τ ) .
+
i =1

And what does the algorithm get?

n ^
ALG ≥ τ · Pr[ Xmax ≥ τ ] + ∑ E[( Xi − τ )+ ] · Pr[ ( X j < τ )]
i =1 j ≤i
n
≥ τ · Pr[ Xmax ≥ τ ] + ∑ E[( Xi − τ )+ ] · Pr[ Xmax < τ ]
i =1

But both these probability values equal half, and hence ALG ≥
1/2 E[ Xmax ].

While a beautiful proof, it is somewhat mysterious, and difficult

to generalize. Indeed, suppose we are allowed to choose up to k However, a recent paper of Shuchi
variables to maximize the sum of their realizations? The above proof Chawla, Nikhil Devanur, and Thodoris
Lykouris gives an extension of Samuel-
seems difficult to generalize, but the following LP-based one will. Cahn’s proof to the multiple item
setting.
 prophets and secretaries 297

24.1.2 The LP-based Algorithm

The second proof is due to Shuchi Chawla, Jason Hartline, David
Malec, and Balu Sivan, and to Saeed Alaei. Define pi as the proba- If you know of an earlier reference for
this proof, please let me know.
bility that element Xi takes on the highest value. Hence ∑i pi = 1.
Moreover, suppose τi is such that Pr[ Xi ≥ τi ] = pi , i.e., the pith per-
centile for Xi . Define

vi ( pi ) := E[ Xi | Xi ≥ τi ]

as the value of Xi conditioned on it lying in the top pith percentile.

Clearly, E[ Xmax ] ≤ ∑i vi ( pi ) · pi . Here’s an algorithm that gets value
1/4 ∑ vi ( pi ) · pi ≥ 1/4 E[ Xmax ]:
i

If we have not chosen an item among 1, . . . , i − 1, when looking at item i, pass

with probability half without even looking at Xi , else accept it if Xi ≥ τi .

Lemma 24.2. The algorithm achieves a value of 1/4E[ Xmax ].

Proof. Say we “reach” item i if we’ve not picked an item before i. The
expected value of the algorithm is
n
ALG ≥ ∑ Pr[reach item i] · 1/2 · Pr[Xi ≥ τi ] · E[Xi | Xi ≥ τi ]
i =1
n
= ∑ Pr[reach item i] · 1/2 · pi · vi ( pi ). (24.1)
i =1

Since we pick each item with probability pi /2, the expected number
of items we choose is half. So, by Markov’s inequality, the probability
we pick no item at all is at least half. Hence, the probability we reach
item i is at least one half too, the above expression is 1/4 ∑i vi ( pi ) · pi
as claimed.

Now to give a better bound, we refine the algorithm above: sup-

pose we denote the probability of reaching item i by ri , and suppose
we reject item i outright with probability 1 − qi . Then (24.1) really
shows that
n
ALG ≥ ∑ r i · q i · p i · v i ( p i ).
i =1
Above, we ensured that qi = ri = 1/2, and hence lost a factor of 1/4.
But if we could ensure that ri · qi = 1/2, we’d get the desired result of
1/2E[ Xmax ]. For the first item r1 = 1 and hence we can set q1 = 1/2.

What about future items? Note that since that

r i +1 = r i (1 − q i · p i ) (24.2)

we can just set qi+1 = 2r1 . A simple induction shows that ri+1 ≥
i +1
1/2—indeed, sum up (24.2) to get ri +1 = r1 − ∑
j≤i pi /2—so that
qi+1 ∈ [0, 1] and is well-defined.
298 the prophet problem

24.1.3 The Computational Aspect

If the distribution of the r.v.s Xi is explicitly given, the proofs above
immediately give us algorithms. However, what if the variables Xi
are given via a black-box that we can only access via samples?
The first proof merely relies on finding the median: in fact, finding
an “approximate median” τb such that Pr[ Xmax < τb] ∈ (1/2 − ε, 1/2 + ε)
gives us expected reward 1/2+ε/2E[ Xmax ]. To do this, sample from
Xmax O(ε−2 log δ−1 ) times (each sample to Xmax requires one sample
to each of the Xi s) and take τb to be the sample median. A Hoeffding
bound shows that τb is an “approximate median” with probability at
least 1 − δ.
For the second proof, there are two ways of making it algorithmic.
Firstly, if the quantities are polynomially bounded, estimate pi and vi
by sampling. Alternatively, solve the convex program
 
max ∑ yi · vi ( yi ) | ∑ yi = 1
i i

and use the yi ’s from its solution in lieu of pi ’s in the algorithm

above.
Do we need such good approximations? Indeed, getting samples
from the distributions may be expensive, so how few samples can
we get away with? A paper of Pablo Azar, Bobby Kleinberg, and
Matt Weinberg shows how to get a constant fraction of E[ Xmax ] via
taking just one sample from each of the Xi s. Let us a give a different
algorithm, by Aviad Rubinstein, Jack Wang, and Matt Weinberg.

24.1.4 The One-Sample Algorithm

For the preprocessing, take one sample from the distributions for
each of X1 , X2 , . . . , Xn . (Call these samples S1 , S2 , . . . , Sn .) Set the
threshold τ to be the largest of these sample values. Now when see-
ing the actual items X1 , X2 , . . . , Xn , pick the first item higher than τ.
We claim this one-sample algorithm gives an expected value at least
1/2 E[ Xmax ].

Proof. As a thought experiment, consider taking two independent

samples from each distribution, then flipping a coin Ci to decide
which is Xi and which is Si . This has the same distribution as origi-
nal process, so we consider this perspective.
Now consider all these 2n values together, in sorted order: call
these W1 ≥ W2 ≥ . . . ≥ W2n . We say Wj has index i if it is drawn
from the ith distribution, and hence equal to Xi or Si . Let j∗ be the
smallest position where Wi , Wj∗ +1 have the same index for some
i ≤ j∗ . Observe: the coins Ci for the indices corresponding to the first
 prophets and secretaries 299

j∗ positions are independent, and the coin for the position j∗ + 1 is

the same as one of the previous ones. We claim that

Wi Wj∗ +1
E[ Xmax ] = ∑∗ 2i
+
2j
∗ .
i≤ j

Indeed, Xmax = Wi if all the previous Wi′ s belong to the sample (i.e.,
they are S’s and not X’s), but Wi belongs to the actual values (it is an
X). Moreover, if all the previous values are Ss, then Wj∗ +1 would be
an X and hence the maximum, by our choice of j∗ .
What about the algorithm? If W1 is a sample (i.e., an S-value) then
we don’t get any value. Else if W1 , . . . , Wi are all X values, and Wi+1
is a sample (S value) then we get value at least Wi . If i < j∗ , this hap-
pens with probability 2i1+1 since all the i + 1 coins are independent.
Else if i = j∗ , the probability is 21i = 1j∗ . Hence
2

W Wj∗ W Wj∗ Wj∗ +1

Alg ≥ ∑∗ 2i+i1 + 2j
∗ ≥ ∑∗ 2i+i1 + 2j∗ +1 + 2j
∗ +1 .
i< j i< j

But this is at least half of E[ Xmax ], which proves the theorem.

24.1.5 Extensions: Picking Multiple Items

What about the case where we are allowed to choose k variables
from among the n? Proof #2 generalizes quite seamlessly. If pi is the
probability that Xi is among the top k values, we now have:

∑i pi = k. (24.3)

The “upper bound” on our quantity of interest remains essentially

unchanged:

E[sum of top k r.v.s] ≤ ∑i vi ( pi ) · pi . (24.4)

What about an algorithm to get value 1/4 of the value in (24.4)? The
same as above: reject each item outright with probability 1/2, else
pick i if Xi ≥ τi . Proof #2 goes through unchanged.
For this case of picking multiple items, we can do much better:
√
a result of Alaei shows that one can get within (1 − 1/ k+3) of the
value in (24.4)—for k = 1, this matches the factor
q /2 we showed
1
log k
above. One can, however, get a factor of (1 − O( k )) using a
simple concentration bound.

Proof. Suppose we reduce the rejection probability to δ. Then the

probability that we reach some item i without having picked k items
already is lower-bounded by the probability that we pick at most k
elements in the entire process. Since we reject items with probability
300 the prophet problem

δ, the expected number of elements we pick is (1 − δ)k. Hence, the

probability that we pick less than k items is at least 1 − e−Θ(δ k) , by a
2

Hoeffding boundq for sums of independent random variables. Now

log k
setting δ = O( k ) ensures that the probability of reaching each
item is at least (1 − 1k ), and an argument similar to that in Proof #2
shows that

ALG ≥ ∑in=1 Pr[reach item i ] · Pr[not reject item i ] · Pr[ Xi ≥ τi ] · E[ Xi | Xi ≥ τi ]

q
log k
= ∑in=1 (1 − 1/k) · (1 − O( k )) · pi · vi ( pi ),

q
log k
which gives the claimed bound of (1 − O( k )).

24.1.6 Extensions: Matroid Constraints

Suppose there is a matroid structure M with ground set [n], and

the set of random variables we choose must be independent in this
matroid M. The value of the set is the sum of the values of items Recall that a matroid M = (U, F ) is a
within it. (Hence, the case of at most k items above corresponds to set U is a collection of subsets F ⊆ 2U
that is closed under taking subsets,
the uniform matroid of rank k.) The goal is to make the expected such that if A, B ∈ F and | A| < | B|
value of the set picked by the algorithm close to the expected value of then there exists b ∈ B \ A such that
A ∪ {b} ∈ F . Sets in F are called
the max-weight independent set. independent sets.
Bobby Kleinberg and Matt Weinberg give an algorithm to picks
an independent set whose expected value is at least half the value
of the max-weight independent set, thereby extending the original
single-item prophet inequality seamlessly to all matroids. While their
original proof uses a combinatorial idea, a LP-based proof was subse-
quently given by Moran Feldman, Ola Svensson, and Rico Zenklusen.
The idea is again clever and conceptually clean: find a solution y to
the convex program

∑i vi ( yi ) · yi .
y ∈ the matroid polytope for M

Now given a fractional point y in the matroid polytope, how to get

an integer point (i.e., an independent set). For this they give an ap-
proach called an “online contention resolution” scheme that ensures
that any item i is picked with probability at least Ω(yi ), much like in
the single-item and k-item cases.
There are many other extensions to prophet inequalities: people
have studied more general constraint sets, submodular valuations
instead of just additive valuations, what if the order of items is not
known, what if we are allowed to choose the order, etc. See papers on
arXiv, or in the latest conferences for much more.
 prophets and secretaries 301

24.1.7 Exercises
1. Give a dynamic programming algorithm for the best strategy when we know the
order in which r.v.s are revealed to us. (Footnote 1). Extend this to the case where
you can pick k items.
Open problem: is this “best strategy” problem computationally hard when we are
given a general matroid constraint? Even a laminar matroid or graphical matroid?
2. If we can choose the order in which we see the items, show that we can get ex-
pected value ≥ (1 − 1/e)E[ Xmax ]. (Hint: use proof #2, but consider the elements in
decreasing order of vi ( pi ).)
Open problem: can you beat (1 − 1/e)E[ Xmax ]? A recent paper of Abolhassani et al.
does so for i.i.d. Xi s.

24.2 Secretary Problems

The problem setting: there are n items, each having some intrinsic
non-negative value. For simplicity, assume the values are distinct, but
we know nothing about their ranges. We know n, and nothing else.
The items are presented to us one-by-one. Upon seeing an item, we
can either pick it (in which case the process ends) or we can pass (but
then this item is rejected and we cannot ever pick it again). The goal
is to maximize the probability of picking the item with the largest
value vmax .
If an adversary chooses the order in which the items are presented,
every deterministic strategy must fail. Suppose there are just two
items, the first one with value 1. If the algorithm picks it, the adver-
sary can send a second item with value 2, else it sends one with value
1/2. Randomizing our algorithm can help, but we cannot do much

better than 1/n.

So the secretary problem asks: what if the items are presented in uni-
formly random order? For this setting, it seems somewhat surprising
at first glance that one can pick the best item with probability at least
a constant (knowing nothing other than n, and the promise of a uni-
formly random order). Indeed, here a simple algorithm and proof
showing a probability of 1/4:
Ignore the first n/2 items, and then pick the next item that is better
than all the ones seen so far.

Note that this algorithm succeeds if the best item is in the second
half of the items (which happens w.p. 1/2) and the second-best item
is in the first half (which, conditioned on the above event, happens
w.p. ≥ 1/2). Hence 1/4. It turns out that rejecting the first half of the
items is not optimal, and there are other cases where the algorithm
succeeds that this simple analysis does not account for, so let’s be
more careful. Consider the following 37%-algorithm:
Ignore the first n/e items, and then pick the next item that is better
than all the ones seen so far.
302 secretary problems

Theorem 24.3. As n → ∞, the 37%-algorithm picks the highest number

with probability at least 1/e. Hence, it gets expected value at least vmax /e.
Moreover, n/e is the optimal choice of m among all wait-and-pick algo-
rithms.

Proof. Call a number a prefix-maximum if it is the largest among the

numbers revealed before it. Notice being the maximum is a property
of just the set of numbers, whereas being a prefix-maximum is a
property of the random sequence and the current position. If we
pick the first prefix-maximum after rejecting the first m numbers, the
probability we pick the maximum is
n
∑ Pr[vt is max] · Pr[max among first t − 1 numbers falls in first m positions]
t = m +1
n

(⋆) 1 m m
= ∑ n
·
t − 1
=
n
Hn−1 − Hm−1 ,
t = m +1

where Hk = 1 + 12 + 13 + . . . + 1k is the kth harmonic number. The

equality (⋆) uses the uniform random order. Now using the approxi-
mation Hk ≈ ln k + 0.57 for large k, we get the probability of picking
n −1
the maximum is about m n ln m−1 when m, n are large. This quantity
has a maximum value of 1/e if we choose m = n/e.

Next we show we can replace any strategy (in a comparison-based

model) with a wait-and-pick strategy without decreasing the proba-
bility of picking the maximum.

Theorem 24.4. The strategy that maximizes the probability of picking the
highest number can be assumed to be a wait-and-pick strategy.

Proof. Think of yourself as a player trying to maximize the probabil-

ity of picking the maximum number. Clearly, you should reject the
next number vi if it is not prefix-maximum. Otherwise, you should
pick vi only if it is prefix-maximum and the probability of vi being
the maximum is more than the probability of you picking the maxi-
mum in the remaining sequence. Let us calculate these probabilities.
We use Pmax to abbreviate “prefix-maximum”. For position i ∈
{1, . . . , n}, define

(⋆) Pr[vi is max] (⋆⋆) 1/n i

f (i ) = Pr[vi is max | vi is Pmax] = = = ,
Pr[vi is Pmax] 1/i n

where equality (⋆) uses that the maximum is also a prefix-maximum,

and (⋆⋆) uses the uniform random ordering. Note that f (i ) increases
with i.
Now consider a problem where the numbers are again being re-
vealed in a random order but we must reject the first i numbers. The
 prophets and secretaries 303

goal is to still maximize the probability of picking the highest of the n

numbers. Let g(i ) denote the probability that the optimal strategy for
this problem picks the global maximum.
The function g(i ) must be a non-increasing function of i, else we
could just ignore the (i + 1)st number and set g(i ) to mimic the strat-
egy for g(i + 1). Moreover, f (i ) is increasing. So from the discussion
above, you should not pick a prefix-maximum number at any posi-
tion i where f (i ) < g(i ) since you can do better on the suffix. More-
over, when f (i ) ≥ g(i ), you should pick vi if it is prefix-maximum,
since it is worse to wait. Therefore, the approach of waiting until
f becomes greater than g and thereafter picking the first prefix-
maximum is an optimal strategy.

In keeping with the theme of this chapter, we now give an alter-

nate proof that uses a convex-programming view of the process. We
will write down an LP that captures some properties of any feasible
solution, optimize this LP and show a strategy whose success proba-
bility is comparable to the objective of this LP! The advantage of this
approach is that it then extends to adding other constraints to the
problem.

Proof. (Due to Niv Buchbinder, Kamal Jain, and Mohit Singh.) Let us
fix an optimal strategy. By the first proof above, we know what it is,
but let us ignore that for the time being. Let us just assume w.l.o.g.
that it does not pick any item that is not the best so far (since such an
item cannot be the global best).
Let pi be the probability that this strategy picks an item at posi-
tion i. Let qi be the probability that we pick an item at position i,
p
conditioned on it being the best so far. So qi = 1/ii = i · pi .
Now, the probability of picking the best item is

∑ Pr[ith position is global best and we pick it ]

i
1 i
= ∑ Pr[ith position is global best ] · qi = ∑ qi = ∑ pi . (24.5)
i i
n i
n

What are the constraints? Clearly pi ∈ [0, 1]. But also

pi = Pr[ pick item i | i best so far] · Pr[i best so far]

≤ Pr[ did not pick 1, . . . , i − 1 | i best so far] · (1/i ) (24.6)

But not picking the first i − 1 items is independent of i being the best
so far, so we get

1 
pi ≤ 1 − ∑ pj .
i j <i
304 secretary problems

Hence, the success probability of any strategy (and hence of the

optimal strategy) is upper-bounded by the following LP in variables
pi :

i
max ∑ ·p
i
n i
i · pi ≤ 1 − ∑ p j
j <i

pi ∈ [0, 1].

Now it can be checked that the solution pi = 0 for i ≤ τ and

pi nτ ( i−1 1 − 1i ) for τ ≤ i ≤ n is a feasible solution, where τ is de-
fined by the smallest value such that Hn−1 − Hτ −1 ≤ 1. (By duality,
we can also show it is optimal!)
Finally we can get a stopping strategy whose success probability
matches that of the LP. Indeed, solve the LP. Now, for the ith position
if we’ve not picked an item already and if this item is the best so far,
ip
pick it with probability 1−∑ i p . By the LP constraint, this probabil-
j <i j
ity ∈ [0, 1]. Moreover, removing the conditioning shows we pick an
item at location i with probability pi , and a calculation similar to the
one above shows that our algorithm’s success probability is ∑i ipi /n,
the same as the LP.

24.2.1 Extension: Game-Theoretic Issues

Note that in the optimal strategy, we don’t pick any items in the first
n/e timesteps, and then we pick items with quite varying probabili-
ties. If the items are people interviewing for a job, this gives them an
incentive to not come early in the order. Suppose we insist that for
each position i, the probability of picking the item at position i is the
same. What can we do then?
Let’s fix any such strategy, and write an LP capturing the suc-
cess probabilities of this strategy with uniformity condition as a
constraint. Suppose p ≤ 1/n is this uniform probability (over the
randomness of the input sequence). Again, let qi be the probability
of picking an item at position i, conditioned on it being the best so
far. Note that we may pick items even if they are not the best so far,
just to satisfy the uniformity condition; hence instead of qi = i · p as
before, we have
qi ≤ ip.

Moreover, by the same argument as (24.6), we know that

qi ≤ 1 − (i − 1) p.

And the strategy’s success probability is again ∑ qi /n using (24.5). So

 prophets and secretaries 305

we can now solve the LP

1
max ∑ ·q
i
n i
q i ≤ 1 − ( i − 1) · p
qi ≤ i · p
qi ∈ [0, 1], p ≥ 0

Now the Buchbinder, Jain, and Singh paper shows the optimal value
√
of this LP is at least 1 − 1/ 2 ≈ 0.29; they also give a slightly more
involved algorithm that achieves this success probability.

24.2.2 Extension: Multiple Items

Now back to having no restrictions on the item values. Suppose we
want to pick k items, and want to maximize the expected sum of
these k values. Suppose the set of the k largest values is S⋆ ⊆ [n], and
their total value is V ⋆ = ∑i∈S vi . It is easy to get an algorithm with
expected value Ω(V ⋆ ). E.g., split the n items into k groups of n/k
items, and run the single-item algorithm separately on each of these.
(Why?) Or ignore the first half of the elements, look at the value v̂ of
the (1 − ε)k/2th highest value item in this set, and pick all items in
the second half with values greater than v̂. And indeed, ignoring half
the items must lose a constant factor in expectation.
But here’s an algorithm that gives value V ⋆ (1 − δ) where δ → 0
as k → ∞. We will set δ = O(k−1/3 log k ) and ε = δ/2. Ignore the
first δn items. (We expect δk ≈ k2/3 items from S⋆ fall in this ignored
set.) Now look at the value v̂ of the (1 − ε)δkth -highest valued item
in this ignored set, and pick the first (at most) k elements with values
greater than v̂ along the remaining (1 − δ)n elements.
Why is this algorithm good? There are two failure modes: (i) if v′ = mini∈S⋆ vi
be the lowest value item we care about, then we don’t want v̂ ≤ v′ else we may
pick low valued items, and (ii) we want the number of items from S⋆ in the last
(1 − δ)n and greater than v̂ to be close to k.
Let’s sketch why both bad events happen rarely. For event (i) to happen, fewer
than (1 − ε)δk items from S⋆ fall in the first δn locations: i.e., their number is
less than (1 − ε) times its expectation, which has probability exp(−ε2 δk ) =
1/ poly(k ) by a Hoeffding bound. For event (ii) to happen, more than (1 − ε)δk
of the top (1 − δ)k items from S⋆ fall among the ignored items. This means their
number exceeds (1 + O(ε)) times its expectation, which again has probability
exp(−ε2 δk) = 1/ poly(k).
An aside: the standard concentration bounds we know are for sums of i.i.d.
r.v.s whereas the random order model causes correlations. The easiest way to
handle that is to ignore not the first δn items but a random number of items
∼ Bin(n, δ). Then each item has probability δ of being ignored, independent of
others.

Is this tradeoff optimal? No. Kleinberg showed that one can get
expected value V ⋆ (1 − O(k−1/2 )), and this is asymptotically optimal.
306 secretary problems

In fact, one can extend this even further: a set of vectors a1 , a2 , . . . , an ∈

[0, 1]m is fixed, along with values v1 , v2 , . . . , vn . These are initially un-
known. Now they are revealed one-by-one to the algorithm in a
uniformly random order. The algorithm, on seeing a vector and its
value must decide to pick it or irrevocably reject it. It can pick as
many vectors as it wants, subject to their sum being at most k in each
coordinate; the goal is to maximize the expected value of the picked
vectors. The k-secretary case is the 1-dimensional case when each
ai = (1). Indeed, this is the problem of solving a packing linear pro-
gram online, where the columns arrive in random order. A series of
works have extended the k-secretary case to this online packing LP
p
problem, getting values which are (1 − O( (log m)/k)) times the
optimal value of the LP.

24.2.3 Extension: Matroids

One of the most tantalizing generalizations of the secretary prob-
lem is to matroids. Suppose the n elements form the ground set of
a matroid, and the elements we pick must form an independent
set in this matroid. Babioff, Immorlica, and Kleinberg asked: if the
max-weight independent set has value V ∗ , can we get Ω(V ∗ ) using
an online algorithm? The current best algorithms, due to Lachish,
and to Feldman, Svensson, and Zenklusen, achieve expected value
Ω(V ∗ / log log k), where k is the rank of the matroid. Can we im-
prove this further, say to a constant? A constant factor is known for
many classes of matroids, like graphical matroids, laminar matroids,
transversal matroids, and gammoids.

24.2.4 Other Random Arrival Models

One can consider other models for items arriving online: say a set of
n items (and their values) is fixed by an adversary, and each timestep
we see one of these items sampled uniformly with replacement. (The
random order model is same, but without replacement.) This model,
called the i.i.d. model, has been studied extensively—results in this
model are often easier than in the random order model (due to lack
of correlations). See, e.g., references in a monograph by Aranyak
Mehta.
Do we need the order of items to be uniformly random, or would
weaker assumptions suffice? Kesselhiem, Kleinberg, and Niazadeh
consider this question in a very nice paper and show that much less
independence is enough for many of these results to hold 1 . 1

In general the random-order model is a clean way of modeling the

fact that an online stream of data may not be adversarially ordered.
Many papers in online algorithms have used this model to give better
 prophets and secretaries 307

results than in the worst-case model: some of my favorite ones are

paper of Meyerson on facility location, and this paper of Bahmani,
Chowdhury, and Goel on computing PageRank incrementally.
Again, see online for many many papers related to the secretary
problem: numerous models, different constraints on what sets of
items you can pick, and how you measure the quality of the picked
set. It’s a very clean model, and can be used in many different set-
tings.

Exercises
1. Give an algorithm for general matroids that finds an independent set with expected
value at least an O(1/(log k))-fraction of the max-value independent set.
2. Improve the above result to O(1)-fraction for graphic matroids.