CS 583: Approximation Algorithms Lecture date: February 3, 2016

Instructor: Chandra Chekuri Scribe: CC

Notes updated from Spring 2011.

In the previous lecture we discussed the Knapsack problem. In this lecture we discuss other
packing and independent set problems. We first discuss an abstract model of packing problems.
Let N be a finite ground set. A collection of I ⊂ 2N of subsets of N is said to be down closed if
the following property is true: A ∈ I implies that for all B ⊂ A, B ∈ I. A down closed collection
is also often called and independence system. The sets in I are called independent sets. Given an
independence family (N, I) and a non-negative weight function w : N → R+ the maximum weight
independent set problem is to find maxS∈I w(S). That is, find an independent set in I of maximum
weight. Often we may be interested in the setting where all weights are 1 in which case we wish to
find the maximum cardinality independent set. We discuss some canonical examples.
Example: Independent sets in graphs: Given a graph G = (V, E)
I = {S ⊆ V | there are no edges between nodes in S}. Here the ground set is V .

Example: Matchings in graphs: Given a graph G = (V, E) let I = {M ⊆ E | M is a matching in G}.

Here the ground set is E.

Example: Matroids: A matroid M = (N, I) is defined as a system where I is down closed and
in addition satisfies the following key property: if A, B ∈ I and |B| > |A| then there is an element
e ∈ B \ A such that A ∪ {e} ∈ I. There are many examples of matroids. We will not go into details
here.

Example: Intersections of independence systems: given some k independence systems on the

same ground set (N, I1 ), (N, I2 ), . . . , (N, Ik ) the system defined by (N, I1 ∩ I2 . . . ∩ Ik ) is also an
independence system. Well-known examples include intersections of matroids.

1 Maximum Independent Set Problem in Graphs

A basic graph optimization problem with many applications is the maximum (weighted) independent
set problem (MIS) in graphs.

Definition 1 Given an undirected graph G = (V, E) a subset of nodes S ⊆ V is an independent

set (stable set) iff there is no edge in E between any two nodes in S. A subset of nodes S is a clique
if every pair of nodes in S have an edge between them in G.

The MIS problem is the following: given a graph G = (V, E) find an independent set in G
of maximum cardinality. In the weighted case, each node v ∈ V has an associated non-negative
weight w(v) and the goal is to find a maximum weight independent set. This problem is NP-Hard
and it is natural to ask for approximation algorithms. Unfortunately, as the famous theorem below
shows, the problem is extremely hard to approximate.
 1
Theorem 1 (Håstad [1]) Unless P = N P there is no n1− -approximation for MIS for any fixed
 > 0 where n is the number of nodes in the given graph.

Remark: The maximum clique problem is to find the maximum cardinality clique in a given graph.
It is approximation-equivalent to the MIS problem; simply complement the graph.
The theorem basically says the following: there are a class of graphs in which the maximum
independent set size is either less than nδ or greater than n1−δ and it is NP-Complete to decide
whether a given graph falls into the former category or the latter.
The lower bound result suggests that one should focus on special cases, and several interesting
positive results are known. First, we consider a simple greedy algorithm for the unweighted problem.
Greedy(G):
S←∅
While G is not empty do
Let v be a node of minimum degree in G
S ← S ∪ {v}
Remove v and its neighbors from G
end while
Output S

Theorem 2 Greedy outputs an independent set S such that |S| ≥ n/(∆ + 1) where ∆ is the
maximum degree of any node in the graph. Moreover |S| ≥ α(G)/∆ where α(G) is the cardinality
of the largest independent set. Thus Greedy is a 1/∆ approximation.
Proof: We upper bound the number of nodes in V \ S as follows. A node u is in V \ S because
it is removed as a neighbor of some node v ∈ S when Greedy added v to S. Charge u to v. A
node v ∈ S can be charged at most ∆ times since it has at most ∆ neighbors. Hence we have that
|V \ S| ≤ ∆|S|. Since every node is either in S or V \ S we have |S| + |V \ S| = n and therefore
(∆ + 1)|S| ≥ n which implies that |S| ≥ n/(∆ + 1).
We now argue that |S| ≥ α(G)/∆. Let S ∗ be a largest independent set in G. As in the above
proof we can charge each node v in S ∗ \ S to a node u ∈ S \ S ∗ which is a neighbor of v. The
number of nodes charged to a node u ∈ S \ S ∗ is at most ∆. Thus |S ∗ \ S| ≤ ∆|S ∗ \ S|).
2
n
Exercise: Show that Greedy outputs an independent set of size at least 2(d+1) where d is the
average degree of G.

Remark: The well-known Turan’s theorem shows via a clever argument that there is always an
n
independent set of size (d+1) where d is the average degree of G.

log d
Remark: For the case of unweighted graphs one can obtain an approximation ratio of Ω( d log log d )
where d is the average degree. Surprisingly, under a complexity theory conjecture called the Unique-
2
Games conjecture it is known to be NP-Hard to approximate MIS to within a factor of O( log∆ ∆ )
in graphs with maximum degree ∆ when ∆ is sufficiently large.

Exercise: Consider the weigthed MIS problem on graphs of maximum degree ∆. Alter Greedy
1
to sort the nodes in non-increasing order of the weight and show that it gives a ∆ -approximation.
Can one obtain an Ω(1/d)-approximation for the weighted case where d is the average degree?
LP Relaxation: One can formulate a simple linear-programming relaxation for the (weighted)
MIS problem where we have a variable x(v) for each node v ∈ V indicating whether v is chosen in
the independent set or not. We have constraints which state that for each edge (u, v) only one of
u or v can be chosen.
X
maximize w(v)x(v)
v∈V
subject to x(u) + x(v) ≤ 1 (u, v) ∈ E
x(v) ∈ [0, 1] v∈V

Although the above is a valid integer programming relaxation of MIS when the variabels are
constrained to be in {0, 1}, it is not a particularly useful formulation for the following simple reason.

Claim 3 For any graph the optimum value of the above LP relaxation is at least w(V )/2. In
particular, for the unweighted case it is at least n/2.

Simply set each x(v) to 1/2!

One can obtain a strengthened formulation below by observing that if S is clique in G then any
independent set can pick at most one node from S.

X
maximize w(v)x(v)
v∈V
X
subject to x(v) ≤ 1 S is a clique in G
v∈S
x(v) ∈ [0, 1] v∈V

The above linear program has an exponential number of variables and it cannot be solved in
polynomial time in general but for some special cases of interest the above linear program can
indeed be solved (or approximately solved) in polynomial time and leads to either exact algorithms
or good approximation bounds.

Approximability of Vertex Cover and MIS: The following is a basic fact and is easy to
prove.

Proposition 4 In any graph G = (V, E), S is a vertex cover in G if and only if V \ S is an

independent set in G. Thus α(G) + β(G) = |V | where α(G) is the size of a maximum independent
set in G and β(G) is the size of a minimum vertex cover in G.

The above shows that if one of Vertex Cover or MIS is NP-Hard then the other is as well.
We have seen that Vertex Cover admits a 2-approximation while MIS admits no constant factor
approximation. It is useful to see why a 2-approximation for Vertex Cover does not give any
useful information for MIS even though α(G) + β(G) = |V |. Suppose S ∗ is an optimal vertex
cover and has size ≥ |V |/2. Then a 2-approximation algorithm is only guaranteed to give a vertex
cover of size |V |! Hence one does not obtain a non-trivial independent set by complementing the
approximate vertex cover.
Some special cases of MIS: We mention some special cases of MIS that have been considered
in the literature, this is by no means an exhaustive list.

• Interval graphs; these are intersection graphs of intervals on a line. An exact algorithm can
be obtained via dynamic programming and one can solve more general versions via linear
programming methods.

• Note that a maximum (weight) matching in a graph G can be viewed as a maximum (weight)
independent set in the line-graph of G and can be solved exactly in polynomial time. This
has been extended to what are known as claw-free graphs.

• Planar graphs and generalizations to bounded-genus graphs, and graphs that exclude a fixed
minor. For such graphs one can obtain a PTAS due to ideas originally from Brenda Baker.

• Geometric intersection graphs. For example, given n disks on the plane find a maximum
number of disks that do not overlap. One could consider other (convex) shapes such as axis
parallel rectangles, line segments, pseudo-disks etc. A number of results are known. For
example a PTAS is known for disks in the plane. An Ω( log1 n )-approximation for axis-parallel
1
rectangles in the plane when the rectangles are weighted and an Ω( log log n )-approximation
for the unweighted case.

2 The efficacy of the Greedy algorithm

The Greedy algorithm can be defined easily for an arbitrary independence system. It iteratively
adds the best element to the current independent set while maintaining feasibility. Note that the
implementation of the algorithm requires having an oracle to find the best element to add to a
current independent set S.

Greedy((N, I)):
S←∅
While (TRUE)
Let A ← {e ∈ N \ S | S + e ∈ I}
If A = ∅ break
e ← argmaxe∈A w(e)
S ← S ∪ {e}
endWhile
Output S

Exercise: Prove that the Greedy algorithm gives a 1/2-approximation for the maximum weight
matching problem in a general graph. Also prove that this bound is tight even in bipartite graphs.
Note that max weight matching can be solved exactly in polynomial time.

Remark: It is well-known that the Greedy algorithm gives an optimum solution when (N, I) is a
matroid. Kruskal’s algorithm for min/max weight spanning tree is a special case of this fact.
It is easy to see that Greedy does poorly for MIS problem in general graphs. A natural question
is what properties of I enable some reasonable performance guarantee for Greedy. A very general
result in this context has been established due to Jenkyn’s generalizing several previous results. In
order to state the result we set up some notation. Given an independence system (N, I) we say
that a set A ∈ I is a base if it is a maximal independent set. It is well-known that in a matroid M
all bases have the same cardinality. However this is not true in general independence system.

Definition 2 An independence system (N, I) is a k-system if for any two bases A, B ∈ I, |A| ≤
k|B|. That is, the ratio of the cardinality of a maximum base and the cardinality of a minimum
base is at most k.

The following theorem is not too difficult but not so obvious either.

Theorem 5 Greedy gives a 1/k-approximation for the maximum weight independent set problem
in a k-system.

The above theorem generalizes and unifies all the examples we have seen so far including MIS
in bounded degree graphs, matchings, matroids etc. How does one see that a given independence
system is indeed a k-system for some parameter k? For instance matchings in graphs form a 2-
system. The following simple lemma gives an easy way to argue that a given system is a k-system.

Lemma 6 Suppose (N, I) is an independence system with the following property: for any A ∈ I
and e ∈ N \ A there is a set of at most Y ⊂ A such that |Y | ≤ k and (A \ Y ) ∪ {e} ∈ I. Then I is
a k-system.

We leave the proof of the above as an exercise.

We refer the reader to [3, 4] for analysis of Greedy in k-systems and other special cases.

3 Packing Integer Programs (PIPs)

We can express the Knapsack problem as the following integer program. We scaled the knapsack
capacity to 1 without loss of generality.

n
X
maximize p i xi
i=1
X
subject to s i xi ≤ 1
i
xi ∈ {0, 1} 1≤i≤n

More generally if have multiple linear constraints on the “items” we obtain the following integer
program.

Definition 3 A packing integer program (PIP) is an integer program of the form max{wx | Ax ≤
1, x ∈ {0, 1}n } where w is a 1 × n non-negative vector and A is a m × n matrix with entries in
[0, 1]. We call it a {0, 1}-PIP if all entries are in {0, 1}.
 In some cases it is useful/natural to define the problem as max{wx | Ax ≤ b, x ∈ {0, 1}n } where
entries in A and b are required to rational/integer valued. We can convert it into the above form
by dividing each row of A by bi .
When m the number of rows of A (equivalently the constraints) is small the problem is tractable.
It is some times called the m-dimensional knapsack and one can obtain a PTAS for any fixed
constant m. However, when m is large we observe that MIS can be cast as a special case of
{0, 1}-PIP. It corresponds exactly to the simple integer/linear program that we saw in the previous
section. Therefore the problem is at least as hard to approximate as MIS. Here we show via a
clever LP-rounding idea that one can generalize the notion of bounded-degree to column-sparsity
in PIPs and obtain a related approximation. We will then introduce the notion of width of the
constraints and show how it allows for improved bounds.

Definition 4 A PIP is k-column-sparse if the number of non-zero entries in each column of A is

at most k. A PIP has width W if maxi,j Aij /bi ≤ 1/W .

3.1 Randomized Rounding with Alteration for PIPs

We saw that randomized rounding gave an O(log n) approximation algorithm for the Set Cover
problem which is a canonical covering problem. Here we will consider the use of randomized round-
ing for packing problems. Let x be an optimum fractional solution to the natural LP relaxation of
a PIP where we replace the constraint x ∈ {0, 1}n by x ∈ [0, 1]n . Suppose we apply independent
randomized rounding where we set x0i to 1 with probability 0
P xi . Let x be the resulting integer
solution. The expected weight of this solution is exactly i wi xi which is the LP solution value.
However, x0 may not satisfy the constraints given by Ax ≤ b. A natural strategy to try to satisfy
the constraints is to set x01 to 1 with probability cxi where c < 1 is some scaling constant. This may
help in satisfying the constraints because the
P scaling creates some room in the constraints; we now
have that the expected solution value is c i wi xi , a loss of a factor of c. Scaling by itself does not
allow us to claim that all constraints are satisfied with good probability. A very useful technique in
this context is the technique of alteration; we judiciously fix/alter the rounded solution x0 to force
it to satisfy the constraints by setting some of the variables that are 1 in x0 to 0. The trick is to
do this in such a way as to have a handle on the final probability that a variable is set to 1. We
will illustrate this for the Knapsack problem and then generalize the idea to k-sparse PIPs. The
algorithms we present are from [2].
Rounding for Knapsack: Consider the Knapsack problem. It is convenient to think of this in
the context of PIPs. So we have ax ≤ 1 where ai now represents the size of item i and the knapsack
capacity is 1; wi is the weight of item. Suppose x is a fractional solution. Call an item i “big” if
ai > 1/2 and otherwise it is “small”. Let S be the indices of small items and B the indices of the
big items. Consider the following rounding algorithm.
 Rounding-with-Alteration for Knapsack:
Let x be an optimum fractional solution
Round each i to 1 independently with probability xi /4. Let x0 be rounded solution.
x00 = x0
If (x0i = 1 for exactly one big item i)
00
P j 6= i 0set xj = 0
For each
Else If ( i∈S si xi > 1 or two or more big items are chosen in x0 )
For each j set x00j = 0
End If
Output feasible solution x00

In words, the algorithm alters the rounded solution x0 as follows. If exactly one big item is
chosen in x0 then the algorithm retains that item and rejects all the other small items. Otherwise,
the algorithm rejects all items if two or more big items are chosen in x0 or if the total size of all
small items chosen in x0 exceeds the capacity.
The following claim is easy to verify.

Claim 7 The integer solution x00 is feasible.

analyze the probability of an item i being present in the final solution. Let E1 be
Now let us P
the event that i∈S ai x0i > 1, that is the sum of the sizes of the small items chose in x0 exceeds the
capacity. Let E2 be the event that at least one big item is chosen in x0 .

Claim 8 Pr[E1 ] ≤ 1/4.

Proof: Let Xs = i∈S ai x0i be the random variable that measures the sum of the sizes of the small
P
items chosen. We have, by linearity of expectation, that
X X
E[Xs ] = ai E[x0i ] = ai xi /4 ≤ 1/4.
i∈S i∈S

By Markov’s inequality, Pr[Xs > 1] ≤ E[Xs ]/1 ≤ 1/4. 2

Claim 9 Pr[E2 ] ≤ 1/2.

P P
P the size of each big item in B is at least 1/2, we have 1 ≥ i∈B ai xi ≥ i∈B xi /2.
Proof: Since
Therefore i∈B xi /4 ≤ 1/2. Event E2 happens if some item i ∈ B is chosen
P in the random selection.
Since i is chosen with probability xi /4, by the union bound, Pr[E2 ] ≤ i∈B xi /4 ≤ 1/2. 2

Lemma 10 Let Zi be the indicator random variable that is 1 if x00i = 1 and 0 otherwise. Then
E[Zi ] = Pr[Zi = 1] ≥ xi /16.

Proof: We consider the binary random variable Xi which is 1 if x0i = 1. We have E[Xi ] = Pr[Xi =
1] = xi /4. We write
xi
Pr[Zi = 1] = Pr[Xi = 1] · Pr[Zi = 1 | Xi = 1] = Pr[Zi = 1 | Xi = 1].
4
To lower bound Pr[Zi = 1 | Xi = 1] we upper bound the probability Pr[Zi = 0|Xi = 1], that is, the
probability that we reject i conditioned on the fact that it is chosen in the random solution x0 .
 First consider a big item i that is chosen in x0 . Then i is rejected iff if another big item is chosen
in x0 ; the probability of this can be upper bounded by Pr[E1 ]. If item i is small then it is rejected
if and only if E2 happens or if a big item is chosen which happens with Pr[E1 ]. In either case

Pr[Zi = 0|Xi = 1] ≤ Pr[E1 ] + Pr[E2 ] ≤ 1/4 + 1/2 = 3/4.

Thus,
xi xi
Pr[Zi = 1] = Pr[Xi = 1] · Pr[Zi = 1 | Xi = 1] = (1 − Pr[Zi = 0 | Xi = 1]) ≥ .
4 16
2
One can improve the above analysis to show that Pr[Zi = 1] ≥ xi /8.

Theorem
Pn 11 The randomized algorithm outputs a feasible solution of expected weight at least
i=1 wi xi /16.

Proof: The expected weight of the output is

X X X
E[ wi x00i ] = wi E[Zi ] ≥ wi xi /16
i i i

where we used the previous lemma to lower bound E[Zi ]. 2

Rounding for k-sparse PIPs: We now extend the rounding algorithm and analysis above to
k-sparse PIPs. Let x be a feasible fractional solution to max{wx | Ax ≤ 1, x ∈ [0, 1]n }. For a
column index i we let N (i) = {j | Aj,i > 0} be the indices of the rows in which i has a non-zero
entry. Since A is k-column-sparse we have that |N (i)| ≤ k for 1 ≤ i ≤ n. When we have more
than one constraint we cannot classify an item/index i as big or small since it may be big for
some constraints and small for others. We say that i is small for constraint j ∈ N (i) if Aj,i ≤ 1/2
otherwise i is big for constraint j. Let Sj = {i | j ∈ N (i), and i small for j} be the set of all small
columns for j and Bj = {i | j ∈ N (i), and i small for j} be the set of all big columns for j. Note
that Sj ∩ Bj is the set of all i with Aj,i > 0.

Rounding-with-Alteration for k-sparse PIPs:

Let x be an optimum fractional solution
Round each i to 1 independently with probability xi /(4k). Let x0 be rounded solution.
x00 = x0
For j = 1 to m do
If (x0i = 1 for exactly one i ∈ Bj )
each h ∈ Sj ∪ Bj and h 6= i set x00h = 0
For P
Else If ( i∈Sj Aj,i x0i > 1 or two or more items from Bj are chosen in x0 )
For each h ∈ Sj ∪ Bj set x00h = 0
End If
End For
Output feasible solution x00

The algorithm, after picking the random solution x0 , alters it as follows: it applies the previous
algorithm’s strategy to each constraint j separately. Thus an element i can be rejected at different
constraints j ∈ N (i). We need to bound the total probability of rejection. As before, the following
claim is easy to verify.
Claim 12 The integer solution x00 is feasible.

P the probability of an item i being present in the final solution. Let E1 (j)
Now let us analyze
be the event that i∈Sj Aj,i x0i > 1, that is the sum of the sizes of the items that are small for j in
x0 exceed the capacity. Let E2 (j) be the event that at least one big item for j is chosen in x0 . The
following claims follow from the same reasoning as the ones before with the only change being the
scaling factor.

Claim 13 Pr[E1 (j)] ≤ 1/(4k).

Claim 14 Pr[E2 (j)] ≤ 1/(2k).

Lemma 15 Let Zi be the indicator random variable that is 1 if x00i = 1 and 0 otherwise. Then
E[Zi ] = Pr[Zi = 1] ≥ xi /(16k).
Proof: We consider the binary random variable Xi which is 1 if x0i = 1 after the randomized
rounding. We have E[Xi ] = Pr[Xi = 1] = xi /(4k). We write
xi
Pr[Zi = 1] = Pr[Xi = 1] · Pr[Zi = 1 | Xi = 1] = Pr[Zi = 1 | Xi = 1].
4k
We upper bound the probability Pr[Zi = 0|Xi = 1], that is, the probability that we reject i
conditioned on the fact that it is chosen in the random solution x0 . We observe that
X
Pr[Zi = 0|Xi = 1] ≤ (Pr[E1 (j)] + Pr[E2 (j)] ≤ k(1/(4k) + 1/(2k)) ≤ 3/4.
j∈N (i)

We used the fact that N (i) ≤ k and the claims above. Therefore,
xi xi
Pr[Zi = 1] = Pr[Xi = 1] · Pr[Zi = 1 | Xi = 1] = (1 − Pr[Zi = 0 | Xi = 1]) ≥ .
4k 16k
2
The theorem below follows by using the above lemma and linearity of expectation to compare
the expected weight of the output of the randomized algorithm with that of the fractional solution.

Theorem
Pn 16 The randomized algorithm outputs a feasible solution of expected weight at least
w x
i=1 i i /(16k). There is 1/(16k)-approximation for k-sparse PIPs.

Larger width helps: We saw during the discussion on the Knapsack problem that if all items
are small with respect to the capacity constraint then one can obtain better approximations. For
PIPs we defined the width of a given instance as W if maxi,j Aij /bi ≤ 1/W ; in other words no single
item is more than 1/W times the capacity of any constraint. One can show using a very similar
algorithm and anaylisis as above that the approximation
√ bound improves to Ω(1/k dW e ) for instance
with width W . Thus if W = 2 we get a Ω(1/ k) approximation instead of Ω(1/k)-approximation.
More generally when W ≥ c log k/ for some sufficiently large constant c we can get a (1 − )-
approximation. Thus, in the setting with multiple knapsack constraints, the notion of small with
c
respect to capacities is that in each constraint the size of the item is ≤ log k times the capacity of
that constraint.
References
[1] J. Håstad. Clique is Hard to Approximate within n1− . Acta Mathematica, 182:105–142, 1999.

[2] N. Bansal, N. Korula, V. Nagarajan, A. Srinivasan. On k-Column Sparse Packing Programs.

Proc. of IPCO, 2010. Available at http://arxiv.org/abs/0908.2256.

[3] Julian Mestre. Greedy in Approximation Algorithms. Proc. of ESA, 2006: 528–539.

[4] Moran Feldman, Joseph Naor, Roy Schwartz, Justin Ward. Improved Approximations for
k-Exchange Systems - (Extended Abstract). Proc. of ESA, 2011: 784–798.