A Crash Course on Linear Programs : Part 21

• The Dual Linear Program. For every linear program there is another linear program which lives in a
completely different space but has the same value! In approximation algorithms, the dual is often used
to design and analyze “self-contained” algorithms for problems. By this, I mean algorithms which do
not resort to solving LPs. In this note we brush up on the definitions.

• We begin with minimization programs on n variable. For convenience’s sake, we will differentiate
constraints as “non-trivial” inequalities and “non-negativity” constraints.
n
X
>
lp := minimize c x = cj xj (Linear Program)
j=1

Ax ≥ b, A ∈ Rm×n , b ∈ Rm
x ∈ Rn≥0

• The Lagrangean. The dual, which is not restricted to just linear programs but any program, starts
with what is called the Lagrangean function named after the Italian-French mathematician Joseph-
Louis Lagrange (aka Giuseppe Luis Lagrangia). The main idea of this is to “move all the constraints
to the objective”. Instead of moving all, we move the non-trivial ones. Let us introduce variables
(called Lagrange/dual variables) yi for each of the m constraints/rows of the matrix A. Given this
m-dimensional variable vector y, define
 
 >
y> (b − Ax) 

L(y) := min
n
c x + (Lagrangean)
x∈R≥0  | {z } 
Pm >
= i=1 yi ·(bi −ai x)

One way to think about the above function is the following. For the time being assume yi ≥ 0
and think of it as a rate at which we “penalize” x if it x doesn’t satisfy the ith inequality, that is,
bi > a>i x. In that case, we multiply this “violation” by yi and add it to the function. Since x is trying
to “minimize” the term in the paranthesis, the y’s perhaps nudge the x to becoming more feasible.
The last line is really figurative and shouldn’t be given much attention.
However, a few facts are to be observed.

Fact 1. Suppose x be any feasible solution to (Linear Program). Then, for any y ∈ Rm ≥0 , we
have L(y) ≤ c> x. In particular, this is true if we take the optimal solution x∗ , and if we take the
y which maximizes L(y). Therefore,

max L(y) ≤ lp (1)

y∈Rm
≥0

1
Lecture notes by Deeparnab Chakrabarty. Last modified : 18th Mar, 2022
These have not gone through scrutiny and may contain errors. If you find any, or have any other comments, please email me at
[email protected]. Highly appreciated!

1
 Proof. Because for a feasible x for (Linear Program), we have (b−Ax) ≤ 0 and thus y> (b−Ax) ≤
0 if y ≥ 0. Which in turn means L(y) ≤ c> x.

Fact 2. One can re-write (Lagrangean) as

(
y> b if y> A ≤ c>
L(y) =
−∞ otherwise

Proof. Rearranging gives us L(y) = y> b + minx≥0 c> − y> A x. If (c> − y> A) has any coor-

dinate i negative, then one would choose xi to be as large a positive number and xj = 0 for all other
coordinates to make the minimum be as negative a number as one wants.

• The Dual LP and Weak Duality. The above two facts imply the following: one, that the maximization
of L(y) can be written as a linear program itself, and two, the value of this linear program is a lower
bound on the LP value. This linear program is called the Dual LP.
m
X
dual := maximize b> y = bi yi (Dual Program)
i=1
A> y ≤ c,
y ∈ Rm
≥0

dual ≤ lp (Weak Duality)

A couple of remarks about the dual. One, the dual is a maximization LP while the original LP, which is
called the primal LP, was a minimization one. Therefore the dual value of any feasible dual solution is
a lower bound on the value of the primal LP; this is a very important fact that will be used in algorithm
design and analysis. Second, for every variable xj in the primal there is a constraint in the dual, and
for every constraint in the primal there is a variable yi in the dual. Writing the dual LP is a completely
mechanical process, but experience tells me it takes some time getting used to; the inexperienced
reader is urged to look at the following illustrations and then try taking duals of every LP they see (in
particular, take dual of the dual).

• Two Illustrations. Consider the following LP on n = 3 variables having m = 2 constraints apart

from the non-negativity constraints.

lp := minimize 2x1 + 3x2 − x3 (Illus-primal)

x1 + x2 − x3 ≥ 3, (P1)
x3 − 2x1 ≥ 0, (P2)
x1 , x2 , x3 ≥ 0

Before reading further, can you see any simple lower bound on lp? To me, I can see that the LP
objective is at least x1 + x2 − x3 which is at least 3 by (P1). Therefore, surely lp ≥ 3. Anything larger
doesn’t immediately leap to the eye (the adverb “immediately” is important). Ok, let’s take the dual
now.

2
In the dual LP, we have two variables, let’s call them y1 and y2 corresponding to primal constraints
(P1) and (P2). The objective of the dual LP is to maximize a linear combination of y1 and y2 , and the
coefficients are simply the RHS of the corresponding primal constraints. Thus, it is 3y1 + 0y2 = 3y1 .
There is a dual constraint for each primal variable; therefore, there will be three constraints. Let me
show how to figure out the dual constraint on (y1 , y2 ) corresponding to primal variable x1 . We first
figure out which primal constraints x1 appears in; the corresponding dual variables will appear in the
dual constraint. In this case, x1 appears in both, and so both y1 and y2 will appear. Furthermore,
the coefficient of y1 will be the coefficient of x1 in (P1), and similarly the coefficient of y2 will be
coefficient of x1 in (P2). This forms the LHS of the dual constraint, which in this case is y1 − 2y2 .
The inequality of the constraint is ≤, and the RHS of the constraint is the coeffiecient of x1 in the
primal objective. And therefore, the dual constraint is y1 − 2y2 ≤ 1. We can similarly write the dual
constraints corresponding to x2 and x3 (do you want to try before reading ahead?) Finally, we add
non-negativity constraints on y1 and y2 , and this finishes the dual. Pretty mechanical.

dual := maximize 3y1 (Illus-Dual)

y1 − 2y2 ≤ 2, (D1)
y1 ≤ 3, (D2)
y2 − y1 ≤ −1 (D3)
y1 , y2 ≥ 0

First note that y1 = 3 and y2 = 2 is a feasible solution with dual = 9. And therefore, by (1), we
also get that the optimum value lp of (Illus-primal) is at least 9 as well. Indeed, it is precisely 9 since
(x1 = 0, x2 = 3, x3 = 0) achieves that value; but this was not immediate before the dual, right?
Good. Let’s move to a second and more abstract illustration.
Consider the LP relaxation for the vertex cover problem that we have seen before. Here it is.
X
lp(G) := minimize c(v)xv (Vertex Cover LP)
v∈V
xu + xv ≥ 1, ∀(u, v) ∈ E (2)
xv ≥ 0, ∀v ∈ V

What is the dual of the above LP? Do you want to try writing it before peeking ahead? The dual has a
variable ye per edge of the graph (since the primal has a constraint per edge). The constraint is simply
the sum of these ye ’s since the RHS of the primal constraints is 1. There is a dual constraint per vertex
v ∈ V since there is a primal variable for every v ∈ V . The constraint corresponding to v is a linear
combination of all dual variables ye such that the primal variable xv appears in the eth constraint. In
particular, it is the sum of all the ye ’s for e incident on v. The RHS of the constraint is c(v) since
that is the coefficient of xv int he primal LP. And finally, we have non-negativity constraints on ye ’s.
Done.
X
dual(G) := maximize ye (Vertex Cover Dual)
e∈E
X
ye ≤ c(v), ∀v ∈ V (3)
e:v∈e
ye ≥ 0, ∀e ∈ E

3
• Strong Duality. Here is one of the most magical theorems out there.

Theorem 1 (Strong Duality). dual = lp

Proof. (Sketch) We provide a proof to give an idea of how such a theorem is proven. Indeed, we
consider the special case of non-degenerate feasible regions. That is, the feasible region is full dimen-
sional and every basic feasible solution x has exactly n constraints holding with equality, and the rest
hold with strict inequality. This assumption is not needed, but it gets to the essence of the proof.
Consider an optimal bfs x∗ (recall, such a solution always exists) and let B be the corresponding
basis. So, Bx∗ = bB , that is a> ∗
i x = bi for i ∈ B (we abuse B to denote rows and the index of the
rows), and the rows of B span Rn . In P
particular, the cost vector c can be uniquely written as a linear
combination of the basis vectors; c = i∈B yi ai .
Now consider a candidate solution y to (Dual Program with equalities) where yi P = yi for i ∈ B
and yj = 0 for j ∈
P / B. Observe (a) by definition y A = c , and (b) c x = i∈B yi a>
> > > ∗ ∗
i x =
i∈B yi bi . It seems as if we have found a feasible solution y to the dual LP whose objective equals
c x∗ . Since we already have established weak-duality, this equality would prove theorem. The only
>

nub is that we haven’t establishes y ≥ 0; indeed, we have also not really used x∗ is the optimal
solution. We do so next.
We claim that all the yi ≥ 0 which would complete the proof of the theorem. Suppose not, and say
y1 < 0. Consider a vector v ∈ Rn in the null space of B \ {1} such that a> >
1 v > 0 and ai v = 0 for
i ∈ B \ {1}. This exists since a1 is linearly independent of B \ a1 . Now choose θ > 0 small enough
such that a>
j (θv) > bj for all j ∈
/ B; this is where we are using the non-degeneracy assumption. By
design, therefore, x = x + θv is feasible. And, c> x0 − c> x∗ = θc> v. However,
0 ∗

m
X
c > v = y1 a >
1 v+ yi a>
i v <0
|{z} |{z}
>0 i=2 =0

since y1 < 0. This contradicts x∗ is the optimum solution, completing the proof of strong duality.

• Complementary Slackness. A very interesting feature about the mirroring is captured by the following
observation which, due to its importance, is given a name called complementary slackness. It says, a
dual variable is positive in an optimal dual solution only if the corresponding primal constraint must
be tight, that is hold with equality, in any optimal primal solution. Similarly, a primal variable is
positive in an optimal primal solution only if the corresponding dual constraint is tight.

Lemma 1 (Complementary Slackness.). Let x∗ be any optimal solution of (Linear Program). Let
y∗ be any optimal solution of (Dual Program with equalities). Then, yj∗ > 0 ⇒ a>
j x = bj and
x∗i > 0 ⇒ y> Ai = ci . Her Ai is the ith column of the matrix A.

Proof. For brevity’s sake, let’s call x∗ simply x and y∗ simply y. By Strong Duality, we know that
c> x = y> b, since (x, y) are optimal solutions. We also know that c> ≥ y> A. Therefore, since
x ≥ 0, we get c> x ≥ y> A x. And so,

 
y> b = c> x ≥ y> A x ⇒ y> b ≥ y> (Ax) ⇒ y> (Ax − b) ≤ 0

4
 On the other hand Ax ≥ b, or in other words
Pm if we define the m-dimensional vector v := Ax − b,
vj ≥ 0 for all 1 ≤ j ≤ m. Thus, we get j=1 yj vj ≤ 0 while yj ≥ 0 and vj ≥ 0.
There is only one possibility : we must have m
P
j=1 yj vj = 0. And therefore, whenever yj > 0 we
>
must have vj = 0, that is, a j = bj .
Since y> (Ax − b) = 0, we also get that c> x = y> A x. That is, c> − y> A x = 0. Again, if

we define the n-dimensional vector w := c − A> y, then we get w> x = 0 while both w and x are
non-negative. This would mean that xi > 0 ⇒ wi = 0, that is, y> Ai = ci .

• The Dual of a Maximization LP. The same procedure using the Lagrangean function can be used to
write the dual of a maximization LP as well. So, if the primal LP is
n
X
lp := maximize c> x = cj xj (Max Linear Program)
j=1

Ax ≤ b, A ∈ Rm×n , b ∈ Rm
x ∈ Rn≥0
Then the dual LP also has variables y ∈ Rm corresponding to the constraints in the primal. It is a
minimization LP, and the constraints are of the “≥” type. Weak duality asserts that the value of the
dual is at least the value of the maximizing primal, and strong duality implies they are equal.
m
X
dual := minimize b> y = bi yi (Min Dual Program)
i=1
A> y ≥ c,
y ∈ Rm
≥0

• The Dual with Equality Constraints. Sometimes the primal LP has equality constraints. In that case,
the corresponding dual variables are “free”; that is, they don’t have any non-negativity constraint and
are allowed to be free. Once again, this is not hard to see if one treats the equality constraint as two
sets of inequality constraints, and then writes the dual. In particular, if the primal LP is
n
X
lp := minimize c> x = cj xj (Linear Program with Equalities)
j=1

Ax ≥ b, A ∈ Rm×n , b ∈ Rm
P x = q, P ∈ Rk×n , q ∈ Rk
x ∈ Rn≥0

then its dual has two sets of variables y ∈ Rm corresponding to A and z ∈ Rk corresponding to P .
The program is
dual := maximize b> y + q> z (Dual Program with equalities)
A> y + P > z ≤ c,
y ∈ Rm
≥0 , z ∈ R
k

Note that z has no non-negativity constraints.

5
Notes
Since this is not a course on linear programming, my notes will be short because the alternative is to be
extremely long. All I will say is that everyone who studies linear programming has a favorite source which
enlightened them. For me it was this beautiful text [1] by Bertsimas and Tsitsiklis.

References
[1] D. Bertsimas and J. Tsitsiklis. Introduction to Linear Optimization. Athena-Scientific, 1997.