DDA 5002/CIE 6010 Optimization

Lecture 13 LP Duality Theory

Yuang Chen

School of Data Scienc

The Chinese University of Hong Kong, Shenzhen

March 10, 2025

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 1 / 44

Outline

1 LP Duality

2 Weak and Strong Duality

3 Table of Possibles and Impossibles

4 Complementary Slackness

5 Dual Applications

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 2 / 44

Outline

1 LP Duality

2 Weak and Strong Duality

3 Table of Possibles and Impossibles

4 Complementary Slackness

5 Dual Applications

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 3 / 44

General LP Dual

Primal Dual
minimize cT x maximize bT y
subject to aiT x → bi , i ↑ M1 , subject to yi → 0, i ↑ M1
aiT x ↓ bi , i ↑ M2 , yi ↓ 0, i ↑ M2
aiT x = bi , i ↑ M3 , yi free, i ↑ M3
xj → 0, j ↑ N1 , ATj y ↓ cj , j ↑ N1
xj ↓ 0, j ↑ N2 , ATj y → cj , j ↑ N2
xj free, j ↑ N3 , ATj y = cj , j ↑ N3

aiT is the ith row of A, Aj is the jth column of A

Each primal constraint corresponds to a dual variable
Each primal variable corresponds to a dual constraint
Equality constraints always correspond to free variables

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 4 / 44

Rules to Form Dual Problem

Primal minimize maximize Dual

→ bi →0
Constraints ↓ bi ↓0 Variables
= bi free
→0 ↓ cj
Variables ↓0 → cj Constraints
free = cj

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 5 / 44

Form Dual Example I

max x1 + 2x2 + x3 + x4
s.t. x1 + 2x2 + 3x3 ↓ 2
x2 + x4 ↓ 1
x1 + 2x3 → 1
x1 , x3 → 0, x2 , x4 free

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 6 / 44

Form Dual Example II

(P) ZP = min x1 + 2x2 + 3x3

x1 ,x2 ,x3

s.t. x1 + 5x2 + 4x3 → 6

2x1 + 3x2 ↑ x3 = 3
x1 + x2 ↑ 2x3 ↓ 4
x1 → 0, x2 ↓ 0, x3 free

(D) ZD = max 6y1 + 3y2 + 4y3

y1 ,y2 ,y3

s.t. y1 + 2y2 + y3 ↓ 1
5y1 + 3y2 + y3 → 2
4y1 ↑ y2 ↑ 2y3 = 3
y1 → 0, y2 free, y3 ↓ 0
Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 7 / 44
Form Dual Example III

(P): min cT x
s.t. Ax → b
Dx ↑ d
x ↑0

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 8 / 44

Primal and Dual Pair in Standard Form

(P) min c ↭x (D) max b↭ y

s.t. Ax = b s.t. A↭ y ↑ c
x →0 y free

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 9 / 44

Invariance of Transformations

Theorem
If we transform a linear program to an equivalent one (such as by replacing
free variables, adding slack variables, etc), then the dual of the two
problems will be equivalent.

Theorem
If we transform the primal to its dual, then transform the dual to its dual,
then we will obtain a problem equivalent to the primal problem, that is,
the dual of dual is the primal.

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 10 / 44

Outline

1 LP Duality

2 Weak and Strong Duality

3 Table of Possibles and Impossibles

4 Complementary Slackness

5 Dual Applications

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 11 / 44

Weak Duality Theorem

Primal Dual
min cT x max b T y
s.t. Ax = b, x → 0 s.t. AT y ↑ c

Theorem (Weak Duality Theorem)

If x is feasible to the primal and y is feasible to the dual, then

bT y ↑ c T x

If the primal is a minimization and dual is a maximization, then

Any dual feasible solution will give a lower bound on the primal
optimal value
Any primal feasible solution will give an upper bound on the dual
optimal value
The optimal value of primal is larger than that of dual
Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 12 / 44
Corollary of Weak Duality

If the optimal objective value of the primal minimization problem is

↓↔ (i.e., the primal problem is unbounded), then the dual
maximization problem must be infeasible.
If the optimal cost of the dual maximization problem is +↔ (i.e., the
dual problem is unbounded), then the primal minimization problem
must be infeasible.
Let x → be feasible to the primal problem and y → be feasible to the
dual problem, and suppose c T x → = b T y → , then x → and y → are optimal
solutions to the primal and dual problems, respectively.

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 13 / 44

Strong Duality

Strong Duality Theorem

If a primal linear program (P) has a finite optimal solution x → , then its
dual linear program (D) must also have a finite optimal solution y → , and
the respective optimal objective values are equal, that is c T x → = b T y → .

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 14 / 44

Discussion

Based on the strong duality theorem, we know that (x, y ) is optimal to the
primal and dual respectively if and only if
x is primal feasible
y is dual feasible
They achieve the same objective value

Therefore solving LP is in fact equivalent as solving the following linear

system:
Ax = b, x → 0
AT y ↑ c
bT y = c T x

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 15 / 44

Outline

1 LP Duality

2 Weak and Strong Duality

3 Table of Possibles and Impossibles

4 Complementary Slackness

5 Dual Applications

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 16 / 44

Table of Possibles and Impossibles

The primal and dual LPs can be finite optimal, or unbounded, or

infeasible. So, there are in total 9 combinations. Are all these 9
combinations possible?
Finite Optimal Unbounded Infeasible
Finite Optimal Possible Impossible Impossible
Unbounded Impossible Impossible Possible
Infeasible Impossible Possible Possible
Notice this table is exactly symmetric, because the dual of the dual is the
primal.

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 17 / 44

Figuring out the possibilities

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 18 / 44

Figuring out the possibilities

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 19 / 44

Figuring out the possibilities

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 20 / 44

Outline

1 LP Duality

2 Weak and Strong Duality

3 Table of Possibles and Impossibles

4 Complementary Slackness

5 Dual Applications

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 21 / 44

Complementarity Conditions
Consider the primal-dual pair:
Primal Dual
minimize cT x maximize bT y
subject to aiT x → bi , i ↑ M1 , subject to yi → 0, i ↑ M1
aiT x ↓ bi , i ↑ M2 , yi ↓ 0, i ↑ M2
aiT x = bi , i ↑ M3 , yi free, i ↑ M3
xj → 0, j ↑ N1 , ATj y ↓ cj , j ↑ N1
xj ↓ 0, j ↑ N2 , ATj y → cj , j ↑ N2
xj free, j ↑ N3 , ATj y = cj , j ↑ N3

Theorem
Let x and y are feasible solutions to the primal and dual problems
respectively. Then x and y are optimal if and only if

yi · (aiT x ↔ bi ) = 0, ↗i; xj · (AT

j y ↔ cj ) = 0, ↗j.

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 22 / 44

Complementary Slackness

Let x and y be feasible solutions to the primal and dual problem,

respectively. Then x and y are optimal solutions for the two respective
problems if and only if they satisfy the following conditions:
Primal Complementary Slackness: yi (ai↭ x → bi ) = 0 for all i. In words,
either the i-th primal constraint is active (binding, tight) so ai↭ x = bi ,
or the corresponding dual variable yi = 0.
Dual Complementary Slackness: xj (AT j y → cj ) = 0 for all j. In words,
either the j-th dual constraint is active (binding, tight) so AT j y = cj ,
or the corresponding primal variable xj = 0.

Proof in homework!

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 23 / 44

Example

min 13x1 + 10x2 + 6x3 max 8y1 + 3y2

s.t. 5x1 + x2 + 3x3 = 8 s.t. 5y1 + 3y2 ↑ 13
3x1 + x2 = 3 y1 + y2 ↑ 10
x1 , x2 , x3 → 0 3y1 ↑ 6
Someone solved the dual problem graphically, and told us the dual
optimal solution y1→ = 2, y2→ = 1. We want to use this information and
Complementary Slackness to get the optimal solution of the primal.
If we are given a primal feasible solution x1→ = 1, x2→ = 0; x3→ = 1, can
we verify this is an optimal solution?

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 24 / 44

Outline

1 LP Duality

2 Weak and Strong Duality

3 Table of Possibles and Impossibles

4 Complementary Slackness

5 Dual Applications

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 25 / 44

Applications of Dual

Pricing
Alternative Systems
Max Flow and Min Cut Problems
Robust Optimization

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 26 / 44

Diet problem
Suppose there are n types of foods. Each food j costs cj dollars per unit
to purchase, and has aij units of nutrient i for i = 1, .., m. The goal is to
combine n types of food with the minimum cost to produce an ideal food
combination that contains bi units of nutrient i for i = 1, .., m.
n
!
min c j xj
j=1
!n
s.t. aij xj → bi ↑i = 1, .., m
j=1

xj → 0 ↑j = 1, ..., n
We want to study how the change of right-hand side of the constraint,
namely bi ’s, would a!ect the optimal cost of the diet problem. In
particular, if bi of a specific nutrient i increases by a unit, while holding all
other components of the RHS constant, how would the minimum cost
change? Should it increase or decrease?
Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 27 / 44
Shadow price

Shadow price
The shadow price of a constraint is the change of the objective cost from
the current optimal level if the right-hand side of this constraint is
increased by a unit from the current level.

m
!
max b i yi
i=1
m
!
s.t. aij yi ↓ cj ↑j = 1, .., n
i=1
yi → 0 ↑i = 1, ..., m

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 28 / 44

Shadow price interpretation

A shadow price yi for each constraint i, or nutrient i, is the increase

of the optimal objective cost from the current level by increasing bi
by a unit.
The shadow price yi can be viewed as the unit price of the nutrient i
that you would be willing to purchase, or yi is the value of nutrient i
at the current level of the nutrient requirement bi ’s.
"m
i=1 bi yi is the
"mtotal value of the nutrients of the ideal food
combination. i=1 aij yi for food j is the total cost of nutrients that a
unit of food j provides.

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 29 / 44

Alternative Systems

Given a set of linear inequalities:

AT y ↓ c

An important question is: whether the system has a solution?

It is easy to verify that it has a solution, one only needs to find a
solution (we call it a certificate)
To disprove the existence, can we also have such a certificate?
The answer: Yes.

If we can find a vector x satisfying

Ax = 0, x → 0, cT x < 0

Then there must be no solution to the system AT y ↓ c

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 30 / 44

Farkas’ Lemma

Theorem (Farkas’ Lemma)

Let A ↔ Rm→n and b ↔ Rm . Exactly one of the following two alternatives
hold:
(I) P := {x ↔ Rn : Ax = b, x → 0} =
↗ ↘
(II) Q := {y ↔ Rm : A↑ y → 0, b ↑ y < 0} =
↗ ↘

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 31 / 44

Alternative Systems

One can construct many more pairs of such alternative systems.

It is hard to directly prove something is not feasible.
LP duality provides an alternative approach, transforming the problem
to proving something is feasible.

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 32 / 44

Maximum Flow Problem
The maximum flow problem can be described as follows:
Given a directed, weighted graph G = (V , E ) and a pair of nodes s
and t (V is the set of nodes, E is the set of edges)
One can think this as a tra”c network
There is an edge capacity wij on each edge
Question: What is the largest amount of flow one can send from s to
t, subject to the capacity constraints?

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 33 / 44

A Concrete Example

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 34 / 44

One Transformation

Assume there is an imaginary node o, with edges (o, s) and (t, o). There
is no capacity constraint on those two edges
The problem becomes a closed system. One wants to maximize the
flow from o to s, which we denote by #.

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 35 / 44

LP Formulation

Using this transformation, we can write down the LP formulation. Let xij
denote the amount of flow across edge (i, j).

maximizex,! #
" "
subject to xji ≃ j:(i,j)↓E xij = 0,
j:(j,i)↓E ↑i ↗= s, t
" "
xjs ≃ j:(s,j)↓E xsj + # = 0
"j:(j,s)↓E "
j:(j,t)↓E xjt ≃ j:(t,j)↓E xtj ≃ # = 0
xij ↓ wij , ↑(i, j) ↔ E
xij → 0, ↑(i, j) ↔ E

The first constraint is the flow balancing constraints for all nodes
other than s and t
The second (third, resp.) constraint is the flow balancing constraints
for node s (t, resp.)

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 36 / 44

Dual of the Maximum Flow Problem

We construct the dual problem:

!
minimize wij zij
(i,j)↓E

subject to zij → yi ≃ yj , ↑(i, j) ↔ E

y s ≃ yt = 1
zij → 0

What does the dual problem mean?

First assume all y ’s are 0 or 1. Then

We assign a label (0 or 1) to each node, 1 to s and 0 to t.
If i has a larger label than j for (i, j) ↔ E , there is a cost wij .

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 37 / 44

Interpretation of the Dual

The dual problem is equivalent to finding a subset S of vertices containing

s but not t, that minimizes the weight of the cut, i.e.
!
wij
i↓S,j↔↓S

This is called the min-cut problem

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 38 / 44

Max Flow = Min Cut

Theorem
The maximum flow of the network is equal to the smallest cut size of any
subset S of vertices.

Corollary
If the value of a flow is equal to the value of some cut, then both are
optimal.

One can view the min-cut as the bottleneck of the network.

The maximum flow that can be sent through this network is equal to
the tightest bottleneck of this network.
This is one classical example of dual problems: maximum flow versus
minimum cut.

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 39 / 44

Robust optimization

min c ↭x
s.t. a↭ x ↓ b

Suppose the coe”cient a is not known exactly, but all we know is that a
lives in some polytope U :

U = {a : Ba ↓ d}

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 40 / 44

Example

Suppose there are two assets. We know that the return on asset 1 will be
in between 0.8 and 1.1 and that on asset 2 will be in between 0.75 and
1.25. Also the return on asset 1 will be no less than that of 2.

U = {(r1 , r2 ) : 0.8 ↓ r1 ↓ 1.1, 0.75 ↓ r2 ↓ 1.25, r1 → r2 }

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 41 / 44

Robust constraint

We will make a decision x such that it remains feasible for all variations of
a ↔ U, i.e. find x such that

a↭ x ↓ b ↑a ↔ U

We call such a constraint robust constraint.

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 42 / 44

Reformulation of robust constraint

a↭ x ↓ b, ↑a ↔ U = {a : Ba ↓ d}
↭
⇐ max a x ↓ b
a:Ba↗d
⇐ min d ↭y ↓ b
y :B ↭ y =x,y ↘0

 ↭
d y ↓ b
⇐ B ↭y = x


y →0

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 43 / 44

Reformulation LP

min c ↭x
s.t. a↭ x ↓ b
a ↔ U = {a : Ba ↓ d}

min c ↭x
d ↭y ↓ b
B ↭y = x
y →0

Yuang Chen (SDS/CUHK-SZ) DDA5002/CIE6010 Lecture 13 March 10, 2025 44 / 44