Combinatorial Optimization

Lecture 2: Linear Programming

Le Hai Yen, Institute of Mathematics, VAST

Master IATOM, USTH, Hanoi

2024

Lecture 2: Linear Programming 1 / 65

 Linear programming(LP)
= problem of optimizing a linear function subject to linear constraints.

Lecture 2: Linear Programming 2 / 65

 Linear programming(LP)
= problem of optimizing a linear function subject to linear constraints.

Lecture 2: Linear Programming 2 / 65

 Linear programming(LP)
= problem of optimizing a linear function subject to linear constraints.

Lecture 2: Linear Programming 2 / 65

 Linear programming(LP)
= problem of optimizing a linear function subject to linear constraints.

Lecture 2: Linear Programming 2 / 65

Table of contents

1 Linear Programming
Examples
Formulations

2 Solving LPs
2D Graphical solution
Simplex Method
LP solver

3 Duality
A Motivated Example
Definition and Construction
Weak and Strong Duality
Complementary Slackness Condition

Lecture 2: Linear Programming 3 / 65

Example 1: The Diet Problem

Menu: hamburger, chicken sandwich, fish sandwich, and deluxe

cheeseburger.
Your meal must satisfy certain percentages of the USRDA (United
States Recommended Daily Allowance) of Vitamin A, Vitamin C,
Calcium, and Iron.
Goal: consuming the fewest calories

%A %C %Cal %Iron Calories

Hambuger 4 4 10 15 250
Chicken 8 15 15 8 400
Fish 2 0 15 10 370
Cheeseburger 15 6 30 20 490
Requirements 10 10 15 15

Lecture 2: Linear Programming 4 / 65

Example 1: The Diet Problem

Menu: hamburger, chicken sandwich, fish sandwich, and deluxe

cheeseburger.
Your meal must satisfy certain percentages of the USRDA (United
States Recommended Daily Allowance) of Vitamin A, Vitamin C,
Calcium, and Iron.
Goal: consuming the fewest calories

%A %C %Cal %Iron Calories

Hambuger 4 4 10 15 250
Chicken 8 15 15 8 400
Fish 2 0 15 10 370
Cheeseburger 15 6 30 20 490
Requirements 10 10 15 15

Lecture 2: Linear Programming 4 / 65

Example 1(cont.)

For example,
if one ate the chicken and the cheeseburger, that would amount to
the percentages of 23, 21, 45, and 28, along with 890 calories.
if one ate two fifths of each of the hamburger, chicken, and
cheeseburger, this satisfies the four requirements with only 456
calories.

Question 1
Find a menu that satisfies the requirements while consuming fewer than
456 calories.

Question 2
Is it possible to consume fewer than 400 calories under these conditions?
Less than 350?

Lecture 2: Linear Programming 5 / 65

Example 1(cont.)

For example,
if one ate the chicken and the cheeseburger, that would amount to
the percentages of 23, 21, 45, and 28, along with 890 calories.
if one ate two fifths of each of the hamburger, chicken, and
cheeseburger, this satisfies the four requirements with only 456
calories.

Question 1
Find a menu that satisfies the requirements while consuming fewer than
456 calories.

Question 2
Is it possible to consume fewer than 400 calories under these conditions?
Less than 350?

Lecture 2: Linear Programming 5 / 65

Example 1(cont.)

For example,
if one ate the chicken and the cheeseburger, that would amount to
the percentages of 23, 21, 45, and 28, along with 890 calories.
if one ate two fifths of each of the hamburger, chicken, and
cheeseburger, this satisfies the four requirements with only 456
calories.

Question 1
Find a menu that satisfies the requirements while consuming fewer than
456 calories.

Question 2
Is it possible to consume fewer than 400 calories under these conditions?
Less than 350?

Lecture 2: Linear Programming 5 / 65

Example 1(cont.)

Problem formulation:
Minimize w = 250y1 + 400y2 + 370y3 + 490y4
subject to 4y1 + 8y2 + 2y3 + 15y4 ≥ 10
4y1 + 15y2 + 6y4 ≥ 10
(1)
10y1 + 15y2 + 15y3 + 30y4 ≥ 15
15y1 + 8y2 + 10y3 + 20y4 ≥ 15
y1 , y2 , y3 , y4 ≥ 0.

Lecture 2: Linear Programming 6 / 65

Example 2: The Matching Problem

The Mathematics Department has 10 courses it would like to offer during

a particular time slot. Luckily, there are 10 professors available to teach at
that time. Their qualifications are given in the following table

Lecture 2: Linear Programming 7 / 65

Problem 2 (cont.)

Let’s define a set of variables xi,j for 1 ≤ i ≤ 10, 0 ≤ j ≤ 9,


1 if professor i ends up teaching course j,
xi,j =
0 otherwise.

Problem formulation

max z = 10
P P9
c x
P9 i=1 j=0 i,j i,j
s.t.
P10 j=0 ci,j xi,j ≤ 1 ∀1 ≤ i ≤ 10
i=1 ci,j xi,j ≤ 1 ∀0 ≤ j ≤ 9
xi,j ∈ {0, 1}.

Lecture 2: Linear Programming 8 / 65

Example 3: A transportation problem

m plants, n warehouses
si : supply of the ith plant, i = 1, . . . , m
dj : demand of the jth warehouse, j = 1, . . . , n
cij : cost of transportation from the ith plant to the j warehouse.
Goal: Minimize the cost of transportation.
Variables: xij number of units to send i → j.

Lecture 2: Linear Programming 9 / 65

Example 3: A transportation problem

m X
X n
min cij xij
i=1 j=1
Xm
subject to xij = dj j = 1, . . . , n
i=1
n
X
xij = si i = 1, . . . , m
j=1
xij ≥ 0

Lecture 2: Linear Programming 10 / 65

Example 4: A production problem

A firm produces n different goods using m different raw materials.

bi : avaiable amount of the ith raw material,
aij : the required units of the ith raw material to produce the jth good,
cj : revenue of the jth good per unit.
Goal: Maximize the total revenue.
Variables: xj the amount of the jth good.

Lecture 2: Linear Programming 11 / 65

Example 4: A production problem

A firm produces n different goods using m different raw materials.

bi : avaiable amount of the ith raw material,
aij : the required units of the ith raw material to produce the jth good,
cj : revenue of the jth good per unit.
Goal: Maximize the total revenue.
Variables: xj the amount of the jth good.

Lecture 2: Linear Programming 11 / 65

Example 4: A production problem

A firm produces n different goods using m different raw materials.

bi : avaiable amount of the ith raw material,
aij : the required units of the ith raw material to produce the jth good,
cj : revenue of the jth good per unit.
Goal: Maximize the total revenue.
Variables: xj the amount of the jth good.

Lecture 2: Linear Programming 11 / 65

Example 4: A production problem

max c1 x1 + · · · + cn xn
subject to ai1 x1 + . . . ain xn ≤ bi i = 1, . . . , m,
xj ≥ 0 j = 1, . . . , n.

Lecture 2: Linear Programming 12 / 65

Example 4: Linear classification
Given n examples of objects: xi i = 1, . . . , n;
Each object belongs to one of two classes.

Hình: Data

Lecture 2: Linear Programming 13 / 65

Example 4: Linear classification

Goal: Find a linear classifier. i.e. a vector a and a scalar b such that for
any example of the first class

aT xi ≥ b,

and for any example of the second class

aT xi < b.

Lecture 2: Linear Programming 14 / 65

General LP Problems

Lecture 2: Linear Programming 15 / 65

Standard form

Lecture 2: Linear Programming 16 / 65

Reduction to standard form

Lecture 2: Linear Programming 17 / 65

Example 5

min 2x1 + 4x2

subject to x1 + x2 ≥ 3
3x1 + 2x2 = 14
x1 ≥ 0.

Standard form

max −2x1 − 4x2+ + 4x2−

subject to −x1 − x2+ + x2− ≤ −3
3x1 + 2x2− − 2x2− ≤ 14
−3x1 − 2x2− + 2x2− ≤ −14
x1 , x2+ , x2− ≥ 0.

Lecture 2: Linear Programming 18 / 65

Example 5

min 2x1 + 4x2

subject to x1 + x2 ≥ 3
3x1 + 2x2 = 14
x1 ≥ 0.

Standard form

max −2x1 − 4x2+ + 4x2−

subject to −x1 − x2+ + x2− ≤ −3
3x1 + 2x2− − 2x2− ≤ 14
−3x1 − 2x2− + 2x2− ≤ −14
x1 , x2+ , x2− ≥ 0.

Lecture 2: Linear Programming 18 / 65

Example 5

min 2x1 + 4x2

subject to x1 + x2 ≥ 3
3x1 + 2x2 = 14
x1 ≥ 0.

Standard form

max −2x1 − 4x2+ + 4x2−

subject to −x1 − x2+ + x2− ≤ −3
3x1 + 2x2− − 2x2− ≤ 14
−3x1 − 2x2− + 2x2− ≤ −14
x1 , x2+ , x2− ≥ 0.

Lecture 2: Linear Programming 18 / 65

Matrix formulation

Lecture 2: Linear Programming 19 / 65

Table of contents

1 Linear Programming
Examples
Formulations

2 Solving LPs
2D Graphical solution
Simplex Method
LP solver

3 Duality
A Motivated Example
Definition and Construction
Weak and Strong Duality
Complementary Slackness Condition

Lecture 2: Linear Programming 20 / 65

A small example

Lecture 2: Linear Programming 21 / 65

A small example

Lecture 2: Linear Programming 22 / 65

A small example

Lecture 2: Linear Programming 23 / 65

History

Lecture 2: Linear Programming 24 / 65

History

Lecture 2: Linear Programming 25 / 65

History

Lecture 2: Linear Programming 26 / 65

List of Algorithms for solving LPs

Lecture 2: Linear Programming 27 / 65

Specialities of Simplex Method

Lecture 2: Linear Programming 28 / 65

Idea of Simplex Method

Lecture 2: Linear Programming 29 / 65

Idea of Simplex Method

Lecture 2: Linear Programming 30 / 65

Idea of Simplex Method

Lecture 2: Linear Programming 31 / 65

Idea of Simplex Method

Lecture 2: Linear Programming 32 / 65

Simplex Method: Step 1
Use slack variables to transform inequality constraint to equality constraints

Use simplex algorithm to solve LP:

max z = x1 + x2
s.t. 3x1 + 5x2 ≤ 90
9x1 + 5x2 ≤ 180
x2 ≤ 15
x1 , x2 ≥ 0.

max z = x1 + x2
s.t. 3x1 + 5x2 +x3 = 90,
⇔ 9x1 + 5x2 +x4 = 180,
x2 +x5 = 15
xj ≥ 0 ∀1 ≤ j ≤ 5 .

Lecture 2: Linear Programming 33 / 65

Simplex Method: Step 2
Find an initial vertex of feasible set

Express some variables and objective function in term of the other

variables.
max z = x1 + x2 max z = x1 + x2
s.t. 3x1 + 5x2 + x3 = 90, s.t. x3 = 90 − 3x1 − 5x2 ,
9x1 + 5x2 + x4 = 180, ⇔ x4 = 180 − 9x1 − 5x2 ,
x2 + x5 = 15 x5 = 15 − x2
xj ≥ 0 ∀1 ≤ j ≤ 5. xj ≥ 0 ∀1 ≤ j ≤ 5

Basic variables: variables chosen to express (here x3 , x4 , x5 )

Non-basic variables (or parameters): the remaining variables (here
x1 , x2 )

Lecture 2: Linear Programming 34 / 65

Simplex Method: Step 2 (cont.)
Find an initial vertex of feasible set

Dictionary: formulation expressing basic variables and objective function

in term of non-basic variables
Dictionary 1 Tableau 1
x1 x2 x3 x4 x5 z b
max z = x1 + 2x2
3 5 1 0 0 0 90
s.t. x3 = 90 − 3x1 − 5x2 ,
9 5 0 1 0 0 180
x4 = 180 − 9x1 − 5x2 ,
0 1 0 0 1 0 15
x5 = 15 − x2
xj ≥ 0 ∀1 ≤ j ≤ 5 -1 -1 0 0 0 1 0

Lecture 2: Linear Programming 35 / 65

Question 3
Consider the following LP

max z = −12x1 − 11x2 − 13x3

s.t. −x1 + x2 ≤ −2,
(P) −x2 − x3 ≤ −3,
x1 + x3 ≤ 5
xj ≥ 0 ∀1 ≤ j ≤ 3.

a. Write the initial dictionary.

b. Write the initial tableau.

Lecture 2: Linear Programming 36 / 65

Simplex Method: Step 2 (cont.)
Find an initial vertex of feasible set and formulate a new dictionary

Set all non-basic variables to 0, then compute basic variables.

The obtained solution is called basic solution

(x1 , x2 , x3 , x4 , x5 ) = (0, 0, 90, 180, 15).

Let us denote by β for the set of subscripts of basic variables and π

for the set of subscripts of parameters:

β = {3, 4, 5}, π = {1, 2}.

Feasible basic solution: a basic solution with non-negative values of

basic variables.

Lecture 2: Linear Programming 37 / 65

Question 4
Consider the linear problem (P) from Question 3.
a. Find an infeasible basic solution to (P).
b. Find a feasible nonbasic solution to (P).

Lecture 2: Linear Programming 38 / 65

Step 3
Find a better vertex of feasible set and formulate a new dictionary

We need to set to zero one of the non-zero or basic variables, and

release one of the variables previously set to zero.
In order to figure out which variable should enter the basis: Look at
the objective function z = x1 + x2 , we can increase x1 .
To figure out which variable should leave the basis: Release the
constraints that limits the increase in x1 the most. Constraint 1 limits
x1 to 30, constraint 2 limits x1 to 20: eliminate Constraint 2 (i.e., the
slack variable x4 )

Lecture 2: Linear Programming 39 / 65

Simplex Method: Step 3
Find a better vertex of feasible set and formulate a new dictionary

We get
x (1) = (20, 0, 30, 0, 15)
with z (1) = 20. This produces the new basis
β(1) = {1, 3, 5},
parameter set
π(1) = {2, 4},
and dictionary below
Dictionary 2
max z = 20 + 49 x2 + 91 x4
s.t. x3 = 30 − 10 1
3 x2 + 3 x4 ,
x1 = 20 − 59 x2 − 91 x4 ,
x5 = 15 − x2
xj ≥ 0 ∀1 ≤ j ≤ 5
Lecture 2: Linear Programming 40 / 65
Simplex Method: Step 3
Find a better vertex of feasible set and formulate a new dictionary

In the tableau form:

Tableau 1
choose the column with the x1 x2 x3 x4 x5 z b
most negative entry in row (z).
3 5 1 0 0 0 90
choose the row that limits the 9 5 0 1 0 0 180
most the increase in the 0 1 0 0 1 0 15
corresponding variable. (smallest
-1 -1 0 0 0 1 0
positive ratio test.)

Lecture 2: Linear Programming 41 / 65

Simplex Method: Step 3
Find a better vertex of feasible set and formulate a new dictionary

Then use row

operations to Tableau 1
clear everything x1 x2 x3 x4 x5 z b
in the 0 10/3 1 -1/3 0 0 30
yellow-shaded 1 5/9 0 1/9 0 0 20
column, leaving 0 1 0 0 1 0 15
only a 1 in the 0 -4/9 0 1/9 0 1 20
orange-shaded
cell.

Lecture 2: Linear Programming 42 / 65

Repeat the procedures of Step 3 until
objective value cannot increase anymore (in this case the current
feasible basic solution is optimal)
or objective value tends to 1 (in this case the LP is unbounded)
Solution: z ∗ = 24 andx ∗ = (15, 9, 0, 0, 6).

Lecture 2: Linear Programming 43 / 65

Exercise

Consider the following LP

max z = 4x1 + 5x2

s.t. 14x1 + 11x2 ≤ 154
7x1 + 16x2 ≤ 112
x1 , x2 ≥ 0.

a. Draw the feasible region.

b. Write dictionary D(0), tableau T (0), basis β(0), parameters π(0),
basic solution x(0), and objective value z(0).
c. Write the first pivot operation (i  j) and the corresponding row
operations.
d. Perform the first pivot and write D(1), T (1), β(1), π(1), x(1), and
z(1).

Lecture 2: Linear Programming 44 / 65

Exercise (cont.)

e. Write the second pivot operation and the corresponding row

operations.
f. Perform the second pivot and write D(2), T (2), β(2), π(2), x(2), and
z(2).
g. Use part f to write x ∗ and z ∗ .
h. Plot the points x(k) on your drawing from part a and draw an arrow
from each x(k) to x(k + 1).

Lecture 2: Linear Programming 45 / 65

LP solver: SciPy.linprog (Python)

Colab: https://colab.research.google.com
SciPy.linprog https://docs.scipy.org/doc/scipy/reference/
generated/scipy.optimize.linprog.html

Lecture 2: Linear Programming 46 / 65

Example

Lecture 2: Linear Programming 47 / 65

Table of contents

1 Linear Programming
Examples
Formulations

2 Solving LPs
2D Graphical solution
Simplex Method
LP solver

3 Duality
A Motivated Example
Definition and Construction
Weak and Strong Duality
Complementary Slackness Condition

Lecture 2: Linear Programming 48 / 65

A Motivated Example

Lecture 2: Linear Programming 49 / 65

A Motivated Example

Lecture 2: Linear Programming 50 / 65

Dual of Standard-form LP

Lecture 2: Linear Programming 51 / 65

Dual of Dual =Primal

Lecture 2: Linear Programming 52 / 65

Dual of general-form LP

Lecture 2: Linear Programming 53 / 65

Dual of general-form LP

Lecture 2: Linear Programming 54 / 65

Weak Duality

Weak Duality
For any feasible solution x of primal problem and for any feasible solution
y of dual problem, we have

c t x ≤ bt y .

Primal problem is unbounded ⇔ dual problem is infeasible.

Dual problem is unbounded ⇔ primal problem is infeasible.
Lecture 2: Linear Programming 55 / 65
Example

Lecture 2: Linear Programming 56 / 65

Example

Lecture 2: Linear Programming 57 / 65

Strong Duality

Strong Duality
If primal problem has an optimal solution x ∗ , then dual problem has an
optimal solution y ∗ . Furthermore, we have c t x ∗ = b t y ∗

Lecture 2: Linear Programming 58 / 65

A certificate of optimality
We have
x is a feasible solution of primal problem, and
y is a feasible solution of dual problem, and
c t x = bt y ,
if and only if
x is an optimal solution of primal problem, and
y is an optimal solution of dual problem

Lecture 2: Linear Programming 59 / 65

Example

Lecture 2: Linear Programming 60 / 65

Motivation

Lecture 2: Linear Programming 61 / 65

Complementary Slackness Condition

Theorem
Let x ∗ be a feasible solution of primal problem. and y ∗ be a feasible
solution of dual problem. Then x ∗ and y ∗ are optimal solutions if and only
if
(bi − ai1 x1∗ − ai2 x2∗ − · · · − ain xn∗ )yi∗ = 0 ∀i = 1; . . . ; m;
(a1j y1∗ + a2j y2∗ + · · · + amj ym
∗
− cj )xj∗ = 0 ∀j = 1; . . . ; n
Lecture 2: Linear Programming 62 / 65
Example

Lecture 2: Linear Programming 63 / 65

Example

Lecture 2: Linear Programming 64 / 65

 Thank you for your attention!

Lecture 2: Linear Programming 65 / 65