Linear Programming

Discrete Mathematics and Optimization

Bioinformatics

(ESCI) Linear Programming 1 / 34

The diet problem
Three foods available: broccoli, milk, and oranges
Institute for Health recommends per day at least
3.7 liters water, 1.000mg calcium, 90mg vitamin C
Table listing for each food the nutrient content and cost

Food 100g Vitamin C Calcium (mg) Water (g) Cost

Broccoli 107 47 91 0.381
Whole Milk 0 276 87 0.100
Oranges 53.2 40 87 0.272

Problem: compute a minimum cost diet satisfying the requirements

(ESCI) Linear Programming 2 / 34

The diet problem
Three foods available: broccoli, milk, and oranges
Institute for Health recommends per day at least
3.7 liters water, 1.000mg calcium, 90mg vitamin C
Table listing for each food the nutrient content and cost

Food 100g Vitamin C Calcium (mg) Water (g) Cost

Broccoli 107 47 91 0.381
Whole Milk 0 276 87 0.100
Oranges 53.2 40 87 0.272

Problem: compute a minimum cost diet satisfying the requirements

x1 , x2 , x3 amount of units of each food.
Minimize 0.381x1 + 0.100x2 + 0.272x3 subject to:

107x1 + 53.2x2 ≥ 90
47x1 + 276x2 + 40x3 ≥ 1000
91x1 + 87x2 + 87x3 ≥ 3700

(ESCI) Linear Programming 2 / 34

The Diet Problem

n type of food items F1 , . . . , Fn

Each unit of Fj has a cost of cj
m nutrients (vitamin A, proteins, etc.) N1 , . . . , Nm
aij = amount of nutrient Ni in a unit of food Fj .
A healthy diet should contain at least bi quantity of nutrient Ni

Goal:
Design a diet of minimum cost satisfying health requirements

(ESCI) Linear Programming 3 / 34

Formalisation of a linear programing problem

In order to model it as a linear program one has to identify

1 Decision variables: xi = number of units of Fi in the diet
2 Explicit constraints:
ai1 x1 + · · · + ain xn ≥ bi
where the left-hand side is the amount of Ni in the diet
3 Implicit constraints: xi ≥ 0

LP problem
Minimize cT x (cost of the diet)
Subject to Ax ≥ b (health requirements)
x ≥0 (non-negative amount of amount of food)

(ESCI) Linear Programming 4 / 34

What is Linear Programming?
Optimizing a linear function subject to linear constraints.
n = number of variables
m = number of constraints
x = (x1 , . . . , xn ) is the vector of variables
A linear program in standard form is of the form
Maximize c 1 x1 + · · · + c n xn
Subject to a11 x1 + · · · + a1n xn ≤ b1
..
.
am1 x1 + · · · + amn xn ≤ bm
x1 , . . . , xn ≥ 0
Alternatively, it can be of the form
Minimize c 1 x1 + · · · + c n xn
Subject to a11 x1 + · · · + a1n xn ≥ b1
..
.
am1 x1 + · · · + amn xn ≥ bm
x1 , . . . , xn ≥ 0
(ESCI) Linear Programming 5 / 34
Examples. A simple example
Maximize 2x + 2y
Subject to 2x + 3y ≤ 12
3x + y ≤ 10
x + y ≤ 4.5
x, y ≥ 0
To visualize the regions: www.desmos.com/calculator

(ESCI) Linear Programming 6 / 34

Examples. A simple example
Maximize 2x + 2y
Subject to 2x + 3y ≤ 12
3x + y ≤ 10
x + y ≤ 4.5
x, y ≥ 0
To visualize the regions: www.desmos.com/calculator

(ESCI) Linear Programming 6 / 34

Examples. A simple example
Maximize 2x + 2y
Subject to 2x + 3y ≤ 12
3x + y ≤ 10
x + y ≤ 4.5
x, y ≥ 0
To visualize the regions: www.desmos.com/calculator

(ESCI) Linear Programming 6 / 34

Examples. A simple example
Maximize 2x + 2y
Subject to 2x + 3y ≤ 12
3x + y ≤ 10
x + y ≤ 4.5
x, y ≥ 0
To visualize the regions: www.desmos.com/calculator

(ESCI) Linear Programming 6 / 34

Examples. A simple example
Maximize 2x + 2y
Subject to 2x + 3y ≤ 12
3x + y ≤ 10
x + y ≤ 4.5
x, y ≥ 0
To visualize the regions: www.desmos.com/calculator

(ESCI) Linear Programming 6 / 34

Examples. Data fitting: least absolute deviations
n experimental data (s1 , t1 ), . . . , (sn , tn )
want to find the line ax + b that fits best the data.

In this case the criterion is

n
X
Minimize D(a, b) = |asi + b − ti |.
i=1

This alternative to least squares is more resistant to outliers, but the function to
be minimized is not differentiable.

(ESCI) Linear Programming 7 / 34

Examples. Data fitting: least absolute deviations
In this case the criterion is
n
X
Minimize D(a, b) = |asi + b − ti |.
i=1

D is not linear (nor differentiable,) but. . .

ei = |asi + b − ti | = error for estimating the i-th point:

−ei ≤ asi + b − ti ≤ ei

Minimize e1 + · · · + en (sum of absolute errors)

Subject to ei ≥ asi + b − ti i = 1, . . . , n
ei ≥ −(asi + b − ti ) i = 1, . . . , n
Not in standard form
Variables ei are auxiliary, we only need a and b
Remark. Objective functions or constraints involving absolute values can often be
handled by LP methods by introducing extra variables and/or extra constraints.

(ESCI) Linear Programming 8 / 34

Examples. Flow in a network

Send flow through a network from a source to a target with constraints on the
edges.

Maximize xSA + xSC (flow from S

Subject to 0 ≤ xSA ≤ 10, 0 ≤ xSC ≤ 14, 0 ≤ xAB ≤ 5, 0 ≤ xAT ≤ 5
0 ≤ xCB ≤ 13, 0 ≤ xCD ≤ 10, xBT ≤ 8, 0 ≤ xDT ≤ 12
xSA = xAB + xAT , xBT = xAB + xCB ,
xSC = xCB + xCD , xCD = xDT

(ESCI) Linear Programming 9 / 34

Equational form and basic feasible solutions
Linear programs in equational form

Maximize cT x
Subject to Ax = b
x ≥0

A is m × n of rank m ≤ n,
x is an n-column vector, and
b is an m-column vector.
From a standard LP to an LP in equational form

Introduce slack variables. For each inequality

ai1 x1 + · · · + ain xn ≤ bi

introduce a new variable yi ≥ 0 such that

ai1 x1 + · · · + ain xn + yi = bi .

(ESCI) Linear Programming 10 / 34

Basic feasible solutions

Maximize cT x
Subject to Ax = b
x ≥0

Notation: AB = matrix of columns of A indexed by the subset B ⊂ [n]

A basic feasible solution of the linear program is x such that, for some m–element
subset B ⊂ [n],
x is a feasible solution,
xi = 0 for i ∈
/ B, and
AB is nonsingular

Variables xj with j ∈ B are basic. The remaining variables are nonbasic.

A basic feasible solution is uniquely determined by B.

(ESCI) Linear Programming 11 / 34

A geometric view
The feasible set of a linear program is a polyhedron P in Rn , bounded or not.

A vertex (a tip, a corner) of P is a point v ∈ P, such that there is a hyperplane H

such that v is precisely the intersection of P and H.

Theorem
Let P be the set of feasible solutions of a linear program in equational form. Then
v is a vertex of the polyhedron P if and only if v is a basic feasible solution of the
linear program.

(ESCI) Linear Programming 12 / 34

A geometric view

The feasible set of a linear program is a polyhedron P in Rn , bounded or not.

Theorem
Let P be the set of feasible solutions of a linear program in equational form. Then
v is a vertex of the polyhedron P if and only if v is a basic feasible solution of the
linear program.

Theorem
An optimal solution of a linear program in equational form is achieved by a basic
feasible solution, that is, in a vertex of the polyhedron P of feasible solutions.

This provides a (nonefficient) algorithm to find a solution to an LP problem:

check the values of the objective function on vertices of the polyhedron of feasible
solutions

(ESCI) Linear Programming 13 / 34

The simplex method
Input: a linear program in equational form
Maximize cT x
Subject to Ax = b
x ≥0
Output: a solution x where the objective function is maximum or a certificate of
nonexistence (unboundedness).

Description of the algorithm:

1. Find a basic feasible solution
2. Move to a neighboring basic feasible solution while increasing the value of the
objective function
3. When no further increase is possible, the optimum is reached.

(ESCI) Linear Programming 14 / 34

The simplex method
Input: a linear program in equational form
Maximize cT x
Subject to Ax = b
x ≥0
Output: a solution x where the objective function is maximum or a certificate of
nonexistence (unboundedness).

(ESCI) Linear Programming 14 / 34

An example

Maximize x1 + x2
Subject to −x1 + x2 ≤ 1
x1 ≤ 3
x2 ≤ 2
x1 , x2 ≥ 0

Step 0: Put the problem in equational form.

Introduce Slack variables x3 , x4 , x5

Maximize x1 +x2
Subject to −x1 +x2 +x3 =1
x1 +x4 =3
x2 +x5 =2
x1 , x2 , x3 , x4 , x5 ≥0

(ESCI) Linear Programming 15 / 34

An example (II)

Maximize x1 +x2
Subject to −x1 +x2 +x3 =1
x1 +x4 =3
x2 +x5 =2
x1 , x2 , x3 , x4 , x5 ≥0

Step 1a: Identify a basic feasible condition:

▶ (0, 0, 1, 3, 2) is a basic feasible solution.
▶ x3 , x4 , x5 basic variables, x1 , x2 nonbasic
Step 1b: Organize the input in a tableau (or dictionary) with these data:

x3 =1 +x1 −x2 eq.1

x4 =3 −x1 eq.2
(0, 0, 1, 3, 2) basic feasible solution.
x5 =2 −x2 eq.3
z = x1 +x2 z =0

(ESCI) Linear Programming 16 / 34

An example (III)

x3 =1 +x1 −x2 eq.1

x4 =3 −x1 eq.2
(0, 0, 1, 3, 2) basic feasible solution.
x5 =2 −x2 eq.3
z = x1 +x2 z =0

Step 2: Identify entering and leaving variables: pivot step

▶ entering: one with positive coefficient in z (increasing its value increases the
objective function).
Two choices, x1 , x2 . Choose x2 entering, for instance.
▶ leaving: the one which restricts most the increasing of the entering variable.

x3 ≥ 0 →x2 ≤ 1
x4 ≥ 0 →x2 < ∞
x5 ≥ 0 →x2 ≤ 2

x3 leaving

(ESCI) Linear Programming 17 / 34

An example (IV)

Step 3: Increase the entering variable as much as possible (x2 = 1), update
the tableau

x3 =1 +x1 −x2 x2 =1 +x1 −x3

x4 =3 −x1 x4 =3 −x1
→
x5 =2 −x2 x5 =1 −x1 +x3
z =0 +x1 +x2 z =0 z =1 +2x1 −x3 z =1
(0, 1, 0, 3, 1) basic feasible solution.

(ESCI) Linear Programming 18 / 34

An example (V)
Step 4: Repeat Step 2 and Step 3 while possible.
▶ x1 entering, x5 leaving. Maximum increase x1 = 1

x2 =1 +x1 −x3 x1 = 1 +x3 −x5

x4 =3 −x1 x2 =2 −x5
→
x5 =1 −x1 +x3 x4 = 2 −x3 +x5
z =1 +2x1 −x3 z =1 z = 3 +x3 −2x5 z = 3
(1, 2, 0, 2, 0) basic feasible solution.

▶ x3 entering, x4 leaving. Maximum increase x1 = 1

x1 =1 +x3 −x5 x1 =3 −x4

x2 =2 −x5 x2 =2 −x5
→
x4 =2 −x3 +x5 x3 = 2 −x4 +x5
z =3 +x3 −2x5 z =3 z = 5 −x4 −x5 z = 5
(3, 2, 2, 0, 0) basic feasible solution.

Return x1 = 3, x2 = 2 and z = 5.

(ESCI) Linear Programming 19 / 34

An example (VI): a geometric illustration

Maximize x1 +x2
Subject to −x1 +x2 +x3 =1
x1 +x4 =3
x2 +x5 =2
x1 , x2 , x3 , x4 , x5 ≥0

x1 ≥ 0

x1 ≤ 3

x2 ≤ 2

x2 ≥ 0

(ESCI) Linear Programming 20 / 34

Outline of the simplex algorithm

0 Put the problem in equational form.

1 Find a basic feasible solution. Initialize the Tableau.
2 Pivot step: Select leaving and entering variables.
3 Increase entering variable. Update Tableau and current basic feasible solution.
4 Repeat steps 2 and 3 while possible.
5 Output solution.

(ESCI) Linear Programming 21 / 34

The example again

Initialize x2 in; x3 out

x3 =1 +x1 −x2 x2 =1 +x1 −x3

x4 =3 −x1 x4 =3 −x1
x5 =2 −x2 x5 =1 −x1 +x3
z =0 +x1 +x2 z =0 z =1 +2x1 −x3 z = 1
(0, 0, 1, 3, 2) (0, 1, 0, 3, 1)

x1 in; x5 out x3 in ; x4 out

x1 =1 +x3 −x5 x1 =3 −x4

x2 =2 −x5 x2 =2 −x5
x4 =2 −x3 +x5 x3 =2 −x4 +x5
z =3 +x3 −2x5 z =3 z =5 −x4 −x5 z = 5
(1, 2, 0, 2, 0) (3, 2, 2, 0, 0)

(ESCI) Linear Programming 22 / 34

Outline of the simplex algorithm

0 Put the problem in equational form.

1 Find a basic feasible solution. Initialize the Tableau.

2 Pivot step: Select leaving and entering variables.

3 Increase entering variable. Update Tableau and current basic feasible solution.

4 Repeat steps 2 and 3 while possible.

5 Output solution.

(ESCI) Linear Programming 23 / 34

Outline of the simplex algorithm

Handling troubles
0 Put the problem in equational form.
1 Find a basic feasible solution. Initialize the Tableau.
Infeasibility: no solutions?
2 Pivot step: Select leaving and entering variables.
Rules of choice
3 Increase entering variable. Update Tableau and current basic feasible solution.
Degeneracy: what if no choice increases objective function?
4 Repeat steps 2 and 3 while possible.
Unboundeedness: what if no maximum exists?
5 Output solution.

(ESCI) Linear Programming 23 / 34

Exception handling (I)

Unboundedness

x1 = 1 +x2 −x3
x4 =3 −x3
z = 1 +x2 −x3

Entering variable x2 : no variable restricts its increase.

For any t ≥ 0, (1 + t, t, 0, 3) is feasible solution with value z = 1 + t.

Output: No maximum for the objective function (or maximum is infinity).

(ESCI) Linear Programming 24 / 34

Exception handling (I)

Unfeasibility

Maximize c T x
Subject to Ax ≤ b, x ≥ 0

If b ≥ 0, introduction of m slack variables always allows for an initial basic feasible

solution.

But:

Maximize c T x
Subject to Ax = b, x ≥ 0

(ESCI) Linear Programming 25 / 34

Exception handling (II)
Infeasibility

Maximize c T x
Subject to Ax = b, x ≥ 0

Assume all bi ≥ 0
Introduce an auxiliary variable yi for each equation.
(if we set all variables to 0, by how much they fail to satisfy the equation.)
Write a Linear Programming problem for maximization of

−y1 − y2 · · · − ym

▶ If maximum value is zero, then the values of the variables x1 , . . . , xn provide a

basic feasible solution to start the original problem.
▶ Otherwise the program is infeasible.

(ESCI) Linear Programming 26 / 34

Exception handling (II)
Unfeasibility: Example

Maximize x1 + 2x2
Subject to x1 + 3x2 + x3 = 4
2x2 + x3 = 2
x1 , x2 ≥ 0

Introduce auxiliary variables x4 and x5 and write new Linear Program problem:

Maximize − x4 − x5
Subject to x1 + 3x2 + x3 + x4 =4
2x2 + x3 + x5 = 2
x1 , x2 x3 , x4 , x5 ≥ 0

(ESCI) Linear Programming 27 / 34

Exception handling (II)
Unfeasibility: Example

Maximize x1 + 2x2
Subject to x1 + 3x2 + x3 = 4
2x2 + x3 = 2
x1 , x2 ≥ 0

Basic feasible solution (0, 0, 0, 4, 2). Write Tableau

x4 = 4 − x1 − 3x2 − x3
x5 = 2 − 2x2 − x3
z = −6 + x1 + 5x2 + 2x3

(ESCI) Linear Programming 27 / 34

Exception handling (II)
Unfeasibility: Example

Maximize x1 + 2x2
Subject to x1 + 3x2 + x3 = 4
2x2 + x3 = 2
x1 , x2 ≥ 0

Start pivot steps

▶ x1 in x4 out

x1 = 4 − 3x2 − x3 − x4
x5 = 2 − 2x2 − x3
z = −2 + 2x2 + x3 − x4

▶ x3 in x5 out

x1 = 2 − x2 − x4 + x5
x3 = 2 − 2x2 − x5
z = 0 − x4 − x5

(ESCI) Linear Programming 27 / 34

Exception handling (II)
Unfeasibility: Example

Maximize x1 + 2x2
Subject to x1 + 3x2 + x3 = 4
2x2 + x3 = 2
x1 , x2 ≥ 0

Basic feasible solution (2, 0, 2, 0, 0), z = 0

x1 = 2 − x2 − x4 + x5
x3 = 2 − 2x2 − x5
z = 0 − x4 − x5

Basic feasible solution for initial problem (2, 0, 2)

(ESCI) Linear Programming 27 / 34

Exception handling (III)
Degeneracy
In pivot step no increase of entering variable is possible
Allow for some zero increase of the objective function
Example

Maximize x2
Subject to −x1 + x2 ≤ 0
x1 ≤ 2
x1 , x2 ≥ 0
Tableau:
x3 = +x1 −x2
x4 = 2 −x1
z = +x2
x2 enters and x3 leaves, but the value of x2 can not be increased

(ESCI) Linear Programming 28 / 34

Exception handling (III)
Degeneracy
In pivot step no increase of entering variable is possible
Allow for some zero increase of the objective function
Example

Maximize x2
Subject to −x1 + x2 ≤ 0
x1 ≤ 2
x1 , x2 ≥ 0

Degenerate pivot step x2 in x3 out. Basic feasible solution (0, 0, 0, 2)

x2 = +x1 −x3
x4 = 2 −x1
z = x1 −x3

(ESCI) Linear Programming 28 / 34

Exception handling (III)
Degeneracy
In pivot step no increase of entering variable is possible
Allow for some zero increase of the objective function
Example

Maximize x2
Subject to −x1 + x2 ≤ 0
x1 ≤ 2
x1 , x2 ≥ 0

x1 in and x4 out. Basic feasible solution (2, 2, 0, 0)

x1 =2 −x4
x2 = 2 −x3 −x4
z = 2 −x3 −x4

Otput: (2, 2), z = 2.

(ESCI) Linear Programming 28 / 34

Exception handling (III)
Degeneracy
In pivot step no increase of entering variable is possible
Allow for some zero increase of the objective function
Example

Maximize x2
Subject to −x1 + x2 ≤ 0
x1 ≤ 2
x1 , x2 ≥ 0

It may happen that there are more than one degenerate pivot steps.
It may happen that the degenerate pivot steps cycle to some previous point.

(ESCI) Linear Programming 28 / 34

Pivot rules

More than one possibility for the entering variable

Largest Coefficient. Choose variable with largest coefficient in z
Largest Increase. Largest absolute improvement of z
Steepest edge. Moves current feasible solution in a direction closest to c.
Bland’s rule. Entering variable with smallest index. Same rule for leaving
variable

Theorem The simplex method with Bland’s rule is always finite (cycling is
impossible).

(ESCI) Linear Programming 29 / 34

Efficiency of the simplex method

The simplex algorithm ranges among the most frequently used in real life.

(ESCI) Linear Programming 30 / 34

Efficiency of the simplex method
The simplex algorithm ranges among the most frequently used in real life.
In some examples it may need exponential running time (Klee-Minty
examples, 1972)

(ESCI) Linear Programming 30 / 34

Efficiency of the simplex method

The simplex algorithm ranges among the most frequently used in real life.
In some examples it may need exponential running time (Klee-Minty
examples, 1972)
In practice it performs very satisfactorily: typically between 2m and 3m steps
to reach the optimum.

(ESCI) Linear Programming 30 / 34

Efficiency of the simplex method

The simplex algorithm ranges among the most frequently used in real life.
In some examples it may need exponential running time (Klee-Minty
examples, 1972)
In practice it performs very satisfactorily: typically between 2m and 3m steps
to reach the optimum.
It has been proved that small alterations always brings polynomial time
problems (Spielman-Teng, 2001).

(ESCI) Linear Programming 30 / 34

Efficiency of the simplex method

The simplex algorithm ranges among the most frequently used in real life.
In some examples it may need exponential running time (Klee-Minty
examples, 1972)
In practice it performs very satisfactorily: typically between 2m and 3m steps
to reach the optimum.
It has been proved that small alterations always brings polynomial time
problems (Spielman-Teng, 2001).
Most mathematical software systems (Maple, Matlab, Mathematica,...) have
packages implementing the simplex algorithm.
In Python the library scipy.optimize.linprog has several implemented
algorithms for LP problems
https://docs.scipy.org/doc/scipy/reference/generated/scipy.
optimize.linprog.html

(ESCI) Linear Programming 30 / 34

Summary of the simplex method
1. Convert the input linear program to equational form

maximize c T x subject to Ax = b and x ≥ 0

2. If no feasible basis is available, arrange for b ≥ 0, and solve

Maximize −(xn+1 + · · · + xn+m ) x = (x1 , . . . , xn+m )
Subject to Ax = b A = (A | Im )
x ≥0
If optimal value negative, the program is infeasible; stop.
Otherwise, the first n components of the optimal solution form a basic feasible
solution of the original linear program.
3. For a feasible basis B ⊆ {1, 2, . . . , n} compute

XB =p + QXN
z = z0 + r T XN

4. If r ≤ 0,then return (p, 0) as optimal solution; stop.

(ESCI) Linear Programming 31 / 34

 XB =p + Qx
z = z0 + r T Xn

4. If r ≤ 0,then return (p, 0) as optimal solution; stop.

5. Else select entering variable xv with positive coefficient in r
6. If the column of the entering variable xv is nonnegative, the program is
unbounded; stop.
7. Else select a leaving variable xu .
In all rows where the coefficient of xv is negative, divide the component of the
vector p by that coefficient and change sign.
Choose the row with minimal ratio.
8. Replace the current basis B by B − u + v .
Update the tableau. Go to Step 4.

(ESCI) Linear Programming 32 / 34

Simplex in Python
Optimization library in scipy:
docs.scipy.org/doc/scipy/reference/optimize.linprog-simplex.html

(ESCI) Linear Programming 33 / 34

Simplex in Python
Optimization library in scipy:
docs.scipy.org/doc/scipy/reference/optimize.linprog-simplex.html

(ESCI) Linear Programming 33 / 34

Concluding remarks
The Simplex method was introduced by George Dantzig around 1947.

(ESCI) Linear Programming 34 / 34

Concluding remarks

The Simplex method was introduced by George Dantzig around 1947.

Implementing the simplex method is far from being trivial.
Pivot operations require computing the inverse of AB . This is a costly an
unstable computation and requires advanced techniques from matrix theory,
such as LU decompositions.

(ESCI) Linear Programming 34 / 34

Concluding remarks

The Simplex method was introduced by George Dantzig around 1947.

Implementing the simplex method is far from being trivial.
Pivot operations require computing the inverse of AB . This is a costly an
unstable computation and requires advanced techniques from matrix theory,
such as LU decompositions.
There is a wide range of applications in bioinformatics of Linear
Programming (and Integer Linear Programming), including sequence analysis,
protein structure, haplotyping, microarrays, evolutionary distances...
https://www.researchgate.net/publication/26552925_
Mathematical_Programming_in_Computational_Biology_an_
Annotated_Bibliography

(ESCI) Linear Programming 34 / 34