Linear and Integer Programming

Lecture Notes

Marco Chiarandini

June 18, 2015

2
Contents

1 Introduction 5
1.1 Operations Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Mathematical Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Resource Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3.1 Mathematical model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3.2 General Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.3 Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Diet Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.5 The Mathematical Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.5.1 Solving LP Models in Practice . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.6 A Brief History of Linear Programming (LP) . . . . . . . . . . . . . . . . . . . . . . 15
1.7 Fourier Motzkin elimination method . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2 The Simplex Method 19

2.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.1.1 Linear Programming Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.1.2 Fundamental Theorem of LP . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2 Systems of Linear Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3 Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4 Exception Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.4.1 Unboundedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.4.2 Infinite solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.4.3 Degeneracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.4.4 Pivot Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.4.5 Efficiency of simplex method . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.5 Infeasibility and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3 Duality 49
3.1 Derivation and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.1.1 Bounding approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.1.2 Geometric Interpretation of Duality . . . . . . . . . . . . . . . . . . . . . . . 50
3.1.3 Multipliers Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.1.4 Duality Recipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.2 Duality Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.3 Lagrangian Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.4 Dual Simplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.5 Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

3
4 CONTENTS

3.6 Farkas Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

3.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4 Revised Simplex Method 71

4.1 Efficiency Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.2 More on Polyhedra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.3 More on LP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.3.1 LP: Rational Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.3.2 Interior Point Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.3.3 Further topics in LP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

5 Modeling in Mixed Integer Linear Programming 83

5.1 Introduction to Integer Linear Programming . . . . . . . . . . . . . . . . . . . . . . . 83
5.1.1 Combinatorial Optimization Problems . . . . . . . . . . . . . . . . . . . . . . 84
5.1.2 Solution Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.2 MILP Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.2.1 Assignment Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.2.2 Knapsack Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.2.3 Set Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.2.4 Graph Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.3 Modeling Tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.4 Formulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.4.1 Alternative Formulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

6 Well Solved Problems 99

6.1 Relaxations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.2 Well Solved Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
6.2.1 Separation problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
6.3 Totally Unimodular Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

7 Network Flows 105

8 Cutting Plane Algorithms 115

8.1 Valid Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
8.2 Cutting Plane Algorithms for Integer Programming . . . . . . . . . . . . . . . . . . . 116
8.2.1 Chvátal-Gomory cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
8.2.2 Cutting Plane Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
8.2.3 Gomory’s fractional cutting plane algorithm . . . . . . . . . . . . . . . . . . . 116

9 Branch and Bound 129

Chapter 1

Introduction

1.1 Operations Research

Operations Research (aka, Management Science, Analytics): is the discipline that uses a scientific
approach to decision making. It seeks to determine how best to design and operate a system, usu-
ally under conditions requiring the allocation of scarce resources, by means of quantitative methods.
The solution approaches lay at the intersection between the fields of mathematics, computer science
and statistics. It encompasses a wide range of problem-solving techniques and methods applied in the
pursuit of improved decision-making and efficiency: simulation, mathematical optimization, queuing
theory and other stochastic-process models, Markov decision processes econometric methods, data
envelopment analysis, neural networks, expert systems, decision analysis, and the analytic hierarchy
process.
Some example within mathematical optimization:

• Production Planning and Inventory Control: planning the issue of orders for refilling ware-
houses avoiding stock out and satisfying space capacity limits.
• Budget Investment, given a budget and a number of projects, each with its own foreseen return
and cost of resources, find the projects to fund that would maximize the profit.
• Blending and Refining in the chemical industry
• Energy planning, for example, deciding when to activate cogeneration plant in order to meet
the forecast demand of heat and electricity in the next few hours.
• Manpower Planning, for example scheduling the shifts of a group of nurses such that a depart-
ment of an hospital is manned 24 hours a day with exactly 3 nurses and working agreements
are respected. Or in the airline and railways industries, rostering crews such that geographical
locations and working agreements are satisfied.
• Packing Problems, for example, filling containers with 3D packs without exceeding capacity
and minimizing the free space.
• Cutting Problems: for example, in textile or paper industry. Cutting a paper roll in pieces to
accommodate journals of different sizes while minimizing the waste.
• Routing, for example, in logistics, delivering products (oil, beer, food, etc.) to customers or
retailers such that the total traveling distance is minimized and the capacity of the vehicle
satisfied.

5
6 CHAPTER 1. INTRODUCTION

• Location Decisions, for example, deciding where to open a set of warehouses having to ensure
a satisfactory coverage of a number of retailers.

• Scheduling/Timetabling, for example, in the manufacturing industry, schedule the sequences

jobs in an assembly line or in education, planning courses such that no two courses sharing
students have overlap in time and a number of side constraints are satisfied. The need for
timetables arises also in public transportation.

In all these contexts planning decisions must be made that relate to quantitative issues. For
example, fewest number of people, shortest route. On the other side not all plans are feasible –
there are constraining rules. Moreover there is a limited amount of available resources. It can be
extremely difficult to figure out what to do.
In Figure 1.1, we depict a common scheme of the solution process in applied optimization.

Algorithms
Modeling
(simplex, b&b)
variables,
& Solvers Solution
constraints,
(Gurobi,
obj. func.
CPLEX)

Problem Decision
Crew Scheduling

First we observe the real life system and interview the persons involved to understand the prob-
lem. We then write a problem description in clear, plain English. This is useful to get back to
the comissioners and ensure that there are no misunderstandings. You should challenge your de-
scription by presenting cases that are not valid for the real life situation but that would be allowed
by your description. This procedure is helpful to make the description precise and less prone to
misinterpretations. Then we are ready to introduce mathematical notation, which makes almost
impossible to misinterpret your model and removes all sources of disturbance. The real life objects
are abstracted to sets, graph, networks or other mathematical concepts. Then the model made by
known parameters, unknowns, objectives and constraints is forumlated. Any word description is
at this point removed. Finally, the model is solved on some test data and the solution interpreted
and crosschecked with respect to reality. The central idea in this process is to build a mathematical
model describing exactly what one wants, and what the “rules of the game” are.

1.2 Mathematical Modeling

The first step is to find out exactly what the decision maker needs to know: For example: which
investment? which product mix? which job j should a person i do? For each decision define a
decision variable of suitable type (continuous, integer valued, binary) according to the needs. For
example, the decision whether to send a vehicle through an arc of a network or not can be modelled
by a decision variable that can take only 0 and 1 values, that is, a binary decision variable. Clearly
in this context a value of 0.5 would not have any meaning in the physical world that we are trying
to model. Next, identify the input constants for the model. These are called parametrs and are
values that are known and fixed once the specific instance of the problem is known. The next step
is formulating the objective bunction to compute the benefit/cost in terms of decision variables and
1.3. RESOURCE ALLOCATION 7

parameters. Finally, the constraints indicating the interplay between the different variables must be
expressed in mathematical terms.

1.3 Resource Allocation

In the manufacturing industry, a common decision to take is which product mix to set in production.
It is known as the factory planning problem. Suppose a factory makes two types of yogurts, a Normal
one with a medium fermentation time and bacterial culture added and another one, Long, with a
long fermentation and bacterial culture added. One liter of Normal gives a profit of 6 Dkk while
one liter of Long gives 8 Dkk.
To produce the desired quantity of yogurt two processes are required. The heating of milk at 45
degrees to denaturate its proteins and allow fermentation and the addition of bacterial culture. For
each liter of yogurt produced the amount of hours for the fermentation and the amount og bacterial
cultures in decigrams are given below:
Normal Lite
Fermentation 5 10
Bacterial culture 4 4
The company has the following capacity for the fermentation given by the amount of hours in
which the heating is possible because manned. Heating capacity: 60 hours per week Bacterial culture
capacity: 40 decigrams per week
Q: How much of each type of yogurt, Normal and Long, should the company produce to maximize
the profit?

1.3.1 Mathematical model

Decision Variables
x1 ≥ 0 liters of product Normal
x2 ≥ 0 liters of product Long
Object Function
max 6x1 + 8x2 maximize profit
Constraints
5x1 + 10x2 ≤ 60 heating capacity
4x1 + 4x2 ≤ 40 bacterial culture capacity
Calling processes heating and bacterial culture as A and B. Yogurt types as products 1 and 2.

max 6x1 + 8x2

5x1 + 10x2 ≤ 60
4x1 + 4x2 ≤ 40
x1 ≥ 0
x2 ≥ 0
aij 1 2 bi
A 5 10 60
B 4 4 40
cj 6 8
Graphical Representation:
8 CHAPTER 1. INTRODUCTION

x2

5x1 + 10x2 ≤ 60
x1

6x1 + 8x2 = 16 4x1 + 4x2 ≤ 40

1.3.2 General Model

First we introduce the notation. Let be J = 1, 2, . . . , m indexed by j the set of products and let
I = 1, 2, . . . , n indexed by i be the set of raw materials. For each raw material i ∈ I the factory
has bi units at disposal. To produce a unit product of product j ∈ J aij units of raw material j
are needed. The values aij are called technological coefficients. They determine how much resources
are needed to produce a given product given the current technology. The prevailing market value
of the raw material i is ρi and a unit of the jth product can be sold at the market price σj . The
return from the sell of the items produced is called revenue. The profit is the difference between the
revenue and the total expenses due to the production, ie:

profit = revenue − expenses

P
The profit cj derived from the sell of a unit of product j is given by cj = σj − ni=1 ρi aij .
All the values introduced so far are given and they value for an instance of the problem is fixed.
These are the parameters of data of the problem. We set out to determine the mix of products to
produce. This is equivalent to decide the amount of each product to set in production. We denote
the amount of product j but xj . Since the value of xj , j ∈ I is unknown initially, we call xj the
variables of our problem. Note that a negative value for xj would not have a meaning, hence we
know at least that xj ≥ 0 for all j in J.
We are thus ready to write a general mathematical model for the factory planning problem,
which looks as follows.

max c1 x1 + c2 x2 + c3 x3 + . . . + cn xn = z
subject to a11 x1 + a12 x2 + a13 x3 + . . . + a1n xn ≤ b1
a21 x1 + a22 x2 + a23 x3 + . . . + a2n xn ≤ b2
.. .. ..
. . .
am1 x1 + am2 x2 + am3 x3 + . . . + amn xn ≤ bm
x1 , x2 , . . . , xn ≥ 0

The words “subject to” are often abbreviated to “s.t.”. More concisely the model can be written
in scalar form as:
Pn
max cj xj
j=1
P
n
aij xj ≤ bi , i = 1, . . . , m
j=1
xj ≥ 0, j = 1, . . . , n
1.3. RESOURCE ALLOCATION 9

Further, defining the following vectors and matrices, x ∈ Rn , c ∈ Rn , A ∈ Rm×n , b ∈ Rm :

       
c1 a11 a12 . . . a1n x1 b1
 c2  a21 a22 . . . a2n   x2   b2 
       
c =  . , A= . .. .. ..  , x =  . , b= . 
 ..   .. . . .   ..   .. 
cn a31 a32 . . . amn xn bm

our LP model can then be rewritten in matrix form as:

max z = cT x
Ax ≤ b
x ≥ 0

For our numerical example:

P
n
max cj xj max 6x1 + 8x2
j=1
P
n 5x1 + 10x2 ≤ 60
aij xj ≤ bi , i = 1, . . . , m 4x1 + 4x2 ≤ 40
j=1
x1 , x2 ≥ 0
xj ≥ 0, j = 1, . . . , n
 
  x1
max cT x max 6 8
x2
Ax ≤ b
x ≥ 0     
5 10 x1 60
≤
4 4 x2 40
x ∈ Rn , c ∈ Rn , A ∈ Rm×n , b ∈ Rm
x1 , x2 ≥ 0

1.3.3 Duality
Above we saw the factory planning problem from the perspective of the company owing the raw
materials. We assumed that it was convenient for the company to produce and sell products.
However, a plausible alternative would be to close the factory and sell the raw material to the market.
What would be the price of the raw material such that this becomes feasible and attractive? To
answer this question we have to solve a resource valuation problem. Let’s take the point of view of
an outside company who has to make an offer for buying the raw materials.
From this standpoint the unknowns that are to be determined are the values of a unit of raw
material i, which we indicate by zi , for i = 1, 2, . . . , m. These values
P are the variables of the problem.
The total expenses for buying the raw materials are given by m i=1 bi zi . The buying company is
interested in minimizing precisely this value. From the perspective of the owing company this value
is known as the opportunity cost of owning the raw material. It is the value that could be obtained
by the opportunity of selling all the raw material and closing the factory. The owning company wants
to minimize the lost opportunity cost with respect to producing and selling the products. The value
zi has to be larger than the prevailing unit market value of material i, ρi , otherwise the price would
contradict the market and the owning company would prefer selling to someone else. Similarly, for
each single product j ∈ J the opportunity cost derived from producing a unit of product has to be
larger than the unitary price of σj of the product. If this was not true, then the owning company
would not sell the raw material but rather use it to produce the product and sell that one instead.
10 CHAPTER 1. INTRODUCTION

We can therefore write the model for the resource valuation problem as follows:
m
X
min bi zi (1.1)
i=1
Xm
zi aij ≥ σj , j = 1...n (1.2)
i=1
zi ≥ ρi , i = 1...m (1.3)

Constraints 1.2 and 1.3 ensure that we are not contradicting the market while the objective 1.1 aims
at making the deal appealing for the buying company.
Let yi = zi − ρi be the markup that the owing company would make by reselling the raw material
at the price zi with respect to the price ρi at which it bought it. We can then rewrite the model
above as:

m
X X
min yi bi + ρi bi (1.4)
i=1 i
Xm
yi aij ≥ cj , j = 1...n (1.5)
i=1
yi ≥ 0, i = 1...m (1.6)
P
where in the objective function the term i ρi bi is always constant and does not impact the solution.
The problem we wrote is known as the dual of the previous resource allocation problem, which gets
consequently the name of primal. The two models are one the dual of the other.

P
n P
m
max u = cj xj min w = yi bi
j=1 i=1
P
n P
m
aij xj ≤ bi , i = 1, . . . , m yi aij ≥ cj , j = 1...n
j=1 i=1
xj ≥ 0, j = 1, . . . , n yi ≥ 0, i = 1...m

As we will see the optimal value of the primal problem u∗ is the same as the optimal value of the
dual problem w∗ , ie, u∗ = w∗ .

1.4 Diet Problem

The Diet Problem belongs to the family of blending problems. We wish to select a set of foods that
will satisfy a set of daily nutritional requirement at minimum cost.

min cost/weight
subject to nutrition requirements:
eat enough but not too much of Vitamin A
eat enough but not too much of Sodium
eat enough but not too much of Calories
...
1.5. THE MATHEMATICAL MODEL 11

The problem was motivated in the 1930s and 1940s by the US army. It was first formulated as
a linear programming problem by George Stigler.
Suppose there are:

• 3 foods available, corn, milk, and bread, and

• there are restrictions on the number of calories (between 2000 and 2250) and the amount of
Vitamin A (between 5,000 and 50,000)

Food Cost per serving Vitamin A Calories

Corn $0.18 107 72
2% Milk $0.23 500 121
Wheat Bread $0.05 0 65

1.5 The Mathematical Model

Parameters (given data) :

F = set of foods
N = set of nutrients

aij = amount of nutrient j in food i, ∀i ∈ F , ∀j ∈ N

ci = cost per serving of food i, ∀i ∈ F
Fmini = minimum number of required servings of food i, ∀i ∈ F
Fmaxi = maximum allowable number of servings of food i, ∀i ∈ F
Nminj = minimum required level of nutrient j, ∀j ∈ N
Nmaxj = maximum allowable level of nutrient j, ∀j ∈ N

xi = number of servings of food i to purchase/consume, ∀i ∈ F

Decision Variables

Objective Function : Minimize the total cost of the food

X
Minimize ci xi
i∈F

Constraint Set 1 : For each nutrient j ∈ N , at least meet the minimum required level
X
aij xi ≥ Nminj , ∀j ∈ N
i∈F
12 CHAPTER 1. INTRODUCTION

Constraint Set 2 : For each nutrient j ∈ N , do not exceed the maximum allowable level.
X
aij xi ≤ Nmaxj , ∀j ∈ N
i∈F

Constraint Set 3 : For each food i ∈ F , select at least the minimum required number of
servings
xi ≥ Fmini , ∀i ∈ F

Constraint Set 4 : For each food i ∈ F , do not exceed the maximum allowable number of
servings.
xi ≤ Fmaxi , ∀i ∈ F
All together we obtain the following system of equalities and inequalities that gives the linear
programming problem:

X
min ci xi
i∈F
X
aij xi ≥ Nminj , ∀j ∈ N
i∈F
X
aij xi ≤ Nmaxj , ∀j ∈ N
i∈F
xi ≥ Fmini , ∀i ∈ F
xi ≤ Fmaxi , ∀i ∈ F

The linear programming model by Stigler consisted of 9 equations in 77 variables He guessed

an optimal solution using a heuristic method. In 1947, the National Bureau of Standards used the
newly developed simplex method by Dantzig to solve Stigler’s model. It took 9 clerks using hand-
operated desk calculators 120 man days to solve for the optimal solution. The original instance is
available at: http://www.gams.com/modlib/libhtml/diet.htm

1.5.1 Solving LP Models in Practice

There are two main approaches to pass a model to a solver. A dedicated modelling language allows
to declare the problem in a very similar way to the mathematical model written above and then call
the solver for the solution. Examples are AMPL, ZIMPL, GAMS, GNU MathProg. Alternatively, it
is possible to use libraries from common programming languages. Python offers a good compromise
between these two approaches, in that, even if it is an imperative, procedural language it allows to
define the model in a way very similar to those of modelling languages.
Let’s first have a look at how things look in a modelling language as AMPL. A good way to
proceed is to separate the model from the data in different files.
# diet.mod
set NUTR;
set FOOD;

param cost {FOOD} > 0;

param f_min {FOOD} >= 0;
param f_max { i in FOOD} >= f_min[i];
1.5. THE MATHEMATICAL MODEL 13

param n_min { NUTR } >= 0;

param n_max {j in NUTR } >= n_min[j];
param amt {NUTR,FOOD} >= 0;

var Buy { i in FOOD} >= f_min[i], <= f_max[i]

minimize total_cost: sum { i in FOOD } cost [i] * Buy[i];

subject to diet { j in NUTR }:
n_min[j] <= sum {i in FOOD} amt[i,j] * Buy[i] <= n_max[j];

# diet.dat
data;

set NUTR := A B1 B2 C ;
set FOOD := BEEF CHK FISH HAM MCH MTL SPG TUR;

param: cost f_min f_max :=

BEEF 3.19 0 100
CHK 2.59 0 100
FISH 2.29 0 100
HAM 2.89 0 100
MCH 1.89 0 100
MTL 1.99 0 100
SPG 1.99 0 100
TUR 2.49 0 100 ;

param: n_min n_max :=

A 700 10000
C 700 10000
B1 700 10000
B2 700 10000 ;

param amt (tr):

A C B1 B2 :=
BEEF 60 20 10 15
CHK 8 0 20 20
FISH 8 10 15 10
HAM 40 40 35 10
MCH 15 35 15 15
MTL 70 30 15 15
SPG 25 50 25 15
TUR 60 20 15 10 ;

Below is instead an example in Gurobi Python on a different dataset.

# data.py

from gurobipy import *

categories, minNutrition, maxNutrition = multidict({

’calories’: [1800, 2200],
’protein’: [91, GRB.INFINITY],
’fat’: [0, 65],
14 CHAPTER 1. INTRODUCTION

’sodium’: [0, 1779] })

foods, cost = multidict({

’hamburger’: 2.49,
’chicken’: 2.89,
’hot dog’: 1.50,
’fries’: 1.89,
’macaroni’: 2.09,
’pizza’: 1.99,
’salad’: 2.49,
’milk’: 0.89,
’ice cream’: 1.59 })

# Nutrition values for the foods

nutritionValues = {
(’hamburger’, ’calories’): 410,
(’hamburger’, ’protein’): 24,
(’hamburger’, ’fat’): 26,
(’hamburger’, ’sodium’): 730,
(’chicken’, ’calories’): 420,
(’chicken’, ’protein’): 32,
(’chicken’, ’fat’): 10,
(’chicken’, ’sodium’): 1190,
(’hot dog’, ’calories’): 560,
(’hot dog’, ’protein’): 20,
(’hot dog’, ’fat’): 32,
(’hot dog’, ’sodium’): 1800,
(’fries’, ’calories’): 380,
(’fries’, ’protein’): 4,
(’fries’, ’fat’): 19,
(’fries’, ’sodium’): 270,
(’macaroni’, ’calories’): 320,
(’macaroni’, ’protein’): 12,
(’macaroni’, ’fat’): 10,
(’macaroni’, ’sodium’): 930,
(’pizza’, ’calories’): 320,
(’pizza’, ’protein’): 15,
(’pizza’, ’fat’): 12,
(’pizza’, ’sodium’): 820,
(’salad’, ’calories’): 320,
(’salad’, ’protein’): 31,
(’salad’, ’fat’): 12,
(’salad’, ’sodium’): 1230,
(’milk’, ’calories’): 100,
(’milk’, ’protein’): 8,
(’milk’, ’fat’): 2.5,
(’milk’, ’sodium’): 125,
(’ice cream’, ’calories’): 330,
(’ice cream’, ’protein’): 8,
(’ice cream’, ’fat’): 10,
(’ice cream’, ’sodium’): 180 }

# Model diet.py
1.6. A BRIEF HISTORY OF LINEAR PROGRAMMING (LP) 15

m = Model("diet")

# Create decision variables for the foods to buy

buy = {}
for f in foods:
buy[f] = m.addVar(obj=cost[f], name=f)

# The objective is to minimize the costs

m.modelSense = GRB.MINIMIZE

# Update model to integrate new variables

m.update()

# Nutrition constraints
for c in categories:
m.addConstr(
quicksum(nutritionValues[f,c] * buy[f] for f in foods) <= maxNutrition[c], name
=c+’max’)
m.addConstr(
quicksum(nutritionValues[f,c] * buy[f] for f in foods) >= minNutrition[c], name
=c+’min’)

# Solve
m.optimize()

1.6 A Brief History of Linear Programming (LP)

Releated to linear programming problems are systems of linear equations, which we study in Linear
Algebra.
It is impossible to find out who knew what when first. The Egyptians and Babylonians considered
about 2000 B.C. the solution of special linear equations. But, of course, they described examples
and did not describe the methods in "today’s style". What we call "Gaussian elimination" today
has been explicitly described in Chinese "Nine Books of Arithmetic" which is a compendium written
in the period 2010 B.C. to A.D. 9, but the methods were probably known long before that. Gauss,
by the way, never described "Gaussian elimination". He just used it and stated that the linear
equations he used can be solved "per eliminationem vulgarem"
The origins of Linear Programming date back to Newton, Leibnitz, Lagrange, etc.
• In 1827, Fourier described a variable elimination method for systems of linear inequalities,
today often called Fourier-Motzkin elimination (Motzkin, 1937). It can be turned into an LP
solver but inefficient.
• In 1932, Leontief (1905-1999) studies the Input-Output model to represent interdependencies
between branches of a national economy (1976 Nobel prize).
• In 1939, Kantorovich (1912-1986) layed down the foundations of linear programming. He won
the Nobel prize in economics in 1975 with Koopmans on Optimal use of scarce resources:
foundation and economic interpretation of LP.
• The math subfield of Linear Programming was created by George Dantzig, John von Neumann
(Princeton), and Leonid Kantorovich in the 1940s.
16 CHAPTER 1. INTRODUCTION

• In 1947, Dantzig (1914-2005) invented the (primal) simplex algorithm working for the US Air
Force at the Pentagon. (program=plan)

• In 1954, Lemke describes the dual simplex algorithm. In 1954, Dantzig and Orchard Hays
present the revised simplex algorithm.

• In 1970, Victor Klee and George Minty created an example that showed that the classical
simplex algorithm has exponential worst-case behavior.

• In 1979, L. Khachain found a new efficient algorithm, the Ellipsoid method, for linear pro-
gramming. It was terribly slow.

• In 1984, Karmarkar discovered yet another new efficient algorithm for linear programming,
the interior point method. It proved to be a strong competitor for the simplex method.

Some other important marks in the history of optimization are the following:

• In 1951, Nonlinear Programming began with the Karush-Kuhn-Tucker Conditions

• In 1952, Commercial Applications and Software began

• In 1950s, Network Flow Theory began with the work of Ford and Fulkerson.

• In 1955, Stochastic Programming began

• In 1958, Integer Programming began with cutting planes by R. E. Gomory

• In 1962, Complementary Pivot Theory

1.7 Fourier Motzkin elimination method

Suppose A is a matrix from Qm×n and b a vector from Qn . Does the system of linear inequalities
Ax ≤ b have a solution?
The Fourier Motzkin elimination method works with the following steps:

1. transform the system into another by eliminating some variables such that the two systems
have the same solutions over the remaining variables.

2. reduce to a system of constant inequalities that can be easily decided

Let M = {1 . . . m} be the set that indexes the constraints. For a variable j = 1...n let partition
the rows of the matrix A in those in which xj appears with a positive, negative and null coefficient,
respectively, that is:
N = {i ∈ M | aij < 0}
Z = {i ∈ M | aij = 0}
P = {i ∈ M | aij > 0}
Let xr be the variable to eliminate.
 Pr−1 
 xr ≥ b0ir − Pk=1 a0ik xk , air < 0  xr ≥ Ai (x1 , . . . , xr−1 ), i ∈ N
x ≤ b0ir − r−1 0
k=1 aik xk , air > 0 xr ≤ Bi (x1 , . . . , xr−1 ), i ∈ P
 r 
all other constraints(i ∈ Z) all other constraints(i ∈ Z)
1.7. FOURIER MOTZKIN ELIMINATION METHOD 17

Hence the original system is equivalent to


max{Ai (x1 , . . . , xr−1 ), i ∈ N } ≤ xr ≤ min{Bi (x1 , . . . , xr−1 ), i ∈ P }
all other constraints(i ∈ Z)

which is equivalent to

Ai (x1 , . . . , xr−1 ) ≤ Bj (x1 , . . . , xr−1 ) i ∈ N, j ∈ P
all other constraints(i ∈ Z)

we eliminated xr but: 
|N | · |P | inequalities
|Z| inequalities
d
After d iterations if |P | = |N | = n/2 exponential growth: 1/4(n/2)2

Example
−7x1 + 6x2 ≤ 25
x1 − 5x2 ≤ 1
x1 ≤ 7
−x1 + 2x2 ≤ 12
−x1 − 3x2 ≤ 1
2x1 − x2 ≤ 10
Let x2 be the variable we choose to eliminate:

N = {2, 5, 6}, Z = {3}, P = {1, 4}

We obtain |Z ∪ (N × P )| = 7 constraints.
By adding one variable and one inequality, Fourier-Motzkin elimination can be turned into an
LP solver. How?
18 CHAPTER 1. INTRODUCTION
Chapter 2

The Simplex Method

In this chapter we study the simplex method or (simplex algorithm). It was the first algorithm
to solve linear programming problems proposed in 1947 by George Dantzig in a technical report
“Maximization of a Linear Function of Variables Subject to Linear Inequalities” Dantzig [1951].

2.1 Preliminaries
We lay down some definitions from linear algebra that will be useful to motivate and describe the
simplex algorithm.

• R: set of real numbers

N = {1, 2, 3, 4, ...}: set of natural numbers (positive integers)
Z = {..., −3, −2, −1, 0, 1, 2, 3, ...}: set of all integers
Q = {p/q | p, q ∈ Z, q 6= 0}: set of rational numbers

• We will often useP

matrix notation. Vectors are always meant to be column vectors. The scalar
product: y x = ni=1 yi xi
T

• For a set of k vectors from the vector space Rn , v1 , v2 . . . , vk ∈ Rn , the vector x ∈ Rn is a

linear combination if there exist λ = [λ1 , . . . , λk ]T ∈ Rk such that

k
X
x = λ 1 v1 + · · · + λ k v k = λi vi
i=1

The values λ1 , . . . , λk are called coefficients of the linear combination.

If λ T 1 = 1, the the linear combination is an affine combination. Here, k
Pk 1 is a vector in R with
T
all components equal to 1. Hence, λ 1 = 1 corresponds to saying i=1 λi = 1.
If λ ≥ 0 then the linear combination is a conic combination.
If λ ≥ 0 and λ T 1 = 1 then the linear combination is a convex combination.

• A set S of vectors from Rn is linearly (affine) independent if no element of it can be expressed

as linear (affine) combination of the others
Eg: S ⊆ Rn =⇒ max n lin. indep. (n + 1 lin. aff. indep.)

19
20 CHAPTER 2. THE SIMPLEX METHOD

Figure 2.1:

Figure 2.2:

• For a set of points S ⊆ Rn linear hull (aka linear span), affine hull, conic hull and convex hull
are respectively the sets:
lin(S) = {λ1 v1 + · · · + λk vk |k ≥ 0; v1 , · · · , vk ∈ S; λ1 , · · · , λk ∈ R}
aff(S) = {λ1 v1 + · · · + λk vk |k ≥ 1; v1 , · · · , vk ∈ S; λ1 , · · · , λk ∈ R; λ1 + · · · + λk = 1}
cone(S) = {λ1 v1 + · · · + λk vk |k ≥ 0; v1 , · · · , vk ∈ S; λ1 , · · · , λk ≥ 0}
conv(S) = {λ1 v1 + · · · + λk vk |k ≥ 0; v1 , · · · , vk ∈ S; λ1 , · · · , λk ≥ 0; λ1 + · · · + λk = 1}

See Figure 2.1 for a geometrical interpretation of these concepts. The set of points can be the
vectors made by the columns of an n × m matrix A, hence the previous definitions can refer
to a matrix as well.
• convex set: if x, y ∈ S and 0 ≤ λ ≤ 1 then λx + (1 − λ)y ∈ S
• convex function if its epigraph {(x, y) ∈ R2 : y ≥ f (x)} is a convex set or f : X → R, if
∀x, y ∈ X, λ ∈ [0, 1] it holds that f (λx + (1 − λ)y) ≤ λf (x) + (1 − λ)f (y)
• See Fig 2.2.
• Given a set of points X ⊆ Rn the convex hull conv(X) is the convex linear combination of
P
the points conv(X) = {λ1 x1 + λ2 x2 + . . . + λn xn | xi ∈ X, λ1 , . . . , λn ≥ 0 and i λi = 1}

• rank of a matrix for columns (= for rows)

if (m, n)-matrix has rank = min{m, n} then the matrix is full rank
if (n, n)-matrix is full rank then it is regular and admits an inverse
2.1. PRELIMINARIES 21

• G ⊆ Rn is an hyperplane if ∃ a ∈ Rn \ {0} and α ∈ R:

G = {x ∈ Rn | aT x = α}
• H ⊆ Rn is an halfspace if ∃ a ∈ Rn \ {0} and α ∈ R:
H = {x ∈ Rn | aT x ≤ α}
(aT x = α is a supporting hyperplane of H)
• a set S ⊂ Rn is a polyhedron if ∃ m ∈ Z+ , A ∈ Rm×n , b ∈ Rm :
m
\
P = {x ∈ R | Ax ≤ b} = {x ∈ Rn | Ai· x ≤ bi }
i=1

• a polyhedron P is a polytope if it is bounded: ∃ B ∈ R, B > 0:

P ⊆ {x ∈ Rn | kxk ≤ B}

• Theorem: every polyhedron P 6= Rn is determined by finitely many halfspaces

• General optimization problem: max{ϕ(x) | x ∈ F }, F is feasible region for x
• Note: if F is open, eg, x < 5 then: sup{x | x < 5}
sumpreum: least element of R greater or equal than any element in F
• If A and b are made of rational numbers, P = {x ∈ Rn | Ax ≤ b} is a rational polyhedron
• A face of P is F = {x ∈ P | ax = α}. Hence F is either P itself or the intersection of P with
a supporting hyperplane. It is said to be proper if F 6= ∅ and F 6= P . In Figure 2.3, a face is
a side of the tetrahedron, an edge and a vertex.
• A point x for which {x} is a face is called a vertex of P and also a basic solution of Ax ≤ b
(a vertex of a polytope is a face of dimension 0). There are four vertices in the polytope of
Figure 2.3.
• A facet is a maximal face distinct from P cx ≤ d is facet defining if cx = d is a supporting
hyperplane of P (a facet of a polytope in n has dimension n − 1). In Figure 2.3, the four sides
are the facets of the polytope.

Depending on whether we study systems of linear equalities or inequalities and using integer or
continuous variables we may be in a different field of mathematics:
• Linear algebra studies linear equations
• Integer linear algebra studies linear diophantine equations
• Linear programming studeis linear inequalities (simplex method)
• Integer linear programming studies linear diophantine inequalities
22 CHAPTER 2. THE SIMPLEX METHOD

Figure 2.3: Two examples of polytopes in R3 : a tetrahedron and dodecahedron.

2.1.1 Linear Programming Problem

Input: a matrix A ∈ Rm×n and column vectors b ∈ Rm , c ∈ Rn
Task: Establish one of the three:
1. {x ∈ Rn ; Ax ≤ b} is empty (prob. infeasible), or
2. a column vector x ∈ Rn such that Ax ≤ b and cT x is max can be found
3. for all α ∈ R there is an x ∈ Rn with Ax ≤ b and cT x > α (prob. unbounded )

1. F = ∅
2. F 6= ∅ and ∃ solution
(a) one solution
(b) infinite solution
3. F 6= ∅ and 6 ∃ solution

2.1.2 Fundamental Theorem of LP

Theorem 2.1 (Fundamental Theorem of Linear Programming). Given:

min{cT x | x ∈ P } where P = {x ∈ Rn | Ax ≤ b}

If P is a bounded polyhedron and not empty and x∗ is an optimal solution to the problem, then:

• x∗ is an extreme point (vertex) of P , or

• x∗ lies on a face F ⊂ P of optimal solution

Proof. The first part of the proof shows by contradiction that x∗ must be on the boundary of P .
Then, if x∗ is not a vertex, it is a convex combination of vertices and it shows that all points of the
convex combination are also optimal.
For the first part, suppose, for the sake of contradiction, that x∗ ∈ int(P ), that is, is interior
to P , not a vertex. Then there exists some  > 0 such that the ball of radius  centered at x∗ is
contained in P , that is B (x∗ ) ⊂ P . Therefore,
 c
x∗ − ∈P
2 ||c||
2.2. SYSTEMS OF LINEAR EQUATIONS 23

and substituting in the objective function

 
∗  c  cT c 
cT
x − = cT x∗ − = cT x∗ − ||c|| < cT x∗ .
2 ||c|| 2 ||c|| 2

Hence x∗ is not an optimal solution, a contradiction. Therefore, x∗ must live on the boundary
of P . If x∗ P Pbe the convex combination of vertices of P , say x1 , . . . , xt .
is not a vertex itself, it must
Then x∗ = ti=1 λi xi with λi ≥ 0 and ti=1 λi = 1. Observe that

t
! ! t
! t
X X X
∗ ∗
0=c T
λi xi −x T
=c λi (xi − x ) = λi (cT xi − cT x∗ ).
i=1 i=1 i=1

Since x∗ is an optimal solution, all terms in the sum are nonnegative. Since the sum is equal to zero,
we must have that each individual term is equal to zero. Hence, cT x∗ = cT xi for each xi , so every
xi is also optimal, and therefore all points on the face whose vertices are x1 , . . . , xt , are optimal
solutions.

It follows from the theorem that the optimal solution is at the intersection of hyperplanes sup-
porting halfspaces. Since there are finitely many halfspaces to describe a polyhedron, then there are
finitely many possibilities to look for optimal soltuions.

A solution method could proceed as follows:
n
write all inequalities as equalities and solve all m systems of linear equalities (n # variables, m #
equality constraints). For each point we need then to check if it is feasible and if it is best in cost
(optimality condition). We can solve each system of linear equations by Gaussian elimination.
n


Checking all m may result in a lot of work. Recall that [Cormen et al., 2009, pag. 1097]
   en m 
n
=O
m m

hence the the asymptotic upper bound is an exponential function. The simplex method, as we will
see, tries to be smarter and only visit some of the vertices. Shortly said, it finds a solution that is
at the intersection of some m hyperplanes. Then it tries to systematically produce the other points
by exchanging one hyperplane with another and at each point it checks optimality.

2.2 Systems of Linear Equations

Before we proceed to the treatment of the simplex algorithm we revise the solution methods for a
system of linear equations Ax = b. There are three methods to solve Ax = b if A is n × n and
|A| =
6 0:

1. Gaussian elimination

2. By inverse matrix: find A−1 , then calculate x = A−1 b

3. Cramer’s rule

However, if A is m × n and m 6= n or if |A| = 0 then we can only use Gaussian elimination.

24 CHAPTER 2. THE SIMPLEX METHOD

Gaussian Elimination Gaussian Elimination works in two steps:

1. Forward elimination: reduces the system to triangular (row echelon) form by elementary row
operations

• multiply a row by a non-zero constant

• interchange two rows
• add a multiple of one row to another

2. Back substitution

Alternatively, one can compute the reduced row echelon form (RREF) of A and read immediately
the solution from there.
We illustrate this procedure on the following numerical example.

2x + y − z = 8 (R1)
−3x − y + 2z = −11 (R2)
−2x + y + 2z = −3 (R3)

On the right side we perform the computations. The style is taken from emacs org mode, that offers
a convenient environment for working with tables.
|----+----+----+----+-----|
2x + y − z = 8 (R1) | R1 | 2 | 1 | -1 | 8 |
−3x − y + 2z = −11 (R2) | R2 | -3 | -1 | 2 | -11 |
| R3 | -2 | 1 | 2 | -3 |
−2x + y + 2z = −3 (R3) |----+----+----+----+-----|
|---------------+---+-----+------+---|
| R1’=1/2 R1 | 1 | 1/2 | -1/2 | 4 |
2x + y − z = 8 (R1) | R2’=R2+3/2 R1 | 0 | 1/2 | 1/2 | 1 |
| R3’=R3+R1 | 0 | 2 | 1 | 5 |
+ 21 y + 12 z = 1 (R2) |---------------+---+-----+------+---|
+ 2y + 1z = 5 (R3)
|-------------+---+-----+------+---|
2x + y − z = 8 (R1) | R1’=R1 | 1 | 1/2 | -1/2 | 4 |
1 1 | R2’=2 R2 | 0 | 1 | 1 | 2 |
+ 2y + 2z = 1 (R2) | R3’=R3-4 R2 | 0 | 0 | -1 | 1 |
− z = 1 (R3) |-------------+---+-----+------+---|

2x + y − z = 8 (R1) |---------------+---+-----+---+-----|
1 1 | R1’=R1-1/2 R3 | 1 | 1/2 | 0 | 7/2 |
+ 2y + 2z = 1 (R2) | R2’=R2+R3 | 0 | 1 | 0 | 3 |
− z = 1 (R3) | R3’=-R3 | 0 | 0 | 1 | -1 |
|---------------+---+-----+---+-----|
x = 2 (R1)
y = 3 (R2) |---------------+---+---+---+----+
| R1’=R1-1/2 R2 | 1 | 0 | 0 | 2 | => x=2
z = −1 (R3) | R2’=R2 | 0 | 1 | 0 | 3 | => y=3
| R3’=R3 | 0 | 0 | 1 | -1 | => z=-1
|---------------+---+---+---+----+
Gaussian elimination can be implemented in polynomial time O(n2 m) but some care must be
applied to guarantee that all the numbers during the run can be represented by polynomially bounded
bits.

By inverse matrix
Ax = b

x = A−1 b
2.3. SIMPLEX METHOD 25

Calculating the inverse of a matrix is computationally very expensive. In practice (that is, in
computer systems) the computation is rather performed via LU factorization: each matrix can be
expressed as the product of a permutation matrix P , a lower triangular matrix L (with diagonal
elements equal to 1) and upper triangular matrix U .

A = P LU

For our previous example:     

2 1 −1 x 8
−3 −1 2  y  = −11
−2 1 2 z −3
    
2 1 −1 1 0 0 u11 u12 u13
A = −3 −1 2  = P l21 1 0  0 u22 u23 
−2 1 2  l31 l32 1 0 0  u33 
1 0 0 1 0 0 2 1 −1
= 0 1 0 −3/2 1 0 0 1/2 1/2
0 0 1 −1 4 1 0 0 −1
The LU factorization can be computed efficiently by a recursive algorithm. Then to solve Ax = b,
we note that

A = P LU
x = A−1 b = U −1 L−1 P T b
z1 = P T b, z2 = L−1 z1 , x = U −1 z2
The last two equations are solved by forward substitution Lz2 = z1 and by backward substitution
U x = z2 .

Cramer’s rule To Do. See wikipedia for now. It is computationally expensive but we will use it
to derive a result later.
Solving Ax = b in practice and at the computer is done:

– via LU factorization (much quicker if one has to solve several systems with the same matrix
A but different vectors b)

– if A is a symmetric positive definite matrix then Cholesky decomposition (twice as fast)

– if A is large or sparse then iterative methods

2.3 Simplex Method

Dantzig introduced the simplex algorithm to solve LP problems that can be written in scalar form
as:
P
n
max cj xj
j=1
P
n
aij xj ≤ bi , i = 1, . . . , m
j=1
xj ≥ 0, j = 1, . . . , n
26 CHAPTER 2. THE SIMPLEX METHOD

or in matrix notation as:

max cT x
Ax ≤ b
x ≥ 0

x ∈ Rn , c ∈ Rn , A ∈ Rm×n , b ∈ Rm

The algorithm first ensures that the problem is in a standard form. Then determines an easy to
find initial solution. We will initially assume that this initial solution is feasible. Next the algorithm
proceeds by iterating through feasible solutions that are vertices of the polyhedron that represents
the feasibility region. Finally, it will use an optimality condition to terminate. A few exceptions
may occur, they determine initial infeasibility, unboundedness, more than one solution and cycling
in case of degenerancies. We will see how this situations are handled.

Standard Form The first step in the algorithm is to put the LP problem in a standard form:
Proposition 1 (Standard form). Each linear program can be converted in the form:

max cT x
Ax ≤ b
x ∈ Rn

c ∈ Rn , A ∈ Rm×n , b ∈ Rm

Proof. If the problem is not in the standard form already we can transform it:

• if there are equation constraints ax = b, then we introduce two constraints, ax ≤ b and ax ≥ b

for each of those constraints.

• if there are inequalities of the type ax ≥ b, then we change them to the form −ax ≤ −b.

• if the objective function is min cT x then we change it to max(−cT x)

For now, we assume that in the standard form b ≥ 0. As we will see, if this is true then finding
an initial feasible solution is not trivial.
Proposition 2 (Equational standard form). Each LP problem can be converted to the form:

max cT x
Ax = b
x ≥ 0

x ∈ Rn , c ∈ Rn , A ∈ Rm×n , b ∈ Rm

that is, the objective is to maximize, the constraints are all equalities, the variables are all non-
negative.

Proof. Every LP can be transformed in equational standard form:

2.3. SIMPLEX METHOD 27

1. we add one non-negative slack variable xn+i to the left hand side of each constraint i = 1, . . . , m
of the type ≤:
P
n
max cj xj
j=1
P
n
aij xj + xn+i = bi , i = 1, . . . , m
j=1
xj ≥ 0, j = 1, . . . , n
xn+i ≥ 0, i = 1, . . . , m

We assume here that the problem is already in standard form. If it was not then there might
be larger or equal constraints, in which case we would subtract so-called non-negative surplus
variables to make them equality constraints.

2. if some variable l, l ∈ {1, . . . , n} is free, xl R 0, then we introduce two new non-negative

variables:
xl = x0l − x00l
x0l ≥ 0
x00l ≥ 0

3. As above, min cT x ≡ max(−cT x)

4. Again we assume b ≥ 0.

Hence, every LP problem in n×m is converted to an LP problem with at most (m+2n) variables
and m equations (n # original variables, m # constraints).
The relevant form for simplex algorithm is the equational standard form and it is this form most
text books refer to when referring to the standard form. We call the equational standard form
determined by the procedure above canonical if the b terms are all non-negative. It is not always
trivial to put the problem in canonical equational standard form and for infeasible problems it is
simply not possible, as we will see.
From the geometrical point of view the feasibility region of the problem

max{cT x | Ax = b, x ≥ 0}

is the intersection of the set of solutions of Ax = b, which is an affine space (a plane not passing
through the origin) and the nonegative orthant x ≥ 0. For a case in R3 with Ax = b made by
x1 + x2 + x3 = 0 the situation is shown in Figure 2.4 (in R3 the orthant is called octant).
Note that Ax = b is a system of equations that we can solve by Gaussian elimination. Elementary
row operations of A | b such as:
 
• multiplying all entries in some row of A | b by a nonzero real number λ
 
• replacing the ith row of A | b by the sum of the ith row and jth row for some i 6= j
 
do not affect set of feasible solutions. We assume rank( A | b ) = rank(A) = m, ie, rows of A are
linearly independent. Otherwise, we remove the linear dependent rows and change the value of m.
28 CHAPTER 2. THE SIMPLEX METHOD

Figure 2.4: The intersection between Ax = b and x ≥ 0 in R3 .

Basic Feasible Solutions Let B = {1 . . . m} and N = {m + 1 . . . n + m} be a partition of the

set of indices of the columns of the matrix A. We denote by AB the matrix made of columns of A
indexed by B.
Definition 1 (Basic Feasible Solutions). x ∈ Rn is a basic feasible solution of the linear program
max{cT x | Ax = b, x ≥ 0} for an index set B if:
• xj = 0 for all j 6∈ B.

• the square matrix AB is non-singular, ie, all columns indexed by B are lin. indep.

• xB = A−1
B b is non-negative, ie, xB ≥ 0

In the definition, the last condition ensures the feasibility of the solution.
We call xj , j ∈ B basic variables and remaining variables nonbasic variables. Non basic variables
are set to zero in the basic feasible solution determined by B.
Theorem 2.2 (Uniqueness of a basic feasible solution). A basic feasible solution is uniquely deter-
mined by the set B.
Proof.

Ax =AB xB + AN xN = b
xB + A−1 −1
B AN x N = AB b
xB = A−1
B b AB is singular hence one solution

Hence, we call B a (feasible) basis.

Extreme points of a polyhedron and basic feasible solutions are geometric and algebraic mani-
festations of the same concept (See Figure ??):
2.3. SIMPLEX METHOD 29

Figure 2.5: The points p and q represent feasible solutions but they are not extreme points. The
point r is a feasible solution and an extreme point.

Theorem 2.3. Let P be a (convex) polyhedron from LP in std. form. For a point v ∈ P the following
are equivalent:

(i) v is an extreme point (vertex) of P

(ii) v is a basic feasible solution of LP

The proof, not shown here, goes through recognizing that vertices of P are linear independent
combinations of the columns of the matrix A and that such are the columns in AB since AB is
non-singular by definition.

From the previous theorem and the fundamental theorem of linear programming, it follows that

Theorem 2.4. Let LP = max{cT x | Ax = b, x ≥ 0} be feasible and bounded, then the optimal
solution is a basic feasible solution.

We have thus learned how to find algebraically vertices of the polyhedron. The idea for a solution
algorithm is therefore to examine all basic solutions. From what we saw, this corresponds to generate
different sets B of the indices of the columns of A and checking whether the conditions for being a
basic feasible solution hold. For a matrix A that after trasnformation to standard eqautional form
has n + m columns there are finitely many possible subsets B to examine, precisely
 
m+n
m.

If n = m, then 2m m
m ≈ 4 . Hence, even though at each iteration it might be easy to retrieve the
value of the corresponding solutions, we are still left with exponentially many iterations to perform
in the worst case that we have to see all vertices of the polyhedron.

We are now ready to start working at a numerical example. Let’s consider our previous problem
from resource allocation. In scalar form:

max 6x1 + 8x2

5x1 + 10x2 ≤ 60
4x1 + 4x2 ≤ 40
x1 , x2 ≥ 0
30 CHAPTER 2. THE SIMPLEX METHOD

and in matrix form:  

  x1
max 6 8
x2
    
5 10 x1 60
≤
4 4 x2 40

x1 , x2 ≥ 0
We put the problem in canonical equational standard form:
5x1 + 10x2 + x3 = 60
4x1 + 4x2 + x4 = 40
or, equivalently, in matrix form:
 

 x1
max z= 6 8
x2
 
  x1  
5 10 1 0  
x2  = 60
4 4 0 1 x3  40
x4
x1 , x2 , x3 , x4 ≥ 0
If the equational standard form is canonical one decision variable is isolated in each constraint and
it does not appear in the other constraints nor in the objective function and the b terms are positive.
The advantage of the canonical form is evident: it gives immediately a basic feasible solution:

x1 = 0, x2 = 0, x3 = 60, x4 = 40

The basis of this solution is B = {3, 4}. Consequently, N = {1, 2}. If this solution is also optimal
then the algorithm can terminate and return the solution. Is this solution optimal?
Looking at signs in z it seems not: since they are positive, if we can increase the variables x1 and
x2 to become larger than zero then the soltion quality would improve. Let’s then try to increase a
promising variable, i.e., one with positive coefficient in z. Let’s take x1 and let’s consider how much
we can increase its value looking at the first constraint. Since x2 stays equal to zero, this variable
does not appear in the constraint.
5x1 + x3 = 60
Isolating first x1 and then x3 we can plot the line represented by this constraint:
x3
x1 = 60
5 − 5
x3 = 60 − 5x1 ≥ 0

x3

x1
5x1 + x3 = 60
2.3. SIMPLEX METHOD 31

From the explicit form we see that if we increase x1 more than 12 then x3 becomes negative and
thus the whole solution infeasible. This constraint imposes therefore an upper bound of 12 to the
increase of x1 . Let’s analyze the second constraint now:

4x1 + x4 = 40

Isolating x1 and x4 and plotting the line we obtain:

x4
x1 = 40
4 − 4
x4 = 40 − 4x1 ≥ 0

x4

x1
4x1 + x4 = 40

For a similar reasoning as above we observe that this constraint imposes an upper bound of 10 to
the increase of x1 .
It follows that the value of x1 can be increased at most up to 10. Increasing x1 to 10 makes
x4 = 0 because of the second constraint. Hence, we want that x4 exits the basis while x1 enters
in it. In order to bring the problem back in canonical standard form after the increase of x1 we
need to perform elementary row operations. To this end it is convenient to work with a particular
organization of the data that is called simplex tableau (plural tableaux).

x1 x2 x3 x4 −z b
x3 5 10 1 0 0 60
x4 4 4 0 1 0 40
6 8 0 0 1 0

The variables that label the columns remain fixed throughout the iterations, while the labels in
the rows changes depending on which variables are in basis. The column −z will never change
throughout the iterations of the algorithm. The last row is given by the objective function. Note
that some text books put this row as the first row on the top. With the new basis the new tableau
that correspond to a canonical standard form looks like this:

x1 x2 x3 x4 −z b
x3 0 ? 1 ? 0 ?
x1 1 ? 0 ? 0 ?
0 ? 0 ? 1 ?

that is, there is a permuted identity matrix whose last column, −z, remains fixed while the other
two columns indicate which variable is in basis.
The decisions that we have done so far: to select a variable to increase, the amount of the
increase, which variable has to decrease and putting the tableau the new form, can be written in
general terms as the following pivot operations.
Definition 2 (Pivot operations). The following operations are done at each iteration of the simplex:
32 CHAPTER 2. THE SIMPLEX METHOD

1. Determine a pivot:

column: a column s with positive coefficient in objective function

row: consider the ratio between the coefficients b and the coefficients as of the pivot column
and choose the one with smallest ratio:
 
bi
r = argmini : ais > 0
ais

2. elementary row operations to update the tableau around the pivot.

Note that the choice of the row of the pivot gives us also the increase value θ of entering variable,
that is,  
bi
θ = min : ais > 0
i ais

Let’s get back to our numerical example and perform the simplex iterations:

• x1 enters the basis and x4 leaves the basis. The pivot is element at position (1, 4) ie, 4. The
elementary row operations to put the tableau in the new form are:

– Divide the pivot row by the value of the pivot

– Send to zero the coefficient in the pivot column of the first row
– Send to zero the coefficient of the pivot column in the third (cost) row
| | x1 | x2 | x3 | x4 | -z | b |
|---------------+----+----+----+------+----+-----|
| I’=I-5II’ | 0 | 5 | 1 | -5/4 | 0 | 10 |
| II’=II/4 | 1 | 1 | 0 | 1/4 | 0 | 10 |
|---------------+----+----+----+------+----+-----|
| III’=III-6II’ | 0 | 2 | 0 | -6/4 | 1 | -60 |

From the last row we read: 2x2 − 3/2x4 − z = −60, that is: z = 60 + 2x2 − 3/2x4 . Since x2
and x4 are nonbasic we have z = 60 and x1 = 10, x2 = 0, x3 = 10, x4 = 0.

• Are we done? No, there are still positive coefficients in the objective row! Let x2 enter the
basis. We determine the pivot, which is 5, hence x3 is the variable that exists. After the row
operations we obtain:
| | x1 | x2 | x3 | x4 | -z | b |
|--------------+----+----+------+------+----+-----|
| I’=I/5 | 0 | 1 | 1/5 | -1/4 | 0 | 2 |
| II’=II-I’ | 1 | 0 | -1/5 | 1/2 | 0 | 8 |
|--------------+----+----+------+------+----+-----|
| III’=III-2I’ | 0 | 0 | -2/5 | -1 | 1 | -64 |

• Are we done? Yes! The variables not in basis have negative coefficients in the objective
function that corresponds to the tableau we reached. Hence if we increased them, we would
worsen the objective function. The solutions we have found is therefore the optimal one.

Definition 3 (Reduced costs). The coefficients in the objective function of the nonbasic variables,
c̄N , are called reduced costs.
Note that basic variables have always coefficients in the last row equal to zero.
2.3. SIMPLEX METHOD 33

x2 x2

?
?
x1 ?
?
x1

Figure 2.6: The search process of the simplex.

Definition 4 (Improving variables). An improving variable is a non basic variable with positive
reduced cost.
Proposition 3 (Optimality condition). A basic feasible solution is optimal when the reduced costs in
the corresponding simplex tableau are nonpositive, ie:

c̄N ≤ 0

In Figure 2.6 left, we represent graphically the solution process executed by the simplex algo-
rithm. Starting from the vertex (0, 0), we moved to (10, 0) and finally reached the optimal solution
in (8, 2). For this problem the other path with x2 increased before x1 would have been of the same
length and hence lead to the same number of iterations of the simplex algorithm. However, the
situation is not always like this. In Figure 2.6 right, and in Figure 2.7, we see that choosing one
direction of increase rather than another may influence considerably the efficiency of the search.
We said earlier that trying all points implies approximately 4m iterations. This is an asymptotic
upper bound. On the other hand to find an asymptotic lower bound we should apply the clairvoyant’s
rule, that is, using the shortest possible sequence of steps for any pair of vertices we may choose as
starting and optimal solutions. However, the length of this path for a general polyhedron in Rn is
not known. Hirsh conjectures O(n) but best known result is n1+ln n .
In practice, the simplex algorithm runs in between 2m and 3m iterations. (Hence, relevant to
note, the number of iterations depends on the number of constraints.)

Figure 2.7: The search process in a generic polyhedron in R3 .

34 CHAPTER 2. THE SIMPLEX METHOD

Tableaux and Dictionaries We chose the use the tableau representation which is the original
Dantzig representation of the simplex algorithm. An alternative representation by means of dictio-
naries due to Chvatal is equally spread and used in text books. The tableau representation is more
amenable to implementations at the computer than the dictionary one. However, efficient code use
the revised simplex method and hence not either a tableaux representation.
Let’s consider the general LP problem:
P
n
max cj xj
j=1
P
n
aij xj ≤ bi , i = 1, . . . , m
j=1
xj ≥ 0, j = 1, . . . , n
The equational standard form can be written, perhaps more intuitively, also by isolating the slack
variables:
P
n
max z = cj xj
j=1
P
n
xn+i = bi − aij xj , i = 1, . . . , m
j=1
xj ≥ 0, j = 1, . . . , n
xn+i ≥ 0, i = 1, . . . , m
This form gives immediately the dictionary representation. We compare this representation side
by side with the tableau representation:
Tableau
  Dictionary
  P
  x r = b̄r − ārs xs , r ∈ B
 I ĀN 0 b̄ 
  P s6 ∈B
  z = d¯ + c̄s xs
 
  s6∈B
0 c̄N 1 −d¯

The last row of the dictionary gives us the same objective function as we have seen that it can
be derived from the last row of tableau, namely:
X X
0xr + ¯
c̄N xN − z = −d.
r∈B s6∈B

Decisions in the two cases must correspond. In the dictionary the Pivot operations are given by:
1. Determine a pivot:
column choose the column s with reduced cost strictly positive
row choose the row i with negative coefficients such that the ratio b/asi is minimal
2. update: express the entering variable and substitute it in the other rows
Example 2.1.
max 6x1 + 8x2
5x1 + 10x2 ≤ 60
4x1 + 4x2 ≤ 40
x1 , x2 ≥ 0
2.4. EXCEPTION HANDLING 35

x1 x2 x3 x4 −z b
x3 = 60 − 5x1 − 10x2
x3 5 10 1 0 0 60
x4 = 40 − 4x1 − 4x2
x4 4 4 0 1 0 40
z = + 6x1 + 8x2
6 8 0 0 1 0

After two iterations:

x1 x2 x3 x4 −z b
x1 = 2 − 1/5x3 + 1/4x4
x2 0 1 1/5 −1/4 0 2
x2 = 8 + 1/5x3 − 1/2x4
x1 1 0 −1/5 1/2 0 8
z = 64 − 2/5x3 − 1x4
0 0 −2/5 −1 1 −64

2.4 Exception Handling

So far we have seen the simplex only on a problem that has a unique solution. An LP problem with
a feasibility region F can have the following outcomes:
1. F 6= ∅ and ∃ solution
(a) one solution
(b) infinite solution
2. F = ∅
3. F 6= ∅ and 6 ∃ solution
Therefore, we will now look at the behaviour of the simplex in conditions of:
1. Unboundedness
2. More than one solution
3. Infesibility
Under the last point we will see that the assumption of a starting basic feasible solution does not
always hold and that this needs saome particular handling.
Moreover, the simplex algorithm can incurr in degenerancies that may be benign if they resolve
by themselves or malign if the final result is cycling.

2.4.1 Unboundedness
We consider the following LP problem instance:
max 2x1 + x2
x2 ≤ 5
−x1 + x2 ≤ 1
x1 , x2 ≥ 0
• We write the initial tableau
| | x1 | x2 | x3 | x4 | -z | b |
|----+----+----+----+----+----+---|
| x3 | 0 | 1 | 1 | 0 | 0 | 5 |
| x4 | -1 | 1 | 0 | 1 | 0 | 1 |
|----+----+----+----+----+----+---|
| | 2 | 1 | 0 | 0 | 1 | 0 |
36 CHAPTER 2. THE SIMPLEX METHOD

x2

?
?
x1

Figure 2.8: The solution process of an unbounded problem.

and perform the pivot opertations:

• x2 entering, x4 leaving

| | x1 | x2 | x3 | x4 | -z | b |
|-------------+----+----+----+----+----+----|
| II’=II-I’ | 1 | 0 | 1 | -1 | 0 | 4 |
| I’=I | -1 | 1 | 0 | 1 | 0 | 1 |
|-------------+----+----+----+----+----+----|
| III’=III-I’ | 3 | 0 | 0 | -1 | 1 | -1 |

The second row corresponds to the constraint:

−x1 + x2 + x4 = 1

, where x2 being in the basis is set to zero. Hence, x1 can increase without restriction. This
is why when writing the maximum allowed increase, we enforced ais > 0: θ = min{ abisi :
ais > 0, i = 1 . . . , n}. If ais ≤ 0 then the variable can increase arbitrarily.

• x1 entering, x3 leaving

| | x1 | x2 | x3 | x4 | -z | b |
|--------------+----+----+----+----+----+-----|
| I’=I | 1 | 0 | 1 | -1 | 0 | 4 |
| II’=II+I’ | 0 | 1 | 1 | 0 | 0 | 5 |
|--------------+----+----+----+----+----+-----|
| III’=III-3I’ | 0 | 0 | -3 | 2 | 1 | -13 |

x4 was already in basis but for both I and II

x1 + x3 − x4 = 4
x2 + x3 + 0x4 = 5

x4 can now be increased arbitrarily: in the first constraint it will be compensated by x1 (x3 is
non basic and hence 0) and in the second constraint it does appear at all.

We are therefore in the condition of an unbounded problem. We recognise this when a variable
chosen to enter in the basis is not upper bounded in its increase. Figure 2.8 provides the geometrical
view of the solution process for this example.
2.4. EXCEPTION HANDLING 37

2.4.2 Infinite solutions

Consider the following LP problem instance:

max x1 + x2
5x1 + 10x2 ≤ 60
4x1 + 4x2 ≤ 40
x1 , x2 ≥ 0

• The initial tableau is

| | x1 | x2 | x3 | x4 | -z | b |
|----+----+----+----+----+----+----|
| x3 | 5 | 10 | 1 | 0 | 0 | 60 |
| x4 | 4 | 4 | 0 | 1 | 0 | 40 |
|----+----+----+----+----+----+----|
| | 1 | 1 | 0 | 0 | 1 | 0 |

we proceed to the pivot operations:

• x2 enters, x3 leaves

| | x1 | x2 | x3 | x4 | -z | b |
|-------------+-----+----+------+----+----+----|
| I’=I/10 | 1/2 | 1 | 1/10 | 0 | 0 | 6 |
| II’=II-4Ix4 | 2 | 0 | -2/5 | 1 | 0 | 16 |
|-------------+-----+----+------+----+----+----|
| III’=III-I | 1/2 | 0 | -1/6 | 0 | 1 | -6 |

• x1 enters, x4 leaves

| | x1 | x2 | x3 | x4 | -z | b |
|----------------+----+----+------+------+----+-----|
| I’=I-II’/2 | 0 | 1 | 1/5 | -1/4 | 0 | 2 |
| II’=II/2 | 1 | 0 | -1/5 | 1/2 | 0 | 8 |
|----------------+----+----+------+------+----+-----|
| III’=III-II’/2 | 0 | 0 | 0 | -1/4 | 1 | -10 |

The corresponding solution is x1 = (8, 2, 0, 0), z = 10. Applying the optimality condition we
see that the solution is optimal. However, we are used to see that nonbasic variables have
reduced costs not equal to 0. Here x3 has reduced cost equal to 0. Let’s make it enter the
basis.

• x3 enters, x2 leaves

| | x1 | x2 | x3 | x4 | -z | b |
|----------------+----+----+----+------+----+-----|
| I’=5I | 0 | 5 | 1 | -5/4 | 0 | 10 |
| II’=II+I’/5 | 1 | 1 | 0 | 4 | 0 | 10 |
|----------------+----+----+----+------+----+-----|
| III’=III | 0 | 0 | 0 | -1/4 | 1 | -10 |

We find a different solution that has the same value: x2 = (10, 0, 10, 0), z = 10. Note that we
use a subscript to differentiate from the first soltution.
Hence we found two optimal solutions. If we continued from here we would again bring x2 in
the basis and x3 out, thus cycling.
38 CHAPTER 2. THE SIMPLEX METHOD

If more than one solution is optimal, then we saw that also all their convex combinations are
optimal solutions. Let’s then express all optimal solutions. The convex combination is:

2
X
x= αi xi
i=1
αi ≥ 0 ∀i = 1, 2
2
X
αi = 1
i=1

In our case we have:

xT1 = [8, 2, 0, 0]
xT2 = [10, 0, 10, 0]

Any vector x resulting from the convex combination with coefficients α1 = α and α2 = 1 − α is
given by:      
x1 8 10
x2  2 0
  = α   + (1 − α)  
x3  0 10
x4 0 0
or
x1 = 8α + 10(1 − α)
x2 = 2α
x3 = 10(1 − α)
x4 = 0.
A problem has infinite solutions when the objective hyperplane is parallel to one of the faces of the
feasibility region with dimension larger than 0. The example is depicted in Figure 2.9. A face could
have larger dimensions and the simplex would find all its extreme vertices before looping between
them.

2.4.3 Degeneracy
Let this time the LP problem instance be:
max x2
−x1 + x2 ≤ 0
x1 ≤ 2
x1 , x2 ≥ 0
• The initial tableau is
| | x1 | x2 | x3 | x4 | -z | b |
|----+----+----+----+----+----+---|
| x3 | -1 | 1 | 1 | 0 | 0 | 0 |
| x4 | 1 | 0 | 0 | 1 | 0 | 2 |
|----+----+----+----+----+----+---|
| | 0 | 1 | 0 | 0 | 1 | 0 |
2.4. EXCEPTION HANDLING 39

x2

x1

Figure 2.9: The example with infinite solutions. The objective function is parallel with the edge of
the feasibility region. The solutions found by the simplex are the two extremes of the segment.

The novel element here is that a right-hand side coefficient is zero, ie, b1 = 0. In the pivot
operations, a null b term will make such that the entering variable will not be increased.
Indeed, a null b term will make the increase value θ null.
Definition 5 (Degenerate pivot step). We call degenerate pivot step a pivot step in which the
entering variable stays at zero.

• Let’s proceed and make x2 enter the basis and x3 leave it.

| | x1 | x2 | x3 | x4 | -z | b |
|---+----+----+----+----+----+---|
| | -1 | 1 | 1 | 0 | 0 | 0 |
| | 1 | 0 | 0 | 1 | 0 | 2 |
|---+----+----+----+----+----+---|
| | 1 | 0 | -1 | 0 | 1 | 0 |

• in the next step we end up avoding the constraint with the null b term and the step is not
degenerate anymore. We exit from the degeneracy state and reach an optimal tableau:

| | x1 | x2 | x3 | x4 | -z | b |
|---+----+----+----+----+----+----|
| | 0 | 1 | 0 | 1 | 0 | 2 |
| | 1 | 0 | 0 | 1 | 0 | 2 |
|---+----+----+----+----+----+----|
| | 0 | 0 | -1 | -1 | 1 | -2 |

where the solution is x1 = 2, x2 = 2 and the objective value is z is 2.

The situation is represented graphically in Figure 2.10. If n is the number of original variables,
degenerancies arise when n + 1 or more constraints meet at a vertex. In other terms, there are
polyhedra that have vertices that are overdetermined, that is, the number of facets that meet in
those vertices is larger than dim(P ). In this case, every dim(P ) inequalities that define these facets
determine a basis that produce a basis solution. In linear algebra terms, for n + m variables of an
LP problem in equational standard form, a basis solution that belongs to a basis B has n variables
set to zero and the remaining m variables set to A− B 1b. In a degenerate basis solution there are more
than n variables set to zero. It follows that the same solution x is solution of more than one regular
n × n subsystem.
Degenerancy may lead to cycling in the simplex.
40 CHAPTER 2. THE SIMPLEX METHOD

x2

x1

Figure 2.10: In the origin 3 constraints meet. In that vertex the simplex method encounters a
degeneracy, which in this case is resolved and another vertex reached.

Theorem 2.5. If the simplex fails to terminate, then it must cycle.

Proof. A tableau is completely determined by specifying which variables are basic and which are
nonbasic. There are only  
n+m
m
different possibilities. The simplex method always moves to non-worsening tableaux. If the simplex
method fails to terminate, it must visit some of these tableaux more than once. Hence, the algorithm
cycles.

Degenerate conditions may appear often in practice but cycling is rare and some pivoting rules
prevent cycling. (Ex. 7 of Sheet 3 shows the smallest possible example.)
Under certain pivoting rules cycling may happen. So far we chose an arbitrary improving variable
to enter the basis.

2.4.4 Pivot Rules

Pivot rules are rules for breaking ties in selecting improving variables to enter in the basis. Ties
may occur also in selecting the leaving variables and the rules can regulate also how to break those
ties. However, the choice of the entering variables is more important than the choice of the leaving
variables.

• Largest Coefficient: select the improving variable with largest coefficient in last row of the
tableau, ie, reduced cost. This is the original Dantzig’s rule, and it was shown that it can lead
to cycling.

• Largest increase: select the improving variable that leads to the best absolute improvement,
ie, argmaxj {cj θj }. This rule is computationally more costly.

• Steepest edge select the improving variable that if brought in the basis, would move the current
basic feasible solution in a direction closest to the direction of the vector c (ie, maximizes the
cosine of the angle between the two vectors):

cT (xnew − xold )
a · b = kak kbk cos θ =⇒ max
k xnew − xold k
2.4. EXCEPTION HANDLING 41

Figure 2.11: Klee and Minty’s example

• Bland’s rule chooses the improving variable with the lowest index and, if there are more than
one leaving variable, the one with the lowest index. This rule prevents cycling but it is slow.

• Random edge select an improving variable uniformly at random.

• Perturbation method perturb the values of bi terms to avoid bi = 0, which must occur for
cycling. To avoid cancellations: 0 < m  m−1  · · ·  1  1. It can be shown to be the
same as lexicographic method, which prevents cycling

2.4.5 Efficiency of simplex method

• The asymptotic upper bound is given by trying all basis, which is ≈ 4m . Klee and Minty 1978
constructed an example that requires the simplex with Dantzig’s pivot rule to visit all 2n − 1
vertices of a polyhedron and hence the maximum number of iterations. See Figure 2.11.

• It is unknown if there exists a pivot rule that leads to polynomial time. The best would be
the Clairvoyant’s rule: that is, choose the pivot rule that gives the shortest possible sequence
of steps. This corresponds to determining the diameter of the m dimensional polytope. The
diameter of a polytope P is the maximum distance between any two vertices in the edge graph
of P (Figure 2.12). The diameter gives a lower bound for any pivoting rule for the simplex
algorithm. Hirsch conjectured (1957) that the diameter of any n-facet convex polytope in
d-dimensional Euclidean space is at most n − d. Kalai and Kleitman (1992) gave an O(nlog n )
upper bound on the diameter namely n1+ln n . Hirsh conjecture was disproved in May 2010
by Francisco Santos from the University of Cantabria in Santander. He constructed a 43-
dimensional polytope with 86 facets and diameter bigger than 43. [Documenta Math. 75 Who
Solved the Hirsch Conjecture? Günter M. Ziegler]. In general terms he showed the existence
of polytopes with diameter (1 + )(n − d). It remains open whether the diameter is polynomial,
or even linear, in n and d.

• In practice the simplex runs in between 2m and 3m number of iterations, hence the running
time seems to be dependent on the number of constraints.

• Positive results are of smoothed complexity type: that is, average case on slight random
perturbations of worst-case inputs. D. Spielman and S. Teng (2001), Smoothed analysis of
algorithms: why the simplex algorithm usually takes polynomial time
O(max(n5 log2 m, n9 log 4 n, n3 σ −4 ))
42 CHAPTER 2. THE SIMPLEX METHOD

Figure 2.12: . The shortest path between any two vertices of a polyhedron may contain an expo-
nentially growing number of vertices as the dimension grows.

• One of the most prominent mysteries in Optimization remains the question of whether a lin-
ear program can be solved in strongly-polynomial time. A strongly polynomial-time method
would be polynomial in the dimension n and in the number m of inequalities only, whereas
the complexity of the known weakly-polynomial time algorithms for linear programming, like
the ellipsoid method or variants of the interior-point method, also depend on the binary en-
coding length of the input. The simplex method, though one of the oldest methods for linear
programming, still is a candidate for such a strongly polynomial time algorithm. This would
require the existence of a pivoting rule that results in a polynomial number of pivot steps.
Since the famous Klee-Minty example, many techniques for deriving exponential lower bounds
on the number of iterations for particular pivoting rules have been found.

Some very important pivoting rules, however, have resisted a super-polynomial lower-bound
proof for a very long time. Among them the pivoting rules Random Edge (uniformly random
improving pivots) Randomized Bland’s Rule (random shuffle the indexes + lowest index for
entering + lexicographic for leaving) Random-Facet and Zadeh’s pivoting rule (least-entered
rule: enter the improving variable that has been entered least often – it minimizes revisits).
Random-Facet has been shown to yield sub-exponential running time of the simplex method
independently by Kalai as well as by Matousek, Sharir and Welzl. For every linear program
with at most n√variables and at most m constraints, the expected number of pivot steps is
bounded by eC m ln n , where C is a (not too large) constant. (Here the expectation means
the arithmetic average over all possible orderings of the variables.) O. Friedmann together
with Hansen and Zwick have shown super-polynomial (but subexponential) lower bounds for
Random Edge, Randomized Bland’s rule and Zadeh’s pivoting rules in 2011. The same authors
in 2015 proposed an improved version of the Random-Facet rule that achieves the best known
sub-exponential running time. These results are unrelated to the diameter of the polytope.
(Sources: Mathematical Optimization Society, 2012 Tucker Prize Citation; Thomas Dueholm
Hansen, Aarhus University, http://cs.au.dk/~tdh/;

2.5 Infeasibility and initialization

So far we have assumed that the b terms in the standard form were positive. If this is not the case,
it might be not trivial to obtain a canonical equational standard form. Let’s consider the following
2.5. INFEASIBILITY AND INITIALIZATION 43

example:
max x1 − x2
x1 + x2 ≤ 2
2x1 + 2x2 ≥ 5
x1 , x2 ≥ 0

The second constraint is of larger and equal type. To make it smaller and equal we multiply left-hand
side and right-hand side by -1, yielding a negative right-hand side term. The equational standard
form becomes:
max x1 − x2
x1 + x2 + x3 = 2
−2x1 + −2x2 + x4 = −5
x1 , x2 , x3 , x4 ≥ 0

We can now make the b terms all positive:

max x1 − x2
x1 + x2 + x3 = 2
2x1 + 2x2 − x4 = 5
x1 , x2 , x3 , x4 ≥ 0

However, when we write the corresponding tableau we observe that it is not in canonical form,
that is, we cannot recognize an identity submatrix.

| | x1 | x2 | x3 | x4 | -z | b |
|----+----+----+----+----+----+---|
| x3 | 1 | 1 | 1 | 0 | 0 | 2 |
| x4 | 2 | 2 | 0 | -1 | 0 | 5 |
|----+----+----+----+----+----+---|
| | 1 | -1 | 0 | 0 | 1 | 0 |

We note that, similarly to the canonical form, one decision variable is isolated in each constraint
and it does not appear in the other constraints nor in the objective function but for x4 the coefficient
is −1. If we take x3 and x4 in basis then reading from the tableau, their value is x3 = 2 and x4 = −5.
This does not comply with the definiton of basic feasible solution that asks all variables to be non-
negative. Hence, we do not have an initial basic feasible solution!
In general finding any feasible solution is as difficult as finding an optimal solution, otherwise we
could do binary search on the values of the objective function (that is, solving a sequence of systems
of linear inequalities, one of which being the constrained objective function).
To find an initial feasible solution we formulate an auxiliary problem and solve it.

Auxiliary Problem (Phase I of the Simplex) We construct an auxiliry problm by introducing

one non-negative auxiliary variable for each constraint in which there is not an isolated variable with
coefficient +1. This will solve the problem of having an initial fesibile basis but the problem would
be different. If however we find a solution in which the new variables introduced are set to zero
then this solution will be valid also for the original problem. The goal of the auxiliary problem is
therefore to find a solution where the auxiliary variables are zero, or in other terms, to minimize the
sum of the auxiliary variables. To remain consistent with our maximization form, we can rewrite
the problem as a maximization problem.
44 CHAPTER 2. THE SIMPLEX METHOD

In our example above we introduce the auxiliary non-negative variable x5 in the second constraint
and minimize its value:

w∗ = min x5 = max −x5

x1 + x2 + x3 = 2
2x1 + 2x2 − x4 + x5 = 5
x1 , x2 , x3 , x4 , x5 ≥ 0

If w∗ = 0 then x5 = 0 and the two problems are equivalent, if w∗ > 0 then not possible to set x5 to
zero and the original problem does not have a feasible solution.
Let’s solve this auxiliary problem.

• In the initial tableau we introduce a new row at the bottom for the new objective function
to maximize and a new column denoted by −w. We keep the row for the original objective
function and the column −z. In the pivot operations we will keep −z and −w always in basis.
will keep

| | x1 | x2 | x3 | x4 | x5 | -z | -w | b |
|----+----+----+----+----+----+----+----+---|
| | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 2 |
| | 2 | 2 | 0 | -1 | 1 | 0 | 0 | 5 |
| z | 1 | -1 | 0 | 0 | 0 | 1 | 0 | 0 |
|----+----+----+----+----+----+----+----+---|
| w | 0 | 0 | 0 | 0 | -1 | 0 | 1 | 0 |

• The initial tableau is not yet in canonical form but it can very easily be made such by letting
x5 enter the basis:

| | x1 | x2 | x3 | x4 | x5 | -z | -w | b |
|-------+----+----+----+----+----+----+----+---|
| | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 2 |
| | 2 | 2 | 0 | -1 | 1 | 0 | 0 | 5 |
| z | 1 | -1 | 0 | 0 | 0 | 1 | 0 | 0 |
|-------+----+----+----+----+----+----+----+---|
| IV+II | 2 | 2 | 0 | -1 | 0 | 0 | 1 | 5 |

Now we have a basic feasible solution. It is [0, 0, 2, 0, 5] and its objective value is w = −5. It
is not optimal and therefore we proceed to find the optimal solution.

• x1 enters the basis and x3 leaves it:

| | x1 | x2 | x3 | x4 | x5 | -z | -w | b |
|--------+----+----+----+----+----+----+----+----|
| | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 2 |
| II-2I’ | 0 | 0 | -2 | -1 | 1 | 0 | 0 | 1 |
| III-I’ | 0 | -2 | -1 | 0 | 0 | 1 | 0 | -2 |
|--------+----+----+----+----+----+----+----+----|
| IV-2I’ | 0 | 0 | -2 | -1 | 0 | 0 | 1 | 1 |

The tableau is optimal. The optimal value can be read from the last row of the tableau:
w∗ = −1. Hence, we see that x5 = 1 and it cannot be decreased further. Then no solution
with X5 = 0 exists and there is no feasible solution for our initial problem.
2.5. INFEASIBILITY AND INITIALIZATION 45

x2

x1

Figure 2.13: The feasibility region is the intersection of the half-spaces described by the constraints.
In this case it is empty.

The original problem is infeasible. We can appreciate this also graphically from Figure 2.13
where we see that the intersection between the half-spaces that define the problem is empty.
Let’s change the right-hand side of the second constraint in the prvious example to be 2 instead
of 5.

max x1 − x2
x1 + x2 ≤ 2
2x1 + 2x2 ≥ 2
x1 , x2 ≥ 0
The equational standard form becomes:

max x1 − x2
x1 + x2 + x3 = 2
2x1 + 2x2 − x4 = 2
x1 , x2 , x3 , x4 ≥ 0

Since it is not canonical we resort to the Phase I of the simplex by formulating the auxiliary problem:

w = min x5 = max −x5

x1 + x2 + x3 = 2
2x1 + 2x2 − x4 + x5 = 2
x1 , x2 , x3 , x4 , x5 ≥ 0

• The initial tableau for the auxiliary problem is:

| | x1 | x2 | x3 | x4 | x5 | -z | -w | b |
|----+----+----+----+----+----+----+----+---|
| | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 2 |
| | 2 | 2 | 0 | -1 | 1 | 0 | 0 | 2 |
| z | 1 | -1 | 0 | 0 | 0 | 1 | 0 | 0 |
|----+----+----+----+----+----+----+----+---|
| w | 0 | 0 | 0 | 0 | -1 | 0 | 1 | 0 |

We do not have yet an initial basic feasible solution.

46 CHAPTER 2. THE SIMPLEX METHOD

• we can set the problem in canonical form by making x5 entering the basis:

| | x1 | x2 | x3 | x4 | x5 | -z | -w | b |
|-------+----+----+----+----+----+----+----+---|
| | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 2 |
| | 2 | 2 | 0 | -1 | 1 | 0 | 0 | 2 |
| z | 1 | -1 | 0 | 0 | 0 | 1 | 0 | 0 |
|-------+----+----+----+----+----+----+----+---|
| IV+II | 2 | 2 | 0 | -1 | 0 | 0 | 1 | 2 |

The basic feasible solution is [0, 0, 2, 0, 2] and yields an objective value w = −2. This solution
is not optimal.

• x1 enters the basis and x5 leaves it:

| | x1 | x2 | x3 | x4 | x5 | -z | -w | b |
|----+----+----+----+------+------+----+----+----|
| | 0 | 0 | 1 | 1/2 | -1/2 | 0 | 0 | 1 |
| | 1 | 1 | 0 | -1/2 | 1/2 | 0 | 0 | 1 |
| z | 0 | -2 | 0 | 1/2 | -1/2 | 1 | 0 | -1 |
|----+----+----+----+------+------+----+----+----|
| w | 0 | 0 | 0 | 0 | -1 | 0 | 1 | 0 |

The solution is optimal and w∗ = 0 hence x5 = 0 and we have a starting feasible solution for
the original problem.

From now we can proceed with the Phase II of the simplex method, which works on the original
problem as we have learned.

• First we rewrite the tableau that we reached by keeping only what we need:

| | x1 | x2 | x3 | x4 | -z | b |
|----+----+----+----+------+----+----|
| | 0 | 0 | 1 | 1/2 | 0 | 1 |
| | 1 | 1 | 0 | -1/2 | 0 | 1 |
|----+----+----+----+------+----+----|
| z | 0 | -2 | 0 | 1/2 | 1 | -1 |

• x4 enters the basis and x3 leaves it:

| | x1 | x2 | x3 | x4 | -z | b |
|----+----+----+----+----+----+----|
| | 0 | 0 | 2 | 1 | 0 | 2 |
| | 1 | 1 | 1 | 0 | 0 | 2 |
|----+----+----+----+----+----+----|
| z | 0 | -2 | -1 | 0 | 1 | -2 |

The tableau is not optimal and the corresponding solution is: x1 = 2, x2 = 0, x3 = 0, x4 = 2,

z = 2.

The solution process is geometrically depicted in Figure 2.14. Phase I starts from the origin,
which for this problem is not a feasible solution, it then jumps to the first feasible solution and from
there with the Phase II to the optimal solution.
2.5. INFEASIBILITY AND INITIALIZATION 47

x2

x1

Figure 2.14: The geometric representation of the feasible example. The blue line represents the
objective function.

Dictionary form In dictionary form, the auxiliary problem can be seen below:

max x1 − x2
x3 = 2 − x1 − x2
x1 + x2 ≤ 2
x4 = −5 + 2x1 + 2x2
2x1 + 2x2 ≥ 5
z = x1 + x2
x1 , x2 ≥ 0
We introduce corrections of infeasibility

max −x0 ≡ min x0

x3 = 2 − x1 − x2
x1 + x2 ≤ 2
x4 = −5 + 2x1 + 2x2 + x0
2x1 + 2x2 − x0 ≥ 5
z = − x0
x1 , x2 , x0 ≥ 0

This new problem is still infeasible but it can be made feasible by letting x0 enter the basis. Which
variable should leave? The most infeasible one: the variable that has the negative b term with the
largest absolute value.
48 CHAPTER 2. THE SIMPLEX METHOD
Chapter 3

Duality

In the previous chapter we saw that the economical interpretation of a given LP problem leads us
to define a dual problem. In this chapter we look at the theory of duality from a more general
and systematic perspective. We present four analytic ways to derive a dual problem. Then we look
at four important theorems that are at the foundation of linear programming. Finally, we present
important practical uses of duality such as the dual simplex method, the sensitivity analysis and
infeasibility proofs.

3.1 Derivation and Motivation

We consider three alternative ways to derive a dual problem for a given LP problem. The bounding
approach, that we see first, is the most intuitive one and gives the opportunity to discuss a geo-
metrical interpretation of duality. The multiplier approach is based on the theory of the simplex
method. We can then give the recipe approach, which is a mechanical way to write the dual. The
last derivation is an application of the Lagrangian relaxation approach, which is a general method
that is often used for solving difficult problems and which is therefore worth learning.

3.1.1 Bounding approach

Suppose we have the following instance of LP problem:

max 4x1 + x2 + 3x3 = z

x1 + 4x2 ≤ 1
3x1 + x2 + x3 ≤ 3
x1 , x2 , x3 ≥ 0

Any feasible solution to this problem provides a lower bound to the objective value. By attempts,
we can get

(x1 , x2 , x3 ) = (1, 0, 0) z∗ ≥ 4
(x1 , x2 , x3 ) = (0, 0, 3) z∗ ≥ 9

Which is the best one? Clearly the largest lower bound 9 is the best lower bound. If we knew that
we cannot do better then we could claim the solution (0, 0, 3) optimal. How can we know what is the
best we can do? We could look at upper bounds. Let’s try to derive one upper bound. We multiply

49
50 CHAPTER 3. DUALITY

left and right hand sides of the constraint inequalities by a positive factor and sum the inequalities.
This will not change the sense of the inequality.

2 · ( x1 + 4x2 ) ≤ 2·1
+3 · ( 3x1 + x2 + x3 ) ≤ 3·3
4x1 + x2 + 3x3 ≤ 11x1 + 11x2 + 3x3 ≤ 11

In the left-most side of the last row we rewrote the objective function of our problem. The left-hand
side of the inequality obtained by summing left and right hand sides is certainly larger than this
function, indeed, the three variables must all be non-negative and their coefficients in the objective
function are one by one smaller than their corresponding coefficients in the left-hand side of the
obtained inequality. Hence z ∗ ≤ 11. Is this the best upper bound we can find?
To obtain this lower bound we chose two arbitrary multipliers y1 , y2 ≥ 0 that preserve the sign
of the inequalities and made a linear combination of the inequalities:

y1 · ( x1 + 4x2 ) ≤ y1 (1)
+y2 · ( 3x1 + x2 + x3 ) ≤ y2 (3)
(y1 + 3y2 )x1 + (4y1 + y2 )x2 + y2 x3 ≤ y1 + 3y2

We aim at
cT x ≤ y T Ax ≤ y T b.
hence we have to impose some restrictions on the multipliers, namely that the coefficients of the
variables xi , i = 1, 2, 3 in the left-hand side of the linear combination of the constraints doe not
exceed the coefficient of the same variables in the objective function:

y1 + 3y2 ≥ 4
4y1 + y2 ≥ 1
y2 ≥ 3

Thus
z = 4x1 + x2 + 3x3 ≤ (y1 + 3y2 )x1 + (4y1 + y2 )x2 + y2 x3 ≤ y1 + 3y2 .
Then to attain the best upper bound we need to solve the following problem:

min y1 + 3y2 = w
y1 + 3y2 ≥ 4
4y1 + y2 ≥ 1
y2 ≥ 3
y1 , y2 ≥ 0

This is the dual problem of our original instance of LP problem. We will soon prove that z ∗ = w∗ .

3.1.2 Geometric Interpretation of Duality

Let’s consider the following example:

max x1 + x2 = z
2x1 + x2 ≤ 14
−x1 + 2x2 ≤ 8
2x1 − x2 ≤ 10
x1 , x2 ≥ 0
3.1. DERIVATION AND MOTIVATION 51

The feasibility region and the objective function are plotted in Figure 3.1, left. The feasible solution
x∗ = (4, 6) yields z ∗ = 10. To prove that this solution is optimal we need to show that no other
feasible solution can do better. To do this we need to verify that y ∗ = (3/5, 1/5, 0) is a feasible
solution of the dual problem:
min 14y1 + 8y2 + 10y3 = w
2y1 − y2 + 2y3 ≥ 1
y1 + 2y2 − y3 ≥ 1
y1 , y2 , y3 ≥ 0
and that w∗ = 10. To put it differently, we multiply the constraints 2x1 + x2 ≤ 14 by 3/5 and
multiply −x1 + 2x2 ≤ 8 by 1/5. Since the sum of the resulting two inequalities reads x1 + x2 ≤ 10,
we conclude that every feasible solution x1 , x2 of the original problem satisfies x1 + x2 ≤ 10,
3
5· 2x1 + x2 ≤ 14
1
5· −x1 + 2x2 ≤ 8
x1 + x2 ≤ 10
Interpreted geometrically, this conclusion means that the entire region of feasibility is contained in
the half plane x1 + x2 ≤ 10; this fact is evident from Figure 3.1, center. Actually, we have support
for a stronger conclusion: the set of all points (x1 , x2 ) satisfying
2x1 + x2 ≤ 14
(3.1)
−x1 + 2x2 ≤ 8
is a subset of the half plane x1 + x2 ≤ 10. Now let us consider linear combinations of these two
inequalities, Each of these combinations has the form:

2(v − w)x1 + (v + 2w)x2 ≤ 14v + 8w (3.2)

for some non-negative v and w. Geometrically, it represents a half-plane that contains the set
represented by (3.1) and whose boundary line passes through the point (4, 6). Examples of these
boundary lines are

v = 1, w = 0 =⇒ 2x1 + x2 = 14
v = 1, w = 1 =⇒ x1 + 3x2 = 22
v = 2, w = 1 =⇒ 3x1 + 4x2 = 36

and are shown in Figure 3.1, right.

The family of all lines 3.2 may be thought of as a single line that is attached to a hinge at
(4, 6) but is free to rotate on the hinge. Continuous changes of the multipliers v and w amount to
continuous rotations of the line. One can chose non-negative v and w so as to make ?? coincide with
the line x1 + x2 = 10. This claim will be shown by the Strong Duality Theorem.

We consider an alternative geometrical interpretation of duality for a general linear programming

problem. This interpretation follows the same lines as the physical interpretation in the text book.
Consider the LP problem:
max{cT x | Ax ≤ b}
and let P = {x | Ax ≤ b} be the feasible region, which is a polyhedron. Finding the maximum of
(3.1.2) can be seen as shifting the hyperplane orthogonal to the vector c (recall from linear algebra
that an hyperplane cT x = d has vector c orthogonal to it), as long as it contains points in P .
52 CHAPTER 3. DUALITY

x2 x2 x2

−x1 + 2x2 ≤ 8
2x1 − x2 ≤ 10

x1 + 3x2 = 22
x1 x1 x1
x1 + x2 x1 + x2 ≤ 10 3xx1 2+
x1 + ≤4x
102 = 36
2x1 + x2 ≤ 14 2x1 + x2 = 14

Figure 3.1:

Suppose the maximum is finite, say its value is z ∗ and attained by the element x∗ of P . Let

a1 x ≤ b1
..
.
ak x ≤ bk

be the inequalities from Ax ≤ b satisfied with equality by x∗ .

Now, geometric insight tells us that cx = z ∗ is a non-negative combination of a1 x = b1 , . . . , ak x =
bk . Say
c = y1∗ a1 + . . . + yk∗ ak
z ∗ = y1∗ b1 + . . . + yk∗ bk
for y1∗ . . . yk∗ ≥ 0. This implies

max{cT x | Ax ≤ b} = z ∗ = y1∗ b1 + . . . + yk∗ bk ≥ min{bT y | y ≥ 0, AT y = c}

(the inequality follows since yi∗ give a feasible solution for the minimum). The inequality holds also
in the opposite verse:
cT x = yAx ≤ bT y
This yields the LP-duality equation

max{cT x | Ax ≤ b} = min{bT y | y ≥ 0, AT y = c}.

3.1.3 Multipliers Approach

An alternative way to derive the dual problem arises from the simplex method. Throughout its
iterations the simplex method performs linear combinations of the rows of the tableau aiming at a
final tableau that is optimal. Given the LP problem max{cT x | Ax ≤ b, x ≥ 0}, the conditions of
optimality for the tableau are c̄ ≤ 0. The multipliers of the rows of the tableau that have to be
subtracted to the last row can be aggregated throughout the iterations of the simplex. At the end
the last row of the final tableau will be a linear combination of the rows of the initial tableau. Let’s
call π1 , . . . , πm , πm+1 the multipliers of the linear combination from the first to the last tableau.
 
π1 a11 a12 . . . a1n a1,n+1 a1,n+2 . . . a1,m+n 0 b1
..  .. .. 
.  .
. 

πm  am1 am2 . . . amn am,n+1 am,n+2 . . . am,m+n 0 bm 
πm+1 c1 c2 . . . cn 0 0 ... 0 1 0
3.1. DERIVATION AND MOTIVATION 53

Working columnwise, the conditions of optimality since c̄k ≤ 0 for all k = 1, . . . , n+m can be written
as 

 π1 a11 + π2 a21 . . . + πm am1 + πm+1 c1 ≤ 0

 . . ..

 .. .. .



 π a + π a . . . + π a + π c ≤ 0

 1 1n 2 2n m mn m+1 n

π1 a1,n+1 , π2 a2,n+1 , . . . πm am,n+1 ≤ 0
.. .. .. .. .. .. (3.3)

 . . . . . .





 π1 a1,n+m , π2 a2,n+m , . . . πm am,n+m ≤ 0



 πm+1 = 1

π1 b1 + π2 b2 ... + πm bm (≤ 0)
For the columns that correspond to the variables in basis (eg, from n + 1 to n + m, ie, the second
block of constraints in the system (3.3)) we will have an identity matrix and hence πi ≤ 0, i = 1..m.
From the last row of the final tableau we read z = −πb. Since we want to maximize z then we would
try to solve for min(−πb) or equivalently for max πb. We can then rewrite:
max π1 b1 + π2 b2 . . . + πm bm
π1 a11 + π2 a21 . . . + πm am1 ≤ −c1
.. .. ..
. . .
π1 a1n + π2 a2n . . . + πm amn ≤ −cn
π1 , π2 , . . . πm ≤ 0
and setting y = −π and substituting we obtain:
max −y1 b1 + −y2 b2 . . . + −ym bm
−y1 a11 + −y2 a21 . . . + −ym am1 ≤ −c1
.. .. ..
. . .
−y1 a1n + −y2 a2n . . . + −ym amn ≤ −cn
−y1 , −y2 , . . . − ym ≤ 0
which leads to the same dual as we saw in the previous section:
min bT y = w
AT y ≥ c
y ≥ 0
Example 3.1. We sketch a numerical example:
max 6x1 + 8x2
5x1 + 10x2 ≤ 60
4x1 + 4x2 ≤ 40
x1 , x2 ≥ 0


 5π1 + 4π2 + 6π3 ≤ 0

 10π1 + 4π2 + 8π3 ≤ 0



1π1 + 0π2 + 0π3 ≤ 0

 0π1 + 1π2 + 0π3 ≤ 0



 0π1 + 0π2 + 1π3 = 1

60π1 + 40π2
which we transform by setting:
y1 = −π1 ≥ 0
y2 = −π2 ≥ 0
54 CHAPTER 3. DUALITY

Figure 3.2:

3.1.4 Duality Recipe

We saw earlier that any problem can be put in the standard form and now we derived the dual
problem for this form. By using the same transformation techniques we can write the mechanical
rules to write the dual of a certain LP problem. They are listed in Figure 3.2, which is taken from
our textbook.

3.2 Duality Theory

Theorem 3.1 (Symmetry of duality). The dual of the dual is the primal.

Proof. Let’s write the dual pair:

Primal problem: Dual problem:

max z = cT x min w = bT y
(P) Ax ≤ b (D) AT y ≥ c
x ≥ 0 y ≥ 0.

We proceed to derive the dual of (D). We put first the dual in the standard form:

min bT y ≡ − max −bT y

−AT y ≤ −c
y ≥ 0
3.2. DUALITY THEORY 55

then we use the same rules used to derive (D) from (P) to derive the dual of (D)

− min cT x
−Ax ≥ −b
x ≥ 0

From the derivations we saw already that the dual problem yields upper bounds (to maximization
primal problems). This is true in general:

Theorem 3.2 (Weak Duality Theorem). Given:

(P) max{cT x | Ax ≤ b, x ≥ 0}
(D) min{bT y | AT y ≥ c, y ≥ 0}

for any feasible solution x of (P) and any feasible solution y of (D):

cT x ≤ bT y
P
Proof. In scalar form, from the feasibility of x in (P) we have that nj=1P
aij xi ≤ bi for i = 1..m and
xj ≥ 0 for j = 1..n. From the feasibility of y in (D) we have that cj ≤ m i=1 yi aij for i = 1..m and
yi ≥ 0 for j = 1..n. It follows that:
!  
n
X n
X m
X m
X n
X m
X
cj xj ≤ yi aij xj =  aij xi  yi ≤ bi yi .
j=1 j=1 i=1 i=1 j=1 i=1

The following theorem is due to Von Neumann and Dantzig, 1947, and Gale, Kuhn and Tucker,
1951.

Theorem 3.3 (Strong Duality Theorem). Given:

(P) max{cT x | Ax ≤ b, x ≥ 0}
(D) min{bT y | AT y ≥ c, y ≥ 0}

exactly one of the following occurs:

1. (P) and (D) are both infeasible

2. (P) is unbounded and (D) is infeasible

3. (P) is infeasible and (D) is unbounded

4. (P) has feasible solution x∗ = [x∗1 , . . . , x∗n ]T , (D) has feasible solution y∗ = [y1∗ , . . . , ym
∗ ]T and

c T x ∗ = bT y ∗
56 CHAPTER 3. DUALITY

Proof. All other combinations of the 3 possible outcomes (Optimal, Infeasible, Unbounded) for (P)
and (D) are ruled out by the weak duality theorem. For example, the combination (P) unbounded
(+∞) and (D) unbounded (−∞) would clearly violate the weak duality theorem. To show 4, we use
the simplex method (other proofs independent of the simplex method exist, eg, via Farkas Lemma
and convex polyhedral analysis.)
To prove the statement 4. we need to exhibit for an optimal solution x∗ a dual feasible solution y∗
satisfying cT x∗ = bT y∗ . Suppose we apply the simplex method. We know that the simplex method
produces an optimal solution whenever one exists, and in statement 4. we are assuming that one
does indeed exist. Let x∗ be this optimal solution. The final dictionary will be an optimal dictionary
for the primal problem. The objective function in this final dictionary is ordinarily written (see page
34) as

n+m
X X X
z = d¯ + c̄k xk = d¯ + c̄r xr + c̄s xs
k=1 r∈B s6∈B

But, since this is the optimal dictionary and we prefer stars to bars for denoting optimal “stuff”, let us
¯ Also, the reduced costs of the basic variables will be zero and nonbasic variables
write z ∗ instead of d.
will generally consist of a combination of original variables as well as slack variables. Instead of using
c̄k for the coefficients of all these variables, let us differentiate and use c̄j for the objective coefficients
corresponding to original variables, and let us use c̄i for the objective coefficients corresponding to
slack variables. Also, for those original variables that are basic we put c̄j = 0, and for those slack
variables that are basic we put c̄i = 0. With these new notations, we can rewrite the objective
function as
n
X m
X
∗
z=z + c̄j xj + c̄n+i xn+i (3.4)
j=1 i=1

In addition, since we know that x∗ is the optimal solution to the primal:

n
X
∗
z = cj x∗j
j=1

(This is just the original objective function with the substitution of the optimal value for the original
variables.)
We now define
yi∗ = −c̄n+i , i = 1, 2, . . . , m.

and we claim that y∗ = [y1∗ , y2∗ , . . . , ym

∗ ]T is a dual feasible solution satisfying cT x∗ = bT y∗ .

Let’s verify the claim. We substitute in (3.4):

Pn
• z= j=1 cj xj ,

• c̄n+1 = −yi∗ for i = 1, 2, . . . , m and

Pn
• xn+i = bi − j=1 aij xj for i = 1, 2, . . . , m (from the definition of the m slack variables)
3.2. DUALITY THEORY 57

and obtain
 
n
X n
X m
X n
X
cj xj = z ∗ + c̄j xj − yi∗ bi − aij xj 
j=1 j=1 i=1 j=1
m
! n m
!
X X X
= z∗ − yi∗ bi + c̄j + aij yi∗ xj
i=1 j=1 i=1

This must hold for every [x1 , x2 , . . . , xn ] hence:

m
X
∗
z = bi yi∗ (3.5)
i=1
m
X
cj = c̄j + aij yi∗ , j = 1, 2, . . . , n (3.6)
i=1

Equation (3.5) implies that y∗ satisfies cT x∗ = bT y∗ . Since c̄k ≤ 0 for all k = 1, 2, . . . , n + m

(optimality condition for the final tableau), for the original variables we have:
m
X m
X
c̄j ≤ 0 =⇒ cj − yi∗ aij ≤ 0 =⇒ yi∗ aij ≥ cj j = 1, 2, . . . , n (3.7)
i=1 i=1

and for the slack variables:

c̄n+i ≤ 0 =⇒ yi∗ = −ĉn+i ≥ 0 =⇒ yi∗ ≥ 0 i = 1, 2, . . . , m (3.8)

Equations (3.7)-(3.8) imply that y ∗ is also dual feasible solution.

Theorem 3.4 (Complementary Slackness). Given a feasible solution x∗ for (P) and a feasible
solution y∗ for (D), necessary and sufficient conditions for the optimality of both are:
m
!
X
cj − yi∗ aij x∗j = 0, j = 1, . . . , n
i=1
P
This
P ∗ implies that for any j = 1..n, if x∗j 6= 0 then yi∗ aij = cj (no slack nor surplus) and if
∗
yi aij > cj then xj = 0.
Proof. From the weak duality theorem:

z ∗ = cT x∗ ≤ y∗T Ax∗ ≤ bT y∗ = w∗ .

Then, from the strong duality theorem z ∗ = w∗ , hence:

cT x∗ − y∗T Ax∗ = 0

In scalars: !
n
X m
X
cj − yi∗ aij x∗j = 0
j=1 i=1
|{z}
| {z } ≥0
≤0

Hence each term of the sum for j = 1..n must be = 0.

58 CHAPTER 3. DUALITY

Application: Economic Interpretation

max 5x0 + 6x1 + 8x2

6x0 + 5x1 + 10x2 ≤ 60
8x0 + 4x1 + 4x2 ≤ 40
4x0 + 5x1 + 6x2 ≤ 50
x0 , x1 , x2 ≥ 0

final tableau:
x0 x1 x2 s1 s2 s3 −z b
0 1 0 5/2
1 0 0 7
0 0 1 2
−1/5 0 0 −1/5 0 −1 −62

• Which are the values of the original variables? (0, 7, 5/2)

• Which are the values of the reduced costs? (−1/5, 0, 0, −1/5, 0, −1)

• Which are the values of the tableau multipliers? (−1/5, 0, −1)

• Which are the values of the values of dual variables? (1/5, 0, 1) - for the proof of the Strong
duality theorem.

• Which are the values of the shadow prices (or marginal values of the resources)? (1/5, 0, 1)

• Which is the value of the slack variables? (0, 2, 0)

• If one slack variable > 0 then there is overcapacity, that is, the constraint to which the slack
variable belongs is not tight. This can be assessed also via complementary slackness theorem:
y2 is dual variable associated with the second constraint, y2 = 0 from the tableau, hence the
second constraint is not binding.

• How many products can be produced at most? at most m = 3.

Game: Suppose two economic operators:

• P owns the factory and produces goods

• D is the market buying and selling raw material and resources

• D asks P to close and sell him all resources

• P considers if the offer is convenient

• D wants to spend least possible

• y are prices that D offers for the resources

P
• yi bi is the amount D has to pay to have all resources of P
P
• yi aij ≥ cj , ie, the total value to make produce j must be larger than the price per unit of
product
P
• P either sells all resources yi aij or produces product j and earns cj
3.3. LAGRANGIAN DUALITY 59

P
• without constraint yi aij ≥ cj there would not be negotiation because P would be better off
producing and selling
• at optimality the situation is indifferent (strong duality theorem)
• resource 2 that was not totally utilized in the primal has been given value 0 in the dual.
(complementary slackness theorem) Plausible, since we do not use all the resource, likely to
place not so much value on it
P
• for product 0, yi aij > cj hence not profitable producing it (complementary slackness theo-
rem)

3.3 Lagrangian Duality

Relaxation: if a problem is hard to solve then find an easier problem resembling the original one
that provides information in terms of bounds. Then search strongest bounds.

min 13x1 + 6x2 + 4x3 +12x4

2x1 + 3x2 + 4x3 + 5x4 = 7
3x1 + + 2x3 + 4x4 = 2
x1 , x2 , x3 , x4 ≥ 0
We wish to reduce it to a problem easier to solve, ie:
min c1 x1 + c2 x2 + . . . +cn xn
x1 , x2 , . . . , xn ≥ 0
This problem is solvable by inspection: if c < 0 then x = +∞, if c ≥ 0 then x = 0.
Let’s then relax the constraints of our problem by adding to the function to minimize a measure
of the violations of each constraint, ie:
7 − (2x1 + 3x2 + 4x3 + 5x4 )
2 − (3x1 + + 2x3 + 4x4 )
We relax these measures in the objective function with Lagrangian multipliers y1 , y2 . We obtain a
family of problems:
 
 13x1 + 6x2 + 4x3 + 12x4 
P R(y1 , y2 ) = min +y1 (7 − ( 2x1 + 3x2 + 4x3 + 5x4 ))
x1 ,x2 ,x3 ,x4 ≥0  
+y2 (2 − ( 3x1 + + 2x3 + 4x4 ))
The following two properties hold:

1. for all y1 , y2 ∈ R : opt(P R(y1 , y2 )) ≤ opt(P )

2. maxy1 ,y2 ∈R {opt(P R(y1 , y2 ))} ≤ opt(P )

(The second statement can be also seen as a proof of the weak duality theorem.) The problem
PR is easy to solve:
 

 (13 − 2y1 − 3y2 ) x1 


 
 + (6 − 3y1 ) x2 

P R(y1 , y2 ) = min + (4 − 4y1 − 2y2 ) x3
x1 ,x2 ,x3 ,x4 ≥0 
 

 + (12 − 5y1 − 4y2 ) x4 


 
+ 7y1 + 2y2
60 CHAPTER 3. DUALITY

if the coefficients of xi is < 0 then the bound is −∞ and hence useless to our purposes of finding
the strongest bound. Hence,
(13 − 2y1 − 3y2 ) ≥ 0
(6 − 3y1 ) ≥ 0
(4 − 4y1 − 2y2 ) ≥ 0
(12 − 5y1 − 4y2 ) ≥ 0
If they all hold then we are left with 7y1 + 2y2 because all other terms go to 0. Thus,

max 7y1 + 2y2

2y1 + 3y2 ≤ 13
3y1 ≤ 6
4y1 + 2y2 ≤ 4
5y1 + 4y2 ≤ 12

which is the dual problem we were trying to derive.

The general formulation in vector notation is the following:

min z = cT x c ∈ Rn
Ax = b A ∈ Rm×n , b ∈ Rm
x ≥ 0 x ∈ Rn

max { min {cT x + yT (b − Ax)}}

y∈Rm x∈Rn
+

max { min {(c − yA)x + yb}}

y∈Rm x∈Rn
+

max bT y
AT y ≤ c
y ∈ Rm

3.4 Dual Simplex

We saw that as the simplex method solves the primal problem, it also implicitly solves the dual
problem. Indeed, the value of the dual variables is the value of the reduced cost of the slack
variables with a change of sign. This fact is a consequence of the strong duality theorem or of the
multiplier method for the derivation of the dual. The idea is then to apply the simplex method to
the dual problem and observe what happens in the primal tableau. We obtain a new algorithm for
the primal problem: the dual simplex (Lemke, 1954). The dual simplex corresponds to the primal
simplex applied to the dual.
The derivation of this new algorithm can be obtained by following step by step the meaning of
the pivot iterations in the primal and their corresponding meaning in the dual when it is put in
standard form:

max{cT x | Ax ≤ b, x ≥ 0} = min{bT y | AT y ≥ cT , y ≥ 0}
= − max{−bT y | −AT x ≤ −cT , y ≥ 0}

The resulting pivot operation is given here in the right column:

3.4. DUAL SIMPLEX 61

Figure 3.3: The figure shows an iteration of the simplex in the primal problem and the corresponding
step in the dual problem.

Primal simplex on primal problem: Dual simplex on primal problem:

1. pivot > 0 1. pivot < 0

2. col cj with wrong sign 2. row bi < 0

n o (condition of feasibility)
3. row: min abiji : aij > 0, i = 1, .., m n  o
c 
3. col: min  aijj  : aij < 0, j = 1, 2, .., n + m
(least worsening solution)
The dual simplex can sometimes be preferable to the primal simplex. Since the running time of
the primal simplex is in practice between 2m and 3m iterations, then if m = 99 and n = 9 then
better to transform the problem in its dual and solve by the primal algorithm, or even better leave
the problem as it is but solve it with the dual simplex.

An alternative view: we can solve the primal problem with the primal simplex and observe what
happens in the dual problem. This has an important application. Since the last terms of the tableau
become the right hand side terms of the constraints, whenever a tableau in the primal problem is not
optimal, the corresponding tableau in the dual problem is non canonical (or infeasible). Iterating in
the two algorithms we observe that:

• the primal simplex works with feasible solutions towards optimality

• the dual simplex works with optimal solutions towards feasibility.

While in the primal simplex we increase a variable that can improve the objective function, in
the dual simplex we take a constraint that is not yet satisfied and use it to diminish the value of a
variable until the constraint becomes satisfied. See Figure 3.3.
Hence, the dual simplex applied on the primal problem can be used for resolving an infeasible
start. This yields a dual based Phase I algorithm (Dual-primal algorithm) (see Ex 10, Sheet 3)

Example 3.2. Consider the following LP problem with its dual:

max −x1 − x2
min 4y1 − 8y2 − 7y3
−2x1 − x2 ≤ 4
−2y1 − 2y2 − y3 ≥ −1
−2x1 + 4x2 ≤ −8
−y1 + 4y2 + 3y3 ≥ −1
−x1 + 3x2 ≤ −7
y1 , y2 , y3 ≥ 0
x1 , x2 ≥ 0
62 CHAPTER 3. DUALITY

We solve in parallel both problems:

• The initial tableau • The initial tableau (min by ≡ − max −by)
| | x1 | x2 | w1 | w2 | w3 | -z | b |
|---+----+----+----+----+----+----+----| | | y1 | y2 | y3 | z1 | z2 | -p | b |
| | -2 | -1 | 1 | 0 | 0 | 0 | 4 | |---+----+----+----+----+----+----+---|
| | -2 | 4 | 0 | 1 | 0 | 0 | -8 | | | 2 | 2 | 1 | 1 | 0 | 0 | 1 |
| | -1 | 3 | 0 | 0 | 1 | 0 | -7 | | | 1 | -4 | -3 | 0 | 1 | 0 | 1 |
|---+----+----+----+----+----+----+----| |---+----+----+----+----+----+----+---|
| | -1 | -1 | 0 | 0 | 0 | 1 | 0 | | | -4 | 8 | 7 | 0 | 0 | 1 | 0 |

exhibits an infeasible start. has a feasible start (thanks to −x1 − x2 )

Hence, we use the dual simplex. Hence, we use the primal simplex.

• x1 enters, w2 leaves • y2 enters, z1 leaves

| | x1 | x2 | w1 | w2 | w3 | -z | b | | | y1 | y2 | y3 | z1 | z2 | -p | b |
|---+----+----+----+------+----+----+----| |---+----+----+-----+-----+----+----+-----|
| | 0 | -5 | 1 | -1 | 0 | 0 | 12 | | | 1 | 1 | 0.5 | 0.5 | 0 | 0 | 0.5 |
| | 1 | -2 | 0 | -0.5 | 0 | 0 | 4 | | | 5 | 0 | -1 | 2 | 1 | 0 | 3 |
| | 0 | 1 | 0 | -0.5 | 1 | 0 | -3 | |---+----+----+-----+-----+----+----+-----|
|---+----+----+----+------+----+----+----| | | -4 | 0 | 3 | -12 | 0 | 1 | -4 |
| | 0 | -3 | 0 | -0.5 | 0 | 1 | 4 |

• w2 enters, w3 leaves (note that we kept • y3 enters, y2 leaves

cj < 0, ie, optimality)
| | y1 | y2 | y3 | z1 | z2 | -p | b |
| | x1 | x2 | w1 | w2 | w3 | -z | b | |---+-----+----+----+----+----+----+----|
|---+----+----+----+----+----+----+----| | | 2 | 2 | 1 | 1 | 0 | 0 | 1 |
| | 0 | -7 | 1 | 0 | -2 | 0 | 18 | | | 7 | 2 | 0 | 3 | 1 | 0 | 3 |
| | 1 | -3 | 0 | 0 | -1 | 0 | 7 | |---+-----+----+----+----+----+----+----|
| | 0 | -2 | 0 | 1 | -2 | 0 | 6 | | | -18 | -6 | 0 | -7 | 0 | 1 | -7 |
|---+----+----+----+----+----+----+----|
| | 0 | -4 | 0 | 0 | -1 | 1 | 7 |

The Phase I is thus terminated. We note that the tableaux are optimal also with respect to the
Phase II hence we are done.

3.5 Sensitivity Analysis

The sensitivity analysis is done when the final tableau has been reached. Hence, it is also called
postoptimality analysis. It aims at assessing the robustness of the solution or the effects of small
changes to the input data.
We recall the economic interpretation of an LP problem. We reconsider our starting example of
resource allocation problem to determine the best product mix.

max 5x0 + 6x1 + 8x2

6x0 + 5x1 + 10x2 ≤ 60
8x0 + 4x1 + 4x2 ≤ 40
4x0 + 5x1 + 6x2 ≤ 50
x0 , x1 , x2 ≥ 0

The final tableau for this problem is the following (we show only the numbers that are relevant for
3.5. SENSITIVITY ANALYSIS 63

our analysis):
x0 x1 x2 s1 s2 s3 −z b
0 1 0 5/2
1 0 0 7
0 0 1 2
−1/5 0 0 −1/5 0 −1 −62

What-if anaylsis: what changes in the solution if some input data changes? Instead of solving
each modified problems from scratch, exploit results obtained from solving the original problem.

• How much more expensive a product not selected should be? Look at reduced costs, we want:
ci − πaj > 0 hence we must increase the original cost ci .

• What is the value of extra capacity of manpower? Adding 1 + 1 units of the first and second
resource we obtain an increase in objective value of 1/5 + 1, respectively.

Let’s consider the possibilities for a general case:

max{cT x | Ax = b, l ≤ x ≤ u} (3.9)

with optimal solution x∗ .

(I) changes to coefficients of objective function: max{c̃T x | Ax = b, l ≤ x ≤ u}

x∗ of (3.9) remains feasible hence we can restart the simplex from x∗ (primal iteration).

(II) changes to RHS terms: max{cT x | Ax = b̃, l ≤ x ≤ u}

from x∗ optimal feasible solution of (3.9) construct a basic sol x̄ of (II): x̄N = x∗N , AB x̄B =
b̃ − AN x̄N . x̄ is dual feasible and we can start the dual simplex from there (dual iteration).
If b̃ differs from b only slightly it may be we are already optimal.

(III) introduce a new variable:

P6 P7
max j=1 cj xj max j=1 cj xj
P6 P7
j=1 aij xj= bi , i = 1, . . . , 3 j=1 aij xj = bi , i = 1, . . . , 3
lj ≤ xj ≤ uj , j = 1, . . . , 6 lj ≤ xj ≤ uj , j = 1, . . . , 7
[x∗1 , . . . , x∗6 ] feasible [x∗1 , . . . , x∗6 , 0] feasible

A new fesaible solution is easily derived by setting the new variable to zero. We need to check
whether it is worth increasing it (primal iteration).

(IV) introduce a new constraint:

P6 (x∗1 , . . . , x∗6 ) optimal

a4j xj = b4
Pj=1 (x∗1 , . . . , x∗6 , x∗7 , x∗8 ) feasible
6 P
j=1 a5j xj = b5 x∗7 = b4 − 6j=1 a4j x∗j
lj ≤ xj ≤ uj j = 7, 8 P
x∗8 = b5 − 6j=1 a5j x∗j

It may render x∗ infeasible (dual iteration).

64 CHAPTER 3. DUALITY

Example 3.3. (I) Variation of a coefficient in the objective function.

x1 x2 x3 x4 −z b
x3 5 10 1 0 0 60
max 6x1 + 8x2 x4 4 4 0 1 0 40
5x1 + 10x2 ≤ 60 6 8 0 0 1 0
4x1 + 4x2 ≤ 40 x1 x2 x3 x4 −z b
x1 , x2 ≥ 0 x2 0 1 1/5 −1/4 0 2
x1 1 0 −1/5 1/2 0 8
0 0 -2/5 -1 1 −64

The last tableau on the right gives the possibility to estimate the effect of variations. For a
variable in basis the perturbation goes unchanged in the reduced costs. Eg:

2
max(6 + δ)x1 + 8x2 =⇒ c̄1 = − · 5 − 1 · 4 + 1(6 + δ) = δ.
5
If δ > 0 then the variable must enter in basis and we need to bring the tableau in canonical form
for the new basis and hence δ changes the obj value. For a variable not in basis, if it changes the
sign of the reduced cost then it is worth bringing in basis. The δ term propagates to other columns
via pivot operations.

Example 3.4. (II) Changes in the right-hand side (RHS) terms

x1 x2 x3 x4 −z b
x3 5 10 1 0 0 60 + δ
x4 4 4 0 1 0 40 + 
6 8 0 0 1 0

x1 x2 x3 x4 −z b
x2 0 1 1/5 -1/4 0 2 + 1/5δ − 1/4
x1 1 0 −1/5 1/2 0 8 − 1/5δ + 1/2
0 0 -2/5 -1 1 −64 − 2/5δ − 

Looking at the cell in the bottom-left corner of the tableau, −64 − 2/5δ − , we see what would
be the contribution to the objective value of an increase of δ and  of the resources. If both δ =  = 1
then it would be more convenient to augment the second resources.
Let’s analyze the situation when only one of the resources changes, ie, let  = 0. If 60 + δ
=⇒all RHS terms change and we must check feasibility. Which are the multipliers for the first
row?π1 = 51 , π2 = − 41 , π3 = 0.
I: 1/5(60 + δ) − 1/4 · 40 + 0 · 0 = 12 + δ/5 − 10 = 2 + δ/5
II: −1/5(60 + δ) + 1/2 · 40 + 0 · 0 = −60/5 + 20 − δ/5 = 8 − 1/5δ
Risk that the RHS becomes negative. Eg: if δ = −20 then the tableau stays optimal but not
feasible. We need to apply the dual simplex and the increase in the objective value would therefore
be less than what prospected by the marginal values. In Figure 3.4, we plot the objective value as
a function of the increase δ.

Example 3.5. (III) Introduction of a new variable

3.5. SENSITIVITY ANALYSIS 65

f.o.

60 + 2/5δ

δ
-10 40

Figure 3.4: The influence of δ on the objective value.

max 5x0 + 6x1 + 8x2

6x0 + 5x1 + 10x2 ≤ 60
8x0 + 4x1 + 4x2 ≤ 40
x0 , x1 , x2 ≥ 0
What will be the reduced cost of x0 in the final tableau?
X 2 27
cj + πi aij = +1 · 5 − · 6 + (−1)8 = −
5 5
To make the variable worth entering in basis:
• increase its profit

• decrease the amount in constraint II: −2/5 · 6 − a20 + 5 > 0

Example 3.6. (IV) Introduction of a new constraint

max 6x1 + 8x2

5x1 + 10x2 ≤ 60
4x1 + 4x2 ≤ 40
5x1 + 6x2 ≤ 50
x1 , x2 ≥ 0
Final tableau not in canonical form, need to iterate

x1 x2 x3 x4 x5 −z b
x2 0 1 1/5 −1/4 0 2
x1 1 0 −1/5 1/2 0 8
0 0 5/5 6/4 1 0 −2
0 0 −2/5 −1 0 1 −64
66 CHAPTER 3. DUALITY

Example 3.7. (V) Change in a technological coefficient:

x1 x2 x3 x4 −z b
x3 5 10 + δ 1 0 0 60
x4 4 4 0 1 0 40
6 8 0 0 1 0

• first effect on its column

• then look at c

• finally look at b

x1 x2 x3 x4 −z b
x2 0 (10 + δ)1/5 + 4(−1/4) 1/5 −1/4 0 2
x1 1 (10 + δ)(−1/5) + 4(1/2) −1/5 1/2 0 8
0 −2/5δ −2/5 −1 1 −64

The dominant application of LP is mixed integer linear programming. In this context it is

extremely important being able to begin with a model instantiated in one form followed by a sequence
of problem modifications (such as row and column additions and deletions and variable fixings)
interspersed with resolves

3.6 Farkas Lemma

Farkas Lemma gives us another proof of the strong duality theorem. Moreover, it provides a way to
certificate the infeasibility of an LP instance.
Lemma 1 (Farkas). Let A ∈ Rm×n and b ∈ Rm . Then,

either I. ∃x ∈ Rn : Ax = b and x ≥ 0
or II. ∃y ∈ Rm : yT A ≥ 0T and yT b < 0

Easy to see that both I and II cannot occur together:

(0 ≤) yT Ax = yT b (< 0)

Geometric interpretation of Farkas Lemma Linear combination of ai with nonnegative terms

generates a convex cone:
{λ1 a1 + . . . + λn an , | λ1 , . . . , λn ≥ 0}
Polyhedral cone: C = {x | Ax ≤ 0}, intersection of many ax ≤ 0. Convex hull of rays pi =
{λi ai , λi ≥ 0}
Either point b lies in convex cone C
or ∃ hyperplane h passing through point 0 h = {x ∈ Rm : yT x = 0} for y ∈ Rm such that all vectors
a1 , . . . , an (and thus C) lie on one side and b lies (strictly) on the other side (ie, yT ai ≥ 0, ∀i = 1 . . . n
and yT b < 0).
3.6. FARKAS LEMMA 67

Variants of Farkas Lemma

Corollary. (i) Ax = b has sol x ≥ 0 ⇐⇒ ∀y ∈ Rm with yT A ≥ 0T , yT b ≥ 0

(ii) Ax ≤ b has sol x ≥ 0 ⇐⇒ ∀y ≥ 0 with yT A ≥ 0T , yT b ≥ 0

(iii) Ax ≤ 0 has sol x ∈ Rn ⇐⇒ ∀y ≥ 0 with yT A = 0T , yT b ≥ 0

Proof. We show only that i) =⇒ ii) since it will be useful in our proof of the duality theorem.
Ā = [A | Im ]
Ax ≤ b has sol x ≥ 0 ⇐⇒ Āx̄ = b has sol x̄ ≥ 0
By (i):
∀y ∈ Rm
yT b ≥ 0, yT Ā ≥ 0
yT A ≥ 0
y≥0
relation with Fourier & Moutzkin method

Strong Duality by Farkas Lemma

(P ) max{cT x | Ax ≤ b, x ≥ 0}

Assume P has opt sol x∗ with value z ∗ . We find that D has opt sol as well and its value coincide
with z ∗ .
Opt value for P:
γ = c T x∗
We know by assumption:
and ∀ > 0
Ax ≤ b Ax ≤ b
has sol x ≥ 0 has no sol x ≥ 0
cT x ≥ γ cT x ≥ γ + 
Let’s define:
   
A b
 = b̂ =
−cT −γ − 

and consider Âx ≤ b̂0 and Âx ≤ b̂

68 CHAPTER 3. DUALITY

We apply variant (ii) of Farkas’ Lemma:

For  = 0, Âx ≤ b̂0 has sol x ≥ 0 For  > 0, Âx ≤ b̂ has no sol x ≥ 0
is equivalent to: is equivalent to:
there exists ŷ = (u, z) ∈ Rm+1 , there exists ŷ = (u, z) ∈ Rm+1 ,

ŷ ≥ 0 ŷ ≥ 0
ŷT Â ≥ 0 ŷT Â ≥ 0
ŷT b0 ≥ 0 ŷT b < 0

Then Then
AT u ≥ zc AT u ≥ zc
T
b u ≥ zγ bT u < z(γ + )
Hence, z > 0 or z = 0 would contradict the separation of cases.
We can set v = z1 u ≥ 0 By weak duality γ is lower bound for D. Since D
bounded and feasible then there exists y∗ :
T
A v≥c
bT v < γ +  γ ≤ bT y ∗ < γ +  ∀ > 0

v is feasible sol of D with objective value < γ +  which implies bT y∗ = γ

Certificate of Infeasibility Farkas Lemma provides a way to certificate infeasibility.

Theorem 4. Given a certificate y∗ it is easy to check the conditions (by linear algebra):

AT y ∗ ≥ 0
by∗ < 0

Why would y∗ be a certificate of infeasibility?

Proof. (By contradiction)

Assume, AT y∗ ≥ 0 and by∗ < 0.
Moreover assume ∃x∗ : Ax∗ = b, x∗ ≥ 0,then:

(≥ 0) (y∗ )T Ax∗ = (y∗ )T b (< 0)

Contradiction

General form:
infeasible ⇔ ∃y ∗
max cT x
A1 x = b1 bT1 y1 + bT2 y2 + bT3 y3 > 0
A2 x ≤ b2 AT1 y1 + AT2 y2 + AT3 y3 ≤ 0
A3 x ≥ b3 y2 ≤ 0
x ≥ 0 y3 ≥ 0
Example 3.8.
max cT x
x1 ≤ 1
x1 ≥ 2
bT1 y1 + bT2 y2 > 0
AT1 y1 + AT2 y2 ≤ 0
y1 ≤ 0
y2 ≥ 0
3.7. SUMMARY 69

y1 + 2y2 > 0
y1 + y2 ≤ 0
y1 ≤ 0
y2 ≥ 0
y1 = −1, y2 = 1 is a valid certificate.
Note that the Farkas’ infeasibility certificate is not unique! It can be reported in place of the
dual solution because they have the same dimension. To repair infeasibility we should change the
primal at least so much so that the certificate of infeasibility is no longer valid. Only constraints
with yi 6= 0 in the certificate of infeasibility cause infeasibility.

3.7 Summary
In this chapter we have presented the following topics regarding LP duality:

• Derivation:
1. bounding
2. multipliers
3. recipe
4. Lagrangian
• Theory:
– Symmetry
– Weak duality theorem
– Strong duality theorem
– Complementary slackness theorem
– Farkas Lemma: Strong duality + Infeasibility certificate
• Dual Simplex
• Economic interpretation
• Geometric Interpretation
• Sensitivity analysis

The main advantages of considering the dual formulation are:

• proving optimality (although the simplex tableau can already do that)

• checking the correctness of results easily
• attaining an alternative solution method (ie, primal simplex on dual)
• making analysis of sensitivity with respect to the parameters.
• solving P or D we solve the other for free
• attaining a certificate of infeasibility
70 CHAPTER 3. DUALITY
Chapter 4

Revised Simplex Method

The running time of the simplex algorithms depends on the total number of pivot operations and by
the cost of each single operation. We have already discussed in the previous chapter what is known
about the number of iterations. Let’s now focus on the cost of a single iteration.
The complexity of a single pivot operation in the standard simplex is determined by:

• entering variable: O(n)

• leaving variable: O(m)

• updating the tableau: O(mn)

Hence, the most costly operation in the simplex is updating the tableau in the pivot operation.
We can observe that we are doing operations that are not actually needed. For example, in
the tableau the only columns that really matters is the one of the entering variable. Moreover,
we have space issues: we need to store the whole tableau, that is, O(mn) floating point numbers,
this can become a lot: for 1000 constraints and 50000 variables in double precision floating point
storing the whole tableau yields 400MB. Further, most problems have sparse matrices (they contain
many zeros). Sparse matrices are typically handled efficiently by special storing ways and specialized
operators. Instead, the standard simplex immediately disrupts sparsity by problem with an iterated
method like the simplex is that floating point errors accumulate and may become very important.
There are several ways to improve the efficiency of the pivot operation. To gain a better insight
we need a matrix description of simplex. As (we have mostly tried) in the previous chapter, all
vectors are column vectors and denoted by lowercase letters in bold face. Matrices are denoted in
upper case letters.
We consider a general LP problem in standard form
P
n
max cj xj
j=1
P
n
aij xj ≤ bi i = 1..m
j=1
xj ≥ 0 j = 1..n

After the introduction of the slack variables xn+1 , xn+2 , . . . , xn+m the problem can be written in
vector form as
max cT x
Ax = b
x ≥ 0

71
72 CHAPTER 4. REVISED SIMPLEX METHOD

or
max{cT x | Ax = b, x ≥ 0}
where x ∈ Rn+m , A ∈ Rm×(n+m) , c ∈ R(n+m) , and b ∈ Rm .
We aim at understanding the relationship between each tableau and the initial data. We saw
that every tableau corresponds to a basic feasible solution (the reverse is not true because of the
possible degenerancies). For each basic feasible solution:

• B = {1 . . . m} is denoted as the basis

• N = {m + 1 . . . m + n} is the set of indices of the variables not in basis

• AB = [a1 . . . am ] is the basis matrix (or basis)

• AN = [am+1 . . . am+n ] is the remaining matrix after removal of AB from A.

Moreover,

• xN = 0

• xB ≥ 0

We can represent this information in tableau form:

 
 
 
 AN AB 0 b 
 
 
 
 
cTN cTB 1 0

We can now rewrite Ax = b as:

Ax = AN xN + AB xB = b (4.1)
AB xB = b − AN xN (4.2)

Theorem 4.1. Basic feasible solution ⇐⇒ AB is non-singular (ie, the rows are linearly independent
and det(A) 6= 0).

Proof. We have already shown previously that if AB is non-singular then there is a unique basic
feasible solution given by xB = A−1 xB . Now we set out to prove that if x is a basic feasible solution
for B then AB is non-singular. Since a basic feasible solution x satisfies Ax = b and xN = 0 then
it satisfies AB xB = b − AN xN = b. Hence, AB xB = b. From linear algebra we know that if xB is
unique then the matrix of the system AB xB = b must be full rank, or, equivalently, non-singular.
To verify that there are no other basic feasible solutions for B, consider an arbitrary vector x̃ such
that AB x̃ = b and x̃N = 0. Since the resulting verctor satisfies Ax̃ = AB x̃B + AN x̃N = b, it must
satisfy the top m equations in the tableau representing x. But then x̃N = 0 implies x̃ = xB .

Hence, having shown that AB is non-singular, we can write from (4.1):

xB = A−1 −1
B b − AB AN xN (4.3)
 73

and for the objective function:

z = cT x = cTB xB + cTN xN .

Substituting for xB from (4.3):

z = cTB (A−1 −1 T
B b − AB AN xN ) + cN xN =
= cTB A−1 T T −1
B b + (cN − cB AB AN )xN

and collecting together:

xB = A−1 −1
B b − AB AN x N
−1
z = cTB AB b + (cTN − cTB A−1
B AN )xN

which is the dictionary corresponding to a basis B. In tableau form, for a basic feasible solution B
we have:  
 
 
 A−1
B AN I 0 A−1
B b 
 
 
 
 
cTN − cTB A−1
B AN 0 1 −cTB A−1
B b

The identity matrix I of size n × n occupies the columns of the variables that are in basis. The other
terms of the matrix A are given byĀ = A−1 B AN .
The cost of one iteration of the revised simplex in a trivial implementation is determined by the
matrix operations needed to write the values in the tableau. These operations are:

• Inverting the matrix AB : in O(m3 ) by Gauss-Jordan

• Compute A−1 2
B AN : O(m n)

• Compute AB b: O(m2 )

The overall complexity is O(m2 (m+n)). This is apparently more than the standard simplex, however
smart implementations can be more efficient. The most important gain can be achieved by noting
that at each iteration of the simplex we do not need to compute all elements of Ā.

Let’s see this with an example.

Example 4.1. The LP problem is given on the left. Its equational standard form is derived on the
max x1 + x2
max x1 + x2
−x1 + x2 ≤ 1
−x1 + x2 + x3 = 1
right: x1 ≤ 3
x1 + x4 = 3
x2 ≤ 2
x2 + x5 = 2
x1 , x2 ≥ 0
x1 , x2 , x3 , x4 , x5 ≥ 0
The initial tableau is:
x1 x2 x3 x4 x5 −z b
−1 1 1 0 0 0 1
1 0 0 1 0 0 3
0 1 0 0 1 0 2
1 1 0 0 0 1 0
74 CHAPTER 4. REVISED SIMPLEX METHOD

After two iterations this is how the tableau looks like:

x1 x2 x3 x4 x5 −z b
1 0 −1 0 1 0 1
0 1 0 0 1 0 2
0 0 1 1 −1 0 2
0 0 1 0 −2 1 −3

The basic variables are x1 , x2 , x4 and the non basic ones: x3 , x5 . With this information we can
write, looking at the initial tableau:
     
−1 1 0 1 0 x1  
x
AB =  1 0 1 AN = 0 0 xB = x2  xN = 3
x5
0 1 0 0 1 x4
   
cTB = 1 1 0 cTN = 0 0
The tableau is not optimal hence we proceed to the next pivot operation. We describe the operations
of selecting a variable to enter the basis, one to leave the basis and updating the tableau in terms
of matrix calculations.

Entering variable : In the standard simplex we look at the reduced costs in the tableau. In the
revised simplex we need to calculate: cTN − cTB A−1
B AN . This is decomposed into two steps:

Step 1. Find yT by solving the linear system yT AB = cTB . It is possible to calculate yT = cTB A−1
B
but the system can be solved more efficiently without calculating the inverse of AB .

Step 2. Calculate cTN − yT AN (each term cj − yT aj can be calculated independently and for some
pivoting rules one does not need to calculate them all).

Let’s carry out these two steps in our example:

Step 1: we solve the linear system yT AB = cTB , which looks like:

 
  −1 1 0  
y1 y2 y3  1 0 1 = 1 1 0
0 1 0

via cTB A−1 T

B =y :    
  −1 0 1 −1
1 1 0  0 0 1 =  0
1 1 −1 2

Step 2: we calculate cTN − yT AN

 
    1 0  
0 0 − −1 0 2 0 0 = 1 −2
0 1

The first element is the vector is positive and the calculation can stop. It corresponds to
the variable x3 , which therefore has a positive reduced cost and is selected to enter in the
basis.
 75

Leaving variable In the standard simplex we need now to determine the largest feasible amount
θ to increase the entering variable without producing an infeasible solution. We do the constraint
analysis, looking at the tableau and knowing that x5 will remain non-basic and hence zero:

R1: x1 − x3 + x5 = 1 x1 = 1 + x3 ≥ 0
R2: x2 + 0x3 + x5 = 2 x2 = 2
R3: x3 + x4 − x5 = 2 x4 = 2 − x3 ≥ 0

Hence, the first and the second constraints do not pose any limit to the increase of x3 . The third
constraint is the most restrictive. It determines that the largest increase θ is 2.
Translating these operations in matrix operations we observe that they can be expressed as:

xB = x0B − A−1
B AN x N

where x0 is the current solution and x the adjacent solution to which we are moving. Since in the
new solution only one non basic variable changes its value, the selected one x3 , then not all the
terms of A−1
B AN xN need to be calculated but only those that correspond to the entering variable.
Let denote by d the column of A−1B AN and by a the column of AN that correspond to this variable.
We have
d = A−1
B a

and we calculate the update simply by

xB = x0B − dθ

We can thus describe the calculation that we have to carry out in the revised simplex to find θ
such that xB stays positive.

Step 3. Find d by solving AB d = a. It is possible to calculate d = A−1 B a by the system can be

solved more efficiently without calculating the inverse of AB .

Step 4. Determine the largest θ such that xB = x0B − dθ ≥ 0. If there is no such θ, then the
problem is unbounded. Otherwise, at least one component of x0B − dθ equals zero and the
corresponding variable is leaving the basis.

In our numerical example, these two steps yield the following:

Step 3:       
d1 −1 0 1 1 −1
d2  =  0 0 1 0 =⇒ d =  0
d3 1 1 −1 0 1

Step 4:    
1 −1
xB = 2 −  0 θ ≥ 0
2 1

The first two terms do not pose any limit, while for the third term it must be 2 − θ ≥ 0,
which implies θ ≤ 2 Hence, it is x4 that goes to zero and thus leaves the basis.
76 CHAPTER 4. REVISED SIMPLEX METHOD

Updating the tableau This part is the computationally heaviest part of the standard simplex.
In the revised simplex, instead, this step comes for free.

Step 5. The update of xB is done by setting the value found for θ in x0B − dθ ≥ 0 and replacing
x0B with xB . AB is updated by replacing the leaving column by the entering column.

In our example this step yields:

Step 5.      
x1 − d1 θ 3 −1 1 1
xB = x2 − d2 θ = 2
0    AB =  1 0 0 .
θ 2 0 1 0

Incidentally, note that the order of the columns of AB is not important as long as it matches the
order of the components of xB . Hence, the next iteration could just as well be entered with
   
2 1 0 0
x0B = 3 AB = −1 1 1 .
2 0 1 0
The basis heading is an ordered list of basic variables that specifies the actual order of the m
columns of AB . For simplicity, during the solution process the basis heading is updated by replacing
the leaving variable with the entering variable.

The revised simplex allows to save many operations especially if there are many variables! Also
in terms of space the revised simplex is convenient, note indeed that we do not need to store the
matrix AB but a vector containing the basis heading is enough. Special ways to call the matrix A
from memory help to provide then further speeds up. Finally, the revised simplex provides a better
control over numerical issues since A−1
B can be recomputed at the end once the variables that are in
basis are known.
There are different implementations of the revised simplex, depending on how yT AB = cTB and
AB d = a are solved. They are in fact solved from scratch every time. The next section provides the
general idea behind these implementations.

4.1 Efficiency Issues

We saw that to computing the inverse of a matrix is a costly operation and likely to introduce
numerical inaccuracies. If you do not remember the details, go back to your notes from Linear
Algebra. Hence, the two linear systems that must be solved in the revised simplex, AB x = b and
AB d = a are solved without computing A−1
B .

Eta Factorization of the Basis Let AB = B and let’s consider the kth iteration. The matrix
Bk will differ from the matrix Bk−1 by the column p. The column p is the a column appearing in
Bk−1 d = a solved in Step 3. Hence:
Bk = Bk−1 Ek
where Ek is the eta matrix differing from identity matrix only in one column:
    
−1 1 1 −1 1 0 1 −1
 1 0 0 =  1 0 1  1 0
0 1 0 0 1 0 1
4.1. EFFICIENCY ISSUES 77

No matter how we solve yT Bk−1 = cTB and Bk−1 d = a, their update always relays on Bk =
Bk−1 Ek with Ek available. Moreover, when the initial basis is made of slack variables then B0 = I
and B1 = E1 , B2 = E1 E2 , . . .:

Bk = E1 E2 . . . Ek eta factorization

((((yT E1 )E2 )E3 ) · · · )Ek = cTB , uT E4 = cTB , vT E3 = uT , wT E2 = vT , yT E1 = wT

(E1 (E2 · · · Ek d)) = a, E1 u = a, E2 v = u, E3 w = v, E4 d = w

When B0 6= I:
Bk = B0 E1 E2 . . . Ek eta factorization
((((y B0 )E1 )E2 ) · · · )Ek = cTB
T

(B0 (E1 · · · Ek d)) = a

In this case, it helps having the LU factorization of B0 .

LU Factorization To solve the system Ax = b by Gaussian Elimination we put the A matrix in row
echelon form by means of elemntary row operations. Each row operation corresponds to multiply left and
right side by a lower triangular matrix L and a permuation matrix P . Hence, the method throughout its
iterations is equivalent to:

Ax = b
L1 P1 Ax = L1 P1 b
L2 P2 L1 P1 Ax = L2 P2 L1 P1 b
..
.
Lm Pm . . . L2 P2 L1 P1 Ax = Lm Pm . . . L2 P2 L1 P1 b

thus
U = Lm Pm . . . L2 P2 L1 P1 A triangular factorization of A
where U is an upper triangular matrix whose entries in the diagonal are ones (if A is nonsingular such
triangularization is unique).
For a square matrix A the LU decomposition is given by:

A = LU
PA = LU

From an LU decomposition it is easy to solve Ax = LU x = b: set y = U x then

1. Ly = b can be solved easily by forward substitution
2. U x = y can be solved easily by backward substitution.

We can compute the triangular factorization of B0 before the initial iterations of the simplex:

Lm Pm . . . L2 P2 L1 P1 B0 = U

We can then rewrite U as

U = Um Um−1 . . . , U1
Hence, for Bk = B0 E1 E2 . . . Ek :

Lm Pm . . . L2 P2 L1 P1 Bk = Um Um−1 . . . U1 E1 E2 · · · Ek

Then yT Bk = cTB can be solved by first solving:

((((yT Um )Um−1 ) · · · )Ek = cTB

78 CHAPTER 4. REVISED SIMPLEX METHOD

and then replacing yT by ((yT Lm Pm ) · · · )L1 P1 . To show this, we can express

−1
Bk = (Lm Pm · · · L1 P1 ) Um · · · Ek
| {z } | {z }
L U

and in this notation

yL−1 U = c
wU = c
−1
w = yL =⇒ y = Lw.
T
In the literature, solving y Bk = cTB
is also called backward transformation (BTRAN) while solving
Bk d = a is also called forward transformation (FTRAN).

Some further remarks:

• The Ek matrices can be stored by only storing the column and the position
• If sparse columns then they can be stored in compact mode, ie only nonzero values and their indices
• The triangular eta matrices Lj , Uj can also be stored efficiently due to their sparsity.
• To store Pj just two indices are needed

Efficient Implementations
• Dual simplex with steepest descent
• Linear Algebra:

– Dynamic LU-factorization using Markowitz threshold pivoting (Suhl and Suhl, 1990)
– sparse linear systems: Typically these systems take as input a vector with a very small number
of nonzero entries and output a vector with only a few additional nonzeros.
• Presolve, ie problem reductions: removal of redundant constraints, fixed variables, and other extraneous
model elements.
• dealing with degeneracy, stalling (long sequences of degenerate pivots), and cycling:

– bound-shifting (Paula Harris, 1974)

– Hybrid Pricing (variable selection): start with partial pricing, then switch to devex (approximate
steepest-edge, Harris, 1974)
• A model that might have taken a year to solve 10 years ago can now solve in less than 30 seconds
(Bixby, 2002).

4.2 More on Polyhedra

Basic Geometric Facts From Linear Algebra we know that in 2D we need 2 lines to intersect to give
us a point. In 3D, we need three planes. In 4D, we need 4 hyperplanes to intersect to give a point. In 2D,
lines can also be parallel or overlapping and hence they do not necessarily intersect in a point. In 3D planes
can also be intersecting in a line or not intersecting at all if parallel. Similarly in 4D, 4 hyperplanes do not
necessarily intersect in a point.
On the other hand, like a point in 3D can be described by more than 3 intersecting planes, think of the
top vertex of a pyramid, similarly a point in 4D can be described by more than 4 hyperplanes.
These considerations can be generalized to n dimensions. We cannot anymore visualize the situation
but we can use abstract algebra to understand what is going on. In n dimensions we need n hyperplanes to
determine a point. They uniquely identify a point when the rank of the matrix A of the linear system is n
(or A is nonsingular)
4.2. MORE ON POLYHEDRA 79

Vertices of Polyhedra A vertex of a polyhedron in Rn is a point that is a feasible solution to the

system:
a11 x1 + a12 x2 + · · · + a1n xn ≤ b1
a21 x1 + a22 x2 + · · · + a2n xn ≤ b2
.. ..
. .
am1 x1 + am2 x2 + · · · + amn xn ≤ bm
For a point x̄ we say that a constraint, ie an inequality of the system, is active or tight or binding when
the left hand side →−
a i x̄ is equal to the right hand side bi (→
−
x i is the ith row of the matrix A).
m×n
In a vertex of a polyhedron Ax ≤ b, A ∈ R , x ∈ Rn there are at least n active constraints. This
implies that the rank of the matrix of active constraints is n. The viceversa is not necessarily true. A point
x̄ that activates n constraints does not necessarily form a vertex of the polyhedron. Some of these points
may be not feasible, ie, the intersection of the n supporting hyperplanes of the polyhedron happens outside
of the polyhedron itself.
As we for the pyramid in 3D, a vertex can activate more than n constraints. The rank of the matrix of
active constraints is still n.
If there are more constraints than variables, ie m > n, then we can find a subset and determine the
intersecting point of the corresponding supporting hyperplanes. But what happens if there are more variables
than constraints, ie m < n, can we have a vertex? Not necessarily. In LP we deal with this issue by adding
slack variables, they make us choose arbitrarily a vertex. Consider the case of one constraint for two variables:
x1 + x2 ≤ 0
We add x3 and find that two variables are not in basis, ie, their value is 0. Geometrically, this corresponds
to select a point in the line represented by the supporting hyperplane of the constraint.
To define a cube we need 6 cosntraints and there are 23 vertices. For an n-hypercube we need 2n con-
straints and there are 2n constraints. This shows that the number of vertices can give rise to a combinatorial
explosion. The upper bound to the number of vertices when there are m constraints and n variables, m > n,
is given by the number of possibilities of having n active constraints, ie
 
m
.
n
This is an upper bound because:
• some combinations of constraints will not define a vertex, ie, if the rows of the corresponding matrix
are not independent
• some vertices may activate more than n constraints and hence the same vertex can be given by more
than n constraints

Tableaux and Vertices For each tableau is associated to exactly one vertex of the feasibility region.
The reverse is not always true. One vertex of the feasibility region can have more than one tableau associated.
For example degenerate vertices have several tableaux associated.

max 6x1 + 8x2 x1 x2 x3 x4 −z b

5x1 + 10x2 ≤ 60 x2 0 1 1/5 −1/4 0 2
4x1 + 4x2 ≤ 40 x1 1 0 −1/5 1/2 0 8
x1 , x2 ≥ 0 0 0 −2/5 −1 1 −64
The slack variables (x3 , x4 ) = (0, 0) are non basic hence the corresponding constraints are active. Indeed, it
means that no slack is needed between the left hand side and the right hand side of the constraints for which
they have been introduced.

max 6x1 + 8x2 x1 x2 x3 x4 −z b

5x1 + 10x2 ≤ 60 x3 0 0 1 1/2 0 1
4x1 + 4x2 ≤ 40 x1 1 1 0 −1/2 0 1
x1 , x2 ≥ 0 0 −2 0 1/2 1 −1
80 CHAPTER 4. REVISED SIMPLEX METHOD

Now, the non basic variables are (x2 , x4 ) = (0, 0). The constraints that are active are 4x1 + 4x2 ≤ 40 and
x2 ≥ 0. Hence, there are still two active constraints when the non basic variables are two. If in the original
space of the problem we have 3 variables and there are 6 constraints we would have 3 constraints active in the
vertices. After we add the slack variables we have 6 variables in all. If any of the slack variables is positive
then some constraints xi ≥ 0 of the original variables are be active, otherwise the corresponding constraint
of the original problem are active. Hence, we can generalize: the non basic variables are always n and they
tell which constraints (among the original and the variable feasibility constraints xi ≥ 0) are active. A basic
feasible solution implies a matrix of active constraints with rank n, some of which may be due to the original
variables being zero. Let a tableau be associated with a solution that makes exactly n + 1 constraints active.
Then, one basic variable is zero.
Definition 6. In a polyhedron in Rn , two vertices are adjacency iff:

• they have at least n − 1 active constraints in common

• rank of common active constraints is n − 1

In terms of tableaux, this condition menas that between two adjacent vertices there are n − 1 variables
in common in basis.

4.3 More on LP
4.3.1 LP: Rational Solutions
• A precise analysis of running time for an algorithm includes the number of bit operations together with
the number of arithmetic operations.
Example 4.2. The knapsack problem aka, budget allocation problem, that asks to choose amont a
set of n investments those that maximize the profit and cost in total less than B, can be solved by
dynamic programming in
O(n|B|)
The number B needs b = log |B| bits hence the running time is exponential in the number of bits
needed to represent B, ie, O(n2b )

• Weakly polynomial time algorithms have running time that are independent on the sizes of the numbers
involved in the problem and hence on the number of bits needed to represent them.
• Strongly polynomial time algorithms: the running time of the algorithm is independent on the number
of bit operations. Eg: same running time for input numbers with 10 bits as for inputs with a million
bits.
• Running time depends on the sizes of numbers. We have to restrict attention to rational instances
when analyzing the running time of algorithms and assume they are coded in binary.
Theorem 4.2 (Rational Solutions). Optimal feasible solutions to LP problems are always rational as
long as all coefficient and constants are rational.

Proof: derives from the fact that in the simplex we only perform multiplications, divisions and sums
of rational numbers
• In spite of this: No strongly polynomial-time algorithm for LP is known.

4.3.2 Interior Point Algorithms

• Ellipsoid method: cannot compete in practice but weakly polynomial time (Khachyian, 1979)
• Interior point algorithm(s) (Karmarkar, 1984) competitive with simplex and polynomial in some ver-
sions
4.3. MORE ON LP 81

Figure 4.1: Source: Bixby, 2002

– affine scaling algorithm (Dikin)

– logarithmic barrier algorithm (Fiacco and McCormick) ≡ Karmakar’s projective method

1. Start at an interior point of the feasible region

2. Move in a direction that improves the objective function value at the fastest possible rate while
ensuring that the boundary is not reached
3. Transform the feasible region to place the current point at the center of it

• because of patents reasons, now mostly known as barrier algorithms

• one single iteration is computationally more intensive than the simplex (matrix calculations, sizes
depend on number of variables)
• particularly competitive in presence of many constraints (eg, for m = 10, 000 may need less than 100
iterations)
• bad for post-optimality analysis crossover algorithm to convert a sol of barrier method into a basic
feasible solution for the simplex

How Large Problems Can We Solve? The speed up due to algorithmic improvements has been
more important than the the one due to technology and machine architecture improvements (Bixby, 2007).

4.3.3 Further topics in LP

• Numerical stability and ill conditioning
• Lagrangian relaxation
• Column generation
• Decomposition methods:

– Dantzig Wolfe decomposition

– Benders decomposition
82 CHAPTER 4. REVISED SIMPLEX METHOD
Chapter 5

Modeling in Mixed Integer Linear

Programming

Often we need to deal with integral inseparable quantities. For example, if we are modeling the presence of
a bus on a line a value of 0.3 would not have a meaning in practice. Sometimes it may be enough to round
fractional solutions to their nearest integers but other times rounding is not a feasible option, as it may be
costly and a feasible solution is not ensured.
Discrete Optimization is the mathematical field that deals with optimization where the variables must
be integer. Combinatorial optimization is also a kind of discrete optimization, although the term refers to
problems that have a particular structure, like selecting subgraphs, patterns, etc.
In this chapter we will study mixed integer linear programming (MILP). The world is not linear but we
will see that MILP constitutes a very powerful tool and that many situations, apparently non linear, can
actually be modeled in linear terms. In other cases it is possible to linearize by approximation. After all
“OR is the art and science of obtaining bad answers to questions to which otherwise worse answers would be
given.”

5.1 Introduction to Integer Linear Programming

An integer linear programming (ILP) problem has linear objective function, linear constraints and integer
variables. A mixed integer linear programming (MILP) problem has linear objective function, linear con-
straints and both integer and real valued variables. A binary programming or 0–1 programming problem
is an ILP problem where variables can take only two values, 0 and 1. Non-linear programming refers to
all problems that although written in mathematical programming terms may have a non linear objective
function and/or non linear constraints.
Here is a non exhaustive list of mathematical programming formulations. We will not see NLP in this
course.

Linear Programming (LP) Integer (Linear) Programming (ILP) Binary Integer Program (BIP)
0–1 Integer Programming
max cT x max cT x
Ax ≤ b Ax ≤ b max cT x
x ≥ 0 x ≥ 0 Ax ≤ b
x integer x ∈ {0, 1}n

83
84 CHAPTER 5. MODELING IN MIXED INTEGER LINEAR PROGRAMMING

Mixed Integer (Linear) Programming (MILP) Non-linear Programming (NLP)

max cT x + hT y max f (x)

Ax + Gy ≤ b g(x) ≤ b
x ≥ 0 x ≥ 0
y ≥ 0
y integer

Recall that the sets of integers are:

• Z set of integers

• Z+ set of positive integer

• Z+ +
0 set of nonnegative integers ({0} ∪ Z )

• N0 set of natural numbers, ie, nonnegative integers {0, 1, 2, 3, 4, ...}

Whenever different types of variables are present, we will try to comply with the convention used in the
MIPLIB 2003 and use the letters:

• x to denote binary variables

• y to denote general integer variables

• z to denote continuous variables

5.1.1 Combinatorial Optimization Problems

Definition 5.1. Combinatorial Optimization Problem (COP)
Input: Given a finite set N = {1, . . . , n} of objects, a weight cj for each j ∈ N , and a collection F of feasible
subsets of N .
Task: Find a minimum weight feasible subset, ie,
 
X 
min cj | S ∈ F .
S⊆N  
j∈S

Note that the definition above is not an MILP formulation. However, many COP can be formulated as
IP or BIP. Typically, one defines an incidence vector of S, xS ∈ Bn such that:
(
1 if j ∈ S
xSj = .
0 otherwise

That is, an element i of N is selected if xSi = 1 and not selected if xSi = 0. Then, one expresses the structural
constraints in function of xS .

5.1.2 Solution Approaches

MILP problems are solved primarily using linear programming relaxation. That is, relaxing the integrality
constraints we obtain a linear programming problem that we can solve can solve with the simplex method or
the barrier method. The solutions will be in the general case rational. Then to derive integer solutions one
can use heuristics or exact methods such as branch and bound or cutting planes.
5.1. INTRODUCTION TO INTEGER LINEAR PROGRAMMING 85

x2 3x1 − 2x2 + 4

x1
x1 + 0.64x2 − 4 50x1 + 31x2 − 250

Figure 5.1: . An unfortunate situation in a rounding approach to MILP solving.

Rounding
A trivial heuristic to find integer solutions to an MILP problem is to relax the integrality constraint, solve
the linear programming problem thus derived and then round up or down each single fractional value of the
solution found.

Example 5.1. Let’s consider the following example:

max 100x1 + 64x2

50x1 + 31x2 ≤ 250
3x1 − 2x2 ≥ −4
x1 , x2 ∈ Z+

The situation is represented in Figure 5.1. The feasibility region is made of the red dots that represent
integer solutions contained in the convex region defined by the constraints. The feasibility region is not
continuous: now the optimum can be on the border (vertices) of the polytope but also internal.
The linear programming relaxation of this problem is obtained by substituting the integrality constraints
on the two variables with the requirements that they must be non-negative, ie, x1 , x2 ≥ 0. The problem
obtained is a linear programming problem that we can solve with the simplex method.
Let’s denote by (ILP) the original problem and by (LPR) its linear relaxation. If the solution to (LPR)
is integer then the (ILP) is solved. On the other hand if the solution is rational then we can try to round
the values of the variables to their nearest integers.
The solution of (LPR) is (376/193, 950/193). The situation is depicted in the figure. The circles filled in
blue represent the solutions obtained by rounding down or up the values of the variables. For each rounded
solution we need to test whether it is feasible. If we are in R2 then there are 22 possible roundings (up or
down) of the variables and solutions to test. If we are in Rn then there are 2n possible solutions. Hence,
in large problems checking all possible roundings may become computationally costly. Moreover, rounding
does not guarantee that a feasible solution is found and it can be arbitrarily bad with respect to the optimal
solution. In our example, the optimum of (ILP) is (5, 0) (the red circle in the figure), while any rounded
solution is quite far from that.

There are two main techniques to solve MILP problem exactly: branch and bound and cutting planes.

Cutting Planes: sketch

Example 5.2. Suppose we have to solve the problem:

max x1 + 4x2
x1 + 6x2 ≤ 18
x1 ≤ 3
x1 , x2 ≥ 0
x1 , x2 integers
86 CHAPTER 5. MODELING IN MIXED INTEGER LINEAR PROGRAMMING

x2

x1 = 3

x1 + 6x2 = 18

x1 + x2 = 5
x1
x1 + 4x2 = 2

Figure 5.2: Branching in a branch and bound approach for MILP.

We solve the linear programming relaxation. The situation is depicted in Figure 5.2.The optimal solution
is fractional. If we knew the constraint represented by the dashed line in the figure then solving the linear
programming relaxation would give us an integer value, which would be optimal for the original problem. If
we do not know that constraint we can devise a method to add constraints to our problem that would cut
out the rational solution of the current linear relaxation but not cut out any integer feasible solution. This
would bring us closer to the integer optimal solution. An example of such an added cut is given in the figure
by the line just below the constraint x1 + 6x2 ≤ 18. Iterating the procedure a needed number of times would
eventually lead us to an integer solution.

Branch and Bound: sketch

We sketch here the branch and bound procedure and we postpone to a later chapter its full treatment.

Example 5.3. Suppose we have to solve the problem:

max x1 + 2x2
x1 + 4x2 ≤ 8
4x1 + x2 ≤ 8
x1 , x2 ≥ 0, integer

The branch and bound technique works by solving the linear relaxation and then branching on the
fractional values. Branching corresponds to splitting the problem into two subproblems each one taking
a different part of the feasibility region. The solution to the linear relaxation is cut out by restricting a
fractional variable to be:
• larger than the smallest integer larger than the fractional value of the variable (floor), or
• smaller than largest integer smaller than the fractional value of the variable (ceil).
Each new subproblem is then a linear programming problem with an added constraint and we have seen
in sensitivity analysis how a solution can be derived from an optimal tableau after the introduction of a
constraint. When at a node the solution of the linear programming relaxation is integer, then we can stop
branching on that node. The optimal solution will be the best one found at the leaves of the branch and
bound tree.
The branch and bound process for our example problem is shown in Figure 5.3. Each node bears the
information of the best feasible encountered in its subtree and the value of the linear relaxation, which in
this case represents the best possible that can be achieved. The optimal solution has value 4 and is shown
on the right hand side of the corresponding node.
5.2. MILP MODELING 87

x2

x1 + 4x2 = 8
x1
x1 + 2x2 = 1
4x1 + x2 = 8

4.8
x1 ≤ 1 x1 ≥ 2

x2 x2
x1 = 1

x1 + 4x2 = 8 x1 + 4x2 = 8
x1 x1
x1 + 2x2 = 1 x1 + 2x2 = 1
4x1 + x2 = 8 4x1 + x2 = 8

4.8
−∞
x2 ≤ 1 x1 ≥ 2

4.5 2 x1 =2
−∞ 2 x2 =0
x2 ≤ 1 x2 ≥ 2

x1 =1 3 4 x1 =0
x2 =1 3 4 x2 =2

x2 x2

x1 + 4x2 = 8 x1 + 4x2 = 8
x1 x1
x1 + 2x2 = 1 x1 + 2x2 = 1
4x1 + x2 = 8 4x1 + x2 = 8

Figure 5.3: Branching in a branch and bound approach for MILP.

88 CHAPTER 5. MODELING IN MIXED INTEGER LINEAR PROGRAMMING

5.2 MILP Modeling

Find out exactly what the decision maker needs to know. For example: which investment, which product
mix, which job j should a person i do. Then, define:

• the Parameters that represent the values that are fixed and known;

• the decision variables that answer the questions of the decision maker. They must be of a suitable
type according to the decisions they represent (continuous, integer valued, binary).

Finally, using parameters and variables, formulate mathematically:

• the objective function computing the benefit/cost;

• the constraints indicating the interplay between the different variables.

It is helpful, in order to formulate constraints, to first write down the relationship between the variables
in plain words. Then, these constraints can be transformed in logical sentences using connectives such as
and, or, not, implies. Finally, logical sentences can be converted to mathematical constraints.

Example 5.4. The decisions to take are whether to produce in a time period or not. This can be modeled
by using a binary integer variables, xi = 1 or xi = 0, for any period i. With these variables we can formulate
the constraint in three steps:

• Plain English: “The power plant must not work in both of two neighboring time periods”

• Logical sentence: xi = 1 implies ⇒ xi+1 = 0

• Mathematical constraint: xi + xi+1 ≤ 1

We now provide the formulation of a series of relevant problems with several real life applications.

5.2.1 Assignment Problem

Common application: assignees are to be assigned to perform tasks. Suppose we have n persons and n jobs
Each person has a certain proficiency at each job. Formulate a mathematical model that can be used to find
an assignment that maximizes the total proficiency.

Parameters: We use the letter I to indicate the set of persons, indexed by i = 1..n and the letter J to
indicate the set of jobs, indexed by j = 1..n. We represent the proficiency by numerical value δij , the higher
the value is the higher the proficiency for the job.

Decision Variables: We use binary variables:


1 if person i is assigned job j
xij = for i, j = 1, 2, . . . , n
0 otherwise,

Objective Function:
n X
X n
max ρij xij
i=1 j=1

where ρij is person i’s proficiency at job j

5.2. MILP MODELING 89

Constraints:
Each person is assigned one job:
n
X
xij = 1 for all i ∈ I
j=1

e.g. for person 1 we get x11 + x12 + x13 + · · · + x1n = 1

Each job is assigned to one person:

n
X
xij = 1 for all j ∈ J
i=1
e.g. for job 1 we get x11 + x21 + x31 + · · · + xn1 = 1.

5.2.2 Knapsack Problem

Definition 5.2 (Knapsack Problem).
Input: a set of n item types, each with a value vi and weight wi for i = 1, . . . , n.
Task: determine the number of items per type to include in a collection so that the total weight is less than
a given limit, W , and the total value is as large as possible.
Without
P loss of generality (why?) we can assume that we can take at most one item per type. We also
assume i wi > W . The problem is has applications in, for example, capital budgeting, project selection,
etc.
Next we present a mathematical model to determine which items give the largest value.

Parameters: vi the value (or profit) of an item; wi the weight of item i; W the knapsack capacity.

Decision Variables: 
1 if item i is taken
xi = for i = 1, 2 . . . , n
0 otherwise,

Objective Function: we want to maximize the total value of the selected items:
n
X
max vi x i
i=1

Constraints: we cannot exceed knapsack capacity:

n
X
w i xi ≤ W
i=1

5.2.3 Set Problems

Consider the following application:
Example 5.5. Given: a set of regions, a set of possible construction locations for emergency centers, for
each location a set of regions that can be served in less than 8 minutes, and the cost of installing an emergency
center in that location.
Task: decide where to install a set of emergency centers such that the total cost is minimized and all regions
are safely served.
For example you may think of the following numerical data:
90 CHAPTER 5. MODELING IN MIXED INTEGER LINEAR PROGRAMMING

– regions: M = {1, . . . , 5}
– centers: N = {1, . . . , 6}
– cost of centers: cj = 1 for j = 1, . . . , 6
– coverages: S1 = (1, 2), S2 = (1, 3, 5), S3 = (2, 4, 5), S4 = (3), S5 = (1), S6 = (4, 5)

We can model the problem as a combinatorial optimization problem.

Let M = {1, . . . , m} be the set of regions, N = {1, . . . , n} be the set of centers (objects) with costs
N
c1 , ..., cn , and Sj ⊆ M regions serviced by j ∈ N in 8 min. Let S F ⊆ 2 be a collection of sets of centers
each of which would guarantee to safely serve all regions, i.e., j∈T Sj = M for all T ∈ F, T ⊆ N . The the
problem of finding the cheapest set of centers to safely serve all regions can be formalized as:
 
X [ 
min cj | Sj = M .
T ⊆N  
j∈T j∈T

Let’s now formulate the problem as a BIP problem.

Parameters: The set of regions M , the set of centers N , the coverage of regions for each center, Sj , the
cost of each center cj .

Variables: x ∈ Bn , xj = 1 if center j is selected, 0 otherwise

Objective:
n
X
min cj x j
j=1

Constraints: all regions must be safely served: We define an incidence matrix A of size m × |F |:
(
1
aij =
0

The constraint can then be expressed as:

n
X
aij xj ≥ 1.
j=1

In our numerical example the incidence matrix would look like:

x1 x2 x3 x4 x5 x6
S1 S2 S3 S4 S5 S6
 
1 1 1 0 0 1 0
2 1 0 1 0 0 0
A=  
3 0 1 0 1 0 0
4 0 0 1 0 0 1
5 0 1 1 0 0 1

where we labeled the columns with the variables and the sets they represent and the rows by the region
identifier. A feasible solution is a selection of the columns of the matrix such that they identify a submatrix
that has at least a 1 in each row, or in other terms such that all rows are covered by the selected columns.
The one we formulated is a set covering problem. Two other variants, set packing and set partitioning
are also relevant for real life applications. We sketch the formulations here.
5.2. MILP MODELING 91

Definition 5.3 (Set covering). Cover each of M at least once:

1. min, ≥ type constraints min cT x
2. all RHS terms are 1 Ax ≥ 1
x ∈ Bn
3. all matrix elements are 0 or 1
Definition 5.4 (Set packing). Cover as many of M without overlap:
1. max, ≤ type constraints max cT x
2. all RHS terms are 1 Ax ≤ 1
x ∈ Bn
3. all matrix elements are 0 or 1
Definition 5.5 (Set partitioning). Cover exactly once each element of M :
1. max or min, = type constraints max cT x
2. all RHS terms are 1 Ax = 1
x ∈ Bn
3. all matrix elements are 0 or 1

These problems can be generalized to the cases where the coverage must be larger than 1, that is, where
the right hand side of the constraints are larger than 1.
These problems have several applications. Examples are:
• Aircrew scheduling: the legs to cover define the set M and the rosters (crew shifts during which a crew
can cover a number of legs) defines the set N .

• Vehicle routing: the customers to visit define the set M , the routes that visit customers define the set
N.
Here is an example that can be modeled as a generalized set covering.

Example 5.6 (Manpower Planning).

Input A set of workers, a set of 15 working hours per day with a required staffing per hour. Each person
work in shifts that covers 7 hours. A person starting in hour i contributes to the workload in hours i, . . . , i + 6
(Eg: A person starting in hour 3 contributes to the workload in hours 3,4,5,6,7,8,9).
Task: Determine the number of people needed to cover the workload.

Decision Variables:
• xi ∈ N0 : number of people starting to work in hour i (i = 1, . . . , 15). For easiness of expressing the
constraints we also define the variables xi , i = −5, ..., −1, 0.

Objective Function:
9
X
min xi
i=1

Constraints:
• Demand:
i=t
X
xi ≥ dt for t = 1, . . . , 15
i=t−6

• Bounds:
x−5 , . . . , x0 = 0
92 CHAPTER 5. MODELING IN MIXED INTEGER LINEAR PROGRAMMING

5.2.4 Graph Problems

Matching
Definition 5.6 (Matching Theory – Terminology). A Matching M of a graph G = (V, E) is a set of pairwise
non adjacent edges. A vertex is covered by the matching M if it is incident to an edge in M . A matching is
perfect if it covers all vertices in G. A matching is maximal if it cannot be extended any further. A maximum
matching is a matching that covers as many vertices as possible. A graph G is matchable it has a perfect
matching.

For a graph with weights on the edges, the weight of a matching is the sum of the weights on the edges
of the matching.

Definition 5.7 (Maximum weighted matching problem). Given a graph G = (V, E) and weights we on the
edges e ∈ E find the matching of maximum weight.

The MILP formulation of the maximum weighted matching problem is:

P
max we xe
P
v∈V
xe ≤ 1 ∀v ∈ V
e∈E:v∈e
xe ∈ {0, 1} ∀e ∈ E

Binary variables indicate whether an edge is selected or not. The constraint ensures that for each vertex the
number of selected edges that are incident to the vertex are not more than 1.
A particular case is a bipartite matching that arises when the graph is bipartite. A bipartite matching is
equivalent to an assignment problem.

Vertex Cover
Definition 5.8 (Vertex cover problem). Given a graph G, select a subset S ⊆ V such that each edge has at
least one end vertex in S.

The MILP formulation is

P
min xv
v∈V
xv + xu ≥ 1 ∀u, v ∈ V, uv ∈ E
xv ∈ {0, 1} ∀v ∈ V

Roughly said, an approximation algorithm is an algorithm that runs in polynomial time and that guaratees
in the worst case a certain approximation ratio with respect to the optimal solution. Formally, if OP T (π)
is the optimal solution of an instance π of a minimization problem, and A(π) is the solution found by the
approximation algorithm, the approximation ratio AR is defined as:

A(π)
AR = max
π OP T (π)

An approximation algorithm for vertex cover can be easily derived from the linear programming solution.
Let x∗ be the optimal solution of the linear programming relaxation of the MILP formulation of the vertex
cover problem. Then, a cover SLP can be constructed by selecting the vertices whose variables received a
value larger than 1/2, that is:
SLP R = {v ∈ V : x∗v ≥ 1/2}.
The set SLP R is a cover since x∗v + x∗u ≥ 1 implies x∗v ≥ 1/2 or x∗u ≥ 1/2.

Proposition 5.1. The LP rounding approximation algorithm described above gives a 2-approximation:
|SLP | ≤ 2|SOP T | (at most as bad as twice the optimal solution)
5.2. MILP MODELING 93

P P
Proof. Let x̄ be the optimal solution for the MILP formulation of the vertex cover. Then x∗v ≤ x̄v .
Moreover,
X X
|SLP | = 1≤ 2x∗v
v∈SLP v∈V

since x∗v ≥ 1/2 for each v ∈ SLP and thus

X X
|SLP | ≤ 2 x∗v ≤ 2 x̄v = 2|SOP T |.
v∈V v∈V

Maximum independent Set

Definition 5.9 (Maximum independent set problem). Given a graph G = (V, E), find the largest subset
S ⊆ V such that the induced graph, i.e., the graph (S, {uv : uv ∈ E, u, v ∈ S}) has no edges.

The MILP formulation is:

P
max xv
v∈V
xv + xu ≤ 1 ∀u, v ∈ V, uv ∈ E
xv = {0, 1} ∀v ∈ V

Also in this case we could design an algorithm that rounds the LP relaxation of the MILP formulation. The
optimal solution to this LP problem sets xv = 1/2 for all variables and has value |V |/2. This fact implies
that the LP relaxation rounding algorithm gives an O(n)-approximation (almost useless). (To prove this
fact think about the worst possible instance which is a complete graph. What is the optimal integer max
independent set solution for a complete graph?)

Traveling Salesman Problem

The traveling salesman problem has several applications. Here is one. Find the cheapest movement for a
drilling, welding, drawing, soldering arm as, for example, in a printed circuit board manufacturing process
or car manufacturing process.

Definition 5.10 (Traveling salesman problem). Given a set of n locations and costs cij of travelling from
one location i to another location j, find the cheapest tour that visits all locations.

The problem is modeled in graph terms by defining a directed graph D = (V, A). In this context a tour
that visit all vertices is called Hamiltonian tour. Note that if the costs are symmetric everywhere then the
graph can be undirected.
The problem can be formulated as a MILP problem as follows.

Parameters The set of locations identified by 1, ..., n indexed by i and j and the set of traveling costs cij .

Variables (
1
xij =
0

Objective
n X
X n
cij xij
i=1 j=1
94 CHAPTER 5. MODELING IN MIXED INTEGER LINEAR PROGRAMMING

Constraints Each location must have an entering and an outgoing arc:

X
xij = 1 ∀i = 1, . . . , n
j:j6=i
X
xij = 1 ∀j = 1, . . . , n
i:i6=j

The previous constraints alone do not remove the possibility that subtours are found. To eliminate this
possibility there are two ways:
• cut set constraints
XX
xij ≥ 1 ∀S ⊂ N, S 6= ∅
i∈S j6∈S

• subtour elimination constraints

XX
xij ≤ |S| − 1 ∀S ⊂ N, 2 ≤ |S| ≤ n − 1
i∈S j∈S

The problem with these constraints is that there are exponentially many (look at the quantifiers on the
right side). In can learn how to deal with this issue in one of the assignments.

5.3 Modeling Tricks

In this section we review a few modeling tricks. This material is taken from Chapter 9 of Williams [2013].
MILP problem can be defined also when the objective function and/or constraints do not appear to be linear
at first sight. Consider for example the following cases:
• Absolute values
• Minimize the largest function value
• Maximize the smallest function value
• Constraints include variable division
• Constraints are either/or
• A variable must take one of several candidate values

Modeling Trick I Minimize the largest of a number of function values:

min max{f (x1 ), . . . , f (xn )}
Introduce an auxiliary variable z:
min z
s. t. f (x1 ) ≤ z
f (x2 ) ≤ z
Example 5.7.
min max 3y1 + 4y2 + 2y3
Reformulate as:
min z
s. t. 3y1 ≤ z
4y2 ≤ z
2y3 ≤ z
5.3. MODELING TRICKS 95

max min 3y1 + 4y2 + 2y3

Reformulate as:
max z
s. t. 3y1 ≥ z
4y2 ≥ z
2y3 ≥ z

Modeling Trick II Constraints include variable division:

a1 x + a2 y + a3 z
≤b
d1 x + d2 y + d3 z
Rearrange:
a1 x + a2 y + a3 z ≤ b(d1 x + d2 y + d3 z)
which gives:
(a1 − bd1 )x + (a2 − bd2 )y + (a3 − bd3 )z ≤ 0
Example 5.8. Constraint of the form
3x + 4y + 6z
≤ 10
x+y+z
Rearrange:
3x + 4y + 6z ≤ 10(x + y + z)
which gives:
7x + 6y + 4z ≥ 0

Modeling Trick III: “Either/Or Constraints” In conventional mathematical models, the solution
must satisfy all constraints. Suppose that your constraints are of the type “either/or”:
a1 x1 + a2 x2 ≤ b1 or
d1 x1 + d2 x2 ≤ b2
Introduce a new variable y ∈ {0, 1} and a large number M :
a1 x1 + a2 x2 ≤ b1 + M y if y = 0 then this is active
d1 x1 + d2 x2 ≤ b2 + M (1 − y) if y = 1 then this is active
Hence, binary integer programming allows to model alternative choices. For example, we can model the
case of tow disjoint feasible regions, ie, disjunctive constraints, which are not possible in LP.
We introduce an auxiliary binary variable y and M , a big number:
Ax ≤ b + M y if y = 0 then this is active
A0 x ≤ b0 + M (1 − y) if y = 1 then this is active
Example 5.9. At least one of the two constraints must be satisfied:
3x1 + 2x2 ≤ 18 or x1 + 4x2 ≤ 16
Introduce new variable y ∈ {0, 1} and a large number M :
3x1 + 2x2 ≤ 18 + M y
x1 + 4x2 ≤ 16 + M (1 − y)
If y = 1 then x1 + 4x2 ≤ 16 is the active constraint and the other is always satisfied.
If y = 0 then 3x1 + 2x2 ≤ 18 is the active constraints and the other is always satisfied.
96 CHAPTER 5. MODELING IN MIXED INTEGER LINEAR PROGRAMMING

Modeling Trick IV: “Either/Or Constraints” We can generalize the previous trick to the case
where Exactly K of the N constraints:
a11 x1 + a12 x2 + a13 x3 + . . . + a1m xm ≤ d1
a21 x1 + a22 x2 + a23 x3 + . . . + a2m xm ≤ d2
..
.
am1 x1 + aN 2 x2 + aN 3 x3 + . . . + aN m xm ≤ dN
must be satisfied. We need to introduce binary variables y1 , y2 , . . . , yN and a large number M and impose:
a11 x1 + a12 x2 + a13 x3 + . . . + a1m xm ≤ d1 + M y1
a21 x1 + a22 x2 + a23 x3 + . . . + a2m xm ≤ d2 + M y2
..
.
am1 x1 + aN 2 x2 + aN 3 x3 + . . . + aN m xm ≤ dN + M yN

y1 + y2 + . . . yN = N − K
Since in a feasible solution K of the y-variables will be 0, then K constraints will be satisfied.
P
n
Similarly we can model the case where at least h ≤ k of aij xj ≤ bi , i = 1, . . . , k must be satisfied. We
j=1
introduce yi , i = 1, ..., k auxiliary binary variables and impose:
n
X
aij xj ≤ bi + M yi
j=1
X
yi ≤ k − h
i

Modeling Trick V: “Possible Constraints Values” A constraint must take on one of N given
values:
a1 x1 + a2 x2 + a3 x3 + . . . + am xm = d1 or
a1 x1 + a2 x2 + a3 x3 + . . . + am xm = d2 or
..
.
a1 x1 + a2 x2 + a3 x3 + . . . + am xm = dN
We introduce the binary variables y1 , y2 , . . . , yN and impose:
a1 x1 + a2 x2 + a3 x3 + . . . + am xm = d1 y1 + d2 y2 + . . . dN yN

y1 + y2 + . . . yN = 1
Example 5.10. The constraint must equal 6 or 8 or 12:
• 3x1 + 2x2 = 6 or
• 3x1 + 2x2 = 8 or
• 3x1 + 2x2 = 12 or
Reformulate with auxiliary variables y1 , y2 , y3 ∈ {0, 1}:
• 3x1 + 2x2 = 6y1 + 8y2 + 12y3 and
• y1 + y2 + y3 = 1
Example 5.11 (Dijunctive constraints in scheduling). Two tasks, P and Q, must be performed by the same
person. The duration of P (resp. Q) is dp units. The start time of P (resp. Q) is denoted as sp (sq ).
We want to enforce either sp + dp ≤ sq or sq + dq ≤ sp .
Trick: Define binary variable ipq , indicating if P precedes Q Introduce the following constraints
sp + dp ≤ sq + M (1 − ipq )
sq + dq ≤ sp + M ipq
5.4. FORMULATIONS 97

5.4 Formulations
Problems can have more than one MILP formulation. Let’s start by considering the Uncapacited Facility
Location problem.

Definition 5.11 (Uncapacited Facility Location (UFL)).

Given:
• depots N = {1, . . . , n}
• clients M = {1, . . . , m}
• fj fixed cost to use depot j
• transport cost for all orders cij
Task: Determine which depots it is most convient to open and which depots serve which client.

An MILP formulation for this problem is:

Variables (
1 if depot open
yj = ,
0 otherwise
xij fraction of demand of i satisfied by j

Objective XX X
min cij xij + fj yj
i∈M j∈N j∈N

Constraints
n
X
xij = 1 ∀i = 1, . . . , m
j=1
X
xij ≤ myj ∀j ∈ N
i∈M

An alternative formulation for the last constraints is the following:

xij ≤ yj ∀i ∈ M, j ∈ N

Then which formulation should we prefer?

5.4.1 Alternative Formulations

Definition 5.12 (Formulation). A polyhedron P ⊆ Rn+p is a formulation for a set X ⊆ Zn × Rp if and only
if X = P ∩ (Zn × Rp )

That is, if it does not leave out any of the solutions of the feasible region X.

There are infinite formulations.

Definition 5.13 (Convex Hull). Given a set X ⊆ Zn the convex hull of X is defined as:

n t
X t
X
i
conv(X) = x : x = λi x , λi = 1, λi ≥ 0, for i = 1, . . . , t,
i=1 i=1

for all finite subsets {x1 , . . . , xt } of X
98 CHAPTER 5. MODELING IN MIXED INTEGER LINEAR PROGRAMMING

Proposition 5.2. conv(X) is a polyhedron (ie, representable as Ax ≤ b)

Proposition 5.3. Extreme points of conv(X) all lie in X

Hence:
max{cT x : x ∈ X} ≡ max{cT x : x ∈ conv(X)}
This is an important result, it means that we can solve the integer programming problem by solving its
linear programming problem relaxation. However the description of the convex hull conv(X) may require an
exponential number of inequalities to describe and it may be not knwon.
What makes a formulation better than another? Let’s suppose that we have two formulations P1 and P2
and that
X ⊆ conv(X) ⊆ P1 ⊂ P2
Then we can conclude that:

P1 is better than P2

Definition 5.14. Given a set X ⊆ Rn and two formulations P1 and P2 for X, P1 is a better formulation
than P2 if P1 ⊂ P2 .

We can now get back to our two alternative formulations for the UFL problem.

Example 5.12 (Formulations for the UFL).

P
• P1 = UFL with i∈M xij ≤ myj ∀j ∈ N
• P2 = UFL with xij ≤ yj ∀i ∈ M, j ∈ N

We show that
P2 ⊂ P1

P
• P2 ⊆ P1 because summing xij ≤ yj over i ∈ M we obtain i∈M xij ≤ myj

• P2 ⊂ P1 because there exists a point in P1 but not in P2 : for example, let m = 6 = 3 · 2 = k · n The
following solution
x10 = 1, x20 = 1, x30 = 1,
x41 = 1, x51 = 1, x61 = 1
under formulation P1 would admit a fractional value for y0 and y1
P
Pi xi0 ≤ 6y0 y0 = 1/2
i xi1 ≤ 6y1 y1 = 1/2

while under the formulation P2 the variables y could not take a fractional value. Since they must be
integer for their proper use in the obejctive function, then we showed that there is a solution in P1 but
not in P2 while not removing any feasible solution.
Chapter 6

Well Solved Problems

6.1 Relaxations
Suppose we have the following ILP problem:

z = max{c(x) : x ∈ X ⊆ Zn }

The set X represents the set of all feasible solutions. In an ILP this set is a subset of Zn . Since the problem
is a maximization problem a any feasible solution x∗ of value z gives a lower bound to z. Then, to prove the
optimality of a feasible solution we need also an upper bound, z. The if z = z, the solution of that gives z is
optimal. Alternatively, we can stop our search process when z − z ≤  for a given reasonably small .

z
z
z

The concepts, roles and determination of upper and lower bounds are linked to the sense of the optimiza-
tion function. In a minimization problem their roles are exchanged. To avoid this dependency on the sense
of the objective function, the following concepts are instead used.

• Primal bounds: every feasible solution gives a primal bound. In some problems it may be easy or hard
to find feasible solutions. Heuristics are used to provide such type of solutions.
• Dual bounds: They are obtain through relaxations of the problem formulation.

In our initial maximization problem, the lower bounds are primal bounds and the upper bounds are dual
bounds.
Given a primal bound pb and a dual bound db it is possible to calculate the optimality gap:
|pb − db|
gap = · 100
pb + 
The  is added to avoid division by zero when pb = 0. To avoid a confusing behaviour when 0 lies in between
pb and db a different definition, which includes the above one, is often used. For a minimization problem,
this is:
pb − db
gap = (·100)
inf{|z|, z ∈ [db, pb]}
If pb ≥ 0 and db ≥ 0 then pb−db
db . If db = pb = 0 then gap = 0. If no feasible solution is found or db ≤ 0 ≤ pb
then the gap is not computed.

99
100 CHAPTER 6. WELL SOLVED PROBLEMS

Proposition 6.1.
(RP ) z R = max{f (x) : x ∈ T ⊆ Rn } is a relaxation of
(IP ) z = max{c(x) : x ∈ X ⊆ Rn } if :

(i) X ⊆ T or

(ii) f (x) ≥ c(x) ∀x ∈ X

In other terms:  
maxx∈T c(x)
max f (x) ≥ ≥ max c(x)
x∈T maxx∈X f (x) x∈X

• T is the set of candidate solutions;

• X ⊆ T is the set of feasible solutions;

• f (x) ≥ c(x)

Proposition 6.2. (i) If a relaxation RP is infeasible, the original problem IP is infeasible.

(ii) Let x∗ be optimal solution for RP. If x∗ ∈ X and f (x∗ ) = c(x∗ ) then x∗ is optimal for IP.

There are at least four ways to construct relaxations.

1. Linear relaxation Given

(IP) : max{cT x : x ∈ P ∩ Zn }, where P = {c ∈ Rn : Ax ≤ b}

the linear relaxation is

(LPR) : max{cT x : x ∈ P }

Better formulations give better bounds (P1 ⊆ P2 )

Combinatorial relaxations Some complicating constraints are removed leaving a problem easy (that
is, in polyniomail time) to solve. For example, the TSP can be reduced to an Assignment problem by dropping
the subtour elimination constraints.

Lagrangian relaxation it is obtained by bringing all or some constraints in the objective function via
multipliers. That is,

IP : z = max{cT x : Ax ≤ b, x ∈ X ⊆ Zn }
LR : z(u) = max{cT x + u(b − Ax) : x ∈ X}

The for all u ≥ 0 it is z(u) ≥ z.

For both combinatorial and Lagrangian relaxations the constraints that is worth trying to relax are the
ones that
• do not worsen too much the quality of bound (tightness of relaxation)

• leave a remaining problem that can be solved efficiently

• have multipliers that can be found efficiently

• are difficult to formulate mathematically

• are too expensive to write up

6.2. WELL SOLVED PROBLEMS 101

Duality the concept of duality works only for linear programming. We can adapt the definitions of dual
problems to integer programming as follows:

Definition 6.1. Two problems:

z = max{c(x) : x ∈ X} w = min{w(u) : u ∈ U }

form a weak-dual pair if c(x) ≤ w(u) for all x ∈ X and all u ∈ U .

When z = w they form a strong-dual pair.

Proposition 6.3. z = max{cT x : Ax ≤ b, x ∈ Zn+ } and wLP = min{ubT : uA ≥ c, u ∈ Rm

+ } (ie, the dual
of the linear relaxation) form a weak-dual pair.

Proposition 6.4. Let IP and D be a weak-dual pair. Then:

(i) If D is unbounded, then IP is infeasible
(ii) If x∗ ∈ X and u∗ ∈ U satisfy c(x∗ ) = w(u∗ ) then x∗ is optimal for IP and u∗ is optimal for D.

An advantage of ILP with respect to LP is that once we have a dual problem we do not need to solve an
LP like in the LP relaxation to have a bound, any feasible dual solution gives a dual bound for the primal
problem.
Here are some examples of dual pairs in ILP:

Matching: z = max{1T x : Ax ≤ 1, x ∈ Zm+}

Example 6.1. Vertx cover: w = min{1T y : yT A ≥ 1, y ∈ Zn+ }

are weak dual pairs. Indeed, it is easy to see that LP relaxations of these two problems are dual of each
other, then
z ≤ z LP = wLP ≤ w.
The two problems are strong-dual pairs when the graphs are bipartite.

Packing: z = max{1T x : Ax ≤ 1, x ∈ Zn+ }

Example 6.2.
Set covering: w = min{1T y : AT y ≥ 1, y ∈ Zm +}
Are weak-dual pairs, which is provable again via the duality of the linear relaxation.

6.2 Well Solved Problems

6.2.1 Separation problem
We have seen that
max{cT x : x ∈ X} ≡ max{cT x : x ∈ conv(X)}
where X ⊆ Zn and X = P ∩ Zn , P a polyhedron P ⊆ Rn .

Definition 6.2 (Separation problem for a COP). Given x∗ ∈ P is x∗ ∈ conv(X)? If not find an inequality
ax ≤ b satisfied by all points in X but violated by the point x∗ .

Farkas’ lemma states the existence of such an inequality.

The following four properties often go together:

(i) Efficient optimization property: there exists a polynomial time algorithm for max{cx : x ∈ X ⊆ Rn }
(ii) Strong duality property: there exists a strong dual D min{w(u) : u ∈ U } that allows to quickly verify
optimality
(iii) Efficient separation property: there exists an efficient algorithm for the separation problem
(iv) Efficient convex hull property: there is a compact description of the convex hull available.
102 CHAPTER 6. WELL SOLVED PROBLEMS

Note that if the explicit convex hull property is true, then the strong duality property and the efficient
separation property hold (just description of conv(X)).
Problems that are easy have typically all four properties satisfied.
Polyhedral analysis is the field of theoretical analysis to prove results about: the strength of certain
inequalities that are facet defining and the descriptions of convex hull of some discrete X ⊆ Z∗ (we see one
way to do this in the next section).

Example 6.3. Let

m
X
X = {(x, y) ∈ Rm
+ ×B : 1
xi ≤ my, xi ≤ 1 for i = 1, . . . , m}
i=1

P = {(x, y) ∈ Rn+ × R1 : xi ≤ y for i = 1, . . . , m, y ≤ 1}.

The polytope P describes conv(X).

6.3 Totally Unimodular Matrices

When the LP solution to this problem

IP : max{cT x : Ax ≤ b, x ∈ Zn+ }

with all data integer will have integer solution?

Let’s look back at the simplex in matrix notation.
 
 
 
 AN AB 0 b 
 
 
 
 
cTN cTB 1 0

AB x B + AN x N = b
xN = 0 AB xB = b,
AB m × m non singular matrix
xB ≥ 0
Cramer’s rule for solving systems of linear equations:
    
a b x e
=
c d y f
   
 e b a e 
  
f d c f
x=    y =  
 a b 
a b  
 c d c d

Aadj
B b
x = A−1
B b=
det(AB )

Definition 6.3. • A square integer matrix B is called unimodular (UM) if det(B) = ±1

• An integer matrix A is called totally unimodular (TUM) if every square, nonsingular submatrix of A
is UM

Proposition 6.5. • If A is TUM then all vertices of R1 (A) = {x : Ax = b, x ≥ 0} are integer if b is

integer
6.3. TOTALLY UNIMODULAR MATRICES 103

• If A is TUM then all vertices of R2 (A) = {x : Ax ≤ b, x ≥ 0} are integer if b is integer.

 
Proof. if A is TUM then A I is TUM  
Any square, nonsingular submatrix C of A I can be written as
 
B 0
C=
D Ik

where B is square submatrix of A. Hence det(C) = det(B) = ±1

Proposition 6.6. The transpose matrix AT of a TUM matrix A is also TUM.

Theorem 6.7 (Sufficient condition). An integer matrix A with is TUM if

1. aij ∈ {0, −1, +1} for all i, j
Pm
2. each column contains at most two non-zero coefficients ( i=1 |aij | ≤ 2)
3. if the rows can be partitioned into two sets I1 , I2 such that:
• if a column has 2 entries of same sign, their rows are in different sets
• if a column has 2 entries of different signs, their rows are in the same set

 
  0 1 0 0 0
  1 −1 −1 0
  1 −1 0 0 1 1 1 1
1 −1 −1 0 0 1  
0 1 1   1 0 1 1 1
1 1  0 1 0 −1  
1 0 1 1 0 0 1 0
0 0 1 0
1 0 0 0 0
Proof: by induction
Basis: one matrix of one element {+1, −1} is TUM
Induction: let C be of size k.
If C has column with all 0s then it is singular.
If a column with only one 1 then expand on that by induction
If 2 non-zero in each column then X X
∀j : aij = aij
i∈I1 i∈I2

but then linear combination of rows and det(C) = 0

Other matrices with integrality property:

• TUM
• Balanced matrices
• Perfect matrices
• Integer vertices

Defined in terms of forbidden substructures that represent fractionating possibilities.

Proposition 6.8. A is always TUM if it comes from

• node-edge incidence matrix of undirected bipartite graphs
(ie, no odd cycles) (I1 = U, I2 = V, B = (U, V, E))
• node-arc incidence matrix of directed graphs (I2 = ∅)

Eg: Shortest path, max flow, min cost flow, bipartite weighted matching
104 CHAPTER 6. WELL SOLVED PROBLEMS
Chapter 7

Network Flows

105
 models arise in broader problem contexts and how the algorithms that we have
developed for the core models can be used in conjunction with other methods to
solve more complex problems that arise frequently in practice. In particular, this
discussion permits us to introduce and describe the basic ideas of decomposition
methods for several important network optimization models-constrained shortest
paths, the traveling salesman problem, vehicle routing problem, multicommodity
flows, and network design.
Since the proof of the pudding is in the eating, we have also included a chapter
on some aspects of computational testing of algorithms. We devote much of our
discussion to devising the best possible algorithms for solving network flow prob-
lems, in the theoretical sense of computational complexity theory. Although the
theoretical model of computation that we are using has proven to be a valuable guide
for modeling and predicting the performance of algorithms in practice, it is not a
perfect model, and therefore algorithms that are not theoretically superior often
perform best in practice. Although empirical testing of algorithms has traditionally
been a valuable means for investigating algorithmic ideas, the applied mathematics,
computer science, and operations research communities have not yet reached a
consensus on how to measure algorithmic performance empirically. So in this chapter
we not only report on computational experience with an algorithm we have pre-
sented, but also offer some thoughts on how to measure computational performance
and compare algorithms.

1.2 NETWORK FLOW PROBLEMS

In this section we introduce the network flow models we study in this book, and in
the next section we present several applications that illustrate the practical impor-
tance of these models. In both the text and exercises throughout the remaining
chapters, we introduce many other applications. In particular, Chapter 19 contains
a more comprehensive summary of applications with illustrations drawn from several
specialties in applied mathematics, engineering, lpgistics, manufacturing, and the
physical sciences.

Minimum Cost Flow Problem

The minimum cost flow model is the most fundamental of all network flow problems.
Indeed, we devote most of this book to the minimum cost flow problem, special
cases of it, and several of its generalizations. The problem is easy to state: We wish
to determine a least cost shipment of a commodity through a network in order to
satisfy demands at certain nodes from available supplies at other nodes. This model
has a number of familiar applications: the distribution of a product from manufac-
turing plants to warehouses, or from warehouses to retailers; the flow of raw material
and intermediate goods through the various machining stations in a production line;
the routing of automobiles through an urban street network; and the routing of calls
through the telephone system. As we will see later in this chapter and in Chapters
9 and 19, the minimum cost flow model also has many less transparent applications.
In this section we present a mathematical programming formulation of the
minimum cost flow problem and then describe several of its specializations and

4 Introduction Chap. 1
variants as well as other basic models that we consider in later chapters. We assume
our readers are familiar with the basic notation and definitions of graph theory; those
readers without this background might consult Section 2.2 for a brief account of this
material.
Let G = (N, A) be a directed network defined by a set N of n nodes and a
set A of m directed arcs. Each arc (i, j) E A has an associated cost Cij that denotes
the cost per unit flow on that arc. We assume that the flow cost varies linearly with
the amount of flow. We also associate with each arc (i, j) E A a capacity Uij that
denotes the maximum amount that can flow on the arc and a lower bound lij that
denotes the minimum amount that must flow on the arc. We associate with each
node i E N an integer number b(i) representing its supply/demand. If b(i) > 0, node
i is a supply node; if b(i) < 0, node i is a demand node with a demand of - b(i); and
if b(i) = 0, node i is a transshipment node. The decision variables in the minimum
cost flow problem are arc flows and we represent the flow on an arc (i,}) E A by
Xij. The minimum cost flow problem is an optimization model formulated as follows:
Minimize 2 CijXij (l.la)
(i,j)EA

subject to
2 Xij - 2 Xj; = b(i) for all i E N, (l.Ib)
{j:(i,j)EA} {j:(j,i)EA}

for all (i,}) E A, (l.Ic)

where 27= 1 b(i) = O. In matrix form, we represent the minimum cost flow problem
as follows:
Minimize cx (l.2a)
subject to
Xx = b, (l.2b)
I :5 X :5 U. (l.2c)
In this formulation, X is an n x m matrix, called the node-arc incidence matrix
of the minimum cost flow problem. Each column X ij in the matrix corresponds to
the variable Xij. The column X ij has a + 1 in the ith row, a -1 in the jth row; the
rest of its entries are zero.
We refer to the constraints in (l.Ib) as mass balance constraints. The first
term in this constraint for a node represents the total outflow of the node (i.e., the
flow emanating from the node) and the second term represents the total inflow of
the node (i.e., the flow entering the node). The mass balance constraint states that
the outflow minus inflow must equal the supply/demand of the node. If the node is
a supply node, its outflow exceeds its innow; if the node is a demand node, its inflow
exceeds its outflow; and if the node is a transshipment node, its outflow equals its
inflow. The flow must also satisfy the lower bound and capacity constraints (1.1 c),
which we refer to asflow bound constraints. The flow bounds typically model phys-
ical capacities or restrictions imposed on the flows' operating ranges. In most ap-
plications, the lower bounds on arc flows are zero; therefore, if we do not state
lower bounds for any problem, we assume that they have value zero.

Sec. 1.2 Network Flow Problems 5

 In most parts of the book we assume that the data are integral (i.e., all arc
capacities, arc costs, and supplies/demands of nodes are integral). We refer to this
assumption as the integrality assumption. The integrality assumption is not restric-
tive for most applications because we can always transform rational data to integer
data by mUltiplying them by a suitably large number. Moreover, we necessarily need
to convert irrational numbers to rational numbers to represent them on a computer.
The following special versions of the minimum cost flow problem playa central
role in the theory and applications of network flows.

Shortest path problem. The shortest path problem is perhaps the simplest
of all network flow problems. For this problem we wish to find a path of minimum
cost (or length) from a specified source node s to another specified sink node t,
assuming that each arc (i, j) E A has an associated cost (or length) Cij' Some of the
simplest applications of the shortest path problem are to determine a path between
two specified nodes of a network that has minimum length, or a path that takes least
time to traverse, or a path that has the maximum reliability. As we will see in our
later discussions, this basic model has applications in many different problem do-
mains, such as equipment replacement, project scheduling, cash flow management,
message routing in communication systems, and traffic flow through congested cities.
If we set b(s) = 1, b(t) = - 1, and b(i) = 0 for all other nodes in the minimum
cost flow problem, the solution to the problem will send 1 unit of flow from node s
to node t along the shortest path. The shortest path problem also models situations
in which we wish to send flow from a single-source node to a single-sink node in an
uncapacitated network. That is, if we wish to send v units of flow from node s to
node t and the capacity of each arc of the network is at least v, we would send the
flow along a shortest path from node s to node t. If we want to determine shortest
paths from the source node s to every other node in the network, then in the minimum
cost flow problem we set b(s) = (n - 1) and b(i) = - 1 for all other nodes. [We
can set each arc capacity Uij to any number larger than (n - 1).] The minimum cost
flow solution would then send unit flow from node s to every other node i along a
shortest path.

Maximum flow problem. The maximum flow problem is in a sense a com-

plementary model to the shortest path problem. The shortest path problem models
situations in which flow incurs a cost but is not restricted by any capacities; in
contrast, in the maximum flow problem flow incurs no costs but is restricted by flow
bounds. The maximum flow problem seeks a feasible solution that sends the max-
imum amount of flow from a specified source node s to another specified sink node
t. If we interpret uijas the maximum flow rate of arc (i,j), the maximum flow problem
identifies the maximum steady-state flow that the network can send from node s to
node t per unit time. Examples of the maximum flow problem include determining
the maximum steady-state flow of (1) petroleum products in a pipeline network, (2)
cars in a road network, (3) messages in a telecommunication network, and (4) elec-
tricity in an electrical network. We can formulate this problem as a minimum cost
flow problem in the following manner. We set b(i) = 0 for all i E N, Cij = 0 for all
(i, j) E A, and introduce an additional arc (t, s) with cost C ts = - 1 and flow bound
U ts = 00. Then the minimum cost flow solution maximizes the flow on arc (t, s); but

6 Introduction Chap. J
since any flow on arc (t, s) must travel from node s to node t through the arcs in A
[since each b(i) = 0], the solution to the minimum cost flow problem will maximize
the flow from node s to node t in the original network.

Assignment problem. The data of the assignment problem consist of two

equally sized sets Nt and N z (i.e., / Nt / = / N z /), a collection of pairs A Nt x
N z representing possible assignments, and a cost cij associated with each element
(i, j) E A. In the assignment problem we wish to pair, at minimum possible cost,
each object in Nt with exactly one object in N z . Examples of the assignment problem
include assigning people to projects, jobs to machines, tenants to apartments, swim-
mers to events in a swimming meet, and medical school graduates to available in-
ternships. The assignment problem is a minimum cost flow problem in a network
G = (Nt U N 2 , A) with b(i) = 1 for all i E N l , b(i) = -1 for all i E N 2 , and
uij = 1 for all (i, j) E A.

Transportation problem. The transportation problem is a special case of

the minimum cost flow problem with the property that the node set N is partitioned
into two subsets NJ and N z (of possibly unequal cardinality) so that (1) each node
in Nl is a supply node, (2) each node N z is a demand node, and (3) for each arc
(i,j) inA, i E Nt andj E N z . The classical example of this problem is the distribution
of goods from warehouses to customers. In this context the nodes in N 1 represent
the warehouses, the nodes in N2 represent customers (or, more typically, customer
zones), and an arc (i, j) in A represents a distribution channel from warehouse i to
customer j.

Circulation problem. The circulation problem is a minimum cost flow prob-

lem with only transshipment nodes; that is, b(i) = 0 for all i E N. In this instance
we wish to find a feasible flow that honors the lower and upper bounds lij and Uij
imposed on the arc flows Xij' Since we never introduce any exogenous flow into the
network or extract any flow from it, all the flow circulates around the network. We
wish to find the circulation that has the minimum cost. The design of a routing
schedule of a commercial airline provides one example of a circulation problem. In
this setting, any airplane circulates among the airports of various cities; the lower
bound lij imposed on an arc (i, j) is 1 if the airline needs to provide service between
cities i and j, and so must dispatch an airplane on this arc (actually, the nodes will
represent a combination of both a physical location and a time of day so that an arc
connects, for example, New York City at 8 A.M. with Boston at 9 A.M.).

In this book, we also study the following generalizations of the minimum cost
flow problem.
Convex cost flow problems. In the minimum cost flow problem, we assume
that the cost of the flow on any arc varies linearly with the amount of flow. Convex
cost flow problems have a more general cost structure: The cost is a convex function
of the amount of flow. Flow costs vary in a convex manner in numerous problem
settings, including (1) power losses in an electrical network due to resistance, (2)
congestion costs in a city transportation network, and (3) expansion costs of a com-
munication network.

Sec. J.2 Network Flow Problems 7

 Generalized flow problems. In the minimum cost flow problem, arcs con-
serve flows (Le., the flow entering an arc equals the flow leaving the arc). In gen-
eralized flow problems, arcs might "consume" or "generate" flow. If Xij units of
flow enter an arc (i, j), then jJ.ijXij units arrive at node j; jJ.ij is a positive multiplier
associated with the arc. If 0 < jJ.ij < I, the arc is lossy, and if I < jJ.ij < 00, the arc
is gainy. Generalized network flow problems arise in several application contexts:
for example, (I) power transmission through electric lines, with power lost with
distance traveled, (2) flow of water through pipelines or canals that lose water due
to seepage or evaporation, (3) transportation of a perishable commodity, and (4)
cash management scenarios in which arcs represent investment opportunities and
multipliers represent appreciation or depreciation of an investment's value.

Multicommodity flow problems. The minimum cost flow problem models

the flow of a single commodity over a network. Multicommodity flow problems arise
when several commodities use the same underlying network. The commodities may
either be differentiated by their physical characteristics or simply by their origin-
destination pairs. Different commodities have different origins and destinations, and
commodities have separate mass balance constraints at each node. However, the
sharing of the common arc capacities binds the different commodities together. In
fact, the essential issue addressed by the multicommodity flow problem is the al-
location of the capacity of each arc to the individual commodities in a way that
minimizes overall flow costs. Multicommodity flow problems arise in many practical
situations, including (I) the transportation of passengers from different origins to
different destinations within a city; (2) the routing of nonhomogeneous tankers (non-
homogeneous- in terms of speed, carrying capability, and operating costs); (3) the
worldwide shipment. of different varieties of grains (such as corn, wheat, rice, and
soybeans) from countries that produce grains to those that consume it; and (4) the
transmission of messages in a communication network between different origin-
destination pairs.

Other Models
In this book we also study two other important network models: the minimum span-
ning tree problem and the matching problem. Although these two models are not
flow problems per se, because of their practical and mathematical significance and
because of their close connection with several flow problems, we have included
them as part of our treatment of network flows.

Minimum spanning tree problem. A spanning tree is a tree (i.e., a con-

nected acyclic graph) that spans (touches) all the nodes of an undirected network.
The cost of a spanning tree is the sum of the costs (or lengths) of its arcs. In the
minimum spanning tree problem, we wish to identify a spanning tree of minimum
cost (or length). The applications of the minimum spanning tree problem are varied
and include (1) constructing highways or railroads spanning several cities; (2) laying
pipelines connecting offshore drilling sites, refineries, and consumer markets; (3)
designing local access networks; and (4) making electric wire connections on a con-
trol panel.

8 Introduction Chap. 1
 Matching problems. A matching in a graph G = (N, A) is a set of arcs
with the property that every node is incident to at most one arc in the set; thus a
matching induces a pairing of (some 00 the nodes in the graph using the arcs in A.
In a matching, each node is matched with at most one other node, and some nodes
might not be matched with any other node. The matching problem seeks a matching
that optimizes some criteria. Matching problems on a bipartite graphs (i.e., those
with two sets of nodes and with arcs that join only nodes between the two sets, as
in the assignment and transportation problems) are called bipartite matching prob-
lems, and those on nonbipartite graphs are called nonbipartite matching problems.
There are two additional ways of categorizing matching problems: cardinality match-
ing problems, which maximize the number of pairs of nodes matched, and weighted
matching problems, which maximize or minimize the weight of the matching. The
weighted matching problem on a bipartite graph is also known as the assignment
problem. Applications of matching problems arise in matching roommates to hostels,
matching pilots to compatible airplanes, scheduling airline crews for available flight
legs, and assigning duties to bus drivers.

1.3 APPLICATIONS
Networks are pervasive. They arise in numerous application settings and in many
forms. Physical networks are perhaps the most common and the most readily iden-
tifiable classes of networks; and among physical networks, transportation networks
are perhaps the most visible in our everyday lives. Often, these networks model
homogeneous facilities such as railbeds or highways. But on other occasions, they
correspond to composite entities that model, for example, complex distribution and
logistics decisions. The traditional operations research "transportation problem" is
illustrative. In the transportation problem, a shipper with inventory of goods at its
warehouses must ship these goods to geographically dispersed retail centers, each
with a given customer demand, and the shipper would like to meet these demands
incurring the minimum possible transportation costs. In this setting, a transportation
link in the underlying network might correspond to a complex distribution channel
with, for example, a trucking shipment from the warehouse to a railhead, a rail
shipment, and another trucking leg from the destination rail yard to the customer's
site.
Physical networks are not limited to transportation settings; they also arise in
several other disciplines of applied science and engineering, such as mathematics,
chemistry, and electrical, communications, mechanical, and civil engineering. When
physical networks occur in these different disciplines, their nodes, arcs, and flows
model many different types of physical entities. For example, in a typical commu-
nication network, nodes will represe'nt telephone exchanges and'transmission facil-
ities, arcs will denote copper cables or fiber optic links, and flow would signify the
transmission of messages or of data. Figure 1",1 shows some typical associations
for the nodes, arcs, and flows in a variety of physical networks.
Network flow problems also arise in surprising ways for problems that on the
surface might not appear to involve networks at all. Sometimes these applications
are linked to a physical entity, and at other times they are not. Sometimes the nodes
and arcs have a temporal dimension that models activities that take place over time.

Sec. 1.3 Applications 9

 4.10 Minimum Cost Flows 169

has non-negative cost by Theorem 4.10.1. Using that P is a minimum cost

(s, t)-path in N (x), we conclude that each of R, Q has cost at least c(P )
implying that c(P ′ ) ≥ c(P ). Hence (4.24) holds. #
⊓

4.10.3 The Assignment and the Transportation Problem

In this section we briefly discuss two special cases of the minimum cost flow
problem, both of which occur frequently in practical applications. For a more
detailed discussion see, e.g., [91, Section 3.12].
In the Assignment Problem, the input consists of a set of persons
P1 , P2 , . . . , Pn , a set of jobs J1 , J2 , . . . , Jn and an n × n matrix M = [Mij ]
whose entries are non-negative integers. Here Mij is a measure for the skill of
person Pi in performing job Jj (the lower the number the better Pi performs
job Jj ). The goal is to find an assignment ! π of persons to jobs so that each
n
person gets exactly one job and the sum i=1 Miπ(i) is minimized. Given
any instance of the assignment problem, we may form a complete bipartite
graph B = (U, V ; E) where U = {P1 , P2 , . . . , Pn }, V = {J1 , J2 , . . . , Jn } and
E contains the edge Pi Jj with the weight Mij for each i ∈ [m], j ∈ [n]. Now
the assignment problem is equivalent to finding a minimum weight perfect
matching in B. Clearly we can also go the other way and hence the assignment
problem is equivalent to the weighted bipartite matching problem. It
is also easy to see from this observation that the assignment problem is a
(very) special case of the minimum cost flow problem. In fact, if we think
of Mij as a cost and orient all edges from U to V in the bipartite graph
above, then what we are seeking is an integer-valued flow of minimum cost
so that the value of the balance vector equals 1 for each Pi , i = 1, 2, . . . , m,
and equals -1 for each Jj , j = 1, 2, . . . , n.
Inspecting the description of the buildup algorithm above, it is not hard
to see that the following holds (Exercise 4.53).

Theorem 4.10.8 The buildup algorithm solves the assignment problem for
a bipartite graph on n vertices in time O(n3 ). ⊓
#

In the transportation problem we are given a set of production plants

S1 , S2 , . . . , Sm that produce a certain product to be shipped to a set of re-
tailers T1 , T2 , . . . , Tn . For each pair (Si , Tj ) there is a real-valued cost cij of
transporting one unit of the product from Si to Tj . Each plant produces ai ,
i = 1, 2, . . . , m, units per time unit and each retailer needs! bj , j = 1, 2,
!.n. . , n,
m
units of the product per time unit. We assume below that i=1 ai = j=1 bj
(this is no restriction of the model as shown in Exercise 4.54). The goal is
to find a transportation schedule for the whole production (i.e., how many
units to send from Si to Tj for i = 1, 2, . . . , m, j = 1, 2, . . . , n) in order to
minimize the total transportation cost.
Again the transportation problem is easily seen to be a special case of the
minimum cost flow problem. Consider a bipartite network N with partite sets
170 4. Flows in Networks

S = {S1 , S2 , . . . , Sm } and T = {T1 , T2 , . . . , Tn } and all possible arcs from S

to T , where the capacity of the arc Si Tj is ∞ and the cost of sending one unit
of flow along Si Tj is cij . Now it is easy to see that an optimal transportation
schedule corresponds to a minimum cost flow in N with respect to the balance
vectors

b(Si ) = ai , i = 1, 2, . . . , m, and b(Tj ) = −bj , j = 1, 2, . . . , n.

Again we could solve the transportation problem by the buildup algorithm

but in this case we would not be guaranteed a polynomial running time since
the running time would depend on the required balance values. Applying
Theorem 4.10.4, we obtain a strongly polynomial algorithm for the problem.
Clearly one would expect the existence of an algorithm of better complex-
ity for the transportation problem (being a restricted version of the general
minimum cost flow problem). Such an algorithm was given by Kleinschmidt
and Schannath.
Theorem 4.10.9 [600] The transportation problem with m suppliers and n
consumers can be solved in time O(min{n, m}(n + m)2 log(n + m)). ⊓
#
For much more material on the assignment and transportation problems,
including a survey of various complexities, the reader may consult Chapters
17 and 21 of Schrijver’s book [803].

4.11 Applications of Flows

In this section we illustrate the applicability of flows to a large spectrum of
problems both of theoretical and practical nature. For further applications
see, e.g., Chapters 5, 13 and 17. Since we will need these results in later
chapters the main focus is on finding certain substructures in digraphs.

4.11.1 Maximum Matchings in Bipartite Graphs

Let G = (V, E) be an undirected graph. Recall that a matching in G is a set

of edges from E, no two of which share a vertex, and a maximum match-
ing of G is a matching of maximum cardinality among all matchings of G.
Matching problems occur in many practical applications such as the following
scheduling problem. We are given a set T = {t1 , t2 , . . . , tr } of tasks (such as
handling a certain machine) to be performed and a set P = {p1 , p2 , . . . , ps }
of persons, each of which is capable of performing some of the tasks from
T . The goal is to find a maximum number of tasks such that each task can
be performed by some person who does not at the same time perform any
other task and no task is performed by more than one person. This can be
formulated as a matching problem as follows. Let B = (P, T ; E) be the bi-
partite graph whose vertex set is P ∪ T and such that for each i, j such that
114 CHAPTER 7. NETWORK FLOWS
Chapter 8

Cutting Plane Algorithms

8.1 Valid Inequalities

Given an interger programming problem:
(IP ) : z = max{cT x : x ∈ X}, X = {x : Ax ≤ b, x ∈ Zn+ }
We have argued that if we knew the convex hull description of X then we could solve (IP) by linear pro-
gramming. As we said, it can be proved that: conv(X) = {x : Ãx ≤ b̃, x ≥ 0}. That is, the convex hull can
be described by the intersection of halfplanes and it is therefore a polyhedron. Hence the best formulation
for (IP) would be:
LP : z = max{cT x : Ãx ≤ b̃, x ≥ 0.
If this formulation is not knwon then we can try to approximate it.
Definition 8.1 (Valid inequalities). An inequality ax ≤ b is a valid inequality for X ⊆ Rn if ax ≤ b, for all
x∈X
Which are useful inequalities? How can we find them? How can we use them?
The following examples show how inequalities can be deduced in a pre-processing stage.
Example 8.1.
X = {(x, y) : x ≤ 999y; 0 ≤ x ≤ 5, y ∈ B1 }
The constraint
x ≤ 5y
is implied by the formulation above. Indeed if y is 0 then x ≤ 0, if y is 1 then x ≤ 5. Hence x will never be
larger than 5.
Example 8.2. X = {x ∈ Zn+ : 13x1 + 20x2 + 11x3 + 6x4 ≥ 72}
Dividing left and right hand side by 11 we get:
13 20 6 72 6
2x1 + 2x2 + x3 + x4 ≥ x1 + x2 + x3 + x4 ≥ =6+
11 11 11 11 11
which gives the implied constraint:
2x1 + 2x2 + x3 + x4 ≥ 7.
Example 8.3. Capacitated facility location:
X
xij ≤ bj yj ∀j ∈ N xij ≤ bj yj
i∈M
X
xij = ai ∀i ∈ M xij ≤ ai
j∈N

xij ≥ 0, yj ∈ B n xij ≤ min{ai , bj }yj

115
116 CHAPTER 8. CUTTING PLANE ALGORITHMS

8.2 Cutting Plane Algorithms for Integer Programming

8.2.1 Chvátal-Gomory cuts
Let X ∈ P ∩ Zn+ be a feasibility region with P = {x ∈ Rn+ : Ax ≤ b}, A ∈ Rm×n . Let also u ∈ Rm + be a
vector of multipliers. Further, recall the notation {a1 , a2 , . . . an } to represent the columns of A.
The following procedure for cut generation Chvátal-Gomory generates valid inequalities:
n
X
1) uaj xj ≤ ub valid: u ≥ 0
j=1
n
X X X
2) buaj cxj ≤ ub valid: x ≥ 0 and buaj cxj ≤ uaj xj
j=1
Xn
3) buaj cxj ≤ bubc valid for X since x ∈ Zn
j=1

Theorem 8.1. Applying this CG procedure a finite number of times every valid inequality for X can be
obtained.

8.2.2 Cutting Plane Algorithms

Let X ∈ P ∩Zn+ be a feasible region for which we are given a family of valid inequalities F : aT x ≤ b, (a, b) ∈ F.
We do not find and use them all a priori. We are only interested in those close to optimum. We achieve this
with the following procedure:

Init.: t = 0, P 0 = P

Iter. t: Solve z̄ t = max{cT x : x ∈ P t }

let xt be an optimal solution
if xt ∈ Zn stop, xt is opt to the IP
if xt 6∈ Zn solve separation problem for xt and F
if (at , bt ) is found with at xt > bt that cuts off xt

P t+1 = P ∩ {x : ai x ≤ bi , i = 1, . . . , t}

else stop (P t is in any case an improved formulation)

8.2.3 Gomory’s fractional cutting plane algorithm

Cutting plane algorithm + Chvátal-Gomory cuts
• max{cT x : Ax = b, x ≥ 0, x ∈ Zn }

• Solve LPR to optimality

 
 I ĀN = A−1 
 B AN 0 b̄ 
 
c̄B c̄N (≤ 0) 1 −d¯

P
xu = b̄u − āuj xj , u∈B
P j∈N
z = d¯ + c̄j xj
j∈N
8.2. CUTTING PLANE ALGORITHMS FOR INTEGER PROGRAMMING 117

• If basic optimal solution to LPR is not integer then ∃ some row u: b̄u 6∈ Z1 .
The Chvatál-Gomory cut applied to this row is:
X
x Bu + bāuj cxj ≤ bb̄u c
j∈N

(Bu is the index in the basis B corresponding to the row u) (cntd)

P
• Eliminating xBu = b̄u − āuj xj in the CG cut we obtain:
j∈N
X
(āuj − bāuj c)xj ≥ b̄u − bb̄u c
| {z } | {z }
j∈N 0<fu <1
0≤fuj <1
X
fuj xj ≥ fu
j∈N

fu > 0 or else u would not be row of fractional solution. It implies that x∗ in which x∗N = 0 is cut out!

• Moreover: when x is integer, since all coefficient in the CG cut are integer the slack variable of the cut
is also integer: X
s = −fu + fuj xj
j∈N

(theoretically it terminates after a finite number of iterations, but in practice not successful.)

max x1 + 4x2
x1 + 6x2 ≤ 18
x1 ≤ 3
x1 , x2 ≥ 0
x1 , x2 integer

x2

x1 = 3
x1 + 6x2 = 18

x1
x1 + 4x2 = 2

| | x1 | x2 | x3 | x4 | -z | b |
|---+----+----+----+----+----+----|
| | 1 | 6 | 1 | 0 | 0 | 18 |
| | 1 | 0 | 0 | 1 | 0 | 3 |
|---+----+----+----+----+----+----|
| | 1 | 4 | 0 | 0 | 1 | 0 |

| | x1 | x2 | x3 | x4 | -z | b |
|---+----+----+----+----+----+----|
| | 0 | 6 | 1 | -1 | 0 | 15 |
| | 1 | 0 | 0 | 1 | 0 | 3 |
|---+----+----+----+----+----+----|
| | 0 | 4 | 0 | -1 | 1 | -3 |

| | x1 | x2 | x3 | x4 | -z | b |
|---+----+----+------+------+----+------|
| | 0 | 1 | 1/6 | -1/6 | 0 | 15/6 |
| | 1 | 0 | 0 | 1 | 0 | 3 |
|---+----+----+------+------+----+------|
| | 0 | 0 | -2/3 | -1/3 | 1 | -13 |
118 CHAPTER 8. CUTTING PLANE ALGORITHMS

x2 = 5/2, x1 = 3
Optimum, not integer

• We take the first row: | | 0 | 1 | 1/6 | -1/6 | 0 | 15/6 |

P 1
• CG cut j∈N fuj xj ≥ fu 6 x3 + 56 x4 ≥ 1
2

• Let’s see that it leaves out x∗ : from the CG proof:

1/6 (x1 + 6x2 ≤ 18)

5/6 (x1 ≤ 3)
x1 + x2 ≤ 3 + 5/2 = 5.5

since x1 , x2 are integer x1 + x2 ≤ 5

• Let’s see how it looks in the space of the original variables: from the first tableau:

x3 = 18 − 6x2 − x1
x4 = 3 − x1

1 5 1
(18 − 6x2 − x1 ) + (3 − x1 ) ≥ x1 + x2 ≤ 5
6 6 2

• Graphically:
x2

x1 = 3
x1 + 6x2 = 18
x1 + x2 = 5
x1
x1 + 4x2 = 2

• Let’s continue:
| | x1 | x2 | x3 | x4 | x5 | -z | b |
|---+----+----+------+------+----+----+------|
| | 0 | 0 | -1/6 | -5/6 | 1 | 0 | -1/2 |
| | 0 | 1 | 1/6 | -1/6 | 0 | 0 | 5/2 |
| | 1 | 0 | 0 | 1 | 0 | 0 | 3 |
|---+----+----+------+------+----+----+------|
| | 0 | 0 | -2/3 | -1/3 | 0 | 1 | -13 |

We need to apply dual-simplex

(will always be the case, why?)

c
ratio rule: min | aijj |

• After the dual simplex iteration:

| | x1 | x2 | x3 | x4 | x5 | -z | b |
|---+----+----+------+----+------+----+-------|
| | 0 | 0 | 1/5 | 1 | -6/5 | 0 | 3/5 |
| | 0 | 1 | 1/5 | 0 | -1/5 | 0 | 13/5 |
| | 1 | 0 | -1/5 | 0 | 6/5 | 0 | 12/5 |
|---+----+----+------+----+------+----+-------|
| | 0 | 0 | -3/5 | 0 | -2/5 | 1 | -64/5 |
8.2. CUTTING PLANE ALGORITHMS FOR INTEGER PROGRAMMING 119

We can choose any of the three rows.

4
Let’s take the third: CG cut: 5 x3 + 15 x5 ≥ 2
5

• In the space of the original variables:

4(18 − x1 − 6x2 ) + (5 − x1 − x2 ) ≥ 2
x1 + 5x2 ≤ 15

x2

x1

• ...
128 CHAPTER 8. CUTTING PLANE ALGORITHMS
Chapter 9

Branch and Bound

129
References

T. Cormen, C. Leiserson, R. Rivest, and Stein. Introduction to algorithms. MIT press, Cambridge, Mas-
sachusetts, USA, third edition, 2009.

G. Dantzig. Maximization of a linear function of variables subject to linear inequalities. In T. Koopmans,

editor, Activity Analysis of Production and Allocation, pages 339–347. John Wiley & Sons, New York,
1951.

H. P. Williams. Model building in mathematical programming. John Wiley & Sons, Chichester, fifth edi-
tion, 2013. ISBN 9780471997887. URL http://site.ebrary.com/lib/sdub/docDetail.action?docID=
10657847.

141