Lec11p1, ORF363/COS323

Instructor:
This lecture: Amir Ali Ahmadi
Fall 2014
• Linear Programming (LP)
○ Applications of linear programming TAs: Y. Chen,
G. Hall,
○ History of linear programming J. Ye
• Geometry of linear programming
○ Geometric and algebraic definition of a vertex and their equivalence
○ Optimality of vertices
• The simplex method
○ The idea behind the simplex algorithm
○ An example of the simplex algorithm in use
○ Initialization

• LP is an important and beautiful topic covered in much more depth in ORF

307 [Van14]. We'll only be scratching the surface here.
• Our presentation in this lecture is mostly based on [DPV08], with some
elements from [Ber09], [BT97], [CZ13], [Van14].

Linear Programming

Linear programming is a subclass of convex optimization problems in which

both the constraints and the objective function are linear functions.

• A linear program is an optimization problem of the form:

minimize
subject to

where and This is called a linear program in

standard form.
• Not all linear programs appear in this form but we will see later that they
can all be rewritten in this form using simple transformations.

• Linear programming is truly about solving systems of linear inequalities. In

this sense, the subject natural follows the topic of the end of our last
lecture, namely, that of solving systems of linear equations.

Lec11 Page 1
Lec11p2, ORF363/COS323

Applications of linear programming

Example 1: Transportation

• All plants produce product A (in different quantities) and all

warehouses need product A (also in different quantities).

• The cost of transporting one unit of product A from to is

We want to minimize the total cost of transporting product A while still

fulfilling the demand from the warehouses and without exceeding the supply
produced by the plants.

• Decision variables: , quantity transported from to

• The objective function to minimize:

• The constraints are:

○ Not exceed the supply in any factory:

○ Fulfill needs of the warehouses:

○ Quantity transported must be nonnegative:

Lec11 Page 2
Lec11p3, ORF363/COS323

Example 2: The maximum flow problem

Recall from Lecture 1:

• The goal is to ship as much oil as possible from S to T.
• We cannot exceed the capacities on the edges.
• No storage at the nodes: for every node (except S and T),
flow in=flow out.

Lec11 Page 3
Lec11p4, ORF363/COS323

Example 3: LP relaxation for the largest independent set in a graph

• Given an undirected graph, the goal is to find the largest collection of

nodes such that no two nodes share an edge. (Recall that we saw some
applications of this problem in scheduling in Lecture 1.)

• We can write this problem as a linear program with integer constraints.

Such a problem is called an integer program (more precisely a linear
integer program). Many integer programs of interest in practice are in
fact binary programs; i.e., they only have 0/1 constraints. This is such an
example.

Integer programs (IPs) are in general difficult to solve (we will formalize
this statement in a few lecture). However, we can easily obtain the so-
called "LP relaxation" of this problem by replacing the constraint
with

(LP) s.t.

Observe that the optimal solution to the LP (denoted by ), is an

upperbound to the optimal solution to the IP (denoted by ,); i.e.,

.
(Why?)

Lec11 Page 4
Lec11p5, ORF363/COS323
Example 4: Scheduling [Ber09]

This is another example of integer programming and LP relaxation.

A hospital wants to start weekly nightshifts for its nurses. The goal is to
hire the fewest number of nurses possible.

• There is demand for nurses on days j=1,…7.

• Each nurse wants to work 5 consecutive days.

How many nurses should we hire?

• The decision variables here will be where is the number

of nurses hired on day

• The objective is to minimize the total number of nurses:

• The constraints take into account the demand for each day but also
the fact that the nurses want to work 5 consecutive days. This means
that if the nurses work on day 1, they will work all the way through
day 5.

• Naturally, we would want to be positive integers as we do

not want to get fractions of nurses. Such a constraint results in an IP.
The obvious LP relaxation is the following: impose only nonnegativity
constraints
• The optimal value of the LP will give a lower bound on the optimal
value of the IP; i.e., it will say that it is not possible to meet the
demand with less than LPOPT number of nurses (why?).
• If it just so happens that the optimal solution to the LP is an integer
vector, the same solution must also be optimal to the IP (why?).

Lec11 Page 5
Lec11p6, ORF363/COS323

History of Linear Programming

• Solving systems of linear inequalities goes at least as far back as the late 1700s,
when Fourier invented (a pretty inefficient) solution technique, known today
as the "Fourier-Motzkin" elimination method.

• In 1930s, Kantorovich and Koopmans brought new life to linear programming

by showing its widespread applicability in resource allocation problems. They
jointly received the Nobel Prize in Economics in 1975.

• Von Neumann is often credited with the theory of "LP duality" (the topic of our
next lecture). He was a member of the IAS here at Princeton.

• In 1947, Dantzig invented the first practical algorithm for solving LPs: the
simplex method. This essentially revolutionized the use of linear programming
in practice. (Interesting side story: Dantzig is known for solving two open
problems in statistics, mistaking them for homework after arriving late to
lecture. This inspired the first scene in the movie Good Will Hunting.)

John von Neumann (1903-1957) George Dantzig (1914-2005)

• In 1979, Khachiyan showed that LPs were solvable in polynomial time using
the "ellipsoid method". This was a theoretical breakthrough more than a
practical one, as in practice the algorithm was quite slow.

• In 1984, Karmarkar developed the "interior point method", another

polynomial time algorithm for LPs, which was also efficient in practice. Along
with the simplex method, this is the method of choice today for solving LPs.

Leonid Khachiyan (1952 - 2005) Narendra Karmarkar (b. 1957)

Lec11 Page 6
Lec11p7, ORF363/COS323

LP in alternate forms

As mentioned before, not all linear programs appear in the standard form:

min.
s.t.

Essentially, there are 3 ways in which a linear program can differ from the
standard form:

• it is a maximization problem instead of a minimization problem,

• some constraints are inequalities instead of equalities:
• some variables are unrestricted in sign.

There are, however, simple transformations that reduce these alternative

forms to standard form. We briefly showed how this is done in class. The
reader can find these simple transformations in section 7.1.4 of [DPV08].

Solving an LP in two variables geometrically

s.t.

We are trying to find the "largest"

level set of the objective function
that still intersects the feasible
region.

Lec11 Page 7
Lec11p8 ORF363/COS323

All possibilities for an LP

Infeasible case

min.
s.t.

The three regions in the drawing do not intersect : there are no feasible solutions.

Unbounded case

min.
s.t.

The intersection of the green and the blue regions is

unbounded; we can push the objective function as high up as
we want.
Infinite number of optimal
solutions

s.t.

There is an entire "face" of the feasible region that is optimal. Notice that the
normal to this face is parallel to the objective vector.

Unique optimal solution

s.t.

Lec11 Page 8
Lec11p9, ORF363/COS323

Geometry of LP

The geometry of linear programming is very beautiful. The simplex

algorithm exploits this geometry in a very fundamental way. We'll prove
some basic geometric results here that are essential to this algorithm.

Definition of a polyhedron:

• The set where is a nonzero vector in is called a

hyperplane.

• The set where is a nonzero vector in is called a

halfspace.

• The intersection of finitely many half spaces is called a polyhedron.

○ This is always a convex set:
○ Halfspaces are convex (why?) and intersections of convex sets
are convex (why?).

• A set is bounded if such that

• A bounded polyhedron is called a polytope.

Lec11 Page 9
Lec11p10, ORF363/COS323

Extreme points

A point is an extreme point of a convex set if it cannot be written as a

convex combination of two other points in . In other words, there does
not exist , and 0,1] such that

Alternatively, is an extreme point if

or

Which is extreme? What are the extreme points here?

Extreme points are always on the boundary, but not every point on the
boundary is extreme.

Definition. Consider a set of constraints

Given a point we say that a constraint is tight (or active or binding) at

if
Equality constraints are tight by definition.

Definition. Two constraints are linearly independent if the corresponding

are independent.

• With these two definitions, we can now define the notion of a vertex of a
polyhedron.

Lec11 Page 10
Lec11p11, ORF363/COS323

Vertex

A point is a vertex of a polyhedron , if

(i) it is feasible (
(ii) linearly independent constraints that are tight at

• You may be wondering if extreme points and vertices are the same thing.
The theorem below establishes that this is indeed the case.
• Note that the notion of an extreme point is defined geometrically while the
notion of a vertex is defined algebraically.
• The algebraic definition is more useful for algorithmic purposes and is
crucial to the simplex algorithm. Yet, the geometric definition is used to
prove the fundamental fact that an optimal solution to an LP can always be
found at a vertex. This is crucial to correctness of the simplex algorithm.

Theorem 1: Equivalence of extreme point and vertex

Let be a non-empty polyhedron with .

Let Then,

is an extreme point is a vertex.

Proof:

( Let be a vertex. This implies that linearly independent constraints

are tight at Denote by an matrix whose rows are that of associated
with the tight constraints. Similarly let be a vector of size collecting entries
of corresponding to the tight constraints. So

Suppose we could write for some and

Then , We have:

Lec11 Page 11
Lec11p12, ORF363/COS323

If then as is invertible. If then If then

the previous equality forces which means that
as is invertible.

Suppose is not a vertex. Let Since is not a

vertex, there does not exist linearly independent vectors

We claim that there exists a vector

Indeed, take at most linearly independent , We want to
argue that the linear system

has nontrivial solution. But recall that each is of length So this is an

underconstrained linear system. Hence, it has infinitely many solutions,
among which there is at least one nonzero solution, which we take to be

Let and where is some positive scalar.

We claim that for small enough we have
• for , because
• for the claim follows from continuity of the function
and the fact that when

As , this implies that is not an extreme point of .

Corollary. Given a finite set of linear inequalities, there can only be a finite
number of extreme points.

Proof:
We have shown that extreme points and vertices are the same, so we
prove the result for vertices. Suppose we are given a total of
constraints. To obtain a vertex, we need to pick linearly independent
constraints that are tight. There are at most ways of doing this and
each subset of linearly independent constraints gives a unique vertex
(as seen previously, the vertex satisfies where is invertible). As
As a consequence, there are at most vertices.

Lec11 Page 12
Lec11p13, ORF363/COS323

Definition. A polyhedron contains a line if and such

that

Theorem 2: Existence of extreme points

Consider a nonempty polyhedron . The following are equivalent:

1. does not contain a line.
2. has at least one extreme point.

Corollary.
Every bounded polyhedron has an extreme point.

Theorem 3: Optimality of extreme points

Consider the LP:

s.t.

Suppose has at least one extreme point and there exists an optimal
solution, then there exists an optimal solution which is at a vertex.

Proof.
Let Q be the set of optimal solutions (assumed to be nonempty). In other
words, if is the optimal value of the LP, then

Using Theorem 2 and the fact that we know that has an extreme
point has no lines has no lines has an extreme point. Let
be an extreme point of We will show that is also an extreme point of
Once this is proved, since , we would be done.

Lec11 Page 13
Lec11p14, ORF363/COS323

Suppose that is not an extreme point of Then

s.t.
Multiplying by on both sides, we obtain:

Since is optimal, . Combined with the previous equality, this

implies: is not an extreme point of
Contradiction.

Implication of the theorems

• These theorems show that when looking for an optimal solution, it is

enough to examine only the extreme points.

• This leads to an algorithm for solving an LP: if there are constraints in

(the polytope is then pick all possible subsets of linearly
independent constraints out of the Solve (in worst case) systems
of equations of the type where are the restrictions of and
to the subset of constraints. This can be done, e.g., by the conjugate
gradien method from the previous lecture or by Gaussian elimination.
Evaluate the objective function at each solutuion and pick the best.

• Unfortunately, this algorithm, even though correct, is very inefficient. The

reason is that there are too many vertices to explore: the number is
exponential in . For example, consider the constraints
Then we have in general inequalities, but extreme
points:

• The simplex method is an intelligent algorithm for reducing the number of

vertices visited.

Lec11 Page 14
Lec11p15, ORF363/COS323

The Simplex algorithm

Main idea

Consider some generic LP:

s.t.

In a nutshell, this is all the simplex algorithm does:

• start at a vertex,
• while there is a better neighboring vertex, move to it.

Definition. Two vertices are neighbors if they share tight constraints.

Example:
.
s.t.

Here is the feasible set:

Image credit: [Jon06]

Po ts A I re vert ces of the polyhedro Co s der B t s t ght for

constraint (6) as , for constraint (1) as and for
constraint (4) as . These are three independent constraints so B is
indeed a vertex.

Points A and B are neighbors as they both are tight for constraints (6) and (4) but
A is tight for (5) whereas B is tight for (1). The point C is another neighbor of B.

Lec11 Page 15
Lec11p16, ORF363/COS323

Section 7.6 of [DPV08] gives a very neat presentation of the simplex algorithm in
surprisingly few pages. We won't repeat this here since it's done beautifully in the
book and we followed the book closely in lecture. Instead, we just give an outline
of the steps involved and an example. The example is different than the one in the
book. So you can read one and do the other for practice.

At every iteration of the simplex algorithm, we complete two tasks:

1. Check whether the current vertex is optimal (if yes, we're done),
2. If not, determine the vertex to move to next.

• We start the algorithm at the origin. We are assuming for now that the origin
is feasible (i.e., . Note that if is feasible, then it is a vertex (why?)
• Both tasks listed above are easy if the vertex is at the origin:
○ The origin is optimal if and only if (why?).
○ If for some , we can increase until a new constraint becomes
tight. Now we are at a new vertex (why?).
• Once at a new vertex, we move it to the origin by a "change of coordinates".
Then we simply repeat. See Section 7.6 of [DPV08] for details.
• Finally, if the origin is not feasible, to get the algorithm started we first solve
an "auxiliary LP"; see Section 7.6.3 of [DPV].

An example in two dimensions

s.t.

Here is the feasible set:

Lec11 Page 16
Lec11p17, ORF363/COS323

Iteration 1: The origin is feasible and all are positive. Hence we can
pick either or as the variable we want to increase. We pick and
keep

The origin corresponds to the intersection of (4) and (5). By increasing

, we are releasing (4). As we increase , we must make sure we still
satisfy the constraints. In particular, we must have (1) and
which means Note that constraint (1)
becomes tight before constraint (3) (recall that remains at zero as we
increase Hence, the new vertex is at the intersection of (1) and (5),
i.e., D=(200,0).

We now need to bring D to the origin via a change of coordinates from

to : and

Rewriting and in terms of and , we get a new LP:

max.
s.t.

Since the coefficient of is positive, we must continue.

Lec11 Page 17
Lec11p18, ORF363/COS323

Iteration 2: The current vertex corresponds to the intersection of (1)-

(5). As the coefficient of is positive, we pick as the variable
wevincrease, i.e., we release (5). The other constraints have to be
satisfied, namely and (3). This corresponds to and
As constraint (3) is the one becoming tight next.

The new vertex is then at the intersection of (1) and (3). Reverting to
the original LP, we can see that this is C.

We then change coordinates again to bring C to the origin. This is done

by letting and i.e., and

The problem becomes:

max.
s.t. (1)

Since coefficient of is positive, we must continue.

Lec11 Page 18
Lec11p19, ORF363/COS323

Iteration 3: The current vertex is at the intersection (1) and (3). As the
coefficient for is positive, we pick as the variable we will increase
while keeping This means that we are releasing constraint (1).

We have to meet all the constraints, limiting how much we can increase
and So (2) is the constrint becoming
tight next.

The new vertex is the intersection of (3) and (2). This is the point B.

We change coordinates to make B the new origin. This is done by letting

and i.e., and

The problem becomes:

max.
s.t. (1)

Both coefficients are negative. Hence we conclude that vertex B is

optimal. The optimal value of our LP is 1900.

Lec11 Page 19
Lec11p20, ORF363/COS323

Notes:
The relevant chapter form [DPV08] for this lecture is Chapter 7.
To minimize overlap with ORF 307, we skipped the sections on
bimatrix games and the network algorithm for max-flow. Your
[CZ13] book also has a chapter on linear programming, but
reading that is optional.

References:
- [Bert09] D. Bertsimas. Lecture notes on optimization methods
(6.255). MIT OpenCourseWare, 2009.

- [BT97] D. Bertsimas and J. Tsitsiklis, Introduction to Linear

Optimization, Athena Scientific Series in Optimization and Neural
Computation, 1997.

- [CZ13] E. K. P. Chong and S. H. Zak, An Introduction to Optimization,

4th Edition, Wiley Press, 2013.

- [DPV08] S. Dasgupta, C. Papadimitriou and U. Vazirani, Algorithms,

McGraw Hills, 2008.

- [Jon06] J. Jones, Notes on linear programing, 2006.

- [Van14]: R. Vanderbei, Linear Programming: Foundations and

Extensions, Fourth edition, Springer, 2014.

Lec11 Page 20