Algorithmics (CS4000)

Maciej Liśkiewicz

Lecture 2: Algorithms for Linear Programming

October, 2021

1 A Brute-force Algorithm

In this section we show first that, using the results of the previous lecture, an optimum of a LP
in standard form is an extreme point of the polyhedron. As a direct corollary of this fact we
obtain a brute-force algorithm for solving LP. Moreover, we will use this fact to prove that the
simplex algorithm generate an optimum, if it terminates.
Before we give the theorem, let us recall that there are two possibilities when a linear program
maximize: cT x, subject to Ax ≤ b and x ≥ 0 has no solution: the problem can be infeasible,
i.e. {x ∈ Rn : Ax ≤ b, x ≥ 0} = ∅ , or unbounded meaning it has feasible but no optimal
solutions, i.e. ∀α ∈ R ∃x ∈ Rn cT x > α . The theorem below says that if a LP is neither
infeasible nor unbounded than it has an optimum solution and the solution is an extreme point
of the polyhedron {x ∈ Rn : Ax ≤ b} .
Theorem 1 Any bounded LP in standard form

maximize: cT x, subject to: Ax ≤ b, x ≥ 0

has an optimum at a basic feasible solution.

Proof: Assume x is an optimal solution which is not a basic feasible solution. We show how to
“move” x inside the polyhedron to a basic feasible solution of no worse value. The idea is to
move x first to the boundary of the polyhedron and further to move inside a face’s boundary.
Let P be the polyhedron containing feasible solutions of the LP. From Theorem 1 of the
previous lecture, we have that x is not an extreme point. Therefore, there exists y 6= 0 such
that x + y ∈ P and x − y ∈ P . Moving along y we do not change the value of the objective
function, i.e. we have cT y = 0 . Indeed, if cT y > 0 , then we get cT (x + y) > cT x ; if cT y < 0 ,
then we get cT (x − y) > cT x . Thus, we obtain contradictions in both cases. Since cT y = 0
then for any α ∈ R it is true that cT (x + αy) = cT x .
Now consider a constraint ai that is tight for x , i.e. such that (ai )T x = bi . Note that for
any α ∈ R it is true that (ai )T (αy) = 0 . To see this, note that from inequality (ai )T (x+y) ≤ bi
follows that (ai )T y ≤ 0 , and from (ai )T (x − y) ≤ bi follows that (ai )T y ≥ 0 . Thus we get that
(ai )T y = 0 . Moreover, one can conclude that for any α , (ai )T (αy) = 0 . Thus, if a constraint
2 Algorithmics: Algorithms for Linear Programming

ai is tight for x , meaning that x is inside the hyperplane {z : (ai )T z = bi } then moving x
along vector αy we stay inside the hyperplane, for any α .
Now, let λ = max{α ∈ R : x + αy ∈ P and x − αy ∈ P} . To see that λ is defined correctly
we consider any j , with yj 6= 0 . Note that if xj = 0 then yj has to be 0 , too. Otherwise
xj + yj < 0 or xj − yj < 0 what means that x + y 6∈ P or x − y 6∈ P . Therefore, λ is defined
correctly. Using such λ , we move x to x + λy that remains a feasible solution. But then,
the number of tight constraints for this solution is greater that for x . This “moving” can be
repeated, till the solution becomes basic. t
u
Using the idea of the proof we can convert any feasible solution of a LP in standard form
to a basic feasible solution of no worse value. Thus, to solve the problem, it suffices to evaluate
the objective function at all basic feasible solutions (or equivalently all extreme points of the
polyhedron) and take the best one. Since for LP with n variables, any basic feasible solution
m


is defined by a set of n constraints, there are at most n basic feasible solutions, where m
denotes the number of LP’s constraints, including x ≥ 0 . To find a basic feasible solution for
each set of n constraints, first a linear system of inequalities has to be solved, which can be
done by Gaussian elimination in O(n3 ) steps, and next it should be tested if the solution fulfills
all constraints. The algorithm can be summarized as follows.

Brute-force LP Solver (A, b, c)

1. For any subset of n constraints from Ax ≤ b ; x1 ≥ 0, . . . , xn ≥ 0
replace the inequalities by equalities,
for the obtained system of linear equations
if the corresponding matrix is nonsingular then
find a solution x∗ = (x∗1 , . . . , x∗n ) ,
if the constraints Ax∗ ≤ b ; x∗1 ≥ 0, . . . , x∗n ≥ 0 are fulfilled then
add x∗ to the set of basic feasible solutions S .
2. If no basic feasible solution exists (i.e. S = ∅ ) then return infeasible“ and stop.
”
3. Take x∗ from S giving maximum objective value cT x∗ .
4. If there exists an unbounded edge e of {x ∈ Rn : Ax ≤ b, x ≥ 0} which is incident
to x∗ such that for any y 6= x∗ on e we have cT y > cT x∗ then return unbounded“,
”
else return x∗ .

Corollary 1 There exists a brute-force algorithm for solving a given LP in standard form. If the
LP has n variables and
m constraints (including x ≥ 0 ) then the worst-case time complexity
of the algorithm is O( m 3
n n nmL) , where L the size of the binary encoding of A, c and b .

2 Efficient Algorithms for the Linear Programming

The simplex algorithm is the most prominent method for solving the linear programming. The
algorithm was introduced by Geogre B. Danzig in 1947 and has since then been studied exten-
sively. Although its worst case time complexity is exponential, the algorithm is often remarkably
fast in practice.
In 1979, Leonid Khachiyan introduced the first worst-case polynomial-time algorithm for
linear programming. His, so called ellipsoid method, finds the optimum solution in time O(n4 L) ,
M. Liśkiewicz, October 2021 3

where n denotes the number of variables and L the size of the binary encoding of A, c and
b . Though known implementations do not work well in practice, Khachiyan’s achievement was
to prove that the linear programming is solvable in polynomial time at all.
In 1984, Narendra Karmarkar proposed quite different algorithm for linear programming, so
called projective method. This algorithm is faster than Khachiyan’s method and works in time
O(n3.5 L) . Karmarkar’s method was then improved by other authors and currently the best
known time complexity of this method is O(n3 L) .

3 The Simplex Algorithm: an Introduction

From Theorem 1 we know that if a linear program (in standard form) has a solution, then it
must have an extreme point solution. Thus, one can now proceed as follows: find initially a
vertex of a polyhedron P describing the basic feasible solution of the given program. Then in
iterations, using a local search strategy, move along edges of the polyhedron moving to vertices
having no worse objective value until the vertex of the optimum solution is reached. The
challenge, however, is to provide some simple way of identifying extreme points. The simplex
algorithm implements the approach above using a very elegant and efficient method to identify
next extreme points.
For a given linear program in standard form:

maximize cT x
subject to Ax ≤ b (1)
x ≥ 0,

with c ∈ Rn , b ∈ Rm , and A ∈ Mm×n [R] , the algorithm does not search the polyhedron in
space Rn . Instead, using m slack variables xn+1 , . . . , xn+m , the program (1) is transformed
into slack form and the idea behind the algorithm is to search the polyhedron in Rn+m . Below
we will see how the algorithm profits from this to identify, in a very simple way, extreme points
neighbor to currently visited ones.
With each iteration there will be associated a “basic solution” that is easily obtained from
the slack form of the linear program. To find such a solution we partition variables of the linear
program in the slack form into “basic” and and “nonbasic” variables and set each nonbasic
variable to 0, and compute the value of a nonbasic variable from the equality constants.
We start with an example which shows the main idea behind the iteration of the algorithm.
Consider the following linear program in standard form:

maximize 3x1 + x2 + 2x3

subject to x1 + x2 + 3x3 ≤ 30
2x1 + 2x2 + 5x3 ≤ 24 (2)
4x1 + x2 + 2x3 ≤ 36
x1 , x2 , x3 ≥ 0

Fortunately, we can easily find a basic feasible solution for the program setting: x1 = 0, x2 =
0, x3 = 0 (the left lower vertex of the polyhedron in Fig. 1). Using the slack variables x4 , x5 , x6
4 Algorithmics: Algorithms for Linear Programming

Figure 1: The polyhedron specified by the constraints of the linear program described in (2).

for any constraint, we put the LP into slack form. Additionally, we apply a variable z to express
the objective function. Thus, we get the following four equations and six constraints:
z = 3x1 + x2 + 2x3
x4 = 30 − x1 − x2 − 3x3
x5 = 24 − 2x1 − 2x2 − 5x3 (3)
x6 = 36 − 4x1 − x2 − 2x3
x1 , x2 , x3 , x4 , x5 , x6 ≥ 0
Note that program (3) is in slack form, as defined in the previous lecture, with the only difference
that the objective function 3x1 + x2 + 2x3 is represented now by equation z = 3x1 + x2 + 2x3 .
A solution
(x1 , x2 , x3 , x4 , x5 , x6 ) = (0, 0, 0, 30, 24, 36)
fulfills the equations above (ignoring the first one) and the constraints in the last line. Therefore
this is a basic feasible solution of the program (3). Moreover, the value of z = 0 is the objective
value of the original LP (2) for (x1 , x2 , x3 ) = (0, 0, 0) .
Searching for a next vertex, in each iteration step the simplex algorithm uses the program in
slack form (3). When the vertex is founded, the program is updated. To explain how it works,
we introduce the notions of basic and nonbasic variables.
Assume x1 , . . . , xn denote the variables of the initial linear program and let xn+1 , . . . , xn+m
denote the slack variables. The simplex algorithm partitions in each iteration the set of all
variables x1 , x2 , . . . , xn+m into basic variables, which are described by set B , and the set
of nonbasic variables, described by N . The set of basic variables contains m and the set
of nonbasic variables contains n variables and in each iteration we have that the sets are
disjoint. In the algorithm B, N ⊆ {1, 2, . . . , n + m} are considered as sets of indexes of variables
x1 , x2 , . . . , xn+m . Initially, variables x1 , x2 , . . . , xn are nonbasic and xn+1 , xn+2 , . . . , xn+m are
basic. Thus, the algorithm starts with N = {1, . . . , n} and B = {n + 1, . . . , n + m} . In our
working example we have:
N = {1, 2, 3} and B = {4, 5, 6}.
M. Liśkiewicz, October 2021 5

In each iteration step, the simplex algorithm “encodes” the current state of the searching
using a program in slack form:

(F ) The set of variables x1 , . . . , xn+m is partitioned into two (disjoint) sets represented by
sets of indexes N and B . Moreover, the program contains
P
1. equation of the form: z = v + j∈N cj xj ,
P
2. for every i ∈ B , equation of the form xi = bi + j∈N ai,j xj , with bi ≥ 0 ,
3. for all i = 1, . . . , n + m , inequalities xi ≥ 0 ,
where v, cj , bj , ai,j are real constants.

Note that program (3) is in form (F ) . It is easy to see that the following fact is true.
Fact 1 If a linear program P has form (F ) then the solution (x1 , x2 , . . . , xn+m ) defined as:
(
0 if i ∈ N ,
xi =
bi if i ∈ B ,

is a feasible basic solution of P which has objective value v .

Note, that if the solution is defined as above then (x1 , . . . , xn ) is a basic feasible solution of
the original LP and it has the objective value v .
The aim of a single iteration step of the simplex algorithm is to increase, or if it is not possible,
do not decrease, the current objective function represented by z . To this aim we consider basic
variables with positive coefficients in the equation for z . We can choose arbitrary variable - in
our example let us choose x1 .
As we increase x1 , the values of x4 , x5 , x6 all decrease. To keep these variables nonnegative
we increase x1 as follows:
 
30 24 36 36
x1 := min , , = = 9.
1 2 4 4
After this operation one of the basic variables, in our example x6 , takes value 0 . We will
therefore switch the roles of x1 and x6 : We say that x6 leaves basis, therefore it is called
the leaving variable and x1 enters basis, thus, we call it the entering variable. The operation
of switching the roles of leaving and entering variables is called pivot and is performed in two
steps:

• Solve equation containing the leaving variable (in our example x6 ) for the entering variable
(in our example x1 ):
1 1 1
x1 = 9 − x2 − x3 − x6 .
4 2 4
• Rewrite the other equations replacing the entering variable ( x1 ) with the expression com-
puted in the previous step. In our example we get:
1 1 3
z = 27 + 4 x2 + 2 x3 − 4 x6
1 1 1
x1 = 9 − 4 x2 − 2 x3 − 4 x6
3 5 1
x4 = 21 − 4 x2 − 2 x3 + 4 x6 (4)
3 1
x5 = 6 − 2 x2 − 4x3 + 2 x6
x1 , x2 , x3 , x4 , x5 , x6 ≥ 0
6 Algorithmics: Algorithms for Linear Programming

It is true:
Fact 2 After performing the pivot operation we get the program of the same feasible solutions
as before.
In our example we get:

(x1 , x2 , x3 , x4 , x5 , x6 ) = (9, 0, 0, 21, 6, 0) with the objective value 27.

In Fig. 1 we can see the current vertex (9, 0, 0) . The values of slack variables x4 , x5 , x6 are
21, 6, 0 .
In the next iteration step we proceed analogously. Program (4) encodes the current situation
meaning, that N = {2, 3, 6} and B = {1, 4, 5} . Using the notions defined above we can now
summarize the next iteration step as follows:

P
1. Choose the entering variable xj : in equation z = v + j∈N cj xj take a variable xj such
that cj > 0 . In our example, let us select x3 ;
P
2. Choose the leaving variable xi : for equations xi = bi + j∈N ai,j xj , with i ∈ B , select
xi such that −abii,j = min{ −abii,j : ai,j < 0} ;
In our example we select x6 since
 
9 21 6 6 3
min , , = =
1/2 5/2 4 4 2

corresponds to this variable.

3. Perform pivot:
111 1 1 11
z = 4 + 16 x2 − 8 x5 − 16 x6
33 1 1 5
x1 = 4 − 16 x2 + 8 x5 − 16 x6
3 3 1 1
x3 = 2 − 8 x2 − 4 x5 + 8 x6
69 3 5 1
x4 = 4 + 16 x2 + 8 x5 + 16 x6
x1 , x2 , x3 , x4 , x5 , x6 ≥ 0

This finishes the second iteration step. The algorithm reaches extreme point ( 33 3 69
4 , 0, 2 , 4 , 0, 0) .
33 3 69
Again in Fig. 1, you can see point ( 4 , 0, 2 ) corresponding to ( 4 , 0, 0) of slack variables.
Before we show the third iteration, let us consider for a moment, a situation in which for a
selected entering variable xj all coefficients ai,j are positive. This means that we can increase
the value of the entering variable arbitrary, increasing at the same time the objective value. Since
the values of all basis variables increase, the constrains, that all variables remain nonnegative
is fulfilled and such a solution is feasible. This means, however, that the original program is
unbounded and the simplex algorithm returns unbounded“ in such case.
”
In the next iteration step, the only candidate to enter the base is variable x2 . The leaving
variable is x3 since it corresponds to
 
33/4 3/2
min , = 4.
1/16 3/8
M. Liśkiewicz, October 2021 7

After the pivot is finished we have:

1 1 2
z = 28 − 6 x3 − 6 x5 − 3 x6
1 1 1
x1 = 8 + 6 x3 + 6 x5 − 3 x6
8 8 1
x2 = 4 − 3 x3 − 3 x5 + 3 x6
1 1
x4 = 18 − 2 x3 + 2 x5 + 0x6
x1 , x2 , x3 , x4 , x5 , x6 ≥ 0

Finally, we cannot increase the value for z since all coefficients for nonbasic variables are nega-
tive. Thus, we can conclude that we get an optimal solution. In our example we get

(x1 , x2 , x3 , x4 , x5 , x6 ) = (8, 4, 0, 18, 0, 0)

with the value 28. This means that the solution of our linear program (2) is

(x1 , x2 , x3 ) = (8, 4, 0)

(see Fig. 1) and the maximum objective value is 28.

References

[1] W. T. Cormen, C. Leiserson, R. Rivest, C. Stein: Introduction to Algorithms, MIT Press

2009, up 3rd Edition, Chapter 29.

[2] Lecture Notes for “Advanced Algorithms” by E. Demaine and D. Karger (2003 and 2004),
courses.csail.mit.edu/6.854