Lecture 5: Simplex Method

Ming Yan

School of Data Science (SDS)

The Chinese University of Hong Kong, Shenzhen

June 15, 2022

Ming Yan MAT 3007 June 15, 2022 1 / 29

Recap

Extreme points
Basic solutions
Basic feasible solutions
The equivalency between extreme points and basic feasible solutions
(geometry vs. algebraic)
Fundamental LP theorem
Neighboring BFS

Ming Yan MAT 3007 June 15, 2022 2 / 29

Moving from One BFS to Another

We would like to search among BFS for an optimal solution.

Listing all BFS and then ranking them is impossible.
Idea: suppose we can find a BFS. Can we find a neighboring BFS
that improves the current one?

Ming Yan MAT 3007 June 15, 2022 3 / 29

Simplex Method

The simplex method proceeds from one BFS (a corner point of the feasible
region) to a neighboring one to continuously improve the value of the
objective function until reaching optimality.
Invented by George B. Dantzig
We need to define what we mean by an adjacent or neighboring
solution.

0 We need to design an efficient way to find (and move to) the

neighboring BFS with improvement (e.g., we should try to avoid
taking matrix inversions every time).
We need to design a valid stopping criterion for reaching optimality.

Ming Yan MAT 3007 June 15, 2022 4 / 29

A Neighboring BFS

Definition: Neighboring Basic Solutions

Two basic solutions are neighboring (or adjacent) if they di↵er by exactly
one basic (or non-basic) index.

8
For example, a BFS constructed by using the columns {1, 2, 4} is a
neighbor to the BFS constructed by using the columns {1, 2, 5} (but
not {1, 3, 5}).

Ming Yan MAT 3007 June 15, 2022 5 / 29

Example: Moving between BFS

Suppose we start from point A, with basic indices {3, 4, 5}, and are only
allowed to move from one BFS to its neighbor with a smaller objective
value (let’s consider minimize 6x1 4x2 )
x
B x Obj Point
12 4 (2, 4, 0, 1, 0) -28 D
12 5 (3, 3, 0, 0, 1) -30 C

→
13 5 ( 92 , 0, 32 , 0, 7) -27 B
23 4 (0, 16 2 11
3 , 3 , 3 , 0)
64
3 E
34 5 (0, 0, 6, 9, 16) 0 A

If we choose {1, 3, 5} first, then

we move to B, and then C to
• •
reach the optimum:

A = {3, 4, 5} ! B = {1, 3, 5} ! C = {1, 2, 5}

-

Ming Yan MAT 3007 June 15, 2022 6 / 29

Example: Moving between BFS

Suppose we start from point A, with basic indices {3, 4, 5}, and are only
allowed to move from one BFS to its neighbor with a smaller objective
value (let’s change the objective function to min 6x1 4x2 )
B x Obj Point
12 4 (2, 4, 0, 1, 0) -28 D
12 5 (3, 3, 0, 0, 1) -30 C
13 5 ( 92 , 0, 32 , 0, 7) -27 B
23 4 (0, 16 2 11
3 , 3 , 3 , 0)
64
3 E
34 5 (0, 0, 6, 9, 16) 0 A

We can also move to {2, 3, 4}

first, then we move to E, D, and
then C to reach the optimum:

-11--1
A = {3, 4, 5} ! E = {2, 3, 4} ! D = {1, 2, 4} ! C = {1, 2, 5}
-

Ming Yan MAT 3007 June 15, 2022 7 / 29

Simplex Method
min '
CTX
✗
We are given a BFS, whose basis is B(1), . . . , B(m).
- Sil .
A-F-b
We define
×≥ 0
AB = [AB(1) , . . . , AB(m) ], invertible
mxm

which contains m columns from the matrix A. Let AN be the matrix with
remaining columns.
By rearranging the order of variables, we can write

xB
A = [AB , AN ], x = [xB ; xN ] = ,
~
xN

By definition, we have i-ABXB-AIE.no

where xB are the basic variables, and xN are the non-basic variables.

xB = AB 1 b,
= b

xN = 0.

Ming Yan MAT 3007 June 15, 2022 8 / 29

Find a Neighboring BFS

Now, we want to find a neighboring BFS:

Finding a neighbor means changing one of the basic indices.
We want to select a non-basic variable xj to enter the basis.

I
This means, we want to increase xj from the current BFS.

We consider moving x (the current BFS) to x + ✓d, ✓ 0, where:

☒
ur
dj = 1.

-1%21=0
1

2 dj 0 = 0 for all other non-basic indices.

{ ABAN ]

What constraints do we have on d? LAB Aj ][d ] -0

We need to guarantee that the resulting step x + ✓d is still feasible,

that is: ABIB -1A;=0
A(x + ✓d) = b = Ax,
~ = dB = CAN
"Aj
.
___
-

i.e., Ad = 0.
Ming Yan MAT 3007 June 15, 2022 9 / 29
Basic Directions

Now, we write d = [dB ; dN ]. Since dj = 1 and dj 0 = 0 for all other

non-basic indices, we have
✓ ◆
⇥ ⇤ dB
0 = AB AN = AB dB + A·j .
dN

Therefore
dB = AB 1 A·j .
This means that the direction d is uniquely determined:

d = [dB ; dN ] = [ AB 1 A·j ; 0; . . . ; 1; . . . ; 0],

mm
where the 1 is at the jth entry.
We call such d the jth basic direction.

Ming Yan MAT 3007 June 15, 2022 10 / 29

Basic Directions

Moving along the jth basic direction currently only guarantees that the
equality constraint Ax = b still holds.
We also need to consider the constraint x 0. We need to verify that this
still holds (i.e., x + ✓d 0):

=
For non-basic variables i, xi was 0 and di is non-negative (either 1 or
0), therefore, xi + ✓di is still non-negative.
For basic variables, as long as they were all strictly positive, there
must exist a small ✓ such that x + ✓d 0.
Observation: ✗B
Typically, the basic variables of a BFS are all positive (i.e., m positive
entries).
There are cases that some basic variables in a BFS are equal to 0
(degeneracy), we will discuss those cases later.

Ming Yan MAT 3007 June 15, 2022 11 / 29

Example: Finding a Basic Direction

Suppose we move from A to B:

{3, 4, 5} ! {1, 3, 5}, entering
j =1 ↓
non basic
-

Increasing x1 , while x2 remains

=
at 0
Compute what happens to
x3 , x4 , x5 :
2 3 2 3 2 3 2 3 }
91,35
1
d
x3 1 0 0 1 1 • •
4A
x4 5 = 40 1 05 4 25 = 4 25
*
x5 0
in
0 1 2 2 ↓
↓ AB (90,6/9,16)

[ 1%1+4%1=1%-1%1
' ' '
78
"
2 3
◦

00 I
] "

✗ d
Ming Yan MAT 3007 June 15, 2022 12 / 29
Entering Variables and Moving Direction

In this move, the moving direction is

⇥ ⇤
1 0 1 2 2

By design

Ad = 0
= A B dB + A N dN
= AB dB + A·j

dB = AB 1 A·j , j is the entering variable

←

In the example, we select j = 1, which by appearance leads to a

better objective value
Motivation: we need to identify j that leads to improvement

Ming Yan MAT 3007 June 15, 2022 13 / 29

Change in the Objective Value

Question: how does the basic direction a↵ect our objective value?
Remember the objective function of the original LP is c> x. We can
similarly decompose c into basic and non-basic parts, corresponding to the
basic/non-basic indices, i.e.,

c = [cB ; cN ].

Now, we study what happens to the objective function, when we move

from the BFS x to x + ✓d. The change is ✓c> d.
-
-
d- [ ¥3]
If d is the basic direction corresponding to entering column j, we have:

C) 1- CÑDB c > d = cj c>

D
1
B AB A·j =: c̄j
⇐ -
=

We call c̄j the reduced cost of the variable xj .

=

Ming Yan MAT 3007 June 15, 2022 14 / 29

Reduced Costs

The reduced cost

c̄j = cj c> 1
B AB A·j

is a very important concept in the simplex method. It corresponds to the

change of the objective value if one tries to change the basis.

The first term is the cost per unit increase in the variable xj .
The second term is the cost of compensating the change in the basic
variables necessitated by the constraints Ax = b.

Ming Yan MAT 3007 June 15, 2022 15 / 29

Reduced Costs for All variables

c̄> = c> c> 1 > >

B AB A = [cB , cN ] c> 1
B AB [AB , AN ]
- -
⇐
= [c> >
B , cN ] c> 1 • >
B [I, AB AN ] = [0, cN c> 1
B AB AN ]
=
up

11
E. I . .
.
= GG
-
i -

CBTABYA # An '
- ' .

I
= CT _

gtAÉA

Ming Yan MAT 3007 June 15, 2022 16 / 29

Reduced Cost

We can calculate the reduced cost for every non-basic index j 2 N:

c̄j > 0 means incorporating j into the current basis will increase the
objective value (bad).
c̄j < 0 means incorporating j into the current basis will reduce the
objective value (good).
One rule of selecting j is to select the most negative reduced cost.
Once we select j, we call xj the “entering variable”, as j is added to
the set of basic indices.

Question: What is the reduced cost of a basic variable, i.e., j = B(i) for
some i?

c̄B(i) = cB(i) c> 1

B AB AB(i) = cB(i) c>
B ei = cB(i) cB(i) = 0.

Here ei is the vector with 1 at ith component and 0 otherwise.

Ming Yan MAT 3007 June 15, 2022 17 / 29

Reduced Cost and Optimality

Theorem: Stopping Criterion

Consider a basic feasible solution x associated with the basis
B(1), ..., B(m) and let c̄ be the corresponding vector of reduced costs. If
c̄ 0, then x* must be optimal.

Remark:
This theorem gives a stopping criterion for the simplex algorithm: We
stop when all the reduced costs are nonnegative.
It also means that if we can not find a neighbor solution that
improves the objective function, then we must have already found an
optimal solution.

Ming Yan MAT 3007 June 15, 2022 18 / 29

Example: Reduced Cost

minimize 6x1 4x2

x

Suppose we are at A, with basic

A

indices B = {3, 4, 5}, non-basic

indices N = {1, 2}:

c̄1 = c1 cB AB 1 A·1

i.
E.

= 6 [0 0 0]AB 1 A·1 = 6

c̄2 = c2 cB AB 1 A·2
= 4 [0 0 0]AB 1 A·2 = 4

Ming Yan MAT 3007 June 15, 2022 19 / 29

Example: Reduced Cost

Suppose we are at point C, with B = {1, 2, 5} and N = {3, 4}

= -

[c̄3 c̄4 ] = [c3 c4 ] cB AB 1 [A·3 A·4 ]

2 3 12 3
=
=

1 1 0 1 0
= [0 0] [ 6 4
4 0] 2 1 05 4 0 15

3→
2 3 1 0 1
2 2 3

D-
1 1 0 1 0
= [0 0] [ 6 4 0] 24 1 0 5 4 0 15
4 1 1 0 1
= [2 2]
←

Now c̄N 0 and hence the current BFS is optimal.

[
'

:
Ming Yan MAT 3007 June 15, 2022 20 / 29
Change of Basis

Observation: At a certain x, if the reduced costs satisfy c̄j 0 for all

-
=
j 2 N, then x is optimal. What if some c̄j < 0? -

It means that by bringing the jth variable (non-basic) into the basis,
we can decrease the objective value. Thus, we want to go in that
direction.

Assume d is the jth basic direction with c̄j < 0. Moving in this direction
can reduce the objective. But how far can we go?
We need to make sure that x + ✓d 0.
-

We also want to go as far as possible.

Therefore, we choose:

✓⇤ = max{✓ 0 : x + ✓d 0}.

Ming Yan MAT 3007 June 15, 2022 21 / 29

Choosing a Stepsize ✓: Min-Ratio Test

We can move along d as much as

✓⇤ = max{✓ 0 : x + ✓d 0}

without violating any constraints. We consider two cases:

1 If d 0, then ✓⇤ = 1. In this case, one can go unlimitedly far
-
without making the solution infeasible, while keeping the objective
decreasing. Therefore, the original LP is 0
unbounded.
2 If di < 0 for some i, then we can solve:
** *B
O ✓⇤ = min
x
{i:di <0} di
i
. di < ☐ IEB
some
Em
Since di 0 and xi = 0 for i 2 N, we can also write it as:
xi
✓⇤ = min .
{i2B:di <0} di
= #
Ming Yan MAT 3007 June 15, 2022 22 / 29
Example: Determining the Stepsize

Thus,
Suppose we are at A and moving xi
✓⇤ = min
di

°
{i2B:di <0}
towards B:
6 9 16 9
As we have seen before, the = min{ , , }=
1 2 2 2
moving direction is
⇥ ⇤
1 0 1 2 2

The current solution is

2 3 2 3
x3 6
xB = x4 = A B b = 9 5
4 5 1 4
x5 16
.

Ming Yan MAT 3007 June 15, 2022 23 / 29

Moving to a New Basis

Assume ✓⇤ is finite (otherwise the problem is unbounded), then we can

move to another feasible point:

÷
y = x + ✓⇤ d.

Let B(`) 2 {B(1), ..., B(m)} be one index with ✓⇤ = xB(`) /dB(`) (it is
possible for multiple indices to achieve this ! we will discuss this later).
Then we must have:

yB(`) = xB(`) + ✓⇤ dB(`) = 0.

Thus, the basic variable xB(`) has become zero, whereas the non-basic
variable xj is now positive (equal to ✓⇤ ). Meaning the basis has changed to

B(1), ..., B(` 1), j, B(` + 1), ..., B(m).

to
-

- -

We call this xB(`) as the “exiting variable”, as the new xB(`) = 0 and B(`)
-
leaves the set of basic indices.
Ming Yan MAT 3007 June 15, 2022 24 / 29
Example: Moving into a New Basis

2 3
0
607
6 7
x =6
A
667
7
495
16
We moved from point A to
point B:

xB = xA + ✓ ⇤ d
2 3 2 3 293 I

0 1 2
607 6 07 6 7
6 7 96 7 6 03 7
=667+ 6
6 7
6 17 6 7
7 = 627 x1 enters the basis, x4 exits
495 24 2 5 405 the basis.
16 2 7

↑ ↑
Ming Yan MAT 3007 June 15, 2022 25 / 29
An Iteration of Simplex Method

Initialization: We start from a BFS x (with corresponding basis B).

1 We first compute the reduced costs c̄:
c̄j = cj c> 1
B AB A·j .

• If none of the reduced costs is negative, then x is optimal.

• Otherwise choose some j such that c̄j < 0.
2 Compute the jth basic direction d = [ AB 1 A·j ; 0; ...; 1; ...; 0].
• If d 0, then the problem is unbounded, i.e., the optimal value is 1.
• Otherwise, compute ✓⇤ = min { dxii }. 70
i2B,di <0

3 Set y = x + ✓⇤ d. Then, the point y is the new BFS with index j

replacing B(`) in the basis, where B(`) is an index attaining the
minimum in ✓⇤ . The objective function value is changed by
✓⇤ c> d = ✓⇤ c̄j .
=
4 Repeat these procedures.
Ming Yan MAT 3007 June 15, 2022 26 / 29
Simplex Iteration: Discussion

The simplex iteration generates a new feasible point y that has a lower
objective value or it stops with an optimal solution (if the reduced costs
are nonnegative).
Open questions:
How do we find the starting BFS?
What would happen if xB(`) = 0 (0 in BFS x)? ! Degeneracy!
Is the new point y a=
BFS? (We only know that it is feasible).
-

Given several j with c̄j < 0, which xj should we choose to enter the
basis?
Suppose there are multiple ` with the same ✓⇤ = xB(`) /dB(`) , how
should we update the basic indices?

Ming Yan MAT 3007 June 15, 2022 27 / 29

Simplex’s Properties and Convergence

Theorem: Properties of y (Bertsimas Theorem 3.2)

Let x be a nondegenerate BFS (xB > 0) with basic indices B and let
y = x + ✓⇤ d be generated by the simplex iteration. Then, y is a basic
feasible solution associated with the basic indices
{B(1), ..., B(` 1), j, B(` + 1), ..., B(m)}.

Theorem: Convergence (Bertsimas Theorem 3.3)

Assume that the feasible set is nonempty and that every BFS is
nondegenerate. Then, the simplex method terminates after a finite
number of iterations with the following options:
1 The method stops with an BFS which is an optimal solution.
2 We have found a vector d with Ad = 0, d 0, and c> d < 0, and the
optimal value is 1.

Ming Yan MAT 3007 June 15, 2022 28 / 29

Summary

Start with a BFS

Compute reduced cost c̄N
If all reduced costs c̄N 0, optimal; else, select an entering variable xj
Compute dB ; if dB 0, unbounded
Compute ✓⇤ using min-ratio test and select the exiting variable
Create the new basis and compute the new BFS

Ming Yan MAT 3007 June 15, 2022 29 / 29