Lecture 4: The Geometry of Linear Optimization

Ming Yan

School of Data Science (SDS)

The Chinese University of Hong Kong, Shenzhen

June 13, 2022

Ming Yan MAT 3007 June 13, 2022 1 / 43

Recap

Up to now, we have learned how to

formulate linear optimization problems.
transform an LP into the standard form.
Advantages of LP:
Versatile: LP can model many real problems.
Easy to solve: Commercial software can solve LPs with tens of
thousands of variables very easily. It is the easiest one among all
optimization problems.
Fundamental: The LP theories lay the foundation for most
optimization theories.

Ming Yan MAT 3007 June 13, 2022 2 / 43

What’s Next?

some basic structural properties for linear optimization

simplex method for solving linear optimization
linear program duality theory (sensitivity analysis)
a brief introduction of interior point method

Ming Yan MAT 3007 June 13, 2022 3 / 43

 2D Simple LP Examples

Ming Yan MAT 3007 June 13, 2022 4 / 43

A Simple LP Example

It is very helpful to study a small LP from a graphical point of view.

Recall the production problem:

maximize x1 + 2x2
x1 ,x2

subject to x1  100
2x2  200
x1 + x2  150
x1 , x2 0.

There are only two variables. ! Could we plot this problem on a 2D

coordinate system?

Ming Yan MAT 3007 June 13, 2022 5 / 43

Represent LP Using Graphs

We first draw the feasible region.

Ming Yan MAT 3007

±
Decision variables: only on 2 dimensional space
Constraints: forms a polygon feasible region (polyhedron)

June 13, 2022 6 / 43

 Some Definitions: Polyhedron

Polyhedron
A polyhedron is a set that can be written in the form:

{x 2 Rn : Ax b},

where A 2 Rm⇥n and b 2 Rm .

Recall that in the standard form of LP, the feasible set is

Ax = b, x 0.

Is this a polyhedron? Why?

AX ≤ b
I×=✗
Ax≥b < -

AXE b -
× ≥,
identity matrix
Ming Yan MAT 3007 June 13, 2022 7 / 43
Some Definitions: Polyhedron

Polyhedron
A polyhedron is a set that can be written in the form:

{x 2 Rn : Ax b},

where A 2 Rm⇥n and b 2 Rm .

Recall that in the standard form of LP, the feasible set is

Ax = b, x 0.

Is this a polyhedron? Why?

Yes, we can write it as Ax b, Ax  b, Ix 0, where I is the
identity matrix.

Ming Yan MAT 3007 June 13, 2022 7 / 43

Represent LP Using Graphs

:
We draw the line for x1 + 2x2 = c for di↵erent values of c.

Objective function:
::¥ .

All points on one line has the same objective value

We call them iso-profit lines: “iso-” means “single”, “the same”
Ming Yan MAT 3007 June 13, 2022 8 / 43
Optimizing: Moving Iso-profit Lines

We want to maximize our objective function:

/
•

The optimal solution is the highest one among these lines that
touches the feasible region.
Optimal solution: (50, 100). Optimal objective value: 250.

Ming Yan MAT 3007 June 13, 2022 9 / 43

Solving LPs with Two Decision Variables

An LP with two decision variables can be solved in the following way:

1 Plot each constraint as an equation, and then decide which side of
the line is feasible (for an inequality).
2 Find the feasible region.
3 Plot two iso-profit (or iso-cost) lines.
4 Imagine sliding the iso-profit line in the improving direction. The last
point touched in the feasible region is optimal.

Ming Yan MAT 3007 June 13, 2022 10 / 43

Exampling: Solving Another Simple LP

maximize z = 6x1 + 4x2

x1 ,x2
-
subject to x1 + x2  6 (1)

¥%%
2x1 + x2  9 (2)
2x1 + 3x2  16 (3) I

x1 , x2 0.

Feasible region?
Iso-profit lines? 0
Improving direction?
Optimal solution: A or B?

Ming Yan MAT 3007 June 13, 2022 11 / 43

Summary

Geometric representation of linear programs:

Constraints ! polyhedron feasible region
Objective function ! iso-profit line
Max/min ! moving direction

Ming Yan MAT 3007 June 13, 2022 12 / 43

 General LP problems: Extreme Points

Ming Yan MAT 3007 June 13, 2022 13 / 43

Some Observations from 2D Examples

-
ai 0

The feasible region of an LP is a polyhedron.

The optimal solution tends to be a corner of the feasible region.
Some constraints are active at the optimal solution (x2 = 100,
-
x1 + x2 = 150), some are not (x1 < 100).
- -

Ming Yan MAT 3007 June 13, 2022 14 / 43

The High-level Picture for LP Algorithms

The optimal solution to an LP seems to be at the corner of the feasible

sets (we will prove it). Then questions arise:
How to find a corner point (given only the constraints)?
How can we search for the best corner?
We will study algorithms for solving LPs that can:
Guarantee to find the optimal solution.
Run within a certain (reasonable) amount of time.
We introduce extreme points first.

Ming Yan MAT 3007 June 13, 2022 15 / 43

Corner Points: Extreme Points

In an LP, the optimal solution tends to be in one of the corners of the

feasible region. We first formalize this notion.

Definition: Extreme Point

Let P be a polyhedron. A point x 2 P is said to be an extreme point of P

ar t
if we can not find two vectors y, z 2 P with y, z 6= x•* and a scalar
2 [0, 1], such that x = me
y + (1 )z.
_

That is, x cannot be represented as a convex combination of other

points in P (see next slide).
We sometimes call the extreme point the vertex or corner of the
*
polyhedron.
¥¥
✗ .

Ming Yan MAT 3007 June 13, 2022 16 / 43

Convex Combination: Visualization

Convex Combination
Pn any x1 , . . . , xn and 1 , . . . , n 0 satisfying
For 1 +···+ n = 1, we call
@ i=1 i xi a convex combination of x1 , . . . , xn .

÷ % .

Ming Yan MAT 3007 June 13, 2022 17 / 43

Example: Extreme Points

150

100 0 d-

@
50
x2

0 6 •

50

50 0 50 100 150 200

x1
How many extreme points are there in this feasible region?

Ming Yan MAT 3007 June 13, 2022 18 / 43

Example: Extreme Points

150

100

50
x2

50

50 0 50 100 150 200

x1
How many extreme points are there in this feasible region?
Answer: 5

Ming Yan MAT 3007 June 13, 2022 19 / 43

More on Extreme Points

We just introduced extreme points/vertices.

However, this does not tell us how to find those points. We want to have
a good way for finding extreme points.

Preview of the Next Steps:

We need an algebraic way to represent extreme points in high
dimensional spaces.
We will show that it is sufficient to look at extreme points to solve a
linear optimization problem.
Finally, finding the optimal extreme point will lead to the construction
of the simplex algorithm for solving LPs.

Ming Yan MAT 3007 June 13, 2022 20 / 43

Extreme Points: Intersection of Constraints

maximize z = 6x1 + 4x2

x1 ,x2

:
subject to x1 + x2  6 (1)
2x1 + x2  9 (2)
2x1 + 3x2  16 (3)
x1 0 (4)
x2 0. (5)
a

In this 2-D plot, we can see that all

corner points (A-E) are at the
intersection where two constraints co *
are binding.

Ming Yan MAT 3007 June 13, 2022 21 / 43

Extreme Points: Intersection of Constraints

Optimum is achieved at a corner point.

A corner point
✓ ◆is defined by the intersection of two constraints:
5
Maximum = 10 corner points in total.
2
We could enumerate all of them to check if they are feasible/optimal.

Tight/binding constraints Points Objective

(1) & (2) C = (3, 3) 30 ✓

E-
(1) & (3) D = (2, 4) 28
(1) & (4) a- (0,6) infeasible: violates (3)

I
(1) & (5) (6,0) infeasible: violates (2)
(2) & (3) ( 11 7
4 , 2) infeasible: violates (1)
(2) & (4) (0,9) infeasible: violates (1)
(2) & (5) B = ( 92 , 0) 27
(3) & (4) E = (0, 16 3 ) 21 13
(3) & (5) (8,0) infeasible: violates (1)
(4) & (5) A = (0, 0) 0
-

Ming Yan MAT 3007 June 13, 2022 22 / 43

Algebraic Representation of Extreme Points

If we rewrite the LP in its standard

form, we get:

minimize
x1 ,x2 ,x3 ,x4 ,x5
6x1 4x2 '

\
⊖
:O
{
subject to x1 + x2 + x3 = 6
e

2x1 + x2 + x4 = 9 f

0
-

2x1 + 3x2 + x5 = 16
.
. .
.

.
.

x 1 , x2 , x3 , x4 , x5 0. ÷ >

Ming Yan MAT 3007 June 13, 2022 23 / 43

Another Algebraic Representation of Extreme Points

At each corner point we have

two variables equal to 0.
Select three linearly independent
columns to “use”.
2 3

¥
2 3 x1 2 3
1 1 1 0 0 6 7
6 x2 7 6
4 2 1 0 1 05 6 x 3 7 = 4 9 5
6 7
2 3 0 0 1 4 x4 5 16
x5

TE
Ming Yan MAT 3007 June 13, 2022 24 / 43
They Are Equivalent!
✓ ◆
5
We can yield = 10 possibilities in total as well.
3

x vars
x
to use
123 ( 11 7
4 , 2,
1
4 , 0, 0)
1=
24 (2, 4, 0, 1, 0)
125 (3, 3, 0, 0, 1)
134 (8, 0, 2, 7, 0)
135 ( 92 , 0, 32 , 0, 7)
145 (6, 0, 0, 3, 4)
234 (0, 16 2 11
3 , 3 , 3 , 0)
235 (0, 9, 3, 0, 11)
245 (0, 6, 0, 3, 2)
345 (0, 0, 6, 9, 16)

So, from a standard form of LP, we can select columns to enumerate all
corner points.

Ming Yan MAT 3007 June 13, 2022 25 / 43

 Basic Solutions

Ming Yan MAT 3007 June 13, 2022 26 / 43

Finding Extreme Points: Basic Solutions

In the following, we consider an LP in its standard form:

minimize c> x
x
subject to Ax = b
x 0,

where x 2 Rn , A is an m ⇥ n matrix (m < n) and b 2 Rm .

General Assumption:
A has linearly independent rows (or equivalently A has full rank D
m).

What happens if this condition is not satisfied?

Then either there is a redundant constraints (in which case one can
remove it) or the constraints are not consistent (in which case there is
no feasible point).
Ming Yan MAT 3007 June 13, 2022 27 / 43
Basic Solutions

Now, we study the extreme points of an LP in its algebraic form.

Definition: Basic Solution

We call x a basic solution of the LP if and only if
1 Ax = b.
2 There exist indices B(1), . . . , B(m) such that the columns of
7

2 3
-

| | |
4 AB(1) AB(2) · · · AB(m) 5 = AB ×B@
| | |

are linearly independent and xi = 0 for i 6= B(1), ..., B(m). Xp = 0

- ___
_

MXM
KÉ MM
= A✗ = ABXB + Anke
MM
Ming Yan MAT 3007 June 13, 2022 28 / 43
Finding a Basic Solution

Procedure to find a Basic Solution:

1 Choose any m independent columns of A: AB(1) , ..., AB(m) .
2 Let xi = 0 for all i 6= B(1), ..., B(m).
3 Solve the equation Ax = b for the remaining xB(1) , ..., xB(m) .
=D
xB = AB 1 b.
Remarks:
Since AB(1) , ..., AB(m) are linearly independent, the last step must
produce a unique solution.
Basic solution of an LP only depends on its constraints, it has nothing
to do with the objective function.

Ming Yan MAT 3007 June 13, 2022 29 / 43

Questions about Basic Solutions

How many non-zeros could one have in a basic solution (assuming there
are m constraints)?
No more than m!•
Could be anything between 0 to m (typically it is m).

How many basic solutions can one have for a linear program with m
constraints and n variables?
✓ ◆
n
At most = m!(nn! m)! (Combination number).
m
-
Therefore for a finite number of linear constraints, there can only be a
finite number of basic solutions!

Ming Yan MAT 3007 June 13, 2022 30 / 43

Example: Basic Solutions

For the previous LP example, we obtain the basic solutions:

B = {B(1), . . . B(m)} x Feasible solution?
-123 ( 11 7
4 , 2,
1
4 , 0, 0) No, x3 < 0

±
124 (2, 4, 0, 1, 0) Yes
'
125 (3, 3, 0, 0, 1) Yes
134 (8, 0, 2, 7, 0) No, x3 , x4 < 0
135 ( 92 , 0, 32 , 0, 7) Yes
145 (6, 0, 0, 3, 4) No, x4 < 0
234 (0, 16 2 11
3 , 3 , 3 , 0) Yes
235 (0, 9, 3, 0, 11) No, x3 , x5 < 0
245 (0, 6, 0, 3, 2) No, x5 < 0
345 (0, 0, 6, 9, 16) Yes

Obviously, not all basic solutions are feasible. They can violate the
nonnegativity constraints.

Ming Yan MAT 3007 June 13, 2022 31 / 43

Basic Feasible Solutions

Definition: Basic Feasible Solution

If a basic solution x also satisfies that x 0, then we call it a basic
feasible solution (BFS).

How to Find a BFS?

First find a basic solution x.
Check if x 0.

Theorem: Extreme Points and BFS

For the standard LP polyhedron P := {x 2 Rn : Ax = b, x 0}, the
following statements are equivalent:
1 x is an extreme point of P.
2 =
x is a basic feasible solution.

Ming Yan MAT 3007 June 13, 2022 32 / 43

Proof Sketch (1 ! 2

1 ! 2: contradiction from linear dependent columns.

→ Noe 1
=) is not an extreme pt
it ✗ is infeasible ✗ .

BS
it ✗ is Not
ftp.a Apu,
,
. .

ftp.cm ,
pot linearly independent

"
ME
90 7- AXIOS -1 .
.
ABAX =D
we can extend AX such that IÑX)B=A✗
( ¥x7N=0

Act ÑXI =
ABCXB -1 AX )≈ABXb=b
since ✗ [ Bci
)) -1-0

Ming Yan MAT 3007 June 13, 2022 33 / 43

Proof Sketch (2 ! 1)

2 ! 1: contradiction.

Ming Yan MAT 3007 June 13, 2022 34 / 43

Example: Basic Feasible Solutions

We only select the feasible solution from the table to obtain BFS:

B x Feasible? Point
12 4 (2, 4, 0, 1, 0) Yes D
12 5 (3, 3, 0, 0, 1) Yes C
13 5 ( 92 , 0, 32 , 0, 7) Yes B
23 4 (0, 16 2 11
3 , 3 , 3 , 0) Yes E
34 5 (0, 0, 6, 9, 16) Yes A

Exercise: obtain the BFS for the production planning problem following
the same procedure.

Ming Yan MAT 3007 June 13, 2022 35 / 43

Fundamental LP Theorem

Why do we care about basic feasible solutions?

Fundamental LP Theorem
Consider a linear problem in standard form and assume that A has full row
rank m.
1 If the feasible set is nonempty, there is a basic feasible solution.
2 If there is an optimal solution, there is an optimal solution that is also
a basic feasible solution.

! In order to find an optimal solution, we only need to look among

basic feasible solutions!

Ming Yan MAT 3007 June 13, 2022 36 / 43

Proof Sketch (by Construction)

Ming Yan MAT 3007 June 13, 2022 37 / 43

From Basic Feasible Solutions to Optimal Solutions

Consequences:
We only need to search among basic feasible solutions to find the
optimal solution!

How can we search among the basic feasible solutions?

One may suggest to list all the basic feasible solutions and compare
their objective values.
! There might be too many BFS!

For a linear optimization with m constraints and n variables, how

many basic feasible solutions can we have?
✓ ◆
n
! ... if n = 1000, m = 100, then this number is 10143 ...
m

Ming Yan MAT 3007 June 13, 2022 38 / 43

Searching Optimum among BFS

Suppose we can get a BFS, e.g., point A. How should we proceed?

*←É •

Ming Yan MAT 3007 June 13, 2022 39 / 43

Neighboring BFS

Definition: Neighboring Basic Solutions

Two basic solutions are neighboring (or adjacent) if they di↵er by exactly
one basic (or non-basic) index.

not {1, 3, 5}).

.
.
€
For example, a BFS constructed by using the columns {1, 2, 4} is a
neighbor to the BFS constructed by using the columns {1, 2, 5} (but
^

D ({1, 2, 4}) is the neighbor of C ({1, 2, 5}) but not B ({1, 3, 5})

Ming Yan MAT 3007 June 13, 2022 40 / 43

 Neighboring BFS

Ming Yan MAT 3007 June 13, 2022 41 / 43

Checking Neighbors

The main question is how to find a neighboring BFS that improves the
current one (reduces the objective function):
We only need to show how to do it for one step. Then, we can simply
iterate the same methods (until we reach optimal).

A naive way is simply to check all neighbors of the current BFS:

There are ←m(n m) potential neighbors for a BFS (choosing one
index to leave, choosing one to enter). For each one, we need to solve
a linear equation of m variables.
This works, but is likely to be very slow.

We want a more efficient way to do it.

Ming Yan MAT 3007 June 13, 2022 42 / 43

Summary

Geometric representation of linear programs:

2D problems.
Extreme points:
intersection of constraints
independent linear columns
Basic solutions
Basic feasible solutions
Neighboring BFS and searching among them

Ming Yan MAT 3007 June 13, 2022 43 / 43