Lecture 3: Linear Optimization

Ming Yan

School of Data Science (SDS)

The Chinese University of Hong Kong, Shenzhen

June 8, 2022

Ming Yan MAT 3007 June 8, 2022 1 / 42

Linear Optimization (LP)

This lecture will cover

Definition
Standard form
Examples
Transformations to LP

Ming Yan MAT 3007 June 8, 2022 2 / 42

LP definition

A linear optimization problem, or a linear program (LP), is an

optimization problem in which the objective function and all constraint
functions are linear (in the decision variables).

Example (production planning)

maximize x1 + 2x2
x1 ,x2

subject to x1  100
2x2  200
x1 + x2  150
x1 , x2 0

Ming Yan MAT 3007 June 8, 2022 3 / 42

Reformulation using Linear Algebra

maximize x1 + 2x2
x1 ,x2

÷
subject to x1  100
2x2  200
x1 + x2  150
x1 , x2 0
Let x = [x1 , x2 ]> 2 R2⇥1 , c = [1, 2]>
= =

maximize c> x
x
subject to [1, 0]x  100
=
[0, 2]x  200
[1, 1]x  150
x1 , x2 0 × ≥o

Ming Yan MAT 3007 June 8, 2022 4 / 42

General Formulation

A LP can be generally written as

minimize / maximize c> x

x x
subject to a>
j x bj 8j 2 M1
a>
j x  bj 8j 2 M2
a>
j x = bj 8j 2 M3
xi 0 8i 2 N1
xi  0 8i 2 N2

x 2 Rn⇥1 , N1 and N2 are two disjoint subsets of {1, . . . , n}, i.e.,

N1 \ N2 = ;.
M1 , M2 , M3 are three disjoint subsets of {1, . . . ,m
n} satisfying
M1 [ M2 [ M3 = {1, . . . , m}.

Ming Yan MAT 3007 June 8, 2022 5 / 42

Compact Formulation
aix ≥ bi
ai × > bz
A LP can be generally written as I
✓ c> x
minimize / maximize E:# × ≥ [ %)
x x
subject to A1 x b1
←
A>
2 x  b2
I
9
A>
3 x = b3 ✓

:
xi 0 8i 2 N1
xi  0 8i 2 N2

A1 2 Rm1 ⇥n , b1 2 Rm1 ⇥1 , A2 2 Rm2 ⇥n , b2 2 Rm2 ⇥ ,

A3 2 Rm3 ⇥n , b3 2 Rm3 ⇥1 , m1 + m2 + m3 = m.

Ming Yan MAT 3007 June 8, 2022 6 / 42

Standard Form of LP

To study LP systematically, we use a standard (and more compact) form.

A LP is said to be of standard form if it is of the form:

minimize c> x
x
s.t. Ax = b (1)
x 0,

where x 2 Rn , a 2 Rm⇥n (m < n), and b 2 Rm .

In fact, any LP can be written in the standard form, using some “tricks”.
Remark: The definition of “standard form” may di↵er from book to book.
We use (1) as the standard form in this course (consistent with Bertsimas
and Tsitsiklis’s book).

Ming Yan MAT 3007 June 8, 2022 7 / 42

Transform to Standard Form
Max , CTX min .
- CTX

If the objective was maximization

Use c instead of c and change it to minimization

Eliminating inequality constraints Ax  b or Ax b

=D
Write it as Ax + s = b, s
0, or Ax s = b, s 0
We call s the slack variables

If one has xi  0 ×;
- •
Y; replace
Define yi = xi 30
Eliminating “free” variables xi (no constraint on xi ) Xi Xi
,
replace
Define xi = xi+ xi , with xi+ 0, xi 0 X;
In ~

Ming Yan MAT 3007 June 8, 2022 8 / 42

Example 1: Production Planning

maximize x1 + 2x2
x1 ,x2 -

subject to x1  100

:
2x2  200
x1 + x2  150
x1 , x2 0.

Standard form:
minimize x1 2x2
x1 ,x2151 52,53
,

subject to x1 +s1 = 100

2x2 +s2 = 200
x1 +x2 +s3 = 150
x1 , x2 , s1 , s 2 , s 3 0
-
- - -
-

Ming Yan MAT 3007 June 8, 2022 9 / 42

Example 2: SVM

n
X m
X
minimize i + j
i j

s.t. a> xi + b + i 1 8i = 1, . . . , n
→
>
a yj + b j  1 8j = 1, . . . , m
✗

i 0 8i = 1, . . . , n
I j 0 8j = 1, . . . , m
a, b
☆ _
free.

Define a = a+ a , b = b+ -
b , with a+ , a , b + , b
-
0.
= =
-
And add slacks to inequality constraints.

Ming Yan MAT 3007 June 8, 2022 10 / 42

Example 2: SVM Standard Form

min : 2- 8; t -29
i 's

set .
✗i'cat
-

a) + b-1-5-18; -

s; = I

*Fcat _
a- I 1-6+-5 -
6; + t
= -
l
,

at ≥o a- ≥o ¥570
b- Siro ,
si ≥ o
o; ≥o t ≥0
;

Ming Yan MAT 3007 June 8, 2022 11 / 42

Example 2: SVM Standard Form

n
X m
X
minimize i + j
a+ ,a ,b + ,b , , ,s,t
i j

s.t. x>
i a
+
x>
i a +b
+
b + i si = 1 8i
yj> a+ yj> a + b + b j + tj = 1 8j
+ +
a ,a ,b ,b 0
,s 0
,t 0.

Ming Yan MAT 3007 June 8, 2022 11 / 42

Remarks: Standard Form

Standard form is mainly used for analysis purposes. We don’t need to

write a problem in standard form unless necessary. Usually just write in a
way that is easy to understand.

However, being able to transform an LP into the standard form is an

important skill. It is helpful for analyzing LP problems as well as solving
them.

Ming Yan MAT 3007 June 8, 2022 12 / 42

 More LP examples coming!

Ming Yan MAT 3007 June 8, 2022 13 / 42

Staffing Problem

'Ñ day :

X;# the number of nurses on day I Koko has worked

×i too K days at
A hospital wants to make a weekly night shift schedule for its nurses
The demand for the night shift on day j is dj , for j = 1, . . . , 7.
Every nurse works 5 days in a row.
We want to minimize the total number of nurses used while meeting
all demand.
mm É ,
Ignore the integrality constraints for now (i.e., we allow “half” nurse if
necessary)
✗ik = ✗ it , k -11 k= 0, I 2,3 ,

E- ✗in ≥ di

Ming Yan MAT 3007 June 8, 2022 14 / 42

Staffing Problem

¥.
What is your choice of decision variable? How about letting xi be the

number of nurses on day i?

Can’t capture the constraints that nurses have to work 5 days in a row

E-
A better way is to define xi to be the number of nurses starting at day i.
Our objective will be

minimize x 1 + x 2 + x3 + x4 + x5 + x6 + x7
x2R7

Ming Yan MAT 3007 June 8, 2022 15 / 42

M X,
-1 Xx -1×5-1×6 1- ×> ≥d ,

T ×ztX , -1 Xg -1×6-1×7 ≥dz

W X XI Xz
} 46
Xy
'

Th Xx X X} I
, X,
Xy
F ✗5
Xi Xi X} ✗4 "

sa ✗6 Xi
"
× } ✗
¢ Xy
Su
Xy X}
Xp Xs ✗6
Staffing Problem: LP

Ming Yan MAT 3007 June 8, 2022 16 / 42

Staffing Problem: LP

The LP form is
minimize x1 +x2 +x3 +x4 +x5 +x6 +x7
x
subject to x1 +x4 +x5 +x6 +x7 d1

[
"
x1 +x2 +x5 +x6 +x7 d2
x1 +x2 +x3 +x6 +x7 d3
x1 +x2 +x3 +x4 +x7 d4
x1 +x2 +x3 +x4 +x5 d5
x2 +x3 +x4 +x5 +x6 d6
x3 +x4 +x5 +x6 +x7 d7
x1 , x2 , x3 , x4 , x5 , x 6 , x7 0

Ming Yan MAT 3007 June 8, 2022 16 / 42

Staffing Problem: LP in the Matrix form

minimize c> x
x
subject to Ax d, x 0

2 3 2 3 2 3 2 3
x1 1 1 0 0 1 1 1 1 d1
6 x2 7 6 1 7 6 1 1 0 0 1 1 1 7 6 d2 7
6 7 6 7 6 7 6 7
6 x3 7 6 1 7 6 1 1 1 0 0 1 1 7 6 d3 7
6 7 6 7 6 7 6 7
x=6
6 x4 7c = 6
7 6 1 7
7 A=6
6 1 1 1 1 0 0 1 7d = 6
7 6 d4 7
7
6 x5 7 6 1 7 6 1 1 1 1 1 0 0 7 6 d5 7
6 7 6 7 6 7 6 7
4 x6 5 4 1 5 4 0 1 1 1 1 1 0 5 4 d6 5
x7 1 0 0 1 1 1 1 1 d7

Ming Yan MAT 3007 June 8, 2022 17 / 42

Maximin Problems: Air Traffic Control Problem

An air traffic controller needs to control the landing time of0

n aircrafts
Flights must land in the order 1, . . . , n.

.-_
Flight j must land in time interval [aj , bj ].
The objective is to maximize the minimum separation time, which is
the interval between two landings.

tj+1 tj

tj titi
aj '
bj aj+1 bj+1
-

Ming Yan MAT 3007 June 8, 2022 18 / 42

Air Traffic Control Problem: Formulation

Decision variables
tj : the landing time of the flight j
Formulation

maximize min {tj+1 tj }

t j=1,...,n 1

subject to aj  tj  bj j = 1, . . . , n
tj  tj+1 j = 1, . . . , n 1.

The objective function is not linear. We call it a maximin objective.

Ming Yan MAT 3007 June 8, 2022 19 / 42

Air Traffic Control Problem: LP Reformulation?

We introduce to express the minimum separation time:

⇐
min {tj+1 tj }.
j=1,...,n 1

It can be formulated as
max{ :  tj+1 tj 8j = 1, . . . , n 1}
At optimal, must equal the minimal separation (since we try to
maximize ).
I
m¥
.

sit ,
£ -4-11 -
t's
V-j

Ming Yan MAT 3007 June 8, 2022 20 / 42

Air Traffic Control Problem: LP Reformulation?

We introduce to express the minimum separation time:

min {tj+1 tj }.
j=1,...,n 1

It can be formulated as
max{ :  tj+1 tj 8j = 1, . . . , n 1}
At optimal, must equal the minimal separation (since we try to
maximize ).
Formulation
maximize
t,
to

subject to tj+1 tj 8j = 1, . . . , n 1
a j  tj  b j j = 1, . . . , n
tj  tj+1 j 2 J = 1, . . . , n 1.
Ming Yan MAT 3007 June 8, 2022 20 / 42
Remarks

We change min {tj+1 tj } into a max expression.

j=1,...,n 1
Similarly, max{x, y } can be represented as
-

min{z : z x, z y}
.

This technique can be applied to deal with minimax problems.

Ming Yan MAT 3007 June 8, 2022 21 / 42

Minimize Absolute Values: Constrained `1 Regression •
(4
7)
"

:÷
☆

43.4,
Data (x, y ): (1,2), (3,4), (4,7)
x represents temperature (on some scale)
y represents number of Blizzards sold by Dairy Queen (in hundreds)
We want to use the independent variable (x) to predict the dependent
response (y ) using a linear (or rather affine) model, y = ax + b.
Our goal is to find values of a and b that give the “best” fit of the model
to the data.
Minimizing one-norm (`1 ) error: |y (ax + b)|
/
& ,

Prior knowledge: higher temperature leads to higher ice cream

consumption ! a 0

Ming Yan MAT 3007 June 8, 2022 22 / 42

Constrained One-norm Regression: Intuitive Attempt

Decision variables
a slope 8
min ,

b intercept si
s, ≥ latb -2
Formulation ¢ 8, ≥ -

Clatb -27

minimize |1 · a + b 2| + |3 · a + b 4| + |4 · a + b 7|
a,b
we - ur
s.t. a 0, b unrestricted.

1×1 = max { ×, -
✗ ls

Ming Yan MAT 3007 June 8, 2022 23 / 42

Constrained One-norm Regression: LP Reformulation

What is |y (ax + b)|?

|y (ax + b)| = max{y (ax + b), y + (ax + b)}

Use what we have learned in formulating a minimax problem:

minimize e1 + e2 + e3
a,b,e1 ,e2 ,e3 '

s.t. e1 a+b 2
e1 (a + b 2)
}
e2 3a + b 4
e2 (3a + b 4) }
e3 4a + b 7
e3 (4a + b 7) }
a 0

Ming Yan MAT 3007 June 8, 2022 24 / 42

Remarks

An absolute value function:

Similar ideas can be applied for constraints like |a> x + b|  c

•*-
atxtb ≤ c
-

( ✗+ b) ≤c

Ming Yan MAT 3007 June 8, 2022 25 / 42

An Alternative Way to Model Absolute Value Functions

|e1 | = min{e1+ + e1 : e1 = e1+ e1 , e1+ 0, e1 0}

" ^
"
"
-

Lf

3
X
minimize ei+ + ei
a,b,e+ ,e
i=1
s.t. e1+ e1 =a+b 2
e2+ e2 = 3a + b 4
e3+ e3 = 4a + b 7
a, e1+ , e1 , e2+ , e2 , e3+ , e3 0.

Ming Yan MAT 3007 June 8, 2022 26 / 42

Piecewise Linear Convex Function: Energy Management

Given a forecast of the demand for each time period of a particular

day, an energy operator needs to coordinate the energy production
(generation) of a set of generators for each time period
The cost of generating x megawatts (mw) of energy is represented by
-

c(x). A typical cost function usually has an increasing marginal cost

-

~
Piecewise linear function: each cost segment as aj and the marginal
@
cost on each segment as mj maxhm.IT ,
Mi- •?
,

Max -

? }
!

Ming Yan
¥ MAT 3007 June 8, 2022 27 / 42
 Energy Management

We have two generators to satisfy a total energy demand of 20mw.0

Generator 1 and 2 have capacity 16mw and 12mw, respectively. Their
generation cost function can be described as:
generator parameter 1 2 3
a (mw) 8 12 16
1 - -
-

m ($100,000/mw) 2 5 -
10
-

a (mw) 6 8 12
2
m ($100,000/mw) 1 2 -
5-

Goal: determine the amount of generation from each generator in

order to minimize the generation cost while satisfying the demand.

Ming Yan MAT 3007 June 8, 2022 28 / 42

?⃝
Energy Management: LP Reformulation 1

minimize max{f11 (x1 ), f21 (x1 ), f31 (x1 )}+

Motivation: reformulate the x

nonlinear objective function max{f12 (x1 ), f22 (x1 ), f32 (x1 )}+
s.t. x1 + x2 = 20
Piecewise linear convex function
0  x1  16
! maximize over a group of
0  x2  12
linear functions (equivalence)
minimize f1 + f2
x,f

s.t. x1 + x2 = 20
f1 2x1

1-
f1 5x1 24
f1 10x1 84
f2 1x2
f2 2x2 6

E- f2 5x2
0  x1  16
30

0  x2  12

Ming Yan MAT 3007 June 8, 2022 29 / 42

Energy Management: LP Reformulation 2

Another way is to directly model variables based on the physical meaning.

Decision variables
xij is the amount of energy generated by generator i = 1, 2 with cost
on segment j = 1, 2, 3
Formulation

minimize 2x11 + 5x12 + 10x13 + x21 + 2x22 + 5x23

µ§
x ⇐ ← =
s.t. x11 + x12 + x13 + x21 + x22 + x23 = 20

µµ,,µ ,
#
0  x11  8 mi -15×12-110×13

2✗
✗11×12×1
0  x12  4 }

f. 0 ≤ Xi , ≤ 8
0  x13  4 5.

0 ≤ ×, , ≤ ×
0  x21  6
◦ ≤ ✗↳ ≤4
0  x22  2
0  x23  4.

Ming Yan MAT 3007 June 8, 2022 30 / 42

?⃝
Remarks: Why Would It Work?

Why the following solution will not happen?

x11 = 8, x12 = 0, x13 = 4, x21 = 6, x22 = 2, x23 = 0.

Because the cost function is “convex” (increasing marginal cost), it is

optimal to “fill” the cheaper segment before generate from more
expensive ones.
What would happen if the marginal cost decreases when generation
grows, which can be common in representing volume discount?

Ming Yan MAT 3007 June 8, 2022 31 / 42

 More Modeling: Fractional Relationship

Ming Yan MAT 3007 June 8, 2022 32 / 42

Fractional Relationship

Suppose we have a problem:

c> x + d
minimize
x e> x + f
subject to Ax  b.

O O
We assume that e> x + f > 0 for any x satisfying Ax  b.
When do we see this problem?
Production Planning: the objective of a company might be based on
maximizing the ratio: (total profit)/(total production costs).

Ming Yan MAT 3007 June 8, 2022 33 / 42

Linear Fractional Programming

00
Define:
x 1
y= , z= .
e> x +f e> x +f

A- ✗ ≤ b
Ay ≤ 2b

; =
EY 1- 2d

min .
CTL tzd
ÉÉ Ay
.

≤ 2- b
Ey -125 =
I

Ming Yan MAT 3007 June 8, 2022 34 / 42

Linear Fractional Programming

Define:
x 1
y= , z= .
e> x+f e> x+f

Objective. With the definition, we have

c> x + d c> x d ·1
>
= >
+ > = c> y + dz
e x+f e x+f e x+f
Constraints:

Ax  b and e> x + f > 0 () Ay  zb and z > 0

Connection:

e> y + zf = 1

Ming Yan MAT 3007 June 8, 2022 34 / 42

Linear Fractional Programming

We can rewrite our problem as:

minimize c> y + dz
y,z

subject to Ay zb  0
>
e y + fz = 1
z 0

This is an LP!
See Boyd and Vandenberghe for details on the equivalency
(supplemental reading: P151).

Ming Yan MAT 3007 June 8, 2022 35 / 42

Min-Cost Network Flow (MCNF) Problems

The network optimization models we will study are all special cases of
linear programming models.
It is important to be able to formulate network optimization models, when
appropriate because:
network representations make optimization models easier to visualize,
understand, and explain; and,
very efficient network optimization algorithms are available, enabling
us to solve much larger model instances.
If you are interested, check Bertsimas and Tsitsiklis Chapter 7 for more
information.

Ming Yan MAT 3007 June 8, 2022 36 / 42

Min-Cost Network Flow: Problem Descriptions

Let see the following supply chain example for motivation:

Warehouses are in Fuzhou, Lanzhou, and Guiyang.
Customers are in Beijing, Xi’an, Changsha, Zhengzhou, and Kunming.
Each warehouse i has a fixed supply si and each customer j has a
fixed demand dj .
Shipping is only allowed on a specific set of arcs in the network,
limited to 200 units.
Shipping incurs a per unit transportation cost cij between two
locations i and j
We would like to determine an optimal shipping plan, which minimizes
transportation costs.

Ming Yan MAT 3007 June 8, 2022 37 / 42

Min-Cost Network Flow

-
1000

0
-

÷
±

[supply] or [-demand] on each node;

-
=
(shipping cost per unit, lower bound, upper bound) on each arc; (can
-

be abbreviated based on specific problems)

Ming Yan MAT 3007 June 8, 2022 38 / 42

Min-Cost Network Flow: Describing Data

The data for an arc:

arc name origin node termination node lower bound upper bound cost
xij i j `ij uij cij

The data for a node:

node name supply
i si or di
Note that it is our convention to use a supply-oriented system, which
means that supplies are positive and demands are negative supplies.

Ming Yan MAT 3007 June 8, 2022 39 / 42

MCNF: Decisions, Constraints, Objective Function

Decisions: how many units of products we should flow on arc (i, j)

The constraints of the network flow problem are:
conservation of flow at each node:
flow in = flow out, where “flow in” includes external supply and flow
from other nodes and “flow out” includes demand and flow to other
nodes;
flows on each arc must obey the arc’s simple bounds;
Each arc has a per unit cost and the objective is to minimize total
cost.
Network flow models are special types of linear programs.

Ming Yan MAT 3007 June 8, 2022 40 / 42

Min-Cost Network Flow: LP Formulation

minimize 3x21 + 6x24 + 3x25 + 7x26 + 7x31 + · · · + 2x86 + 7x87

xij - -
-
Yip,
- - - -

=]
-

s.t. x21 x31 x51 + it, 200 (2)

=
x21 + x24 + x25 + x26 ←
= 700 (3)
x31 + x34 + x36 + x37 ≤
= 200 (4)
x24 x34 x54 x64 -1k = =
200 (5)
x51 + x54 + x56 + x57 x25 x65 x85 = 300 (6)
..
+ %
. (7)
x85 + x86 + x87 ≤
= 200 (8)
0  xij  200 8 arcs (i, j). (9)

The decision variables are the flows, xij . Constraints (2)-(8) are flow
balance constraints for each of the eight nodes.

Ming Yan MAT 3007 June 8, 2022 41 / 42

Summary

Recap of SVM:
Auxiliary variables
Orientation of the objective function pushing
Correct equivalence
Maximin/minimax
Absolute value
Piecewise linear convex/concave functions

Ming Yan MAT 3007 June 8, 2022 42 / 42