FUNDAMENTALS OF

OPTIMIZATION
Linear Programming
CONTENT
• Linear Programs
• Geometric approach
• Simplex method
Linear Programs

Standard form
f(x) = c1x1 + c2x2 + . . . + cnxn → max
a1,1x1 + a1,2x2 + . . . + a1,nxn ≤ b1
a2,1x1 + a2,2x2 + . . . + a2,nxn ≤ b2
...
am,1x1 + am,2x2 + . . . + am,nxn ≤ bm
x1, x2, . . . xn R, x1, x2, . . . xn  0
Linear Programs
• Convert general linear program forms into standard form
• f(x) → min  -f(x) → max
• g(x)  b  -g(x) ≤ -b
• A = B  (A ≤ B) and (A  B)
• A variable xj  R can be represented by xj = 𝑥𝑗+ - 𝑥𝑗− where 𝑥𝑗+ , 𝑥𝑗−  0
Linear Programs
• Example: Convert general linear program forms into standard form

f(x1, x2) = 3x1 + 2x2 → min

2x1 + x2 ≤ 7
x1 + 2x2 = 8
x1 - x2  2
x1, x2  R, x2  0
Linear Programs
• Example: Convert general linear program forms into standard form
• Represent: x1 = 𝑥1+ - 𝑥1−

f(𝑥1+ , 𝑥1− , x2) = -3 𝑥1+ + 3𝑥1− - 2x2 → max

2 𝑥1+ - 2𝑥1− + x2 ≤ 7
𝑥1+ - 𝑥1− + 2x2 ≤ 8
- 𝑥1+ + 𝑥1− - 2x2 ≤ -8
−𝑥1+ + 𝑥1− + x2 ≤ 2
𝑥1+ , 𝑥1− , x2  R, 𝑥1+ , 𝑥1− , x2  0
Geometric approach
• Constraints (inequalities) form a feasible region
• Optimal points will be one of the corners of the feasible region
Geometric approach

f(x1, x2) = 3x1 + 2x2 → max

2x1 + x2 ≤ 7
x1 + 2x2 ≤ 8
x1 - x2 ≤ 2
x1, x2  0
Geometric approach

2x1 + x2 ≤ 7
x1 + 2x2 ≤ 8
x1 - x2 ≤ 2
x1, x2  0
Geometric approach

2x1 + x2 ≤ 7
x1 + 2x2 ≤ 8
x1 - x2 ≤ 2
x1, x2  0
Geometric approach
• Special cases
• Problem does not have optimal solutions
• Problem does not have feasible solutions

f(x1, x2) = 3x1 + 2x2 → max f(x1, x2) = 3x1 + 2x2 → max
-2x1 - x2 ≤ -7 2x1 + x2 ≤ 7
x1 - x2 ≤ 2 -4x1 - 2x2 ≤ -16
x1, x2  R, x1, x2  0 x1, x2  R, x1, x2  0
Example
• A company must decide to make a plan to produce 2 products P1, P2.
• The revenue received when selling 1 unit of P1 and P2 are
respectively 5$ and 7$
• The manufacturing cost when producing P1 and P2 are
respectively 5$ and 3$
• The storage cost in warehouses for 1 unit of P1 and P2 are
respectively 2$ and 3$
• Compute the production plan so that
• Total manufacturing cost is less than or equal to 200$
• Total storage cost is less than or equal to 150$
• Total revenue is maximal
Simplex method
f(x1, x2,…, xn) = c1x1 + c2x2 + . . . + cnxn → max
a1,1x1 + a1,2x2 + . . . + a1,nxn ≤ b1
a2,1x1 + a2,2x2 + . . . + a2,nxn ≤ b2
. . .
am,1x1 + am,2x2 + . . . + am,nxn ≤ bm
x1, x2, …, xn  0
Simplex method
Add slack variables s1, s2, …, sm  0

f(x1, x2,…, xn, s1, s2, …, sm) = c1x1 + c2x2 + . . . + cnxn → max
a1,1x1 + a1,2x2 + . . . + a1,nxn + s1 = b1
a2,1x1 + a2,2x2 + . . . + a2,nxn + 0 + s2 + . . . = b2
. . .
am,1x1 + am,2x2 + . . . + am,nxn + 0 + . . . + sm = bm
x1, x2, …, xn, s1, s2, …, sm  0
Simplex method

Consider the linear program under augmented form

f(x) = cx → max x = x1 c = (c1, c2, . . ., cn)

Ax = b x2
A = a1,1 a1,2 . . . a1,n b = b1
x0 ...
a2,1 a2,2 . . . a2,n b2
xn
... ...
am,1 am,2 . . . am,n bm

Denote J = (1,2,…, n), I = (1,2,…, m) - set of indices

Basic feasible solutions
 Denote
 A(j) the jth column of matrix A
 If J = (j1, j2, …, jk) – vector of indices (we also use J as a set of
indices), then xJ = (𝑥𝑗1 , . . ., 𝑥𝑗𝑘 ), A(J) = (A(j1), . . ., A(jk)) submatrix of A
composed by selecting columns A(j1), …, A(jk)
 Suppose rank(A) = m
 JB = (j1, j2, …, jm) such that columns A(j1), …, A(jm) are linear
independent, JN – vector of remaining indices of J (out of JB)
 B = A(JB) - basic matrix
 N = A(JN)
 x = (xB, xN), xB = x(JB) = (𝑥𝑗1 , . . ., 𝑥𝑗𝑚 ) such that Bx = b, xB = B-1 b, xN = 0
Basic feasible solutions
Example

f(x) = cx → max x = x1 c = (3, 2, 0, 0, 0)

Ax = b x2
x3 A = 2 1 1 0 0 b= 7
x0
x4 1 2 0 1 0 8
x5 2
1 -1 0 0 1

J = (1,2,3,4,5) – set of variable indices, I = (1,2,3) - set of constraint

indices
JB = (3,4,5), JN = (1,2),
B = 1 0 0 N = 2 1
xB = x3 = 7 xN = x1 = 0
x2 0 0 1 0 1 2
x4 8
x 2 0 0 1 1 -1
5
Simplex method
 Consider another feasible solution x’ = x + x
→ Ax’ = Ax → Ax = 0 → BxB + NxN = 0 → xB = -B-1NxN
→ f = cx = cBxB + cNxN = -(cBB-1N - cN)xN = -NxN =
-σ𝑗𝐽𝑁 j𝑥𝑗 , where u = cBB-1, N = uN - cN
 𝑥𝑗 = 𝑥𝑗  0, jJN
 If j  0, jJN then f ≤ 0: x is a maximizer
Simplex method
 If there exists index p  JN such that p < 0, build another feasible
solution x’ = x + x as follows
 xp =   0
 xj = 0, j  JN \ {p}
 xB = -B-1NxN = -B-1A(p), x’B = xB - B-1A(p),
 f = - p
→ with  > 0 and small, we have f > 0: x is not an optimal solution
because x’ is better than x.
 If B-1A(p) ≤ 0, then x’B = xB - B-1A(p)  0,  > 0. It means that f →
+ when  → + (cannot find maximizer)
Simplex method
 Denote B-1A(p) = (𝑥𝑗 ,𝑝 , 𝑥𝑗 ,𝑝 , . . ., 𝑥𝑗 ,𝑝 )T and
1 2 𝑚
𝑥𝑖
i = 𝑥𝑖 𝑝
if 𝑥𝑖, 𝑝 > 0 i JB
,
+, otherwise
→ select  = q = min{i | i JB} (we have  < +, otherwise the objective
function is unbounded)
Simplex method
JB = (j1, j2, …, jm) s.t. B = A(JB) is a basic
JN – vector of remaining indices of J (out of JB)
While stop condition not reach{
u = cBB-1, N = uN - cN , xN = 0, xB = B-1b
if N  0 then {
print(‘found optimal solution!’), break
} else {
p = argMin{j | j  JN}
Y = B-1A(p)
if Y ≤ 0 then {
print(‘objective is unbounded’), Return null
} else {
q = argMin {xi/Yi | iJB s.t. Yi > 0}
remove q from JB and add p to JB
remove p from JN and add q to JN
B = A(JB), N = A(JN)
}
}
}
Return (xB, xN) where xB = B-1b, xN = 0
Simplex method

Example f(x1, x2) = 3x1 + 2x2 → max

2x1 + x2 ≤ 7
x1 + 2x2 ≤ 8
x1 - x2 ≤ 2
x1, x2  0

Add 3 slack variables x3, x4, x5

f(x1, x2) = 3x1 + 2x2 → max

2x1 + x2 + x3 =7
x1 + 2x2 + x4 =8
x1 - x2 + x5 = 2
x1, x2, x3, x4, x5  0
Simplex method

Init: JB = (3,4,5), JN = (1,2)

B = 1 0 0 N = 2 1
xB = x3 = 7 xN = x1 = 0
x2 0 0 1 0 1 2
x4 8
x5 2 0 0 1 1 -1

f(x) = 0
Simplex method

Step 1: N = (-3 -2) → select p = 1

Y= 2 → select q = 5, JB = (1,3,4), JN = (2,5)

1
1

B = 2 1 0 N = 1 0
xB = x1 = 2 xN = x2 = 0
x5 0 1 0 1 2 0
x3 3
x4 6 1 0 0 -1 1

B-1 = 0 0 1
f(x) = 6
1 0 -2
0 1 -1
Simplex method

Step 2: N = (-5 3) → select p = 2

Y= -1 → select q = 3, JB = (1,2,4), JN = (3,5)

3
3

B = 2 1 0 N = 1 0
xB = x1 = 3 xN = x3 = 0
x5 0 1 2 1 0 0
x2 1
x4 3 1 -1 0 0 1

B-1 = 1/3 0 1/3

f(x) = 11
1/3 0 -2/3
-1 1 1
Simplex method
Step 3: N = (5/3 -1/3) → select p = 5

Y= 1/3 → select q = 4, JB = (1,2,5), JN = (3,4)

-2/3
1

B = 2 1 0 N = 1 0
xB = x1 = 2 xN = x3 = 0
x4 0 1 2 0 0 1
x2 3
x5 3 1 -1 1 0 0

B-1 = 2/3 -1/3 0

f(x) = 12
-1/3 2/3 0
-1 1 1
Simplex method

Step 3: N = (4/3 1/3) > 0 → STOP, found optimal solution

xB = x1 = 2 xN = x3 = 0
x2 3 x4 0
x5 3

f(x) = 12
Simplex method: Tabular form
cB B xB c1 ... cp ... cn

A1 ... Ap ... An

c j1 Aj1 x j1 xj1,1 xj1, p xj1,n j1

... ... ... ... ...

cq Ai0 xq x q ,1 x q, p x q ,n q

... ... ... ... ...

c jm Ajm x jm xjm,1 xjm, p xjm, n jm

 1 ... p ... n
Simplex method: Tabular form
cB B xB c1 ... cp ... cn

A1 ... Ap ... An

c j1 Aj1 x j1 xj1,1 xj1, p xj1,n j1

... ... ... ... ...

cq Ai0 xq x q ,1 x q, p x q ,n q

... ... ... ... ...

c jm Ajm x jm xjm,1 xjm, p xjm, n jm

 1 ... p ... n
Simplex method: Tabular form

 Update schema
 Update xi,j: x’i,j = xi,j – (xq,j * xi,p)/ xq,p,  i  JB \ {q},  j  J \ {p}
 ’j = j – (xq,j * p) / xq,p,  j  J \ {p}
 JB = JB \ {q}  {p}
 x’q = xq / xq,p
 On row q: x’q,j = xq,j / xq,p,  j  J
 On column p: x’q,p = 1, x’i,p = 0,  i  JB \ {q}
 p = 0
Simplex method: Tabular form

Example f(x1, x2,x3,x4,x5) = 3x1 + 2x2 → max

2x1 + x2 + x3 =7
x1 + 2x2 + x4 =8
x1 - x2 + x5 = 2
x1, x2, x3, x4, x5  0
Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

0 A3 7 2 1 1 0 0

0 A4 8 1 2 0 1 0

0 A5 2 1 -1 0 0 1

 -3 -2 0 0 0
Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

0 A3 7 2 1 1 0 0

0 A4 8 1 2 0 1 0

0 A5 2 1 -1 0 0 1

 -3 -2 0 0 0

Select p = 1
Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

0 A3 7 2 1 1 0 0 7/2

0 A4 8 1 2 0 1 0 8

0 A5 2 1 -1 0 0 1 2

 -3 -2 0 0 0

Select p = 1
Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

0 A3 7 2 1 1 0 0 7/2

0 A4 8 1 2 0 1 0 8

0 A5 2 1 -1 0 0 1 2

 -3 -2 0 0 0

Select p = 1 Select q = 5
Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

0 A3 3 2 3 1 0 -2

0 A4 6 1 3 0 1 -1

0 A5 2 1 -1 0 0 1

 -3 -5 0 0 3

Update , xB, xi,j except row A5, and except columns 1

Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

0 A3 3 0 3 1 0 -2

0 A4 6 0 3 0 1 -1

3 A1 2 1 -1 0 0 1

 0 -5 0 0 3

Replace A5 by A1 in the B, Update row corresponding to A1, and column 1

Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

0 A3 3 0 3 1 0 -2

0 A4 6 0 3 0 1 -1

3 A1 2 1 -1 0 0 1

 0 -5 0 0 3

Select p = 2
Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

0 A3 3 0 3 1 0 -2 1

0 A4 6 0 3 0 1 -1 2

3 A1 2 1 -1 0 0 1 +

 0 -5 0 0 3

Select p = 2 Select q = 3
Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

0 A3 3 0 3 1 0 -2

0 A4 3 0 3 -1 1 1

3 A1 3 1 -1 1/3 0 1/3

 0 -5 5/3 0 -1/3

Update , xB, xi,j except row A3, and except columns 2

Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

2 A2 1 0 1 1/3 0 -2/3

0 A4 3 0 0 -1 1 1

3 A1 3 1 0 1/3 0 1/3

 0 0 5/3 0 -1/3

Replace A3 by A2 in B, update xB, xi,j on row A3, and columns 2

Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

2 A2 1 0 1 1/3 0 -2/3 +

0 A4 3 0 0 -1 1 1 3

3 A1 3 1 0 1/3 0 1/3 9

 0 0 5/3 0 -1/3

Select p = 5
Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

2 A2 1 0 1 1/3 0 -2/3 +

0 A4 3 0 0 -1 1 1 3

3 A1 3 1 0 1/3 0 1/3 9

 0 0 5/3 0 -1/3

Select p = 5
Select q = 4
Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

2 A2 3 0 1 -1 2/3 -2/3

0 A4 3 0 0 -1 1 1

3 A1 2 1 0 2/3 -1/3 1/3

 0 0 4/3 1/3 -1/3

Simplex method: Tabular form
cB B xB 3 2 0 0 0

A1 A2 A3 A4 A5

2 A2 3 0 1 -1/3 2/3 0

0 A5 3 0 0 -1 1 1

3 A1 2 1 0 2/3 -1/3 0

 0 0 4/3 1/3 0

STOP, found optimal solution!!!!

Thank you
for your
attentions!