IEDA 3010

Prescriptive Analytics
Linear Optimization 4

IEDA 3010
Linear Optimization
– Algorithms and extensions
Dr. Jin QI
Department of Industrial Engineering and Decision Analytics
Hong Kong University of Science and Technology

Practical Optimization
Solution
Problem Model
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Algorithm
• Simplex Model: Dantzig 1947
– Good in Practice, Bad in Theory
• Ellipsoid Algorithm: Khachiyan 1979
– Good in Theory, Bad in Practice
• Interior Point Method: Karmarkar 1984
– Both good in theory and practice

2
qxd 11/19/08 08:28 AM Page 90
hil76299_ch04_089-160.qxd 11/19/08 08:28 AM Page 90

IEDA 3010
Prescriptive Analytics
Linear Optimization 4

CHAPTER Wyndor Example Revisited

904 SOLVING LINEAR PROGRAMMING PROBLEMS:
CHAPTER 4 SOLVINGTHE SIMPLEX
LINEAR METHOD PROBLEMS: THE
PROGRAMMING

x2 Maximize Z " x2 3x # 5x , Maximize Z " 3x1 # 5x2,

1 2
x1 " 0 subject to x1 " 0 subject to
(0, 9) x1 (0, 9) $ 4 x1 $ 4
2x2 $ 12 2x2 $ 12
3x1 # 2x2 " 18 3x1 # 2x2 $ 183x1 # 2x2 " 18 3x1 # 2x2 $ 18
and and
x1 % 0, x2 % 0 x1 % 0, x2 % 0

(2, 6) (4, 6) (2, 6) (4, 6)

(0, 6) 2x(0,
2 "6)
12 2x2 " 12

x1 " 4 x1 " 4

Feasible (4, 3) Feasible (4, 3)

region region

■ FIGURE 4.1
aries and Constraint boundaries and x "0 x2 " 0
ons for the (0, 0) corner-point solutions for the (0,2 0)
x1 x1
problem. Wyndor Glass Co.(4, 0)
problem. (6, 0) (4, 0) (6, 0)
3
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Wyndor Example Revisited

• Each constraint corresponds to a constraint boundary
• Intersection of constraint boundaries are corner-point
solutions
– Corner-point feasible (CPF) solutions (extreme point)
• (0, 0), (0, 6), (2, 6), (4, 3), and (4, 0)
hil76299_ch04_089-160.qxd 11/19/08 08:28 AM Page 90

– Corner-point infeasible solutions

• (0, 9), (4, 6), and (6, 0)
90 CHAPTER 4 SOLVING LINEAR PROGRAMMING PROBLEMS: THE SIMPLEX ME

x2 Maximize Z " 3x1 # 5x2,

x1 " 0 subject to
(0, 9) x1 $ 4
2x2 $ 12
3x1 # 2x2 " 18 3x1 # 2x2 $ 18
and
x1 % 0, x2 % 0

(2, 6) (4, 6)
(0, 6) 2x2 " 12

x1 " 4

Feasible (4, 3)
region

■ FIGURE 4.1
Constraint boundaries and x2 " 0
4
corner-point solutions for the (0, 0)
(4, 0) (6, 0) x1
Wyndor Glass Co. problem.
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Geometry of Simplex Method

• For an LP with n decision variables, a corner-point
solution is the intersection of n constraint boundaries
• Adjacent CPF solution: “neighbor”
– They share n – 1 constraint boundaries
– They are connected by a line segment on the shared
hil76299_ch04_089-160.qxd 11/19/08 08:28 AM Page 90

constraint boundary
– This line segment is an edge of the feasible region
90 CHAPTER 4 SOLVING LINEAR PROGRAMMING PROBLEMS: THE SIMPLEX ME

x2 Maximize Z " 3x1 # 5x2,

x1 " 0 subject to
(0, 9) x1 $ 4
2x2 $ 12
3x1 # 2x2 " 18 3x1 # 2x2 $ 18
and
x1 % 0, x2 % 0

(2, 6) (4, 6)
(0, 6) 2x2 " 12

x1 " 4

Feasible (4, 3)
region

■ FIGURE 4.1
Constraint boundaries and x2 " 0
5
corner-point solutions for the (0, 0)
(4, 0) (6, 0) x1
Wyndor Glass Co. problem.
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Why Adjacent CPFs?

• Optimality test: assuming an optimal solution exists
– If a CPF solution has no adjacent CPF solutions that are
better, then it must be an optimal solution.

“If you’re better than your neighbors,

then you’re the best in your class.”

– E.g., (2, 6) must be optimal simply because its Z = 36 is

larger than Z = 30 for (0, 6) and Z = 27 for (4, 3)

6
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

92 SolvingCHAPTER
the Example
4 SOLVING LINEAR PROGRAMMING PRO

x2
Z " 30 (2, 6)
(0, 6)
1 2 Z " 36

■ FIGURE 4.2 Feasible (4, 3)

This graph shows the region
sequence of CPF solutions Z " 27
(!, ", #) examined by the
simplex method for the
Wyndor Glass Co. problem.
Z " 12
The optimal solution (2, 6) is 0
found after just three (0, 0)
Z"0 (4, 0) x1
solutions are examined.

7
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Ellipsoid Algorithm
• First polynomial-time algorithm for linear programming
(Khachiyan 1979)
• Idea
– Consider a sequence of smaller and smaller ellipsoids each
with the feasible region inside
– For iteration k: ck= center of Ek
• Eventually ck has to be inside of F, and we are done

Feasible region

F
ck

8
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Ellipsoid Algorithm
• To find the next smaller ellipsoid:
– find most violated constraint ak
• find smallest ellipsoid that includes constraint

Feasible region

F
ck

ak
9
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Linear Optimization
• We can convert nonlinear functions to linear ones

min $ % = max(+,) % + .) )
)
0. 2. 3% ≥ 5

min 6
0. 2. 3% ≥ 5
+,) % + .) ≤ 6

10
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Linear Optimization
• We can convert nonlinear functions to linear ones
– Problem with |.|
– Idea: |x| = max{-x,x}

min $ &% |(% |

%

). +. ,- ≥ / min $ &% 0%
%

). +. ,- ≥ /
(% ≤ 0%
−(% ≤ 0%
11
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Linear Optimization
• We can convert nonlinear functions to linear ones

↔ X1 + 2X2 + X3 ≤ 2X1 + 3X2 + 1

↔ - X1 - X2 + X3 ≤ 1

↔ X1 + 2X2 + X3 ≥ 2X1 + 3X2 - 1

(Because 2X1 + 3X2 - 1 < 0)

Because 2X1 + 3X2 may be > 0 or < 0

No way to convert! Too bad!

12
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Real World LO Problems

• Data are uncertain
– NETLIB example
• PILOT4: Constraint #372

13
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Real World LO Problems

• Data are uncertain
– Most coefficients are “ugly real” numbers and highly
unlikely that real-life parameters are known to high
accuracy. (Except for x880)
• What happened if the accurate solution is only 0.1%
from the optimal solution?

• In the worst case, the constraints can be violated by

450% !! (relative to RHS)

14
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Real World LO Problems

• Data are uncertain
– Small perturbation in coefficients can make the constraint
severely infeasible with respect to the RHS.
– Perturbation in objective can lead to large deviation.

15
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Real World LO Problems

“I work on planning under uncertainty. That's the big

field as far as I'm concerned; that's the future.”

- George Dantzig, 1999 on his 85th Birthday

16
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Real World LO Problems

“In real-world applications of Linear Programming, one
cannot ignore the possibility that a small uncertainty in
the data can make the usual optimal solution
completely meaningless from practical viewpoint.”

Ben-Tal and Nemirovski 2000

17
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Issues of Uncertainty
• Real world data are almost always uncertain
– Estimation error: part of the data is measured/estimated
– Prediction error: part of the data (e.g., future
demands/prices) does not exist when problem is solved
– Implementation error: some components of a solution
cannot be implemented exactly as computed, which in many
models can be mimicked by appropriate data uncertainty.
• E.g. Optimal length = 3.2514265343m

18
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Issues of Uncertainty

• How to account for data uncertainty?

• Computational tractability?
• How to rank objectives that are uncertain?

19
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Classical Robust Optimization

• Family of uncertain LPs:
– Different optimal solutions for set of different parameters
• Which one to choose?

20
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Classical Robust Optimization

• Robust Counterpart Approach
– Worst-case approach (ambiguity aversion)

21
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Classical Robust Optimization

• Consider a simple robust counterpart
– Feasible for all data perturbation

– Equivalent formulation of the robust counterpart

22
 IEDA 3010
Prescriptive Analytics
Linear Optimization 4

Classical Robust Optimization

• Consider a simple robust counterpart
– Extension of the simple robust counterpart

23