Numerical Optimization
Linear Programming
Shirish Shevade
Computer Science and Automation
Indian Institute of Science
Bangalore 560 012, India.
NPTEL Course on Numerical Optimization
Shirish Shevade
Numerical Optimization
Transportation Problem
min
ij cij xij
s.t.
P3
j=1 xij
P2
ai , i = 1, 2
bj , j = 1, 2, 3
xij 0 i, j
i=1 xij
Shirish Shevade
ai : Capacity of the plant Fi
bj : Demand of the outlet Rj
cij : Cost of shipping one unit
of product from Fi to Rj
xij : Number of units of the
product shipped from Fi to Rj
(variables)
The
P objective is to minimize
ij cij xij
P3
j=1 xij ai , i = 1, 2
(constraints)
P2
i=1 xij bj , j = 1, 2, 3
(constraints)
xij 0 i, j (constraints)
Numerical Optimization
The Diet Problem: Find the most economical diet that satisfies
minimum nutritional requirements.
Number of food items: n
Number of nutritional ingredient: m
Each person must consume at least bj units of nutrient j per
day
Unit cost of food item i: ci
Each unit of food item i contains aji units of the nutrient j
Number of units of food item i consumed: xi
Constraint corresponding to the nutrient j:
aj1 x1 + aj2 x2 + . . . + ajn xn bj , xi 0 i
Cost:
c1 x1 + c2 x2 + . . . + cn xn
Shirish Shevade
Numerical Optimization
Problem:
min
c1 x1 + c2 x2 + . . . + cn xn
s.t. aj1 x1 + aj2 x2 + . . . + ajn xn bj j
xi 0 i
Given: c = (c1 , . . . , cn )T , A = (a1 | . . . |an ), b = (b1 , . . . , bm )T .
Linear Programming Problem (LP):
min
cT x
s.t. Ax b
x0
where A Rmn , c Rn and b Rm .
Assumption: m n, rank(A) = m
Linear Constraints can be of the form Ax = b or Ax b
Shirish Shevade
Numerical Optimization
Constraint (Feasible) Set:
Inequality constraint of the type {x : aT x b} or
{x : aT x b} denotes a half space
Equality constraint, {x : aT x = b}, represents an affine
space
Non-negativity constraint, x 0
Constraint set of an LP is a convex set
Polyhedral Set
X = {x : Ax b, x 0}
Polytope: A bounded polyhedral set
Shirish Shevade
Numerical Optimization
Consider the constraint set in R2 :
{(x1 , x2 ) : x1 + x2 2, x1 1, x1 0, x2 0}
Shirish Shevade
Numerical Optimization
Consider the constraint set in R2 :
{(x1 , x2 ) : x1 1, x2 x1 }
Shirish Shevade
Numerical Optimization
Feasible set can be a singleton set
Feasible Set = {(x1 , x2 ) : x1 + x2 = 2, x1 + x2 = 1} = {A}
Shirish Shevade
Numerical Optimization
Feasible set can be empty!
Feasible Set = {(x1 , x2 ) : x1 + x2 2, x1 + x2 1} =
Shirish Shevade
Numerical Optimization
Definition
Let X be a convex set. A point x X is said to be an extreme
point (corner point or vertex) of X if x cannot be represented
as a strict convex combination of two distinct points in X.
Extreme Points: A, B, C and D.
E is not an extreme point.
Shirish Shevade
Numerical Optimization
Extreme Point: A
Shirish Shevade
Numerical Optimization
Constraint Set:
X = {(x1 , x2 ) : x1 + x2 2, x1 1, x1 0, x2 0}
4 constraints in R2
Two constraints are binding (active) at every extreme point
Fewer than two constraints are binding at other points
Shirish Shevade
Numerical Optimization
Consider the constraint set: X = {x : Ax b, x 0} where
A Rmn and rank(A) = m.
m + n hyperplanes associated with m + n halfspaces
m + n halfspaces define X
An extreme point lies on n linearly independent defining
hyperplanes of X
If X is nonempty, the set of extreme points of X is not
empty and has a finite number of points.
An edge of X is formed by intersection of n 1 linearly
independent hyperplanes
Two extreme points of X are said to be adjacent if the line
segment joining them is an edge of X
Shirish Shevade
Numerical Optimization
For example, B and C are adjacent points
Adjacent extreme points have n 1 common binding
linearly independent hyperplanes
Shirish Shevade
Numerical Optimization
Remarks:
Consider the constraint set: X = {x : Ax = b, x 0} where
A Rmn and rank(A) = m.
Let x X be an extreme point of X
m equality constraints are active at x
Therefore, n m additional hyperplanes (from the
non-negativity constraints) are active at x
Shirish Shevade
Numerical Optimization
Geometric Solution of a LP:
min
cT x
s.t. Ax b
x0
where A Rmn , c Rn and b Rm .
Shirish Shevade
Numerical Optimization
Shirish Shevade
Numerical Optimization
Shirish Shevade
Numerical Optimization
Shirish Shevade
Numerical Optimization
Shirish Shevade
Numerical Optimization
Shirish Shevade
Numerical Optimization
Shirish Shevade
Numerical Optimization
Example:
min
s.t.
2x1 x2
x1 + x2 5
x1 + 2x2 6
x1 , x2 0
Shirish Shevade
Numerical Optimization
Example:
min
s.t.
2x1 x2
x1 + x2 5
x1 + 2x2 6
x1 , x2 0
Shirish Shevade
Numerical Optimization
Example:
min
s.t.
2x1 x2
x1 + x2 5
x1 + 2x2 6
x1 , x2 0
Shirish Shevade
Numerical Optimization
Example:
min
s.t.
2x1 x2
x1 + x2 5
x1 + 2x2 6
x1 , x2 0
Shirish Shevade
Numerical Optimization
Example:
min cT x
s.t. x X
Shirish Shevade
Numerical Optimization
Example:
min cT x
s.t. x X
Shirish Shevade
Numerical Optimization
Example:
min cT x
s.t. x X
Shirish Shevade
Numerical Optimization
Example:
min cT x
s.t. x X
Shirish Shevade
Numerical Optimization
Consider a linear programming problem LP:
min
cT x
s.t. aTi x (, =, ) bi , i = 1, . . . , m
x0
Let X = {x : aTi x (, =, ) bi , i = 1, . . . , m, x 0}.
Remarks:
X is a closed convex set
The set of optimal solutions is a convex set.
The linear program may have no solution or a unique
solution or infinitely many solutions.
If x is an optimal solution to LP, then x must be a
boundary point of X. If z = cT x , then {x : cT x = z} is a
supporting hyperplane to X.
If X is compact and if there is an optimal solution to LP,
then at least one extreme point of X is an optimal solution
to the linear programming problem.
Shirish Shevade
Numerical Optimization
LP in Standard Form:
min
cT x
s.t. Ax = b
x0
where A Rmn and rank(A) = m.
Assumption: Feasible set is non-empty
Shirish Shevade
Numerical Optimization
Any linear program can be converted to the Standard Form.
(a) max cT x = min cT x
(b) Constraint of the type
aT x b, x 0
can be written as
aT x + y = b
x 0
y 0
Shirish Shevade
Numerical Optimization
(c) Constraint of the type
aT x b, x 0
can be written as
aT x z = b
x 0
z 0
(d) Free variables (xi R) can be defined as
xi = xi+ xi , xi+ 0, xi 0
Shirish Shevade
Numerical Optimization
Example:
x1 2x2 3x3
x1 + 2x2 + x3 14
x1 + 2x2 + 4x3 12
x1 x2 + x3 = 2
x1 , x2 unrestricted
x3 3constraints
Write the constraints as equality
min
s.t.
x1 + 2x2 + x3 + x4 = 14, x4 0
x1 + 2x2 + 4x3 x5 = 12, x5 0
Define new variables x1+ , x1 , x2+ , x2 and x30 such that
x1 = x1+ x1 , where x1+ 0, x1 0
x2 = x2+ x2 , where x2+ 0, x2 0
x30 = 3 x3 so that x30 0
Shirish Shevade
Numerical Optimization
Therefore, the program
min
x1 2x2 3x3
s.t. x1 + 2x2 + x3 14
x1 + 2x2 + 4x3 12
x1 x2 + x3 = 2
x1 , x2 unrestricted
x3 3
can be converted to the standard form:
min
s.t.
x1+ x1 2(x2+ x2 ) + 3(3 + x30 )
x1+ x1 + 2(x2+ x2 ) x30 + x4 = 17
x1+ x1 + 2(x2+ x2 ) 4x30 x5 = 24
x1+ x1 x2+ + x2 x30 = 5
x1+ , x1 , x2+ , x2 , x30 , x4 , x5 0
Shirish Shevade
Numerical Optimization
Consider the linear program in standard form (SLP):
min
cT x
s.t. Ax = b
x0
where A Rmn , rank(A) = rank(A|b) = m.
Let B Rmm be formed using m linearly independent
columns of A.
Therefore, the system of equations, Ax = b can be written as,
xB
(B N)
= b.
xN
Letting xN = 0, we get
BxB = b xB = B1 b.
(xB : Basic Variables)
(xB 0)T : Basic solution w.r.t. the basis matrix B
Shirish Shevade
Numerical Optimization
Basic Feasible Solution
If xB 0, then (xB 0)T is called a basic feasible solution of
Ax = b
x0
w.r.t. the basis matrix B.
Theorem
Let X = {x : Ax = b, x 0}. x is an extreme point of X if
and only if x is a basic feasible solution of
Ax = b
x 0.
Shirish Shevade
Numerical Optimization
Proof.
(a) Let x be a basic feasible solution of Ax = b, x 0.
Therefore, x = (x1 , . . . , xm , 0, . . . , 0). Let B = (a1 |a2 | . . . |am )
| {z } | {z }
0
nm
where a1 , . . . , am are linearly independent. So,
x1 a1 + . . . + xm am = b.
Suppose x is not an extreme point of X.
Let y, z X, y 6= z and x = y + (1 )z, 0 < < 1.
Since y, z 0, we have
ym+1 = . . . = yn = 0
y a + . . . + ym am = b
and 1 1
zm+1 = . . . = zn = 0
z1 a1 + . . . + zm am = b
Since a1 , . . . , am are linearly independent, x = y = z, a
contradiction. So, x is an extreme point of X.
Shirish Shevade
Numerical Optimization
Proof.(continued)
(b) Let x be an extreme point of X.
x X Ax = b, x 0.
There exist n linearly independent constraints active at x.
m active constraints associated with Ax = b.
n m active constraints associated with n m
non-negativity constraints
x is the unique solution of Ax = b, xN = 0.
Ax = b BxB + NxN = b xB = B1 b 0
Therefore, x = (xB xN )T is a basic feasible solution.
n
Number of basic solutions
m
Enough to search the finite set of vertices of X to get an optimal
Shirish Shevade
Numerical Optimization
solution
Theorem
Let X be non-empty and compact constraint set of a linear
program. Then, an optimal solution to the linear program exists
and it is attained at a vertex of X.
Proof.
Objective function, cT x, of the linear program is continuous
and the constraint set is compact. Therefore, by Weierstrass
Theorem, optimal solution exists.
The set of vertices, {x1 , . . . , xk }, of X is finite.
Therefore, X is the convex hull
P of x1 , . . . , xk .
Hence, for every x X, x = ki=1 i xi where
P
i 0, ki=1 i = 1.
Let z = min
cT xi . Therefore,
x X,
P1ik
Pk for any
k
T
T
z = c x = i=1 i c xi z
i=1 i = z . So, the minimum
value of cT x is attained at a vertex of X.
Shirish Shevade
Numerical Optimization
Consider the constraints:
x1 + x2 2
x1 1
x1 , x2 0
Shirish Shevade
Numerical Optimization
The given constraints
x1 + x2 2
x1 1
x1 , x2 0
can be written in the form, Ax = b, x 0:
=2
x1 +x2 + x3
x1
+x4 = 1
x1 , x2 , x3 ,
x4
0
1 1 1 0
2
Let A =
= (a1 |a2 |a3 |a4 ) and b =
.
1 0 0 1
1
Shirish Shevade
Numerical Optimization
1 1 1 0
1 0 0 1
2
1
= (a1 |a2 |a3 |a4 ) and b =
.
1 1
(1) B = (a1 |a2 ) =
1 0
1
T
xB = (x1 x2 ) = B b = (1 1)T and xN = (x3 x4 )T = (0 0)T .
x = (xB xN )T corresponds to the vertex C.
A=
Shirish Shevade
Numerical Optimization
1 1 1 0
1 0 0 1
2
1
= (a1 |a2 |a3 |a4 ) and b =
.
1 1
(2) B = (a1 |a3 ) =
1 0
1
T
xB = (x1 x3 ) = B b = (1 1)T and xN = (x2 x4 )T = (0 0)T .
x = (xB xN )T corresponds to the vertex B.
A=
Shirish Shevade
Numerical Optimization
A=
1 1 1 0
1 0 0 1
= (a1 |a2 |a3 |a4 ) and b =
(3) B = (a1 |a4 ) =
1 0
1 1
2
1
.
xB = (x1 x4 )T = B1 b = (2 1)T and xN = (x2 x3 )T = (0 0)T .
x = (xB xN )T is not a basic feasible point
Shirish Shevade
Numerical Optimization
1 1 1 0
1 0 0 1
2
1
= (a1 |a2 |a3 |a4 ) and b =
.
1 0
(4) B = (a2 |a4 ) =
0 1
1
T
xB = (x2 x4 ) = B b = (2 1)T and xN = (x1 x3 )T = (0 0)T .
x = (xB xN )T corresponds to the vertex D.
A=
Shirish Shevade
Numerical Optimization
1 1 1 0
1 0 0 1
2
1
= (a1 |a2 |a3 |a4 ) and b =
.
1 0
(5) B = (a3 |a4 ) =
0 1
1
T
xB = (x3 x4 ) = B b = (2 1)T and xN = (x1 x2 )T = (0 0)T .
x = (xB xN )T corresponds to the vertex A.
A=
Shirish Shevade
Numerical Optimization
Example:
min 3x1 x2
s.t. x1 + x2 2
x1 1
x1 , x2 0
Shirish Shevade
Numerical Optimization