Non-linear Programming
Problem
Classical Optimization
Classical optimization theory uses differential
calculus to determine points of maxima and
minima for unconstrained and constrained
functions.
This chapter develops necessary and sufficient
conditions for determining maxima and
minima of unconstrained and constrained
functions.
2
Quadratic forms
Consider the function
f (X ) = a x + a x ++ a x +
2
11 1
2
22 2
2
nn n
a12 x1 x2 + + an1,n xn1 xn
The above function is called the quadratic or quadratic
form in n-variables.
The above function can be written in the form XTAX,
where X = [x1, x2, , xn]T be a n-vector and A = (aij) be a
n n symmetric matrix.
3
Quadratic forms
Then the quadratic form
Q( X ) = X T A X = aii xi2 + 2 aij xi x j
i
1i j n
(or the matrix A (symmetric matrix)) is called
Positive semi definite if XTAX 0 for all X 0.
Positive definite if XTAX > 0 for all X 0.
Negative semi definite if XT A X 0 for all X 0.
Negative definite if XTA X < 0 for all X 0.
The above quantities of the quadratic form depend on the symmetric
matrix A, therefore we have
4
1. Matrix minor test
A necessary and sufficient condition for A (any square
matrix ) to be :
Positive definite (positive semi definite) is that all the n
principal minors of A are > 0 ( 0).
Negative definite (negative semi definite) if kth principal
minor of A has the sign of (-1)k, k = 1, 2, ,n
(kth principal minor of A is zero or has the sign of (-1)k,
k=1,2,,n )
Indefinite, if none of the above cases happen
5
2. Eigenvalue Test
Since matrix A is a real symmetric matrix in XTAX, it
follows that its eigenvalues ( i) are real. Then XTAX is
Positive definite (positive semi definite) if i > 0
(i 0) i =1, 2, ,n.
Negative definite (negative semi definite) if i < 0 ,
(i 0) i=1,2,,n
Indefinite, if A has both positive and negative eigenvalues.
6
Examples : Decide the definiteness of the following
quadratic functions:
(1) x12 + 3 x 22 + 5 x 32
2
1
2
2
2
3
(2) 7 x 10x 7 x + 4 x1x2 2 x1x3 + 4 x2 x3
(3)
2
1
x x
2
2
Local maximum/Local minimum
Let f (X) = f (x1, x2,,xn) be a real-valued function of the n variables
x1, x2, , xn (we assume that f (X) is at least twice differentiable ).
A point X0 is said to be a local maximum of f (X) if there exists an
> 0 such that
Here
"
f ( X 0 + h ) f ( X 0 ) for all h j
"
"
T
0
0
0 T
h = (h1 , h2 ,.., hn ) , X 0 = ( x1 , x2 ,.., xn ) and X 0 + h = ( x10 + h1 , x20 + h2 ,.., xn0 + hn )T .
A point X0 is said to be a local minimum of f (X) if there exists an
> 0 such that
"
f ( X 0 + h ) f ( X 0 ) for all h j
8
Absolute maximum/Absolute minimum
X0 is called an absolute maximum or global maximum of f (X) if
f (X ) f (X0) X.
X0 is called an absolute minimum or global minimum of f (X) if
f (X ) f (X0) X.
Taylor Series expansion for functions of several variables
1 T
f ( X 0 + h) f ( X 0 ) = f ( X 0 )h + h Hh +
2
Theorem
A necessary condition for X0 to be an optimum
point of f (X) is that f ( X 0 ) = 0
(that is all the first order partial derivatives
zero at X0.)
f
xi
are
Definition: A point X0 for which f ( X 0 ) = 0 is called
a stationary point of f (X) (potential candidate for
local maximum or local minimum).
10
Theorem
Let X0 be a stationary point of f (X). A sufficient condition
for X0 to be a
local minimum of f (X) is that the Hessian matrix
H(X0) is positive definite;
local maximum of f (X) is that the Hessian matrix
H(X0) is negative definite.
11
Hessian matrix
2 f
x
1
2 f
x2 x1
H ( x) =
.
.
2
f
xn x1
2 f
x1x2
2 f
x22
2 f
. .
xn x2
2 f
x1xn
2 f
x2 xn
.
2 f
xn2
12
Example 1
Examine the following function for extreme
points
f (x1, x2,x3) = x1 +2 x3 +x2 x3 x12 x 22 x32
13
Definition:
A function f (X)=f (x1,x2,xn) of n variables is said to be
convex if for each pair of points X, Y on the graph, the line
segment joining these two points lies entirely above or on
the graph.
i.e. f((1-) X + Y) (1-) f (X)+ f (Y)
for all 0 1.
14
f is said to be strictly convex if for each pair of points
X,Y on the graph,
f ((1-) X + Y) < (1-) f (X)+ f (Y)
for all such that 0 < < 1.
f is called concave (strictly concave) if f is convex
(strictly convex).
15
Convexity test for function of one variable
A function of one variable f(x) is
convex if
d2 f
0
2
dx
concave if
d2 f
0
2
dx
16
Convexity test for functions of 2 variables
quantity
convex
Strictly
convex
concave
Strictly
concave
fxx fyy - (fxy)2
>0
>0
fxx
>0
<0
fyy
>0
<0
17
Results
(1) The Sum of two convex functions is convex.
(2) Let f(X) = XTAX. Then f(X) is convex if XTAX
is positive semi-definite.
Example: Show that the following function is
convex
f (x1, x2) = -2 x1 x2 +x12 + 2x22
18
Constraint Optimization
Problems
Constrained optimization Problems
KarushKuhnTucker (KKT) conditions:
Consider the problem
maximize z = f(X) = f (x1, x2,, xn)
subject to g(X) 0 [g1(X) 0
g2(X) 0
.
gm(X) 0]
(the non-negativity restrictions, if any, are included in the
above).
20
We define the Lagrangian function
L(X, S, ) = f(X) [g(X) + s2]
where s, s12, s22,..,sm2 ,are the non negative slack variables
added to g1(X) 0 ,. gm(X) 0 to make them into
equalities. Therefore
L(X, S, ) = f(X) [1{g1(X) + s12} + 2{g2(X) +s22}
+ +m{gm(x) + sm2}]
21
KKT conditions
KKT necessary conditions for optimality are given
by ,
L
= f ( X ) g ( X ) = 0
X
L
= 2 i S i = 0, i = 1, 2, ..., m
S i
L
= ( g ( X ) + S 2 ) = 0
22
KKT conditions
The KKT necessary conditions for maximization
problem are :
0
f ( X ) g ( X ) = 0
i g i ( X ) = 0
gi ( X ) 0
i=1,2m
These conditions apply to the minimization case as well,
except that 0.
23
KKT conditions
In scalar notation, this is given by
i 0 i=1,2,.m
f
g1
g 2
g m
1
2
.. m
=0
x j
x j
x j
x j
j = 1, 2, . . . , n
i g i ( X ) = 0 i=1,2,.m
gi ( X ) 0
i=1,2,.m
24
IMPORTANT: The KKT condition can be satisfied at a local minimum (or max.),
a global minimum (or max.) as well as at a saddle point.
We can use the KKT condition to characterize all the stationary points
of the problem, and then perform some additional testing to determine
the optimal solutions of the problem.
25
Sufficiency of the KKT conditions:
Required conditions
Sense of optimization
Objective function Solution space
maximization
Concave
Convex set
minimization
Convex
Convex set
26
Example: Use the KKT conditions to find the optimal
solution for the following problem:
maximize f(x1, x2) = x1+ 2x2 x23
subject to x1 + x2 1
x1
0
x2 0
27
Solution: Here there are three constraints ,
g1(x1,x2) = x1+x2 - 1 0
g2(x1,x2) = -x1
g3(x1,x2) = -x2
0
The KKT necessary conditions for maximization
problem are :
i 0 f ( X ) g ( X ) = 0
i g i ( X ) = 0
gi ( X ) 0
i=1,2m
28
Hence the KKT conditions become
g 3
g1
g 2
f
1
2
3
=0
x1
x1
x1
x1
g 3
g 1
g 2
f
1
2
3
=0
x 2
x 2
x 2
x 2
1g1(x1,x2) = 0
2g2(x1,x2) = 0
(note: f is concave, gi are
3g3(x1,x2) = 0
convex, maximization problem
g1(x1,x2) 0
these KKT conditions are
g2(x1,x2) 0
sufficient at the optimum point)
g3(x1,x2) 0
and 10, 20, 30
29
i.e.
1 1 + 2 = 0
(1)
2 3x22 1 + 3 = 0 (2)
1(x1 + x2 1) = 0
(3)
2 x1
=0
(4)
3 x2
=0
(5)
x1 + x2 1 0
(6)
x1
(7)
x2
(8)
1 0
(10)
and 3 0
(9)
(11)
30
(1) gives 1 = 1 + 2 1 >0 (using 10)
Hence (3) gives x1 + x2 = 1 (12)
Thus both x1, x2 cannot be zero, therefore let x1>0, then (4)
gives 2 = 0. therefore 1 = 1
if now x2 = 0, then (2) gives 2 0 1 + 3 = 0
=> 3 < 0 not possible
Therefore x2 > 0, hence (5) gives 3 = 0 and then (2) gives
x22 = 1/3 => x2 =1/3.
And so x1 = 1- 1/3. Therefore
Max f = 1 - 1/3 + 2/3 1/33 = 1 + 2/33.
31
Example: Use the KKT conditions to derive an optimal
solution for the following problem:
minimize f (x1, x2) = x12+ x2
subject to x12 + x22 9
x1 + x2 1
Solution: Here there are two constraints,
g1(x1,x2) = x12+x22 - 9 0
g2(x1,x2) = x1 + x2 -1 0
32
Thus the KKT conditions are:
1 0, 2 0 as it is a minimization problem
2x1 - 21x1 - 2 = 0
1 - 21x2 - 2 = 0
2
1
2
2
1 ( x + x 9) = 0
2 ( x1 + x2 1) = 0
x12 + x22 9
x1 + x2 1
33
Now 1 = 0
(from 2) gives 2 = 1 Not possible.
Hence 1 0 and so
2
1
2
2
x +x =9
(5)
Assume 2 = 0. So (1st equation of ) (2) gives
2 x1 (1 1 ) = 0
Since 1 0
we get x1= 0
From (5), we get x2 = 3
2nd equation of (2) says (with 1 < 0, 2 = 0 ) x2 = -3
Thus the optimal solution is:
1
x1 = 0, x2 = 3, 1 = , 2 = 0
6
The optimal value is :
z = 3
34
Use the KKT conditions to find an optimal solution of the
following problem:
Maximize f(x) = 20x1 + 10 x2
Subject to x12 + x22 1
x1 + 2x2 2
x1 0, x2 0
max f occurs at x1 = 2/5, x2 = 1/5
35
Quadratic Programming
36
Quadratic Programming
A quadratic programming problem is a non-linear
programming problem of the form
Maximize z = CX + X T DX
subject to AX b, X 0
T
where X = [ x1, x2 ,..., xn ] , b = [b1 , b2 ,..., bn ] , C = [c1, c2 ,..., cn ]
a11 a12 . . a1n
a
a
.
.
a
2n
21 22
A= .
am1 am2 . . amn
d11 d12 . . d1n
d d
.
.
d
2n
21 22
D= .
dn1 dn2 . . dnn
Quadratic Programming
!
Assume that the n n matrix D is symmetric and
negative-definite.
This means that the objective function is strictly
concave.
Since the constraints are linear, the feasible region
is a convex set.
38
Quadratic Programming
In scalar notation, the quadratic programming
problem reads:
n
Maximize z = c j x j + d jj x 2j + 2
j =1
j =1
x xj
ij i
1 i < j n
subject to a11 x1 + a12 x 2 + . . . + a1 n x n b1
a 21 x1 + a 22 x 2 + . . . + a 2 n x n b 2
a m 1 x1 + a m 2 x 2 + . . . + a m n x n b m
x1 , x 2 , . . . , x n 0
39
Wolfes Method to solve a Quadratic Programming
Problem:
!
The solution to this problem is based on the KKT
conditions. Since the objective function is strictly
concave and the solution space is convex, the
KKT conditions are also sufficient for optimum.
Since there are m + n constraints, we have m + n
Lagrange multipliers; the first m of them are
denoted by 1, 2 , , m and the last n of them
are denoted by 1, 2 , , n.
40
Wolfes Method
The KKT (necessary) conditions are:
1. 1, 2 ,. . ., m , 1, 2 ,. . ., n 0
n
i =1
i =1
2 . c j + 2 d ij xi i a ij + j = 0 , j = 1, 2 , . . . , n
3. i aij x j bi = 0, i = 1, 2,. . ., m
j =1
j x j = 0 , j = 1, 2,. . ., n
n
4.
a x
ij
bi , i = 1, 2,. . ., m and x j 0, j = 1, 2,. . ., n
j =1
41
Wolfes Method
Denoting the (non-negative) slack variable for the ith constraint
n
a ij x j b i
j =1
by si, the 3rd condition can be written in an equivalent form
as:
3.
i si = 0, i = 1, 2,. . ., m
j x j = 0 , j = 1, 2,. . ., n
(Referred to as " Restricted Basis " conditions).
42
Wolfes Method
Also condition(s) (2) can be rewritten as:
n
i =1
i =1
2 d ij x i + i a ij j = c j ,
j = 1, 2 , . . . , n
and condition(s) (4) can be rewritten as:
n
4.
ij
x j + si = bi , i = 1, 2,. . ., m
j =1
x j 0 , j = 1, 2,. . ., n
43
Wolfes Method
Thus we have to find a solution of the following m + n
linear equations in the 2n + m unknowns x j , i , j
n
i =1
i =1
2 d ij xi + i a ij j = c j , j = 1, 2 , . . . , n
n
a ij x
+ s i = b i , i = 1, 2 , . . . , m
j =1
i si = 0 ,
i = 1, 2 , . . . , m ,
j x j = 0 , j = 1, 2 , . . . , n
i 0 , si 0 ,
j 0, x j 0 ,
i = 1, 2 , . . . , m ,
j = 1, 2 , . . . , n
44
Wolfes Method
Since we are interested only in a " feasible solution
of the above system of linear equations, we use
Phase-I method to find such a feasible solution. By
the sufficiency of the KKT conditions, it will be
automatically the optimum solution of the given
quadratic programming problem.
45
Example-1:
2
1
2
2
Maximize z = 8x1 + 4x2 x x
subject to x1 + x 2 2
x1 , x 2 0.
46
Denoting the Lagrange multipliers by 1, 1, and 2,
the KKT conditions are:
1. 1 , 1 , 2 0
2.
8 2 x1 1 + 1 = 0
4 2 x2 1 + 2 = 0
i.e. 2 x1 + 1 1
=8
2 x2 + 1
2 = 4
3. x1 + x2 + s1 = 2, 1s1 = 1 x1 = 2 x2 = 0
All variables 0.
47
Introducing artificial variables R1, R2, we thus have to
Minimize
r = R1 + R 2
subject to the constraints
2 x1
+ R1
+ 1 1
2 x 2 + 1
+ R2
x1 + x 2
=8
=4
+ S1 = 2
1 S 1 = 1 x1 = 2 x 2 = 0
All variables 0 (We solve by " Modified Simplex " Algorithm).
48
R1
R2
-1
0
0
-1
0
0
-1
1
0
0
0
-1
0
1
0
0
0
0
0
1
12
0
8
4
2
-1
-1
0
0
-1
0
-1
0
0
1
0
0
0
0
1
0
-2
-2
0
1
8
4
4
2
1
-1
1
0
-1
0
-1
0
-2
1
-1
0
0
0
1
0
2
-2
2
1
0
4
0
49
2
1 1 2
Basic r
x1
x2
r
R1
R2
S1
1
0
0
0
2
0
2
0
1
2
0
0
2
1
2
0
1
1
0
-1
0
-1
0
0
r
R1
R2
x1
1
0
0
0
0
0
0
1
0
-2
2
1
2
1
1
0
1
0
0
0
0
0
0
1
4
-2
4
1
0
1
0
0
1
R2
x1
s1
Sol
Thus we have got the feasible solution
x1 = 2, x2 = 0, 1 = 4, 1 = 0, 2 = 0
and the optimal value is: z = 12
50
Example-2
Maximize
2
1
z = 8x1 x + 2x2 + x3
subject to
x1 + 3 x 2 + 2 x 3 12
x1 , x 2 , x 3 0
51
Example-2: Solution
Denoting the Lagrange multipliers by 1,1, 2, and 3,
the KKT conditions are:
1. 1 , 1 , 2 , 3 0
2. 8 2 x1 1 + 1 = 0
2
31
2 1
+ 2 = 0
+ 3 = 0
i.e. 2 x1 + 1 1
31 2
2 1
=8
=2
3 = 1
52
3. x1 + 3 x2 + 2 x3 + S1 = 12,
1S1 = 0
1 x1 = 2 x2 = 3 x3 = 0
All variables 0.
Solving this by " Modified Simplex Algorithm ", the optimal
solution is:
11
25
x1 = , x2 = , x3 = 0
3
9
and the optimal z =
193
9
53
Basic r
x1
x2
x3 1 1 2
3 R1 R2 R3 S1
Sol
r 1
R1 0
2
0
2
0
0
0
0
0
0
6 -1
0 0
1 -1
-1
0
0
-1
0
0
0 0 0
-1 -1 -1
1 0 0
0
0
0
11
0
8
R2 0
-1
R3 0
-1
S1 0
12
Since 1 S1 = 0 and S1 is in the basis, 1 cannot enter.
So we allow x1 to enter the basis and of course by minimum
ratio test R1 leaves the basis.
54
Basic r
x1
x2
x3 1 1 2
-1
-1
x1 0
0 1/2 -1/2 0
R2 0
-1
R3 0
-1
S1 0
2 -1/2 1/2
3 R1 R2 R3 S1
-1
Sol
1/2 0
-1/2 0
Since 1 S1 = 0 and S1 is in the basis, 1 cannot enter.
So we allow x2 to enter the basis and of course by minimum
ratio test S1 leaves the basis.
55
x1
x2 x3 1 1 2
R1 R2 R3 S1
-1
-1
-1 0
x1 0
0 1/2 -1/2 0
1/2 0
R2 0
-1
R3 0
-1
x2 0
2/3 -1/6 1/6
-1/6 0
1/3 8/3
Basic r
Sol
As S1 is not in the basis, now 1 enters the basis .
And by minimum ratio test R3 leaves the basis.
56
Basic
x1
x2
x3 1 1
-1 3/2
x1 0
0 -1/2
1/4 1/2 0
-1/4 0 15/4
R2 0
-1
3/2
0 1
-3/2 0
1/2
1 0
0 -1/2
1/2
1/2
x2 0
1 2/3 0 1/6
R1
R2 R3
-1 0
0 -1/12 -1/6 0
-5/2
S1
Sol
1/2
1/12 1/3 11/4
Now 3 enters the basis .
And by minimum ratio test R2 leaves the basis.
57
Basic
x1
x2 x3 1 1
x1 0
0 -1/2 1/6
3 0
1 0
0
0
0 0 0
0 0 1
x2 0
1 2/3 0 1/6 -1/18 0
0
0
0 -2/3 1
0 -1/3 0
This is the end of Phase I.
Thus the optimal solution is:
11
25
x1 = , x2 = , x3 = 0
3
9
R1
R2 R3
-1 -1
S1 Sol
-1
1/2 -1/6 0
0 11/3
0 2/3 -1
0 1/3 0
0 1/3
0 2/3
-1/6 1/18 0 1/3 25/9
Thus the optimal
value z is: 193
9
58
Example-3
Maximize
2
1
2
2
z = 6x1 + 3x2 2x 4x1x2 3x
subject to
x1 + x 2 1
2 x1 + 3 x 2 4
x1 , x 2 0
59
Denoting the Lagrange multipliers by 1, 2, 1, and 2,
the KKT conditions are:
1.
1 , 2 , 1 , 2 0
2. 6 4 x1 4 x2 1 2 2 + 1 = 0
3 4 x1 6 x2 1 32 + 2 = 0
i.e. 4 x1 + 4 x2 + 1 + 2 2 1 = 6
4 x1 + 6 x2 + 1 + 32 2 = 3
60
3.
x1 + x2 + S1 = 1,
2 x1 + 3 x2 + S 2 = 4,
1S1 = 2 S 2 = 0
1 x1 = 2 x2 = 0
and all variables 0.
Solving this by " Modified Simplex Algorithm ", the
optimal solution is:
x1 = 1, x2 = 0 and the optimal z = 4.
61
Basic
x 1 x2
1 2 1 2
R1
R2 S1
S2 Sol
r 1
R1 0
8
0
4
10
0
4
2
0
1
5 -1
0 0
2 -1
-1
0
0
0
-1
1
0
-1
0
0
0
0
0
0
0
9
0
6
R2 0
-1
S1 0
S2 0
1 4/3 0
1/3 0
-1
2/3
-5/3
R1 0 4/3 0
1/3 0
-1
2/3
-2/3
x2 0 2/3 1
1/6 1/2 0 -1/6
1/6
1/2
1/2
S1 0 1/3
0 -1/6 1/2 0
1/6
-1/6
S2 0
0 -1/2 -3/2 0
1/2
-1/2
5/2
62
Basic r
x 1 x2
1 2 1 2
R1
R2 S1 S2 Sol
-2
-1
-1
-2
R1 0
-1
x1 0
3/2 1/4 3/4 0 -1/4
1/4
3/4
S1 0
-1/2 -1/43/4 0
1/4
0 -1/4
1/4
S2 0
0 -1/2 -3/2 0
1/2
-1/2
5/2
-1
-1
-4
R1 0
-1
-4
x1 0
2 0 0
-2
1
-1 3 0
0 0 0
1
0
0
0
-1
0
4
-2
0
1
1
2
S2 0
63
Basic r
x 1 x2
1 2 1 2
R1
R2 S1 S2 Sol
-1
-1
1 0
-1
-4
x1 0
2 0 0
-2
0 1
-1
-1
S2 0
-2
Thus the optimum solution is:
x1 = 1, x2 = 0, 1 = 2, 2 = 0, 1 = 0, 2 = 3
And the optimal value is: z = 4
64
Solve the following quadratic programming problem
Maximize
subject to
z = 20x1 + 50x2 20x12 + 18x1 x2 5x22
x1 + x 2 6
x1 + 4 x 2 18
x1 , x 2 0
Max z = 224
Using Excel solver, the opt solution is: x1=2, x2=4
65
Remark:
If the problem is a minimization problem, say, Minimize z,
we convert it into a maximization problem, Maximize -z.
66