Write KKT conditions for

1. max f ( x)
x

on g1 ( x) 0  g 2 ( x) 0 
Solution
Constraunts are
 g1 ( x) 0, g 2 ( x) 0
Lagrangian function is L( x, 1 0, 2 0)  f  x   1  g1 ( x)   2g 2 ( x)
KKT Conditions are
f ( x)=1g1 ( x)  2 g 2 ( x), 1 , 2 0
 g1 ( x) 0, g 2 ( x) 0 or g1 ( x) 0,g 2 ( x) 0 
1g1 ( x) 0, 2 g 2 ( x) 0
Write KKT conditions for
2. min f ( x)
x

on g1 ( x) 0  g 2 ( x) 0 
Solution
Constraunts are
 g1 ( x) 0, g 2 ( x) 0
Lagrangian function is L( x, 1 0, 2 0)  f  x   1  g1 ( x)   2 g 2 ( x)
KKT Conditions are
f ( x)=-1g1 ( x)  2 g 2 ( x), 1 , 2 0
 g1 ( x) 0, g 2 ( x) 0 or g1 ( x) 0,g 2 ( x) 0 
1g1 ( x) 0, 2 g 2 ( x) 0
Write KKT conditions for
3. min f ( x)
x

on g1 ( x) 0  g 2 ( x) 0  g 3 ( x) 0 
Solution
Constraunts are
 g1 ( x) 0, g 2 ( x) 0,g 3 ( x) 0
Lagrangian function is L( x, 1 0, 2 0, 3 )  f  x   1  g1 ( x)   2 g 2 ( x)  3 g3 ( x)
KKT Conditions are
f ( x)=  1g1 ( x)  2 g 2 ( x)  3g 3 ( x), 1 , 2 0
 g1 ( x) 0, g 2 ( x) 0 ,g 3 ( x) 0 or g1 ( x) 0,g 2 ( x) 0,g3 ( x) 0 
1g1 ( x) 0, 2 g 2 ( x) 0
Write KKT conditions for
4. min cT x
x
Note :
on Ax b, x 0 m
 ( Ax  b)  i  Ai x  bi  where Ai is ith row vector of A
T
A is m n i 1

Solution
Sin ce A is m n, Ax  b 0 consists of m constraints
x 0 consists of n constraints
 Lagrangian function is L( x,  0,  0) cT x   T  Ax  b    T x
KKT Conditions are
c = AT    ,   Rm ,   Rn
Ax  b 0, x 0
i  Ai x  bi  0 i : m
 j x j 0 i 1: n
Write KKT conditions for
5. min cT x Note :
x
m
on Ax b, x 0  ( Ax  b)  i  Ai x  bi  where Ai is ith row vector of A
T

i 1
A is m n
Solution
Sin ce A is m n, Ax  b 0 consists of m constraints
x 0 consists of n constraints
 Lagrangian function is L( x,  ,  0)  f  x    T  Ax  b    T x
KKT Conditions are
c = AT    ,   Rn
Ax  b 0 vector , x 0
 j x j 0 i 1: n
Write KKT conditions for
6. max cT x
x

on Ax b, x 0 Note :
m
A is m n  (b  Ax)  i bi  Ai x  where Ai is ith row vector of A
T

i 1
Solution
Sin ce A is m n, b - Ax 0 consists of m constraints
x 0 consists of n constraints
 Lagrangian function is L( x,  ,  0) cT x   T b  Ax    T x
KKT Conditions are
c = AT    ,   Rn
b  Ax 0 vector , x 0
i bi  Ai x  0 i 1: m
 j x j 0 j 1: n
Write KKT conditions for
7. max cT x
x

on Ax b, x 0
A is m n
Solution
Sin ce A is m n, b - Ax 0 consists of m constraints
x 0 consists of n constraints
 Lagrangian function is L( x,  0,  0) cT x   T b  Ax    T x
KKT Conditions are
c = AT    ,   Rn
b  Ax 0 vector , x 0
i bi  Ai x  0 i 1: m
 j x j 0 j 1: n
Lagrangian multiplier and Duality
 Understanding Duality with LP Space of x is R4
max 5x1  6 x2  9 x3  8 x4
s.t x1  2 x2  3 x3  x4 5           (1)
x1  x2  2 x3  3 x4 3           (2)
xi 0, i 1, 2,3, 4
(1)  y1  (2)  y2  y1  x1  2 x2  3x3  x4   y2  x1  x2  2 x3  3x4  5 y1  3 y 2
 y1  y2  x1  2y1 y2  x2  3 y1  2 y2  x3   y1  3y2  x4 5 y1  3 y 2 , y1, y 2 0
5 6 9 8

So if we choose y1 and y2 such that following conditions are met 5 y1  3 y2  5x1  6 x2  9 x3  8 x4

y1 + y2 5 min 5 y1  3 y2

2 y1 + y2 6 s.t
y1 + y2 5
3 y1 + 2y2 9
2 y1 + y2 6
y1 + 3y2 8
3 y1 +2 y2 9
y1 , y2 0 y1 + 3y2 8
y1 and y2 0
 Think algebraically x1  2 x2  3 x3  x4 5,

Intuition and Explanation : x1  x2  2 x3  3 x4 3

xi 0, i 1, 2,3, 4
x1 , x2 , x3 , x4 are chosen such that Constraints are satisfied
Assume that we know the optimal x1* , x2* , x3* , x4*
Computing c1 x1*  c2 x2*  c3 x3*  c4 x4* give maximum value of objective function
5x1  6 x2  9 x3  8 x4
Now if we choose y1 , y2 0 such that
 1  2  1   1  2  2  1   2  3  1   2  1  primal optimal
y  y x *
 2 y  y x *
 3 y  2 y x *
 y  3 y x *

 c1  c2  c3 c4

 5 y1  3 y2  primal optimal for any y1 , y2 0 and satisfying constraints

y1 + y2 5
2 y1 + y2 6 We now try minimizing 5y1+3y2
3 y1 + 2y2 9
y1 + 3y2 8 satisfying the constraints
y1 , y2 0
 x1  2 x2  3 x3  x4 5,
x1  x2  2 x3  3 x4 3
xi 0, i 1, 2,3, 4
Matrix Notation
 x1 
x 
1 2 3 1  5
A   ; b   , x  2 
1 1 2 3  3  x3 
 
 x4 
Ax b, x 0

Let us linearly combine the two inequalities with y1 , y2 0 as multipliers

It is simple in matrix notation : yT Ax  yT b
1 2 3 1  5
 y1 y2   x  y1 y2   ;
1 1 2 3  3
y1 (1x1  2 x2  3x3  x4 )  y2 (1x1  1x2  2 x3  3 x4 ) 5 y1  3 y2
Duality
Using Excel we get following result
* * *
x 1, x 2, x 0, x 0
1 2 3
*
4
y1* 1, y2* 4

5x1*  6 x2*  9 x3*  8 x4* 5 y1*  3 y2* 17

 Getting the Dual in Lagrangian way
max c T x
s.t. Ax b min LD ( y ) bT y

x 0 AT y c
the Lagrangian function is y 0

L = c T x + y T (b  Ax ) +  T x y 0,  0

L( x , y,  )
Lagrangian can be written
c  AT y +  0  c  AT y -  without surplus variable/slack
x
Variables.
LD ( y,  )  AT y    x + y T (b  Ax ) +  T x
T
= yT b = bT y

This function we minimize with respect to y and  such that c  AT y - 

min LD ( y,  ) bT y Such that c  AT y -  y 0,  0

 Complementary Conditions at the Optimal point
max 5x1  6 x2  9 x3  8 x4
s.t x1  2 x2  3 x3  x4 5
x1  x2  2 x3  3 x4 3
xi 0, i 1, 2,3, 4

L( x , y,  ) = 5x1  6 x2  9 x3  8 x4  y1 5   x1  2 x2  3x3  x4   y2 3   x1  x2  2 x3  3x4 

1 x1  2 x2  3 x3  4 x4 y1 , y2 , 1 , 2 , 3 , 4 0

L( x , y,  )
* * * * 5  ( y1  y2 )  1 0
x 1, x 2, x 0, x 0
1 2 3 4
x1
L( x , y,  )
*
y 1, y 4 * 6  (2 y1  y2 )  2 0
1 2 x2
L( x , y,  )
1* 0, 2* 0, 3* 2, 4* 5 x3
9  (3 y1  2 y2 )  3 0

L( x , y,  )
8  ( y1  3 y2 )  4 0
x4
 y1 5   x1  2 x2  3x3  x4 
L( x , y,  ) 5 x16 x2  9 x3 8x4           
1
 y2 3   x1  x2  2 x3  3x4 
          
4
17 0 0

 x  x  x  x
1 1 2 2 3 3 4 4 y* T (b  Ax* ) = 0,  * T x* 0
0 1 0 2 2 0 5 0
Duality theorem in Linear
Programming
Strong Duality : cT x* bT y *

Primal : max cT x ; sub Ax b ; x 0

x

Dual : min bT y ; AT y c;

y

Primal : max cT x ; sub Ax  s b ; x, s 0

x,s
T T
Dual : min b y ; A y  w c; y, w 0
y,w
 Duality theorem in Linear Programming
Strong Duality : cT x* bT y*
To prove this we need the concept of Complementary slackness.
Primal : max cT x ; sub Ax  s b ; x, s 0
x ,s

Dual : min bT y ; AT y  w c; y, w 0

y ,w

Lagrangian for the primal without slack variables is L p (.) cT x  y T b  Ax   xT w

Lagrangian for the dual without surplus variables is LD (.) bT y  xT (c  AT y )  sT y
According to complementarity condition , at the optimal point ( x* , y * , w* , s* )

 y  b  Ax  0; x  w =0;  x  (c  A y ) 0; s 

T T T T
* * * * * T * *
we have y * 0

Premultiplying dual constraint A y  w c with  x 

T
T * * *
, we get

 x*  AT y*  x*  w* x*  c;
T T T

 y*  Ax*  y*  ( Ax*  s* )  y*  b* cT x*

T T T