Mohammed Abdel-Hafez

Outline
 Unconstrained Optimization
▪ Functions of One Variable
o General Ideas of Optimization

o First and Second Order Conditions

o Local v.s. Global Extremum

▪ Functions of Several Variables

o First and Second Order Conditions

o Local v.s. Global Extremum

 Constrained Optimization
▪ Kuhn-Tucker Conditions
▪ Sensitivity Analysis
▪ Second Order Conditions

2/26/2021 Mhafez 2
Unconstrained Optimization
 An unconstrained optimization problem is one where
you only have to be concerned with the objective
function you are trying to optimize.
 An objective function is a function that you are trying to
optimize.
 None of the variables in the objective function are
constrained.

2/26/2021 Mhafez 3
General Ideas of Optimization
 There are two ways of examining optimization.
 Maximization (example: maximize profit)
 In this case you are looking for the highest point on the
function.
 Minimization (example: minimize cost)
 In this case you are looking for the lowest point on the
function.
 Maximization f(x) is equivalent to minimization –f(x)

2/26/2021 Mhafez 4
Graphical Representation of a Maximum
y

16 y = f(x) = -x2 + 8x

4 x
8

2/26/2021 Mhafez 5
Questions Regarding the Maximum
 What is the sign of f '(x) when x < x*?
 Note: x* denotes the point where the function is at a
maximum.
 What is the sign of f '(x) when x > x*?
 What is f '(x) when x = x*?
Definition:
 A point x* on a function is said to be a critical point if
f ' (x*) = 0.
 This is the first order condition for x* to be a
maximum/minimum.

2/26/2021 Mhafez 6
Second Order Conditions
 If x* is a critical point of function f(x), can we decide whether
it is a max, a min or neither?
 Yes! Examine the second derivative of f(x) at x*, f ' '(x*);
 x* is a maximum of f(x) if f ' '(x*) < 0;
 x* is a minimum of f(x) if f ' '(x*) > 0;
 x* can be a maximum, a minimum or neither if f ' '(x*) = 0;

2/26/2021 Mhafez 7
 An Example of f''(x*)=0
 Suppose y = f(x) = x3, then f '(x) = 3x2 and f ''(x) =6x,
 This implies that x* = 0 and f ''(x*=0) = 0.
y=f(x)=x3
y

x*=0 is a saddle point where the point is neither a maximum nor a

minimum

2/26/2021 Mhafez 8
Example of Using First and Second
Order Conditions
 Suppose you have the following function:
 f(x) = x3 – 6x2 + 9x
 Then the first order condition to find the critical
points is:
 f’(x) = 3x2 - 12x + 9 = 0
 This implies that the critical points are at x = 1 and x = 3.
4

-2

-4

-6

-8
-0.5 0 0.5 1 1.5 2 2.5 3 3.5 4
2/26/2021 Mhafez 9
Example of Using First and Second
Order Conditions (Cont.)
 The next step is to determine whether the critical points
are maximums or minimums.
 These can be found by using the second order condition.
 f ' '(x) = 6x – 12 = 6(x-2)

 Testing x = 1 implies:
 f ' '(1) = 6(1-2) = -6 < 0.
 Hence at x =1, we have a maximum.
 Testing x = 3 implies:
 f ' '(3) = 6(3-2) = 6 > 0.
 Hence at x =3, we have a minimum.

 Are these the ultimate maximum and minimum of the

function f(x)?
2/26/2021 Mhafez 10
Local Vs. Global Maxima/Minima
 A local maximum is a point that f(x*) ≥ f(x) for all x in some
open interval containing x* and a local minimum is a point
that f(x*) ≤ f(x) for all x in some open interval containing x*;

 A global maximum is a point that f(x*) ≥ f(x) for all x in the

domain of f and a global minimum is a point that f(x*) ≤ f(x)
for all x in the domain of f.

 For the previous example, f(x) →  as x →  and f(x) → -

as x → -. Neither critical point is a global max or min of f(x).

2/26/2021 Mhafez 11
Local Vs. Global Maxima/Minima (Cont.)

 When f ''(x)≥0 for all x, i.e., f(x) is a convex function, then

the local minimum x* is the global minimum of f(x)

 When f ''(x)≤0 for all x, i.e., f(x) is a concave function, then

the local maximum x* is the global maximum of f(x)

2/26/2021 Mhafez 12
Conditions for a Minimum or a Maximum
Value of a Function of Several Variables
 Correspondingly, for a function f(x) of several
independent variables x
 Calculate f (x ) and set it to zero.
 Solve the equation set to get a solution vector x*.
 Calculate  2 f (x ) .
 Evaluate it at x*.
 Inspect the Hessian matrix at point x*.

H(x ) =  2 f (x )

2/26/2021 Mhafez 13
 Hessian Matrix of f(x)

f (x ) is a C 2 function of n variable s,
  2 f (x )  2 f (x )
  
 x1 2
x1xn 
H(x )   2 f (x ) =     .
  2 f (x )  2 f (x )
  
 xn x1 xn 
2

Since cross - partials are equal for a C 2 function, H(x)

is a symmetric matrix.

2/26/2021 Mhafez 14
Conditions for a Minimum or a Maximum Value of a
Function of Several Variables (cont.)
 Let f(x) be a C2 function in Rn. Suppose that x* is a critical
point of f(x), i.e., f (x *) = 0.

1. If the Hessian H(x *) is a positive definite matrix, then x*

is a local minimum of f(x);
2. If the Hessian H(x *) is a negative definite matrix, then
x* is a local maximum of f(x).
3. If the Hessian H(x *) is an indefinite matrix, then x* is
neither a local maximum nor a local minimum of f(x).

2/26/2021 Mhafez 15
 Example
 Find the local maxs and mins of f(x,y)

f ( x, y ) = x − y + 9 xy
3 3

 Firstly, computing the first order partial derivatives (i.e., gradient of f(x,y))
and setting them to zero

 f 
 
 x   3 x 2
+ 9y 
f ( x, y ) = =  =0
 f  − 3 y 2 + 9 x 
   
 y 
 critical points (x*, y *) is (0,0) and ( 3, -3 ).

2/26/2021 Mhafez 16
 Example (Cont.)
 We now compute the Hessian of f(x,y)

 2 f 2 f 
 2 
 x xy   6 x 9 
 f ( x, y ) =  2  =  9 − 6 y .
2

 f  f   
2

 yx y 2 
 
 The first order leading principal minor is 6x and the second order
principal minor is -36xy-81.
 At (0,0), these two minors are 0 and -81, respectively. Since the
second order leading principal minor is negative, (0,0) is a saddle
of f(x,y), i.e., neither a max nor a min.
 At (3, -3), these two minors are 18 and 243. So, the Hessian is
positive definite and (3,-3) is a local min of f(x,y).
 Is (3, -3) a global min?

2/26/2021 Mhafez 17
Global Maxima and Minima of a Function
of Several Variables
 Let f(x) be a C2 function in Rn, then

 When f(x) is a concave function, i.e.,  2 f (x ) is negative

semidefinite for all x and f (x *) = 0 , then x* is a global
max of f(x);

 When f(x) is a convex function, i.e.,  2 f (x ) is positive

semidefinite for all x and f (x *) = 0 , then x* is a global
min of f(x);

2/26/2021 Mhafez 18
Example (Discriminating Monopolist)
 A monopolist producing a single output has two types of customers. If it
produces q1 units for type 1, then these customers are willing to pay a price
of 50-5q1 per unit. If it produces q2 units for type 2, then these customers
are willing to pay a price of 100-10q2 per unit.
 The monopolist’s cost of manufacturing q units of output is 90+20q.
 In order to maximize profits, how much should the monopolist produce
for each market?
 Profit is:
f (q1 , q2 ) = q1 (50 − 5q1 ) + q2 (100 − 10 q2 ) − (90 + 20(q1 + q2 )).
The critical points are
f f
= 50 − 10q1 − 20 = 0  q1 = 3, = 100 − 20 q2 − 20 = 0  q2 = 4.
q1 q 2
2 f 2 f 2 f 2 f
= −10, = −20, = = 0.
q12
q22
q1q2 q2 q1
  2 f is negative definite  (3,4) is the profit - maximizing supply plan.

2/26/2021 Mhafez 19
Constrained Optimization
 Examples:
 Individuals maximizing utility will be subject to a
budget constraint
 Firms maximising output will be subject to a cost
constraint

 The function we want to maximize/minimize is called

the objective function
 The restriction is called the constraint

2/26/2021 Mhafez 20
Constrained Optimization (General Form)
 A general mixed constrained multi-dimensional
maximization problem is

max f (x ) = f (x 1 ,..., x n )
subject to
g1 (x 1 ,..., x n )  b1 , g 2 (x 1 ,..., x n )  b2 , , g k (x 1 ,..., x n )  bk ,
h1 (x 1 ,..., x n ) = c 1 , h2 (x 1 ,..., x n ) = c 2 , , hm (x 1 ,..., x n ) = c m .

2/26/2021 Mhafez 21
Constrained Optimization (Lagrangian Form)
 The Lagrangian approach is to associate a Lagrange
multiplier i with the i th inequality constraint and μi with
the i th equality constraint.
 We then form the Lagrangian
L (x 1 ,..., x n , 1 ,..., k , 1 ,..., m ) =
k
f (x 1 ,..., x n ) −  i ( g i (x 1 ,..., x n ) − bi )
i =1
m
−  i ( hi (x 1 ,..., x n ) − c i ).
i =1

2/26/2021 Mhafez 22
Constrained Optimization (Kuhn-Tucker
Conditions)
 If x * is a local maximum of f on the constraint set
defined by the k inequalities and m equalities, then,
there exists multipliers 1* , , k* , 1* , , m* satisfying

L( x* ,  * ,  * ) f ( x* ) k * gi ( x* ) m * hi ( x* )

= −  i −  i = 0, j = 1, ,n
x j x j i =1 x j i =1 x j
hi ( x* ) = ci , i = 1, ,m
i* ( gi ( x* ) − bi ) = 0, i = 1, , k
gi ( x* )  bi , i = 1, ,k
i*  0, i = 1, , k

Fin500J Topic 4
2/26/2021 Fall 2008
Mhafez 23
Constrained Optimization (Kuhn-Tucker
Conditions)
 The first set of KT conditions generalizes the
unconstrained critical point condition
 The second set of KT conditions says that x needs to
satisfy the equality constraints
 The third set of KT conditions is

i* ( gi ( x* ) − bi ) = 0, i = 1, ,k
 That is to say
if i*  0 then g i (x * ) = bi
if g i (x * )  bi then i* = 0

Fin500J Topic 4
2/26/2021 Fall 2008
Mhafez 24
Constrained Optimization (Kuhn-Tucker
Conditions)
 This can be interpreted as follows:
 Additional units of the resource bi only have value if the
available units are used fully in the optimal solution, i.e., if
g i (x * )  bi the constraint is not binding thus it does not
make difference in the optimal solution and i*=0.

 Finally, note that increasing bi enlarges the feasible region,

and therefore increases the objective value
 Therefore, i0 for all i

Fin500J Topic 4
2/26/2021 Fall 2008
Mhafez 25
 Example
Form the Lagrangian
max x − y 2
L=x-y 2 −  (x 2 + y 2 − 4) + 1 x + 2 y .
subject to
The first order conditions become:
x +y =4
2 2
L
(1) = 1 − 2  x + 1 = 0,
x  0, y  0. x
L
(2) = −2y − 2  y + 2 = 0,
y
(3)x 2 + y 2 − 4 = 0
(4)1 x = 0, (5)2 y = 0,
(6)1  0, (7)2  0,
(8)x  0, (8)y  0.

Fin500J Topic 4
2/26/2021 Fall 2008
Mhafez 26
 Example (cont.)

By (1), 1 + 1 = 2  x , since 1  0, 1 + 1  0
   0 and x  0, by (4), 1 = 0.
by (2), 2 = 2y (1 +  ), since 1 +   0
 either both y and 2 are zero, or both
are positive, from (5)  y = 0, 2 = 0.
By (3) and (8)  x=2, from (4), 1 = 0,
1 1
by (1),  = . So, (x , y ,  , 1 , 2 ) = (2, 0, , 0, 0).
4 4

Fin500J Topic 4
2/26/2021 Fall 2008
Mhafez 27
Sensitivity Analysis

 We notice that

L (x *, *, *) = f (x *) −  * ( g (x *) − b ) −  * '(h (x *) − c ) = f (x *)

 What happens to the optimal solution value if the

right-hand side of constraint i is changed by a small
amount, say bi , i=1…k. or ci , i=1….m.
 It changes by approximately  * b or i* c i
i i
 i
*is the shadow price of ith inequality constraint and i*
is the shadow price of ith equality constraint

2/26/2021 Mhafez 28
Sensitivity Analysis (Example)
• In the previous example, if we change the first
constraint to x2+y2=3.9, then we predict that the new
optimal value would be 2+1/4(-0.1)=1.975.
• If we compute that problem with this new constraint,
then x-y2= 3.9 = 1.9748.
• If, instead, we change the second constraint from x≥0
to x≥0.1, we do not change the solution or the
optimum value since 1* = 0.

2/26/2021 Mhafez 29
Utility Maximization Example
The utility derived from exercise (X) and watching movies (M)
is described by the function
U ( X,M ) = 100 − e −2 X − e − M
Four hours per day are available to watch movies and exercise.
Our Lagrangian function is
L(X,M,λ) = 100 − e −2 X − e − M −  ( X + M − 4)
First - Order Conditions :
LX = 2e −2 X − λ = 0
LM = e − M − λ = 0
Lλ = X + M − 4 = 0

2/26/2021 Mhafez 30
Utility Max Example Continued

First - Order Conditions :

LX = 2e − 2 X − λ = 0, (1) LM = e − M − λ = 0, (2) Lλ = X + M − 4 = 0 (3)
From (2) we get that λ = e -M . Substituti ng into (1), we get
2e - 2 X − e − M = 0 Solving (3) for M and substituti ng, we get
2e -2X − e −( 4− X ) = 0 → ln(2) - 2X = -4 + X or 3X = ln(2) + 4
ln( 2) + 4 8 − ln( 2)
X* = and M * =
3 3

2/26/2021 Mhafez 31