E1

 251  Linear  and  Nonlinear  

Op2miza2on  
 
Chapter  3:    Condi/ons  of  maxima  and  
minima  

1  
3.1    Defini2on  of  extremum  

3.1.1  Defini2on:    global  minimum  

A point x * is a global minimizer of a function f (x)

if f (x) ≥ f (x * ) for all x.

f (x)

*
x
x 2  
3.1.2  Defini2on:      local  minimum  

A point x * is a local minimizer of a function f (x)

if there exists ε > 0 such that f (x) ≥ f (x * ) for
all x satisfying x − x * < ε .

!2ε 1

2ε 2
f (x)

*
x
x 3  
3.1.3  Defini2on:    strict  local  minimum  

A point x * is a strict local minimizer of a function f (x)

if there exists ε > 0 such that f (x) > f (x * ) for
all x satisfying 0< x − x * < ε .

f (x)

* *
x
x x 4  
3.2  Review  of  single  variable  minimiza2on  

A point x * is a minimizer or maximizer of a function f (x)

d
if (1) f ′(x) = f (x * ) = 0.
dx

If (1) is satisfied:
2
d
(2.a) if f ′′(x * ) = 2 f (x * ) < 0, then x * is a maximizer
dx
(2.b) if f ′′(x * ) > 0, then x * is a minimizer
(2.c) if f ′′(x * ) = 0, then second derivative test is
insufficient.

5  
Can  you  minimize    this  ?  

f (x) = 4 x + 3

f (x)
3

−3 / 4
x

6  
Can  you  minimize    this  ?  

f (x) = x + ax + b
2

f ′(x * ) = 2x * + a = 0 ⇒ x * = −a / 2
f ′′(x) = 2 = f ′′(x * )
Second derivative test is satisfied for x * = −a / 2
⇒ x * = −a / 2 is a minimizer f (x)

2
a
f (x * ) = f (a / 2) = b −
4
⎛ a2 ⎞
2
⎛ a⎞
f (x) = ⎜ x + ⎟⎠ + ⎜ b − ⎟
⎝ 2 ⎝ 4⎠
x
7  
Can  you  minimize    this  ?  

f (x) = x 3
f ′(x) = 3x = 0 ⇒ x = 0 (a candidate minimizer)
2 *

f ′′(x) = 6x
f ′′(x * ) = f ′′(0) = 0 ⇒ x * = 0 is not a minimizer

f (x)

The  func2on  has  no  minimum  at  all.  

8  
Can  you  minimize    this  ?  

⎧ 0, for x ≤ a
⎪
⎪
f (x) = ⎨ (x − a)2 , for x > a
⎪
⎪ (x + a) 2
, for x < −a
⎩

f (x)

−a a x
9  
 f (x)
f ′(x)

f ′′(x)
−a a
−a a

−a a

10  
3.3  Introducing  FONC,  SONC,  and  SOSC  

A point x * is a minimizer or maximizer of a function f (x)

d
if (1) f ′(x) = f (x * ) = 0
dx
(First order necessary condition-FONC)

(2.a) if f ′′(x * ) < 0, then x * is a maximizer

(Second order sufficient condition for strict maximum-SOSC-SMax)
(2.b) if f ′′(x * ) > 0, then x * is a minimizer
(Second order sufficient condition for strict minimum-SOSC-SMin)
(2.c) if f ′′(x * ) = 0, then second derivative test is
insufficient

11  
Second  order  necessary  condi/ons:  

(2.a) if f ′′(x * ) ≤ 0, then x * is a maximizer

(Second order necessary condition for maximum-SONC-Max)
(2.b) if f ′′(x * ) ≥ 0, then x * is a minimizer
(Second order necessary condition for minimum-SONC-Min)

12  
Look  at    these  examples…  

⎧ f3 (x) = x 3
f1 (x) = x 2 0, for x ≤ a
⎪
⎪
f2 (x) = ⎨ (x − a)2 , for x > a
⎪
⎪ (x + a) 2
, for x < −a
⎩
(1) All satisfy FONC and SONC-Min
(2) Only f1 (x) satisfies SOSC-SMin
(3) f1 (x), f2 (x) and f4 (x) have minima and f3 (x) does
not have any minimum
f4 (x) = x 4 (4) f1 (x) and f4 (x) have strict minima.
13  
Examples:  
Least  squares  approxima2on:  
Given a set of numbers {a1 , a2 ,…, aN } , find x minimizing
2

( )
N
f (x) = ∑ x − a j
j =1

Applying FONC:
N
1
( ) ⇒ x* = ∑ a j
N
f ′(x * ) = 2∑ x * − a j = 0
j =1
N j =1
f ′′(x) = 2N ⇒ SOSC-SMin is satisfied
Weighted  Least  squares  approxima2on:  
Given a set of numbers {a1 , a2 ,…, aN } , and their confidence
estimates {c1, c2 ,…, cN } , find x minimizing
2

( )
N
f (x) = ∑ c j x − a j
j =1
14  
3.4  Proofs  for  single  variable  problem  

Recap:    Taylor’s  Theorem  

For any p < m (order of continuity), we can write

h (1) h 2 (2 )
f (b) = f (a) + f (a) + f (a) +
1! 2!
h( p ) ( p )
+ f (a) + o(h p )
p!
where o(h p ) is a term that tends to zero
faster than h p as h tends to zero.

15  
3.4.1  Proof  of    FONC-­‐Min  

Apply definition of local minimum:

f (x * + ε ) − f (x * ) ≥ 0 and f (x * − ε ) − f (x * ) ≥ 0

Apply Taylor's theorem with p=1,a = x * , b = x * + ε , h = ε

f (x * + ε ) − f (x * ) = ε f ′(x * ) + o(ε )
!##"##$ !# #"## $
T1 T2

Apply Taylor's theorem with p=1, a = x * , b = x * − ε , h = − ε

f (x * − ε ) − f (x * ) = − ε f ′(x * ) + o(ε )
!##"##$ !##"##$
T3 T3

ε f ′(x * ) and -ε f ′(x * ) are non-negative ⇒ f ′(x * ) = 0

16  
3.4.2  Proof  of  second  order  necessary  condi/on  (SONC)  

Apply Taylor's theorem and FONC

with p=2,a = x * , b = x * + ε , h = ε
f (x * + ε ) − f (x * ) = (ε 2 / 2) f ′′(x * ) + o(ε 2 )
!##"##$
T1

Apply Taylor's theorem and FONC

with p=2, a = x * , b = x * − ε , h = − ε
f (x * − ε ) − f (x * ) = (ε 2 / 2) f ′′(x * ) + o(ε 2 )
!##"##$
T2

If x * is a minimum, then T1 and T2 are non-negative,

which implies that f ′′(x ) non-negative.
*

17  
3.4.3  Proof  of  second  order  sufficient  condi/on  (SOSC)  

Apply Taylor's theorem and FONC

with p=2,a = x * , b = x * + ε , h = ε
f (x * + ε ) − f (x * ) = (ε 2 / 2) f ′′(x * ) + o(ε 2 )
!##"##$
T1

Apply Taylor's theorem and FONC

with p=2, a = x * , b = x * − ε , h = − ε
f (x * − ε ) − f (x * ) = (ε 2 / 2) f ′′(x * ) + o(ε 2 )
!##"##$
T2

If f′′(x * ) is positive, then T1 and T2 are positive,

which implies that x * is a strict minimum.
18  
3.5  Condi2ons  of  extremum  for  mul2variable  func2on  
3.5.1  Defini/ons  

A point x* is a local minimizer of a function f (x)

if there exists ε > 0 such that f (x) ≥ f (x* ) for
all x satisfying x − x* < ε .

A point x* is a strong(or strict) local minimizer of a function f (x)

if there exists ε > 0 such that f (x) > f (x* ) for
all x satisfying 0< x − x* < ε .

19  
Posi2ve  semi-­‐definite  matrices  

An n × n matrix, M, is called positive semi-definite if

v T Mv ≥ 0, for every v ∈ n

Posi2ve  definite  matrices  

An n × n matrix, M, is called positive definite if

v T Mv > 0, for every v ∈ n

20  
An n × n symmetric matrix, M, is positive semi-definite if
and only if all the eigen values of M are non-negative.

An n × n symmetric matrix, M, is positive definite if

and only if all the eigen values of M are positive.

21  
3.5.2  The  condi/ons  
First  order  necessary  condi/on  (FONC):  

A point x* is a local minimizer of a function f (x)

if there exists ε > 0 such that f (x) ≥ f (x* ) for
all x satisfying x − x* < ε .

x* is a local minimizer only if ∇f (x* ) = 0

22  
Second  order  necessary  condi/on  for  local    
Minimum  (SONC-­‐Min):  

A point x* is a local minimum of a function f (x),

*
only if its Hessian, F(x ), is positive semi-definite.

Second  order  sufficient  condi/on  for      strong  local    

Minimum  (SOSC-­‐Min):  

*
A point x is a strong local minimum of a function f (x),
if ∇f (x ) = 0 and its Hessian, F(x ), is positive
* *

definite. 23  
Exercises    
Find points satisfying FONC and SONC of the function
f (x1 , x2 ) = x12 + x22
f (x1 , x2 ) = x12 − x22
f (x1 , x2 ) = x1 x2
f (x1 , x2 , x3 ) = x1 x2 + x2 x3 + x1 x3

Given a set of vectors {a (1) , a (2 ) ,..., a ( N ) } , find x minimizing

N 2

f (x) = ∑ x − a ( j )
j=1

24  
3.6  Proofs  of  Condi2ons  of  extremum  for  mul2variable  func2on  

3.6.1  Re-­‐interpre/ng  the  defini/on  

A point x* is a local minimizer of a function f (x) if there exists ε > 0 such
that f (x) ≥ f (x* ) for all x satisfying x − x* < ε .

which implies that

A point x* is a local minimizer of a function f (x) if there exists ε > 0 such
that f (x* + α d) ≥ f (x* ) for all d and α satisfying d = 1 and 0 ≤ α ≤ ε .

which again implies that

A point x* is a local minimizer of a function f (x) if the class of
functions g[ x* ,d ] (α ) = f (x* + α d) has a local minimum at α = 0
for all d satisfying d = 1.
25  
3.6.2  The  proofs  
(1) First order necessay condition (FONC)
n
∂
g[′x* ,d ] (0) =∑ dk f (x* ) = 0, ∀ d s. t. d =1
k=1 ∂xk
⇒ d T ∇f (x* ) = 0, ∀ d s. t. d = 1 ⇒ ∇f (x* ) = 0.

2a) Second order necessary condition for minimum (SONC-Min)

n n
∂2
g[′′x,d ] (0) =∑ ∑ d j dk f (x* ) ≥ 0, ∀ d s. t. d =1
j=1 k=1 ∂x j ∂xk
⇒ d T F(x* )d ≥ 0, ∀ d s. t. d = 1 ⇒ Postive semi-definiteness of Hessian at x* .

2b) Second order sufficient condition for strong minimum (SOSC-SMin)

n n
∂2
g[′′x,d ] (0) =∑ ∑ d j dk f (x* ) > 0, ∀ d s. t. d =1
j=1 k=1 ∂x j ∂xk
⇒ d T F(x* )d > 0, ∀ d s. t. d = 1 ⇒ Postive definiteness of Hessian at x* .
26  
3.7  Test  of  posi2ve  definiteness  
3.7.1  Eigen  value  test  

A symmetric matrix is positive definite is all of its Eigen values are

positive.
A symmetric matrix is positive semi-definite is all of its Eigen values
are non-negative.

3.7.2    Sylvester  criterion  

A symmetric matrix is positive definite iff all the leading principal

minors are positive.
A symmetric matrix is positive semi-definite iff all the principal
minors are nonnegative.

27