Numerical Optimization

Lecture 6: Optimality Conditions for Convex Functions and

Existence of Solutions

Sangkyun Lee

The content is from Nocedal and Wright (2006), Bertsekas (2003), and
Sundaram (1996). Topics marked with ** are optional.

The Use of Optimality Conditions

min f (x)

xRn

Note that both necessary and sufficient conditions are required for algorithm
design. That is, for a given point x and an objective function f continuously
differentiable near x , FONC tells that
f (x ) 6= 0 x is NOT a local minimizer,
so an algorithm continues to search for a minimizer. On the other hand,
SOSC tells that (when f is twice continuously differentiable near x ),
f (x ) = 0 2 f (x )  0 x is a strict local minimizer.
Therefore the algorithm can return x as a solution.

1.1

Restrictions of SOSC

SOSC is rather restrictive because it is only able to characterize a strict local

minimizer. Why the points x such that f (x ) = 0 2 f (x )  0 are not
necessarily local minimizers? It is because there could exist a vector d Rn ,
d 6= 0, such that dT 2 f (x )d = 0 (e.g. the eigenvector(s) corresponding
to the eigenvalue 0), for which the quadratic model from Taylors theorem
cannot tell if the function is increasing or decreasing as we move away from
x in the direction of d:
1
f (x + d) f (x ) = f (x )T d + dT 2 f (x )d + o(kdk22 ) = o(kdk22 ).
2
However, when the objective function is convex a much simpler characterization is possible.
1

Properties of Convex Functions

Definition 6.1 (Convex Function). Let C a convex subset of Rn . A function

f : C R is convex if
f (x + (1 )y) f (x) + (1 )f (y), x, y C, [0, 1].
f is called concave if f is convex.
f is called strictly convex if the above inequality is strict for all x, y C
with x 6= y.
For a function f : Rn R, we say f is convex over the convex set C
if the above inequality holds.
**For a function f : C (, ], C is a convex subset of Rn , is also
called convex if the above inequality holds (here we apply extended
arithmetic rules: + = , 0 = 0, = if > 0.
Lemma 6.2 (Jensens Inequality1 ). For a convex function f : C R,
C is a convex subset of Rn , if x1 , . . . , xm C and 1 , . . . , m 0 with
P
m
i=1 i = 1, then
!
m
m
X
X
f
i xi
i f (xi ).
i=1

i=1

This is a special case of a more general inequality,

Z
Z
f (E[X]) E[f (X)], where E[X] =
x p(x)dx,
p(x)dx = 1.

Optimality Conditions for Convex Functions

Theorem 6.3. When f is convex, any local minimizer x is a global minimizer of f .

Proof. For contradiction, suppose that x is a local minimizer but not a
global minimizer of f . Then we can find a point z Rn such that f (z) <
f (x ). Consider a point x in the line segment between x and z,
x = z + (1 )x , for some (0, 1].
By the convexity of f ,
f (x) f (z) + (1 )f (x ) < f (x ).
1

The inequalities of the opposite direction hold when f is a concave function.

(1)

Since any neighborhood N of x contains a piece of the line segment between

x and z, so there will always exist a point x N (x 6= x ) at which (1)
is satisfied. That is, x is not a local minimizer. This contradicts our
assumption that x is a local minimizer. Therefore, x must be a global
minimizer.
Now we discuss the first-order sufficient conditions for convex functions.
Remember that FONC still holds for convex functions.
Theorem 6.4 (FOSC for Convex Functions). If f is convex and differentiable, then any stationary point x is a global minimizer of f .
Proof. Suppose for contradiction that a global minimizer exists but x is
not a global minimizer, even though x is a stationary point. Then we can
find a point z Rn such that f (z) < f (x ). From convexity of f , we get
d
f (x + (z x ))|=0
d
f (x + (z x )) f (x )
= lim
0

f (z) + (1 )f (x ) f (x )
(for (0, 1))
lim
0

= f (z) f (x ) < 0.

(z x )T x f (x ) =

Therefore, f (x ) 6= 0, and so x is not a stationary point.

Note. Theorem 6.42 and FONC3 imply that when f is convex and continuously differentiable, then f (x ) = 0 is a necessary and sufficient condition
for x Rn to be a global minimizer of f : Rn R.

When Does a Minimizer Exist?

Consider the minimization problem

min f (x).

xXRn

We ask if a point x X exists such that

f (x ) f (x) x X, i.e. f (x ) = inf f (x).
xX

If such a point
X exists, then we say f attains a minimum over X at

x , and we write
f (x ) = min f (x), x argminxX f (x).
xX

2
3

Theorem 6.4 does not require continuous differentiability

FONC requires continuous differentiability

If x is known to be the unique minimizer of f over X, then we also write

x = argminxX f (x).
Definition 6.5 (Coercive Functions). A funciton f : X Rn R is called
coercive if for every sequence {xk } X such that kxk k2 , we have
limk f (xk ) = .
Lemma 6.6. 4 Let X be a closed subset of Rn and let f : X R be
continuous at all points in X. Then the level set
L (f ) := {x X : f (x) }
is closed for all Rn .
Lemma 6.7. Let f : X R be continuous at all points of X Rn . If X
is compact, then the image F := {f (x) : x X} is also compact.
Proof. Recall (from lecture 2) that the set F is compact if every sequence
in F has at least one limit point, and all such points are in F .
Pick any sequence {yk } in F . For each k, pick xk X such that f (xk ) =
yk (for each k, at least one such point much exists by the definition of f ; if
more than one exists, pick any one). Then {xk } is a sequence in X, and since
X is compact there exists a subsequence {xki }
i=1 such that limi xki = x
such that x X (x is a limit point of {xk }).
Define y = f (x), and yki = f (xki ). By construction, {yki } is a subsequence of {yk }. Also, since x X, y = f (x) F . Finally, since {xki } x
as i and f is continuous, we have yki = f (xki ) f (x) = y. This shows
that an arbitrary sequence {yk } in F has a limit point which is within F .
Therefore F is compact.
Lemma 6.8. If F R is nonempty and compact, then inf F F and
sup F F .
Proof. We prove for the case of inf. Since F is nonempty and bounded,
inf F (, ). Define intervals Ik = [inf F, inf F + 1/k) and let Fk =
Nk F . Then Fk is nonempty for each k, since otherwise we can find an
lower bound of F which is strictly larger than inf F .
Pick any point fk from Fk . Since fk [inf F, inf F + 1/k) for each k, we
have |fk inf F | < 1/k so that fk inf F as k . Moreover, fk Fk F
and F is a closed set, so the limit of the sequence {fk } must be in F . That
is, inf F F .
4

In fact, the lemma holds for a weaker condition that f is lower semicontinuous.

Theorem 6.9 (Weierstrass Theorem5 ). Let X 6= be a subset of Rn and

let f : X R be continuous at all points of X. If one of the following
conditions holds, then the set of minimizers of f over X is nonempty and
compact.
(a) X is compact.
(b) X is closed and f is coercive.
(c) There exists a scalar such that the level set L (f ) is nonempty and
compact.
Proof. ** (i) Assume that the condition (a) is true. From the proofs of
lemmas 6.7 and 6.8, we observe that we can find a sequence {xk } in X such
that
lim f (xk ) = inf f (x).
xX

Since X is compact, this sequence has at least one limit point x X. In

other words, there exists a subsequence {xki } of {xk } such that limi xki =
x . Moreover, since f is continuous, we have limi f (xki ) = f (x ) =
inf xX f (x). This shows that there exists x X at which f attains a
minimum over X (therefore the set of minimizers is nonempty).
The set of alll minimizers of f over X is the level set {x X : f (x)
inf zX f (z)}, which is closed since f is continuous (Lemma 6.6) and bounded
since X is bounded. Thus it is a compact set.
(ii) Assume that the condition (b) is true. The sequence {xk } in the
proof of part (a) should be bounded, since f is coercive (if {xk } is unbounded, then limk f (xk ) = since f is coercive, but this contradicts
that limk f (xk ) = inf xX f (x) < : finiteness comes from the fact that
f : X R and X is nonempty.) The rest of the proof is similar to that of
part (a).
(iii) Assume that the condition (c) is true. If = inf xX f (x), then the
set of minimizers of f over X is L (f ), and the proof is done since this set is
nonempty and compact by the assumption. If > inf xX f (x), consider the
sequence {xk } in the proof of part (a). Then there exists K > 0 such that
for all k K, xk must belong to L (f ). Since L (f ) is compact, {zk }kK
must be bounded and the proof becomes similar to that of part (a).
Note that the conditions (a), (b), and (c) are only sufficient conditions.

References
Bertsekas, D. P. (2003). Nonlinear Programming. Athena Scientific, second
edition.
5

This theorem holds for a weaker condition that f is lower semicontinuous.

Nocedal, J. and Wright, S. J. (2006). Numerical Optimization. Springer,

second edition.
Sundaram, R. K. (1996). A First Course in Optimization Theory. Cambridge University Press.