Chapter 6 Basics of Set-Constrained and

Unconstrained Optimization
An Introduction to Optimization
Spring, 2014

Wei-Ta Chu

1
Introduction
Consider the optimization problem

The function that we wish to minimize is a real-

valued function called the objective function or cost function.
The vector . The variables are
often referred to as decision variables. The set is a subset of
Rn called the constraint set or feasible set.
Finding the “best” vector over all possible vectors in .
This vector is called the minimizer of over . It is possible
that there may be many minimizers.

2
Introduction
There are also optimization problems that require maximization
of the objective function, in which case seek maximizers.
Minimizers and maximizers are also called extremizers.
Maximization problems can be represented equivalently in the
minimization form because maximizing is equivalent to
minimizing .
This problem is a general form of a constrained optimization
problem. If , then we refer to the problem as an
unconstrained optimization problem.

3
Introduction
The constraint is called a set constraint.
Often, the constraint set takes the form

where and are given functions. We refer to such

constraints as functional constraints.

4
Global and Local Minimizers
Definition 6.1: Suppose that is a real-valued
function defined on some set . A point is a local
minimizer of over if there exists such that
for all and . A point is a global
minimizer of over if for all .
If we replace by , then we have a strict local minimizer
and a strict global minimizer, respectively.

local minimizer

5
strict global minimizer strict local minimizer
Global and Local Minimizers
If is a global minimizer of over , we write
and .
If the minimization is unconstrained, we simply write
or
Example: if is given by , then

Strictly speaking, an optimization problem is solved only when

a global minimizer is found. However, global minimizers are,
in general, difficult to find. Therefore, in practice, we often
have to be satisfied with finding local minimizers.

6
Conditions for Local Minimizers
Given a function , recall that the first-order
derivative of , denoted by , is

The gradient is just the transpose of ; that is,

The second derivative of (also called Hessian of ) is

7
Conditions for Local Minimizers
Example: Let

8
Feasible Direction
Definition 6.2: A vector is a feasible direction at
if there exists such that for all
The directional derivative of in the direction , denoted ,
is the real-valued function defined by

is a feasible direction
is a not feasible direction

9
Directional Derivative
If , then is the rate of increase of at in the
direction .
Suppose that and are given. Then is a function
of , and

Applying the chain rule yields

In summary, if is a unit vector, then is the rate of

increase of at the point in the direction .

10
Example
Define by and let

The directional derivative of in the direction

Note that because , the above is also the rate of increase

of at in the direction

11
Theorem 6.1 First-Order Necessary Condition (FONC)
Let be a subset of Rn and a real-valued function on .
If is a local minimizer of over , then for any feasible
direction at , we have

Proof: Define . Note that .

Define the composite function

Then, by Taylor’s theorem

where . Thus, if , that is, for

sufficiently small values of ( is a local minimizer), then
we have to have .

12
First-Order Necessary Condition (FONC)
does not satisfy the FONC, whereas satisfies the
FONC.
input

feasible
directions

function value

13
First-Order Necessary Condition (FONC)
An alternative way to express the FONC is for all
feasible directions .
In other words, if is a local minimizer, then the rate of
increase of at in any feasible direction in is
nonnegative.
Alternative proof: For any feasible direction, there exists
such that for all ,

Hence, for all

Taking the limit , we conclude

14
Corollary 6.1 Interior Case
Let be a subset of Rn and a real-valued function on .
If is a local minimizer of over and if is an interior
point of , then

Proof: Suppose that has a local minimizer that is an

interior point of . The set of feasible directions at is the
whole of Rn. Thus, for any , and
. Hence, for all , which
implies that .

15
Example
Consider the problem

Problem 1: Is the first-order necessary condition for a local

minimizer satisfied at ?
Solution: At , we have
The point is an interior point of . Hence,
the FONC requires that . Therefore, the point
doest not satisfy the FONC for a local minimizer.

Level sets of this function

16
Example
Consider the problem
On the boundary of

Problem 2: Is the FONC for a local minimizer satisfied at

Solution: At , we have , and hence
, where . For to be feasible at ,
we need and can take an arbitrary value in R. The
point does not satisfy the FONC for a minimizer
because is allowed to be less than zero.
For example, is a feasible direction, but

17
Example
Consider the problem
On the boundary of

Problem 3: Is the FONC for a local minimizer satisfied at

Solution: At , we have , and hence
. For to be feasible, we need and
can take an arbitrary value in R. For example, is
a feasible solution. But . Thus,
does not satisfy the FONC for a local minimizer.

18
Example
Consider the problem
On the boundary of

Problem 4: Is the FONC for a local minimizer satisfied at

Solution: At , we have , and hence
. For to be feasible, we need and .
Hence, satisfies the FONC for a local minimizer.

19
Example
There are two base station antennas, one for the primary base
station and another for the neighboring base station. Both
antennas are transmitting signals to the mobile user, at equal
power.
The power of the received signal is reciprocal of the squared
distance from the associated antenna.
Find the position of the mobile that maximizes the signal-to-
interference ratio.

20
Example
The squared distance from the mobile to the primary antenna is
, while the squared distance from the mobile to the
neighboring antenna is . Therefore, the signal-to-
inference ratio is

By the FONC, at the optimal position we have .

Hence, either or . Evaluating the
objective function at these two candidate points, it’s easy to see
that is the optimal solution.

21
Example
Consider the set-constrained problem
where
Problem 1: Consider a point . Specify all feasible
directions at .
Solution: There are no feasible directions at any .

Problem 2: Which points in satisfy the FONC for this set-

constrained problem?
Solution: Because of the solution for Problem 1, all points in
satisfy the FONC for this set-constrained problem.

22
Example
Problem 3: Based on Problem 2, is the FONC for this set-
constrained problem useful for eliminating local-minimizer
candidates?
Solution: No, the FONC for this set-constrained problem is not
useful for eliminating local-minimizer candidates.

23
Example
Problem 4: Suppose that we use polar coordinates to
parameterize points in terms of a single parameter :
and . Now use the FONC for unconstrained
problems (with respect to ) to derive a necessary condition of
this sort: if is a local minimizer, then for
all satisfying a “certain condition.” Specify what this certain
condition is.
Solution:
Write , where is given by the equations
relating to . Note that . Hence,

Notice that is tangent to at . Alternatively, we

could say that is orthogonal to .

24
Example
Solution:
Suppose that is a local minimizer. Write . Then
is an unconstrained minimizer of . By the FONC for unconstrained
problems, , which implies that for all
tangent to at (or, alternatively, for all orthogonal to )

Same as the Corollary 6.1 Interior Case

25
Theorem 6.2 Second-Order Necessary Condition
(SONC)
Let , a function on , a local minimizer of
over , and a feasible direction at . If , then
Corollary 6.1 Interior Case
where is the Hessian of .
Proof: We prove the result by contradiction. Suppose that there
is a feasible direction at such that and
. Let and define the composite
function .
By Taylor’s theorem,

where by assumption, and

.

26
Theorem 6.2 Second-Order Necessary Condition
(SONC)
For sufficiently small ,

that is,

which contradicts the assumption that is a local minimizer.

Thus,

27
Corollary 6.2 Interior Case
Let be an interior point of . If is a local
minimizer of , , then , and
is positive semidefinite ; that is, for all ,

Proof: If is an interior point, then all directions are feasible.

The result then follows from Corollary 6.1 and Theorem 6.2.

28
Example
Consider a function of one variable
Because and , the point satisfies both
the FONC and SONC. They are necessary conditions, but are
However, is not a minimizer. not sufficient conditions.

29
Example
Consider a function . The FONC
requires that . Thus, satisfies
the FONC.
The Hessian matrix of is
The Hessian matrix is indefinite; that is, for some we
have (e.g., ); and, for some , we have
(e.g., ). Thus, does not satisfy
the SONC, and hence it is not a minimizer.

30
 Theorem 6.3 Second-Order Sufficient Condition
(SOSC), Interior Case
Let be defined on a region in which is an interior
point. Suppose that and . Then, is a
strict local minimizer of .
Proof: Because , we have . Using
assumption 2 and Rayleigh’s inequality it follows that if
then . By the Taylor’s theorem
and assumption 1,

(p. 27)
Hence, for all such that is sufficiently small,

which completes the proof.

31
Example
Let . We have if and only if
. For all , we have

The point satisfies the FONC, SONC, and SOSC. It

is a strict local minimizer.
Actually, is a strict global minimizer.

32
Exercise 6.19
An amphibian vehicle needs to travel from point A (on land) to
point B (in water). The speeds at which the vehicle travels on
land and water are and , respectively.
Suppose that the vehicle traverses a path that minimizes the total time
taken to travel from A to B. Use the FONC to show that for the
optimal path above, the angles and satisfy Snell’s law:

Does the minimizer for the

problem in part a satisfy
the second-order sufficient
condition?

33
Exercise 6.19
Let be the decision variable. Write the total travel time as
which is given by

Differentiating the expression

By the first-order necessary condition, the optimal path

satisfies , which corresponds to

which leads to

34
Exercise 6.19
The second derivative of is given by

Hence, , which shows that the second order sufficient

condition holds.

35
Exercise 6.20
Suppose that you have a piece of land to sell and you have two
buyers. If the first buyer receives a fraction of the piece of
land, the buyer will pay you dollars. Similarly, the
second buyer will pay you dollars for a fraction of of
the land. You goal is to sell parts of your land to the two buyers
so that you maximize the total dollars you receive.
Formulate the problem as an optimization problem of the kind

Suppose that where and are given

positive constants such that . Find all feasible points that
satisfy the first-order necessary condition.
Among those points in the answer of part b, find all that also satisfy
the second-order necessary condition.
36
Exercise 6.20
We have and
We have . Because for all , we
conclude that no interior points satisfy the FONC. Next,
consider any feasible point for . At such a point, the
vector is a feasible direction. But
which means that FONC is violated (recall that the problem is
to maximize ). So clearly the remaining candidates are those
for which . Among these, if , then is a
feasible direction, in which case we have .
This leaves the point . At this point, any feasible
direction satisfies and . Hence, for any feasible
direction, we have

So, the only feasible point satisfies FONC is

37
Exercise 6.20
We have . Hence, any point satisfies SONC (again,
recall that the problem is to maximize ).

38