OPTIMIZATION METHODS

01202N

Lectures Notes

Lecture 7

Fall 2024

1
 Nonlinear optimization
Unconstrained optimization problem
1. Function of one variable:
min 𝒇(𝒙), where 𝒙 ∈ 𝑹

Let 𝑓(𝑥) be a continuous function of a variable 𝑥

• The points 𝑏 and 𝑑 are local points.

• The point 𝑑 is a local min.
• The point 𝑏 is a global min.
Definition: A function 𝑓 has a local minimum value at a point 𝒅 within its domain D
if 𝑓(𝑑) ≤ 𝑓(𝑥), ∀𝑥 ∈ (𝑑 − 𝜀, 𝑑 + 𝜀), for some 𝜀 > 0.
A function 𝑓 has a global minimum value at a point 𝒃 if 𝑓(𝑏) ≤ 𝑓(𝑥) ∀𝑥 ∈ 𝐷.
Theorem 1.
If 𝑓 is continuous function on a defined closed interval, [𝑎, 𝑏] then 𝑓 attains a global
minimum value and a global maximum value.
Theorem 2.
If a function 𝑓 has a local maximum or a local minimum at a point 𝑥 = 𝑐 and if 𝑓′(𝑐)
exists, then 𝑓 ′ (𝑐) = 0

Definition: A critical point of a function 𝑓 is a point 𝑐 in the domain of 𝑓 such that,

either 𝑓 ′ (𝑐) = 0 or 𝑓 ′ (𝑐) doesn’t exist.

Example 1: Find all critical points of 𝑓(𝑥) = 3𝑥 − 𝑥 2 .

Solution: 𝑓(𝑥) = 3𝑥 − 𝑥 2
𝑓 ′ (𝑥) = 3 − 2𝑥
Set 𝑓 ′ (𝑥) = 0 → 𝑥 = 1.5
Therefore, the function has one critical point 𝑥 = 1.5.
Note: 𝑓 ′ (𝑐) = 0 is not enough to say that c is a min. point or max. point.

2
 Example 2: The function 𝑓(𝑥) = 𝑥 3 has a critical point at 𝑥 = 0 because
𝑓 ′ (𝑥) = 3𝑥 2 and 𝑓 ′ (𝑥) = 0 → 𝑥 = 0. But the point 𝑥 = 0 is not a max. point nor a
min. point.

Theorem 3.
If a function 𝑓 has a local maximum or a local minimum at a point 𝑥 = 𝑥0 , then 𝑥0 is a
critical point.
The converse is not true.

Min. or max point critical point

What does 𝒇′ say about 𝒇?

Suppose that 𝒇 is continuous on an interval 𝐼.
1) If 𝑓 ′ (𝑥) > 0 on the interval 𝐼, then 𝑓 is increasing in 𝐼.
2) If 𝑓 ′ (𝑥) < 0 on the interval 𝐼, then 𝑓 is decreasing in 𝐼.

Example 3: Find the critical points of 𝑓(𝑥) = 3𝑥 4 − 4𝑥 3 − 12𝑥 2 + 5 and identify the
intervals of on which 𝑓 is increasing and on which is decreasing.
Solution:
𝑓(𝑥) = 3𝑥 4 − 4𝑥 3 − 12𝑥 2 + 5
𝑓 ′ (𝑥) = 12𝑥 3 − 12𝑥 2 − 24𝑥
= 12𝑥(𝑥 2 − 𝑥 − 2)
= 12𝑥(𝑥 − 2)(𝑥 + 1)
The critical points are 𝑥 = 0, 𝑥 = 2, 𝑎𝑛𝑑 𝑥 = −1.

3
 Intervals −∞ < 𝑥 < −1 −1 < 𝑥 < 0 0<𝑥<2 2<𝑥<∞
12𝑥 - - + +
(𝑥 − 2) - - - +
(𝑥 + 1) - + + +
Sign of𝑓 ′ - + - +
Behavior of 𝑓 Decreasing increasing Decreasing increasing

• Intervals of increase of 𝑓 (−1,0) and (2, ∞)

• Intervals of decrease of 𝑓 (−∞, −1) and (0,2).

50
40
30
20
10 local max
0
-3 -2 -1 -10 0 1 2 3 4
local
min -20
-30
-40 local
min

The first derivative test:

Suppose that 𝑐 is a critical point of a continuous function 𝑓(𝑥), then
a) If 𝑓 ′ changes from negative to positive at 𝑥 = 𝑐, then 𝑓 has a local minimum at 𝑐.
b) If 𝑓 ′ changes from positive to negative at 𝑥 = 𝑐, then 𝑓 has a local maximum at 𝑐.
c) if 𝑓 ′ does not change sign at 𝑥 = 𝑐 (i.e, if 𝑓 ′ is positive on both sides of 𝑐 or
negative on both sides of 𝑐) then 𝑓 has no local maximum or minimum at 𝑐.

Example 4: Consider 𝑓(𝑥) = 𝑥 3 , 𝑓 ′ (𝑥) = 3𝑥 2. 𝑓 ′ (−0.1) > 0 & 𝑓 ′ (0.1) > 0.

𝑓 ′ (𝑥) is positive on both sides of the point 𝑥 = 0, then 𝑥 = 0 is no max. no a min.

Definition: Let 𝑓 be a continuous function defined on an interval I, then

1) 𝑓 is called convex if every line segment joining two distinct points on the graph of
the function is never below the graph.
2) 𝑓 is called concave if every line segment joining two distinct points on the graph
of the function is never above the graph.

4
 What does 𝒇′′ say about 𝒇?
Let 𝑦 = 𝑓(𝑥) be twice differentiable on an interval I
1) If 𝑓 ′′ > 0 on the interval 𝐼, the graph of 𝑓 over I is convex.
2) If 𝑓 ′′ < 0 on the interval 𝐼, the graph of 𝑓 over I is concave.

Example 5: Discuss the convexity of 𝑓(𝑥) = 𝑥 3 − 𝑥 2

Solution:
𝑓(𝑥) = 𝑥 3 − 𝑥 2
𝑓 ′ (𝑥) = 3𝑥 2 − 2𝑥
1
𝑓 ′′ (𝑥) = 6𝑥 − 2 = 6 (𝑥 − )
3

1
− +
3

1 1
So, 𝑓 is convex on ( , ∞) and concave on (−∞, ).
3 3
The curve y = 𝑥 3 − 𝑥 2 in Example 5 changes concavity at the point 𝑥 = 1/3 .
This point is called a point of inflection of the curve.
Definition: Points of Inflection
it is a point where the function changes from being concave to convex, or vice versa.

5
 The second derivative test:
Suppose that 𝑓 ′′ is continuous on an open interval that contains 𝑥 = 𝑐
a) If 𝑓 ′ (𝑐) = 0 𝑎𝑛𝑑 𝑓 ′′ (𝑐) > 0 , then 𝑓 has a local minimum at 𝑐 .
b) If 𝑓 ′ (𝑐) = 0 𝑎𝑛𝑑 𝑓 ′′ (𝑐) < 0 , then 𝑓 has a local maximum at 𝑐
c) If 𝑓 ′ (𝑐) = 0 𝑎𝑛𝑑 𝑓 ′′ (𝑐) = 0 , then the test fails. The function 𝑓 may have a local
maximum, a local minimum, or neither.

Example 6: Discuss the convexity of 𝑓(𝑥) = 𝑥 4 − 4𝑥 3 , and classify all critical points.
Sketch a graph of the function.
Solution:
𝑓(𝑥) = 𝑥 4 − 4𝑥 3
𝑓 ′ (𝑥) = 4𝑥 3 − 12𝑥 2 = 4𝑥 2 (𝑥 − 3)
𝑓 ′′ (𝑥) = 12𝑥 2 − 24𝑥 = 12𝑥(𝑥 − 2)
• The critical points are 𝑥 = 0, 𝑥 = 3
• 𝑓 ′′ (𝑥) = 12𝑥 2 − 24𝑥 = 12𝑥(𝑥 − 2) = 0 → 𝑥 = 0 & 𝑥 = 2
We use these points to define intervals where f is convex or concave.
Intervals 𝑥<0 0<𝑥<2 2<𝑥
12𝑥 - + +
(𝑥 − 2) - - +
′′
Sign of 𝑓 + - +
Behavior of 𝑓 Convex concave convex

The function is convex on (−∞, 0) ∪ (2, ∞) and the function is concave on (0,2).
Using the second derivative test for local extrema, since
𝑓 ′ (3) = 0 and 𝑓 ′′ (3) > 0, then the point 𝑥 = 3 is a local min.
Now, 𝑓 ′ (0) = 0 and 𝑓 ′′ (0) = 0, then no conclusion can be made by the second derivative
test.
By the first derivative test, 𝑓 ′ (0) = 0 and 𝑓 ′ (−0.1) < 0, 𝑓 ′ (0.1) < 0. Then the critical
point 𝑥 = 3 is no max no min.
The general shape of the curve is shown in the accompanying Figure.

6
7
 2. Function of two variables:
Now, we consider the function 𝑓 of two variables, 𝑥 and 𝑦, where 𝑥 ∈ 𝑅 and 𝑦 ∈ 𝑅.
Consider

min 𝑓(𝑥, 𝑦), where 𝑥 ∈ 𝑅 and 𝑦 ∈ 𝑅

Critical point:

The critical point is the point that satisfies

𝜕𝑓 𝜕𝑓
= 0 and =0
𝜕𝑥 𝜕𝑦

Example 1: Find the critical points of 𝑓(𝑥, 𝑦) = 𝑥 2 − 2𝑥𝑦 − 2𝑥 + 2𝑦 2

Solution:
𝜕𝑓
= 0 → 2𝑥 − 2𝑦 − 2 = 0 → (1)
𝜕𝑥
𝜕𝑓
= 0 → −2𝑥 + 4𝑦 = 0 → (2)
𝜕𝑦

Adding (1) & (2), we get

2𝑦 − 2 = 0 → 2𝑦 = 2 → 𝑦 = 1

𝑠𝑢𝑏𝑠𝑡𝑢𝑡𝑖𝑛𝑔 in (2), −2𝑥 + 4 = 0 → −2𝑥 = −4 → 𝑥 = 2

∴ the critical point is (2,1).

Second Derivatives test:

2
Suppose that (𝑐, 𝑑) is a critical point of 𝑓. Let 𝐷(𝑐, 𝑑) = 𝑓𝑥𝑥 (𝑐, 𝑑) 𝑓𝑦𝑦 (𝑐, 𝑑) − 𝑓𝑥𝑦 (𝑐, 𝑑)

1) If 𝐷(𝑐, 𝑑) > 0 and 𝑓𝑥𝑥 (𝑐, 𝑑) > 0, then 𝑓 has a local min. at (𝑐, 𝑑).

2) If 𝐷(𝑐, 𝑑) > 0 and 𝑓𝑥𝑥 (𝑐, 𝑑) < 0, then 𝑓 has a local max. at (𝑐, 𝑑).

3) If 𝐷(𝑐, 𝑑) < 0, then 𝑓 has a saddle point at (𝑐, 𝑑), (see the following Figure)

4) If 𝐷(𝑐, 𝑑) = 0, then the test is inconclusive at (𝑐, 𝑑).

8
 Example 2: classify the critical point of the last example

Solution: from the last example (2,1) is a critical point

We calculate, 𝑓𝑥𝑥 = 2, 𝑓𝑦𝑦 = 4 and 𝑓𝑥𝑦 = −2

2
𝐷(2,1) = 𝑓𝑥𝑥 𝑓𝑦𝑦 − 𝑓𝑥𝑦 = 8 − (−2)2 = 4 > 0 and 𝑓𝑥𝑥 = 2 > 0

∴ The critical point (2,1) is a min. point

Example 3: Find all local minimum, local maximum, and saddle points of

𝑓(𝑥, 𝑦) = 3𝑥 2 + 6𝑥𝑦 + 2𝑦 2 + 4𝑥 + 2𝑦 + 15

Solution:

𝑓𝑥 = 0 → 6𝑥 + 6𝑦 + 4 = 0 (1)

𝑓𝑦 = 0 → 6𝑥 + 4𝑦 + 2 = 0 (2)

(1) − (2) → 2𝑦 + 2 = 0 → 2𝑦 = −2 → 𝑦 = −1
1
Substituting in (2) → 6𝑥 − 4 + 2 = 0 → 6𝑥 = 2 → 𝑥 =
3

1
The critical point is ( , −1)
3

Now, 𝑓𝑥𝑥 = 6, 𝑓𝑦𝑦 = 4 and 𝑓𝑥𝑦 = 6

2
𝐷 = 𝑓𝑥𝑥 𝑓𝑦𝑦 − 𝑓𝑥𝑦 = 6 ∗ 4 − (6)2

= 24 − 36 = −12 < 0

∴ The point is a saddle point

9
10