FENG 346

NUMERICAL METHODS FOR ENGINEERS II

UNCONSTRAINED AND NONLINEAR
OPTIMIZATION PROBLEMS
𝑓 (𝑥 , 𝑦)

Asst. Prof. M. ÖZGÜN GÜLEÇ

Izmir University of Economics
Mechatronics Engineering

0507 705 8644

[email protected]
OUTLINE

•Theory of Maxima and Minima

•Single Variable Problems and Examples

•Multivariable Problems and Examples

2
Theory of Maxima and Minima

The theory of maxima and minima focuses on the solution of nonlinear

parameter optimization problems, based on the differential calculus developed
by Leibniz and Newton at the same period of time, unaware of each other.

The theory can be used both constrained and unconstrained optimization

problems.

3
Single Variable Problems
Minimum
Suppose that is a single degree variable nonlinear
objective function. .
Assume that is a continuous and at least 2 times
differentiable nonlinear function. Let’s determine the
conditions for the existence of an extremum (minimum or
maximum) at the point . Assume that is an infinitesimally
small and positive real number;

and
Then Maximum

4
Single Variable Problems
Minimum
Taylor Series Expansion of (x) around is;

Substitute to find in eqn (1)

Then substitute into

Now, substitute to find in eqn (1) Maximum

Then substitute into

5
Single Variable Problems

Minimum

If is infinitesimely small, the series will converge to the lowest order

term. Hence;

Remember that

Now, it is obvious that; if the point a is an extremum point, the first Maximum

derivative of the function at the point must be equal to zero.

This is proof of why the first derivative must be zero at the extremes. 6
Single Variable Problems
Now, it is obvious that the first derivative at point must be zero if it is an extremum point (minimum or
maximum). This is also the necessary condition for the existence of an extremum point but it is not a
sufficient condition.

: Necessary condition but not sufficient condition. The point may be an inflection point.

Inflection
Point

Minimum

Maximum

7
Single Variable Problems
To determine whether point is an extremum or an inflection point, it is necessary to find where and
converge.

The second terms of both series are same, so it guarantees that

is a minimum. (sufficient condition for existence of an extremum)

is a maximum. (sufficient condition for existence of an extremum)

Both series will converge to the third terms, so, the investigation must be continue with the third terms.
8
 Single Variable Problems
If

Again , to guarantee that for existence of an extremum,the third derivative term must be equal to zero. If it
is zero, there is no sufficient condition again and investigation must be continue with fourth order term. But
if it is not zero, it shows that series and converges opposite directions. Hence this proves that it is an
inflection point.

Necessary but not sufficient condition. Investigation must be continue with fourth order terms.
: is an inflection point. (sufficient condition for existence of an inflection point)
9
Single Variable Problems
If

The fourth terms of both series are same, so it guarantees that

is a minimum. (sufficient condition for existence of an extremum)

is a maximum. (sufficient condition for existence of an extremum)

Both series will converge to the fifth terms, so, the investigation must be continue with the fifth
terms.
10
Single Variable Problems
The investigation of the series continues until finding the sufficient condition for the extremum or the
inflection point. However, it is clear that the rules for even and odd order numbers of derivatives of the
function are same.

11
Single Variable Problems
In General
The necessary condition of an extremum’s existence at point of a function is that number of
odd-order derivatives of the function must be zero.

The sufficient condition is that even-order derivative of the function must be non-zero.

Necessary Condition Sufficient Condition

If both number of odd-order derivatives and same number of even-order derivatives are zero but the order
derivative is non-zero, the related point is an inflection point.

12
Single Variable Problems
Taylor series can only focus around the given point in a function. However, there can be more than one
point that makes the first derivative of the function zero. All points should be investigated seperately to
determine whether it is a local minima or local maxima or an inflection point.

𝑚1= 𝐿𝑜𝑐𝑎𝑙 𝑚𝑖𝑛𝑖𝑚𝑎

𝑚2 = 𝐿𝑜𝑐𝑎𝑙 𝑚𝑖𝑛𝑖𝑚𝑎
𝑀 1= 𝐿𝑜𝑐𝑎𝑙 𝑚𝑎𝑥𝑖𝑚𝑎
𝑀 2= 𝐿𝑜𝑐𝑎𝑙 𝑚𝑎𝑥𝑖𝑚𝑎
𝐼 = 𝐼𝑛𝑓𝑙𝑒𝑐𝑡𝑖𝑜𝑛 𝑃𝑜𝑖𝑛𝑡

Since it is an unbounded function, it cannot be said directly that the point is global maximum and the
point is the global minimum. If there exists the boundaries, we can check all local extremums and
boundaries to determine which one is the global maximum or global minimum. 13
Example
Find the stationary points of the function and show that they are extremum or not?

0   is the unique stationary point.

can be an extremum or an inflection point.

 even-order derivative is greater than zero, it is a local minimum point.

14
Example
Find the stationary points of the function and show that they are extremum or not?

0   is the unique stationary point.

can be an extremum or an inflection point.

at point  It is not non-zero. Continue with third order derivative

 : odd-order derivative is non-zero. Hence, it is an inflection point.

15
Example
In General, find the stationary points of the function and
show that the stationary points are extremum or not?

is the unique stationary point.

Hence,
If , it is an extremum and local minimum.
, it is an inflection point.

16
Example
Find the local minima for the function

Statianory Points

(necessary condition)


Determine the second derivative to check suficient condition

′′
𝑓 ( 𝑥 )=6 𝑥 − 2

Point 
 is a local minimum
The function is not bounded. Hence, there is no
Point  global minimum or global maximum.
 is a local maximum

17
Multivariable Problems

Suppose that is a Multivariable nonlinear objective function. Assume that is a continuous and at least
2 times differentiable nonlinear function.

If the function depends on more than one variable, the problem is called as Multivariable problem.
The number of variables is equal to the degrees of the optimization problem .

Let’s determine the conditions for the existence of an extremum (minimum or maximum) at the point .

18
Multivariable Problems

Suppose that there is an extremum at point . Then, like in single variable problem, let’s write the Taylor
Series Expansion around the point ;

Suppose that is a vector consisting of infinitesimally small real numbers. If there is an extremum point at
point , the signs of the point’s neighbors must be same independent from the selected values.

19
Multivariable Problems

If the equation 1 and 2 are solved together,

Gradient of
Hessian Matrix of

Equation 3 can also be written again in terms of .

20
Multivariable Problems

The serie will converge directly to the first term, so

: is dependent on the randomly selected . Hence

: is the necessary condition for existence of an extremum at point

21
Multivariable Problems

Now the serie will converge to the second term, so

If there is an extremum point at point , must be sign definite. In other words, must be independent from
randomly selected .

Quadratic Form
In Linear algebra, the multiplication of a vector like with a symmetric square matrix like results in a
scalar quantity and is called quadratic form.

22
Multivariable Problems

If and are always same with randomly selected it means that the matrix is sign definite.

• For all if 0 : is positive definite (sufficient condition for local minimum)

• For all if 0 : is negative definite (sufficient condition for local maximum)
• For all if 0 : is positive semidefinite (may be extremum or inflection point. Continue with third
term of the serie)
• For all if 0 : is negative semidefinite (may be extremum or inflection point. Continue with third
term of the serie)
• For some ,0 and, for some , 0 : is indefinite (Saddle point)

23
Multivariable Problems

24
Multivariable Problems

There are two ways to determine the sign definiteness of the Hessian Matrix . The first way is Eigenvalues
method and, the second way is Sylvester Test.

Eigenvalues Method :

: has number of eigenvalues (), where is unit /identity matrix

• ;  is positive definite (local minimum)

• ;  is negative definite (local maximum)
• ;  is positive semidefinite (may be extremum or inflection point)
• ;  is negative semidefinite (may be extremum or inflection point)
• Some and some  is indefinite (Saddle point)
25
Multivariable Problems

Sylvester Test: Based on checking the signs of all principal minor determinants of the Hessian matrix

• ; k=1,2,3,…,n  is positive definite (local minimum)

• ; k=1,3,5,…,n (odd)
is negative definite (local maximum)
; k=2,4,6,…,n (even)
• If there are any ; semidefinite
• If it cannot fit none of them above ; indefinite

26
Example

Find the extremum of the function whose

contour lines are given figure below. Then determine the type of the extremum
(minimum or maximum)

27
Example

28
Example

29
Example

30
Example

Find the extremums of the function whose contour lines are

given figure below. Then determine the type of the extremum (minimum or maximum)

31
Example

32
Example

33
Example

34
Example
Find the extremums of the function whose
contour lines are given figure below. Then determine the type of the extremum
(minimum or maximum)

STEP 1 : Find stationary points

From eqn.(1)
Substitude (3) into (2)

35
Example
STEP 1 : Find stationary points

STEP 2 : Find Hessian Matrix

STEP 3 : Sylvester Test. Find determinants of principal minors.

36
Example
STEP 3 : Sylvester Test. Find determinants of principal minors.

There are 2 minimum points. Let’s determine which one is lower.

37