Scribd
Upload
9 views

Chapter 2 - Unconstrained Optimization

The document discusses unconstrained optimization, which involves finding the best values for variables to minimize or maximize an objective function. It covers key components of optimization problems, including objective functions, decision variables, and constraints, as well as types of optimization models. The document also provides examples of finding local and global minima and maxima, stationary points, and the concept of positive and negative definiteness in quadratic forms.

Uploaded by

Bertrand Manurung
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
9 views

Chapter 2 - Unconstrained Optimization

The document discusses unconstrained optimization, which involves finding the best values for variables to minimize or maximize an objective function. It covers key components of optimization problems, including objective functions, decision variables, and constraints, as well as types of optimization models. The document also provides examples of finding local and global minima and maxima, stationary points, and the concept of positive and negative definiteness in quadratic forms.

Uploaded by

Bertrand Manurung
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 20

Unconstrained Optimization

Dr. Indarsih, M.Si.

Departemen Matematika
FMIPA UGM
What is optimization?
• The process of finding the best value for the variables of a particular
problem to minimize or maximize an objective function.
• Example: How to minimize the risk and maximize profit?
Components of on optimization problem:
• Objective function, express performance of a system, need to be minimized
or maximized.
• Decision variable
A set of unknowns, define the objective function and constraints, can
be continuous, integer or biner.
• Constraints
They are conditions, allows the unknows to take on certain values but
exclude other to render the design to be feasible.
Types of optimization models
The type of objective function or constraints:
• Linear
• Non-Linear

Whether there are constraints or not:


• Optimization with constraints
• Optimization without constraints (unconstrained optimization)
Global Optimum
Local Optimum

𝑓(1) = 5 is a local maximum.


𝑓(3) = −27 is a local minimum.
Local min/max of function of two variables
Optimization on a function of two variables
without constraints
Given 𝑓(𝑥, 𝑦). To minimize or maximize 𝑓 , we must solve
necessary conditions:
𝜕𝑓 𝜕𝑓
= 0 and =0
𝜕𝑥 𝜕𝑦
Second derivatives test
Example
𝑓 𝑥, 𝑦 = 𝑥 ! + 𝑦 ! − 2𝑥 − 6𝑦 + 14
Necessary conditions:
• 𝑓" = 2𝑥 − 2 = 0 ⟹ 𝑥 = 1
• 𝑓# = 2𝑦 − 6 = 0 ⟹ 𝑦 = 3
Check the type:
• 𝑓"" = 2, 𝑓## = 2 , 𝑓"# = 0

• So 𝑓"" > 0, 𝑓## > 0 and 𝑓"" . 𝑓## − (𝑓"# )! > 0, then point
(1.3, 𝑓(1.3)) = (1,3.4) is a local minimum that becomes
the global minimum.
Example
𝑓 𝑥, 𝑦 = 𝑥 ! + 4𝑦 ! − 2𝑥 − 16𝑦 + 𝑥𝑦
Necessary conditions:
• 𝑓" = 2𝑥 − 2 + 𝑦 = 0
• 𝑓# = 8𝑦 − 16 + 𝑥 = 0
Solution of linear equation system 𝑥 = 0, 𝑦 = 2
Check the type:
• 𝑓"" = 2, 𝑓## = 8 , 𝑓"# = 1

• So 𝑓"" > 0, 𝑓## > 0 and 𝑓"" . 𝑓## − (𝑓"# )! > 0, then point
(0,2, 𝑓(0,2)) = (1,3, −16) is a point local minimum that
becomes global minimum.
Example
𝑓 𝑥, 𝑦 = −𝑥 ! − 4𝑦 ! + 6𝑥 + 16𝑦
Necessary conditions:

• 𝑓" = −2𝑥 + 6 = 0 ⟹ 𝑥 = 3

• 𝑓# = −8𝑦 + 16 = 0 ⟹ 𝑦 = 2

Check the type:

• 𝑓"" = −2, 𝑓## = −8 , 𝑓"# = 0

• So 𝑓"" < 0, 𝑓## < 0 and 𝑓"" . 𝑓## − (𝑓"# )! > 0, then point (3,2, 𝑓(3,2)) =
(3,2, 25) is a point local maximum that becomes global maximum.
Optimization of function of 𝑛-variables
Definition
A stationary value of a function 𝑓 over the domain ℜ4 occurs at a point
𝑥 ∗ = (𝑥6∗ ⋯ 𝑥4∗ ) at which the 𝑛-equalities

𝑓7! 𝑥6∗ ⋯ 𝑥4∗ = 0


⋯
𝑓7" 𝑥6∗ ⋯ 𝑥4∗ = 0
or
∇𝑓 𝑥 ∗ = 0
hold simultaneously.
Stationary point
• Minimum point
• Maximum point
• Saddle point
A function f reaches local minimum at 𝑥8 if 𝐻(𝑥8 ) positive definite.
A function f reaches local maximum at 𝑥8 if 𝐻(𝑥8 ) negative definite.
Otherwise 𝑥8 is saddle point.
Matrix 𝐻 is positive definite if all eigenvalues are positive.
Matrix 𝐻 is negative definite if all eigenvalues are negative.
Example
Find the minimum, maximum and saddle point for this function,
𝑓 𝑥 = 𝑥>? + 𝑥!? − 3𝑥> − 12𝑥! + 25.
Solution:
𝑓"! 3𝑥>! − 3
• ∇𝑓 𝑥 = = ! =𝟎
𝑓"" 3𝑥! − 12

• 3𝑥>! − 3 = 0 ⟺ 3 𝑥>! − 1 = 0 ⟺ 𝑥> − 1 𝑥> + 1 = 0 ⟺ 𝑥> = 1 or 𝑥> = −1

• 3𝑥!! − 12 = 0 ⟺ 3 𝑥!! − 4 = 0 ⟺ 𝑥! − 2 𝑥! + 2 = 0 ⟺ 𝑥! = 2 or 𝑥! = −2

• We have four stationary points, (1,2), (-1,2), (1,-2) and (-1,-2).

6𝑥> 0
• Hessian matrix 𝐻 =
0 6𝑥!
Example
6 0 6−𝜆 0
• 𝐻(1,2) = then = 0 is satisfied 𝜆 = 6 and 𝜆 =
0 12 0 12 − 𝜆
12. Two eigenvalues are positive, so 𝐻(1,2) is positive definite. It means
function 𝑓 reaches a minimum value at point (1,2).

−6 0 −6 − 𝜆 0
• 𝐻(−1, −2) = then = 0 is satisfied 𝜆 =
0 −12 0 −12 − 𝜆
− 6 and 𝜆 = −12. Two eigenvalues are negative, so 𝐻(−1, −2) is negative
definite. It means function 𝑓 reaches a maximum value at point (−1, −2).
• Both 𝐻(1, −2) and 𝐻(−1, 2) have are positive and negative eigenvalue, so
points (1, −2) and (−1, 2) are saddle points.
Eigenvalue
The scalar 𝜆 that satisfy characteristic equation
𝐴 − 𝜆𝐼 = 0
is called eigenvalue.
Example:
2 1
Given matrix 𝐴 = .
3 4
2 1 𝜆 0 2−𝜆 1
So 𝐴 − 𝜆𝐼 = − =
3 4 0 𝜆 3 4−𝜆

det 𝐴 − 𝜆𝐼 = 2 − 𝜆 4 − 𝜆 − 1.3 = 𝜆! − 6𝜆 + 5 = (𝜆 − 1)(𝜆 − 5)


det 𝐴 − 𝜆𝐼 = 0 ⟺ 𝜆 = 1 or 𝜆 = 5
Quadratic form
𝑥6 3 1
Let 𝑥 = 𝑥 and 𝐴 = ,
@ 1 1
then
A 3 1 𝑥6
𝑥 𝐴𝑥 = 𝑥6 𝑥@
1 1 𝑥@
= 3𝑥6@ + 2𝑥6 𝑥@ + 𝑥@@
is quadratic form.
In general, quadratic form can be put as
𝑥 A 𝐴𝑥
where 𝑥 = [𝑥6 𝑥@ ⋯ 𝑥4 ]A is the column vector and 𝐴474 is the real
symmetric matrix.
Quadratic form
Example:
7𝑥6@ + 10𝑥@@ + 7𝑥B@ − 4𝑥6 𝑥@ + 2𝑥6 𝑥B − 4𝑥@ 𝑥B
can be put as
7 −2 1 𝑥6
𝑥6 𝑥@ 𝑥B −2 10 −2 𝑥@
1 −2 7 𝑥B
Positive(Negative) Definite
Definition 1
A quadratic form 𝑥 A 𝐴𝑥 is said to be positive definite if 𝑥 A 𝐴𝑥 >0 for all
𝑥 ≠ 0. It is said to be positive semidefinite if 𝑥 A 𝐴𝑥 ≥ 0 for all 𝑥 ≠ 0,
and there is at least one nonzero vector for which 𝑥 A 𝐴𝑥 = 0.

Theorem 1
A quadratic form 𝑥 A 𝐴𝑥 is positive definite (negative) if all eigenvalue of
matrix 𝐴 are positive(negative ).
Excercises
Find all stationary points and identify the type

1. 𝑓 𝑥, 𝑦 = 𝑥 B − 3𝑥 + 𝑥𝑦 @

2. 𝑓 𝑥6 , 𝑥@ , 𝑥B = 2𝑥6@ − 21𝑥6 − 3𝑥6 𝑥@ + 3𝑥@@ − 2𝑥@ 𝑥B + 𝑥B@

You might also like