Scribd
Upload
10 views

Lecture05_multidimensional Gradient Methods

Uploaded by

sx66vgvd2x
Copyright
© © All Rights Reserved
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
10 views

Lecture05_multidimensional Gradient Methods

Uploaded by

sx66vgvd2x
Copyright
© © All Rights Reserved
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 39

Unconstrained Optimization

Multi-Dimensional Unconstrained
Optimization

Chapter 14

Prepared by: Dr Saed Sasi


Libyan Academy for postgraduate studies
Methods for Multi-Dimensional
Unconstrained Optimization
The visual image for 1-D search is riding a roller
coaster where as for 2-D, the visual image is
searching through mountains and valleys.
The methods of Multi-Dimensional Unconstrained
Optimization are classified as;
 Direct or Non-gradient methods: These do not
require derivative evaluation.
 Gradient or Descent or Ascent methods: These
require derivative computation.
Multidimensional Direct or Non-
gradient methods
Random
• Repeatedly evaluateSearch
the function at randomly
selected points. If number of samples is large, the
optimum is eventually located.
• It is a brute-force method that works for even
discontinuous and non-differentiable functions.
• It always finds the global optimum.
• Major shortcoming: As the number of independent
variables grows, the method becomes slower.
• Major shortcoming: It does not take into
consideration the behavior of the underlying
function.
• Other random search techniques: Simulated
annealing, artificial neural networks, genetic
algorithms.
Multidimensional Gradient
Methods
Overview
To fully understand multidimensional
searches, we must first understand how the
first and second derivatives are expressed in
a multidimensional context.
Gradients
Suppose we have a two-dimensional function f(x,
y). An example might be your elevation on a
mountain as a function of your position. Suppose
that you are at a specific location on the mountain
(a, b) and you want to know the slope in an
arbitrary direction. One way to define the direction
is along a new axis h that forms an angle θ with
the x axis. The elevation along this new axis can
be thought of as a new function g(h). If you define
your position as being the origin of this axis (that
is, h= 0), the slope in this direction would be
designated as ǵ(0). This slope, which is called the
directional derivative, can be calculated from the
partial derivatives along the x and y axis by
where the partial derivatives are
evaluated at x =a and y = b.

Assuming that your goal is to gain the most elevation


with the next step, the next logical question would be:
what direction is the steepest ascent? The answer to
this question is provided very neatly by what is
referred to mathematically as the gradient, which is
defined as
Gradients
 The gradient is a vector operator denoted
by  (referred to as “del”)
 When applied to a function , it represents
the functions directional derivatives
 The gradient is the special case where
the direction of the gradient is the
direction of most or the steepest
ascent/descent
 The gradient is calculated by
f f
f  i  j
x y
Aside from defining a steepest path, the first
derivative can also be used to discern whether an
optimum has been reached. As is the case for a
one-dimensional function, if the partial
derivatives with respect to both x and y are zero,
a two-dimensional optimum has been reached.
Hessians
 The Hessian matrix or just the Hessian is
the matrix of second-order partial
derivatives of a function.
 The determinant of the Hessian matrix is
also referred to as the Hessian.
 For a two dimensional function the
Hessian matrix is simply

 2 f 2 f 
 2 
x xy 
H  2
 f 2 f 
 yx y 2 

Hessians cont.
The determinant of the Hessian
matrix denoted
H by can have
three
H  0cases:
2 f / 2 x 2  0 f x, y 
1. If and then has a
local
H minimum.
0 2 f / 2 x 2  0 f x, y 

2. If and then has a


local
H maximum.
0 f x, y 

3. If then has a saddle point.


Steepest Ascent/Descent
Method
Steepest Ascent/Descent strategy
(climbing a hill analogy)
Moving in a fixed path along the initial
gradient until f(x, y) stops increasing,
that is, becomes level along your
direction of travel. This stopping point
becomes the starting point where is
reevaluated and a new direction
followed. The process is repeated until
the summit is reached.
Steepest Ascent/Descent
Method
 Starts from an initial point and
looks for a local optimal solution
along a gradient.
 The gradient at the initial solution
is calculated.
 A new solution is found at the local
optimum along the gradient
 The subsequent iterations involve
using the local optima along the
new gradient as the initial point.
Starting at x0, y0 the coordinates of any
point in the gradient direction can be
expressed as

where h is distance along the h axis.


Advanced Gradient
Approaches

 Conjugate Gradient Method


(Fletcher-Reeves).
 Newton’s Method.
 Marquardt Method
 Quasi-Newton Methods

You might also like