Scribd
Upload
76 views

4 - Multidimensional Gradient Method

The document discusses multidimensional gradient methods for optimization. It defines gradients and Hessians, which are used to determine the direction of steepest ascent/descent for a function. The steepest descent method is then described, which starts at an initial point and iteratively moves in the direction of steepest descent according to the gradient to find a local minimum. An example applies this method to minimize a 2D function, taking iterative steps along the gradient until a minimum is reached.

Uploaded by

Emdad Hossain
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
76 views

4 - Multidimensional Gradient Method

The document discusses multidimensional gradient methods for optimization. It defines gradients and Hessians, which are used to determine the direction of steepest ascent/descent for a function. The steepest descent method is then described, which starts at an initial point and iteratively moves in the direction of steepest descent according to the gradient to find a local minimum. An example applies this method to minimize a 2D function, taking iterative steps along the gradient until a minimum is reached.

Uploaded by

Emdad Hossain
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 14

Multidimensional Gradient Methods

in Optimization

by

Dr. Md. Rajibul Islam


CSE, UAP

10/11/22 1
Steepest Ascent/Descent
Method
Multidimensional Gradient
Methods -Overview
 Use information from the derivatives of
the optimization function to guide the
search
 Finds solutions quicker compared with
direct search methods
 A good initial estimate of the solution is
required
 The objective function needs to be
differentiable
3
Gradients
 The gradient is a vector operator denoted by 
(referred to as “del”)
 When applied to a function , it represents the
functions directional derivatives
 The gradient is the special case where the
direction of the gradient is the direction of most
or the steepest ascent/descent
 The gradient is calculated by

f f
f  i j
x y

4
Gradients-Example
Calculate the gradient to determine the direction of the
steepest slope at point (2, 1) for the function f x, y   x 2 y 2

Solution: To calculate the gradient we would need to


calculate
f f
2 2
 2 xy  2(2)(1)  4  2 x 2 y  2(2) 2 (1)  8
x y
which are used to determine the gradient at point (2,1)
as
f  4i  8 j

5
Hessians
 The Hessian matrix or just the Hessian is the
Jacobian matrix of second-order partial
derivatives of a function.
 The determinant of the Hessian matrix is also
referred to as the Hessian.
 For a two dimensional function the Hessian
matrix is simply

 2 f 2 f 
 2 
x xy 
H  2
 f 2 f 
 yx y 2 

6
Hessians cont.
The determinant of the Hessian matrix
denoted by H can have three cases:
1. If H  0and  2 f /  2 x 2  0then f x, y has a
local minimum.
2. If H  0and  2
f /  x  0 then f x, y has a
2 2

local maximum.
3. If H  0 then f x, y has
 a saddle point.

7
Hessians-Example
Calculate the hessian matrix at point (2, 1) for the
function f x, y   x 2 y 2
Solution: To calculate the Hessian matrix; the partial
derivatives must be evaluated as
2
 f 2 f 2 f 2 f
 2 y 2  2(1) 2  4  2 x 2
 2( 2) 2
8   4 xy  4(2)(1)  8
2 2
 x y 2 xy yx
resulting in the Hessian matrix
 2 f 2 f 
 2 
x xy  4 8
H  2 
 f  f  8 8
2

 yx y 2 

8
Steepest Ascent/Descent Method
 Starts from an initial point and looks for
a local optimal solution along a
gradient.
 The gradient at the initial solution is
calculated.
 A new solution is found at the local
optimum along the gradient
 The subsequent iterations involve using
the local optima along the new gradient
as the initial point.

9
Example
Determine the minimum of the function

  2 2
f x, y  x  y  2 x  4
Use the point (2,1) as the initial estimate of the optimal
solution.

10
Solution
Iteration 1: To calculate the gradient; the partial derivatives
must be evaluated as
f f
 2 x  2  2(2)  2  6  2 y  2(1)  2
x y

f  6i  2 j

Now the function f x, y  can be expressed along the direction


of gradient as
 f f 
f  x0  h, y0  h   f (2  6h,1  2h)  (2  6h) 2  (1  2h) 2  2(2  6h)  4

 x y 

g (h)  40h 2  40h  13

11
Solution Cont.
Iteration 1 continued:
This is a simple function and it is easy to determine h*  0.5
by taking the first derivative and solving for its roots.

This means that traveling a step size of h  0.5 along the


gradient reaches a minimum value for the function in this
direction. These values are substituted back to calculate a
new value for x and y as follows:
x  2  6(0.5)  1
y  1  2(0.5)  0

Note that f 2,1  13 f  1,0  3.0

12
Solution Cont.
Iteration 2: The new initial point is  1,0 .We calculate the
gradient at this point as
f f
 2 x  2  2(1)  2  0  2 y  2(0)  0
x y

f  0i  0 j

This indicates that the current location is a local optimum


along this gradient and no improvement can be gained by
moving in any direction. The minimum of the function is
2 2
f
at point (-1,0) and min  ( 1)  ( 0)  2(1)  4  3 .

13
THE END

You might also like