STEEPEST DESCENT (CAUCHY) METHOD

The use of the negative of the gradient vector as a direction for minimization was first made by Cauchy in 1847.
In this method we start from an initial trial point X1 and iteratively move along the steepest descent directions
until the optimum point is found. The steepest descent method can be summarized by the following steps:

1. Start with an arbitrary initial point X1. Set the iteration number as i = 1.
2. Find the search direction Si as
Si = − ∇fi = − ∇f (Xi)
3. Determine the optimal step length λ∗I in the direction Si and set
Xi+1 = Xi + λ∗i Si = Xi − λ∗i ∇fi
4. Test the new point, Xi+1, for optimality. If Xi+1 is optimum, stop the process.
Otherwise, go to step 5.
5. Set the new iteration number i = i + 1 and go to step 2.

The method of steepest descent may appear to be the best unconstrained minimization technique since each one-
dimensional search starts in the “best” direction. However, owing to the fact that the steepest descent direction is
a local property, the method is not really effective in most problems.

CONJUGATE GRADIENT (FLETCHER–REEVES) METHOD

The convergence characteristics of the steepest descent method can be improved greatly by modifying
it into a conjugate gradient method (which can be considered as a conjugate directions method involving the use
of the gradient of the function). We saw that any minimization method that makes use of the conjugate
directions is quadratically convergent. This property of quadratic convergence is very useful because it ensures
that the method will minimize a quadratic function in n steps or less. Since any general function can be
approximated reasonably well by a quadratic near the optimum point, any quadratically convergent method is
expected to find the optimum point in a finite number of iterations.

We have seen that Powell’s conjugate direction method requires n single-variable minimizations per
iteration and sets up a new conjugate direction at the end of each iteration. Thus it requires, in general, n2 single
variable minimizations to find the minimum of a quadratic function. On the other hand, if we can evaluate the
gradients of the objective function, we can set up a new conjugate direction after every one-dimensional
minimization, and hence we can achieve faster convergence. The construction of conjugate directions and
development of the Fletcher–Reeves method are discussed in this section.

Development of the Fletcher–Reeves Method

The Fletcher–Reeves method is developed by modifying the steepest descent method to make it quadratically
convergent. Starting from an arbitrary point X1, the quadratic function

can be minimized by searching along the search direction S1 = −∇f1 (steepest descent direction) using the step
length

The second search direction S2 is found as a linear combination of S1 and −∇f2:

where the constant β2 can be determined by making S1 and S2 conjugate with respect to [A]. This leads to:
This process can be continued to obtain the general formula for the ith search direction as

Where

Thus the Fletcher–Reeves algorithm can be stated as follows.

Fletcher–Reeves Method

The iterative procedure of Fletcher–Reeves method can be stated as follows:

1. Start with an arbitrary initial point X1.

Where λ∗1 is the optimal step length in the direction S1. Set i = 2 and go to the next step.

4. Find ∇fi = ∇f (Xi ), and set

5. Compute the optimum step length λ∗i in the direction Si , and find the new point

6. Test for the optimality of the point Xi+1. If Xi+1 is optimum, stop the process. Otherwise, set the
value of i = i + 1 and go to step 4.

Remarks:

1. The Fletcher–Reeves method was originally proposed by Hestenes and Stiefel as a method for solving
systems of linear equations derived from the stationary conditions of a quadratic. Since the directions Si used in
this method are A-conjugate, the process should converge in n cycles or less for a quadratic function. However,
for ill-conditioned quadratics (whose contours are highly eccentric and distorted), the method may require much
more than n cycles for convergence. The reason for this has been found to be the cumulative effect of rounding
errors. Since Si is given by Equation, any error resulting from the inaccuracies involved in the determination of
λ∗ i , and from the round-off error involved in accumulating the successive |∇ fi |2Si−1/|∇ fi−1|2 terms, is carried
forward through the vector Si . Thus the search directions Si will be progressively contaminated by these errors.
Hence it is necessary, in practice, to restart the method periodically after every, say, m steps by taking the new
search direction as the steepest descent direction. That is, after every m steps, Sm+1 is set equal to −∇ fm+1
instead of the usual form. Fletcher and Reeves have recommended a value of m = n + 1, where n is the number
of design variables.

2. Despite the limitations indicated above, the Fletcher–Reeves method is vastly superior to the steepest descent
method and the pattern search methods, but it turns out to be rather less efficient than the Newton and the quasi-
Newton (variable metric) methods discussed in the latter sections.
NEWTON’S METHOD

Newton’s method presented in Section 5.12.1 can be extended for the minimization of multivariable functions.
For this, consider the quadratic approximation of the function f (X) at X = Xi using the Taylor’s series
expansion

Where [Ji ] = [J ]|Xi is the matrix of second partial derivatives (Hessian matrix) of f evaluated at the point Xi .
By setting the partial derivatives of Equation equal to zero for the minimum of f (X), we obtain

Equations give

If [Ji ] is nonsingular, Eqs. can be solved to obtain an improved approximation (X = Xi+1) as

Since higher-order terms have been neglected in Equation is to be used iteratively to find the optimum solution
X∗.
The sequence of points X1,X2, . . . ,Xi+1 can be shown to converge to the actual solution X∗ from any initial
point X1 sufficiently close to the solution X∗, provided that [J1] is non-singular. It can be seen that Newton’s
method uses the second partial derivatives of the objective function (in the form of the matrix [Ji ]) and hence is
a second-order method.