Optimization

Nonlinear programming:
One dimensional minimization methods
 Optimality Criteria
• Local optimal point: A point or solution x∗ is said to be a local
optimal point, if there exists no point in the neighbourhood of x∗
which is better than x∗. In the parlance of minimization problems, a
point x∗ is a locally minimal point if no point in the neighbourhood
has a function value smaller than f (x∗).

• Global optimal point: A point or solution x∗∗ is said to be a global

optimal point, if there exists no point in the entire search space
which is better than the point x∗∗. Similarly, a point x∗∗ is a global
minimal point if no point in the entire search space has a function
value smaller than f (x∗∗).

• Inflection point: A point x∗ is said to be an inflection point if the

function value increases locally as x∗ increases and decreases
locally as x∗ reduces or if the function value decreases locally as x∗
increases and increases locally as x∗ decreases.
 One dimensional minimization methods
• Analytical methods (differential calculus methods)
• Numerical methods
Bracketing methods Interpolation methods
Exhaustive search Requiring no derivatives
Bounding Phase Quadratic
Elimination methods Requiring derivatives
Dichotomous search Cubic
Interval halving method Direct root
Fibonacci method Newton
Golden section method Quasi-Newton
Secant
 One dimensional minimization methods
Differential calculus methods:

• Analytical method

• Applicable to continuous, twice differentiable functions

• Calculation of the numerical value of the objective function is

virtually the last step of the process

• The optimal value of the objective function is calculated after

determining the optimal values of the decision variables
 One dimensional minimization methods
Numerical methods:
• The values of the objective function are first found at various
combinations of the decision variables

• Conclusions are then drawn regarding the optimal solution

• Optimal solution is found in two phases :

– First with a crude technique to reach near the optimal solution
– Thereafter, optimal solution is obtained using sophisticated technique

• Elimination methods can be used for the minimization of even

discontinuous functions

• The direct root methods are root finding methods that can be
considered to be equivalent to quadratic interpolation
 Bracketing Method – Exhaustive Search
• The exhaustive search begins from a lower bound on the variable
and three consecutive function values are compared at a time based
on the assumption of unimodality of the function.
• Based on the outcome of comparison, the search is either
terminated or continued by replacing one of the three points by a
new point. The search continues until the minimum is bracketed.

Algorithm [ for min f(x) ]

1. Set x1 = a, ∆x = (b − a)/n (n is the number of intermediate points), x 2 = x1
+ ∆x, and x3 = x2 + ∆x.

2. If f (x1) ≥ f (x2) ≤ f (x3), the minimum point lies in (x1, x3), Terminate;
Else x1 = x2, x2 = x3, x3 = x2 + ∆x, and go to Step 3.

3. Is x3 ≤ b? If yes, go to Step 2; Else no minimum exists in (a, b) or a

boundary point (a or b) is the minimum point.
Example min f(x) = x2 + 54/x in the interval [0,5]
Solution: consider n=10, x = 5/10=0.5

Iter x1 x2 x3 f(x1) f(x2) f(x3)

1 0 0.5 1  108.25 55

2 0.5 1 1.5 108.25 55 38.25

3 1 1.5 2 55 38.25 31

4 1.5 2 2.5 38.25 31 27,85

5 2 2.5 3 31 27.85 27

6 2.5 3 3.5 27.85 27 27.68

x* is bounded [2.5, 3.5] after n/2+2=7 function evaluations.

 Bracketing Method – Bounding Phase
• This method guarantees to bracket the minimum of a unimodal
function. The algorithm begins with an initial guess and thereby finds
a search direction based on two more function evaluations in the
vicinity of the initial guess.
• Thereafter, an exponential search strategy is adopted to reach the
optimum.

Algorithm [ for min f(x) ]

• Choose an initial guess x(0) and an increment ∆. Set k = 0
• If f (x(0) − |∆|) ≥ f (x(0)) ≥ f (x(0) + |∆|), then ∆ is positive; Else if f (x(0) −
|∆|) ≤ f (x(0)) ≤ f (x(0) + |∆|), then ∆ is negative; Else go to Step 1.
• Set x(k+1) = x(k) + 2k ∆.
• If f (x(k+1)) < f (x(k)), set k = k + 1 and go to Step 3;
• Else the minimum lies in the interval (x(k−1), x(k+1)) and Terminate.
Example min f(x) = x2 + 54/x in the interval [0,10]
Solution: Choose initial guess x0 = 0.6, x =0.5.
f(0.6-0.5)= f(0.1) = 540.01, f(0.6)=90.36
f(0.6+0.5)=f(1.1) = 50.3 as f(1.1) < f(0.6) so, x = +0.5

K 0 1 2 3 4
x 0.6 1.1 2.1 4.1 8.1
f(x) 90.36 50.3 30.1 29.98 72.27

x* is bounded [2.0, 8.0] after 6 function evaluations.

Bracketing of minimum poor but requires less number of
function evaluations.
 Region Elimination Method
• All region elimination method assume the objective function
as unimodal.
• A unimodal function is one that has only one peak (maximum)
or valley (minimum) in a given interval
• A function of one variable is said to be unimodal if, given that
two values of the variable are on the same side of the
optimum, the one nearer the optimum gives the smaller
functional value in the case of a minimization problem. This
can be stated mathematically as follows:

A function f (x) is unimodal if

 x1 < x2 < x* implies that f (x2) < f (x1) and
 x2 > x1 > x* implies that f (x1) < f (x2) where x* is the
minimum point
 Unimodal function

• A unimodal function can be a nondifferentiable or even a

discontinuous function

• If a function is known to be unimodal in a given range, the

interval in which the minimum lies can be narrowed down
provided that the function values are known at two different
values in the range.
 Unimodal function
• For example, consider the normalized interval [0,1] and two function
evaluations within the interval as shown:

• There are three possible outcomes:

f1 < f2 f1 > f2 f1 = f2
Unimodal function
• If the outcome is f1 < f2, the
minimum of x can not lie to
the right of x2

• Thus, that part of the interval

[x2,1] can be discarded and a
new small interval of
uncertainty, [0, x2] results as
shown in the figure
Unimodal function

If the outcome is f (x1) > f (x2) ,

the interval [0, x1] can be
discarded to obtain a new
smaller interval of uncertainty,
[x1, 1].
Unimodal function

If f1 = f2 , intervals [0, x1] and

[x2,1] can both be discarded to
obtain the new interval of
uncertainty as [x1,x2]
 Unimodal function

• Furthermore, if one of the experiments (function evaluations in the

elimination method) remains within the new interval, as will be the
situation in Figs (a) and (b), only one other experiment need be placed
within the new interval in order that the process be repeated.

• In Fig (c), two more experiments are to be placed in the new interval in
order to find a reduced interval of uncertainty.

• The assumption of unimodality is made in all the elimination techniques

 Dichotomous search
In the dichotomous search, two experiments are placed as close as possible at
the center of the interval of uncertainty.

Let the positions of the two points be

given by:
L 
x1  0 
2 2
L 
x2  0 
2 2

where L0 is initial interval and  is a

small positive number chosen such that
the two function values give significantly
different results.

Based on the relative values of the objective function at the two points,
almost half of the interval of uncertainty is eliminated.
 Dichotomous Search
• The intervals of uncertainty at the ends of different
pairs of function evaluations are given in the following
table.
Number of 2 4 6

evaluations

Final interval
1  L0     1  L0     
of uncertainty (L0+ )/2     
2 2  2 2 4 2 2

• In general, the final interval of uncertainty after

conducting n function evaluations (n even) is given by:
L0  1 
Ln  n / 2   1  n / 2 
2  2 
Algorithm
1.Choose interval [a, b], tolerance ϵ and x.
2.Set x1 = a+(b – a)/2 - x/2, x2 = a+(b – a)/2 + x/2,
Compute f(x1) and f (x2).
3.If f (x1) < f (x2) set b = x2 else a = x1;
4.If |b-a| < ϵ, Terminate; Else go to Step 2.
Example min f(x) = x2 + 54/x in the interval [2.0,8.0]
x = 0.2
iter a b x1 x2 f(x1) f(x2)
1 2 8 4.9 5.1 35.03041 36.59824
2 2 5.1 3.45 3.65 27.55467 28.11702
3 2 3.45 2.625 2.825 27.46205 27.09567
4 2.625 3.45 2.9375 3.1375 27.01188 27.05506
5 2.625 3.1375 2.78125 2.98125 27.15108 27.00106
 Interval halving method
In the interval halving method, exactly one half of the
current interval of is deleted in every stage. It requires
three function evaluations in the first stage and two
function evaluations in each subsequent stage.

The procedure can be described by the following

steps:
1. Divide the initial interval L0 = [a,b] into four equal parts
and label the middle point x0 and the quarter-interval
points x1 and x2.
2. Evaluate the function f(x) at the three interior points to
obtain f1 = f(x1), f0 = f(x0) and f2 = f(x2).
Case I: f1 < f0 < f2
delete theinterval (x0,b)
x1 = x0 b =x0

Case II: f2 < f0 < f1 Case III: f2 > f0 & f1 > f0

delete the interval (a, x0) delete interval (a, x1) & (x1,b)
x2 = x0 a =x0 a =x1 and b=x2
Algorithm

1.Choose interval [a, b] , tolerance ϵ. Let xo = (a + b)/2,

L = b − a. Compute f (xo).

2.Set x1 = a + L/4, x2 = b−L/4. Compute f(x1) and f(x2).

3.If f (x1) < f (xo) set b = xo; xo = x1; go to Step 5; Else go

to Step 4.

4.If f (x2) < f (xo) set a = xo; xo = x2; go to Step 5; Else set
a = x1, b = x2; go to Step 5.

5.Calculate L = b − a. If |L| < ϵ, Terminate; Else go to

Step 2.
 Example min f(x) = x2 + 54/x in the interval [0,5]

Solution: a=0, b=5, x0 = 2.5, x1 = 1.25, x2 = 3.75,

iter a b x1 xo x2 f(x1) f(x0) f(x2)

1 0 5 1.25 2.5 3.75 44.7625 27.85 28.4625

2 1.25 3.75 1.875 2.5 3.125 32.3156327.85 27.04563

3 2.5 3.75 2.8125 3.125 3.4375 27.11016 27.0456327.5255

4 2.8125 3.4375 2.96875 3.125 3.28125 27.0029527.0456327.22374

5 2.8125 3.125 2.8906252.96875 3.04687527.0367927.0029527.00652

 Optimization
Nonlinear programming:
One dimensional minimization methods
(contd.)
 Fibonacci method
Fibonacci method can be used to find the minimum of a
function of one variable even if the function is not
continuous.

The limitations of the method are:

• The initial interval of uncertainty, in which the optimum lies,
has to be known.
• The function being optimized has to be unimodal in the
initial interval of uncertainty.
• The number of function evaluations to be used in the search
or the resolution required has to be specified before hand.
Fibonacci Search
I1 = I2 + I3
So, we have,
I1 = I 2 + I 3
I2 = I 3 + I 4
.
.
In = In+1 + In+2

Let us assume interval at n+2 iteration becomes zero.

i.e, In+2 = 0
Then,
In+1 = In – In+2 = 1 In Fibonacci Numbers
In = In+1 + In+2 = 1 In 1,1,2,3,5,8,13,21,…
In-1 = In + In+1 = 2 In
In-2 = In-1 + In = 3 In FK = FK-1 + FK-2
In-3 = In-2 + In-1 = 5 In k=2,3,…. and
In-4 = In-3 + In-2 = 8 In F0 = F1 = 1
 In+1 = In – In+2 = 1 In = F0 In
In = In+1 + In+2 = 1 In = F1 In
In-1 = In + In+1 = 2 In = F2 In
In-2 = In-1 + In = 3 In = F3 In
In-3 = In-2 + In-1 = 5 In = F4 In
In-4 = In-3 + In-2 = 8 In = F5 In
.
.
Ik = Ik+1 + Ik+2 = Fn-k+1 In
.
.
I1 = I2 + I3 = Fn In

Which means, after n iterations, In = I1 / Fn

µk+1 = ak+1 + Ik+2
Algorithm
• Set a lower bound a and an upper bound b. Set L = b −
a. Set the desired number of function evaluations to be n.
Set k = 1.

• Compute L*k = (Fn−k-1/Fn-k+1)L. Set x1 = a + L*k and x2 =

b − L*k .

• Compute one of f (x1) or f (x2). If f(x1) < f(x2), b=x2, x2=x1,

x1=a+(b-x2) else a=x1, x1=x2, x2=b – (x1-a)

• set k = k + 1. Is k + 1 = n? yes, Terminate else go to

Step 3;
 Minimize

f(x)=0.65-[0.75/(1+x2)]-0.65 x tan-1(1/x) in the interval [0,3]

by the Fibonacci method using n=6.

Solution: Here n=6 and L0=3.0, which yield:

Fn  2 5
L1*  L0  (3.0)  1.153846
Fn 13
Thus, the positions of the first two points are given by
x1=1.153846 and x2=3.0-1.153846 =1.846154 with f1=f(x1) =
-0.207270 and f2=f(x2)=-0.115843. Since f1 is less than f2, we can
delete the interval [x2,3] by using the unimodality assumption.
 Fibonacci method
Solution:
 Fibonacci method
x3=0+ (x2-x1)=1.846154-1.153846=0.692308,
f3=-0.291364.
Since f1 is greater than f3, we can delete the interval [x1,x2]
 Fibonacci method
x4=0+ (x1-x3)=1.153846-0.692308=0.461538,
f4=-0.309811.
f4 is less than f3, we can delete the interval [x3,x1]
 Fibonacci method
x5=0+ (x3-x4)=0.692308-0.461538=0.230770,
f5=-0.263678.
Since f4 is less than f5, we can delete the interval [0,x5]
 Fibonacci method
x6=x5+ (x3-x4)=0.230770+(0.692308-0.461538)=0.461540
f6=-0.309810.
Since f6 > f4 , we delete the interval [x6, x3] and obtain the
final interval as L6 = [x5, x6]=[0.230770,0.461540].
 Fibonacci method
The ratio of the final to the initial interval is
𝐿6 0.461540 − 0.230770
= = 0.076923
𝐿0 3.0

This value can be compared with

𝐿𝑛 𝐹1 1
= =
𝐿0 𝐹𝑛 𝐹𝑛
which states that if n experiments (n=6) are planned,
a resolution no finer than 1/Fn= 1/F6=1/13=0.076923
can be expected from the method.
 Golden Section Method
• The golden section method is same as the Fibonacci
method except that the ratio of two adjacent intervals are
constant i.e,

• Since Ik = Ik+1 + Ik+2 , so, r = 1+ 1/r  r2 – r -1 = 0

• This gives r = (1+√5)/2 = 1.618 ( golden ratio)
• So, Ik+1 = 1/1.618 Ik = 0.618 Ik
• In the Fibonacci method, the total number of function
evaluations to be conducted has to be specified before
beginning the calculation, whereas this is not required in
the golden section method.
Algorithm
 Choose a lower bound a and an upper bound b. Also
choose a small number ϵ. Set k = 1.

 Compute L = b – a. Set x1 = a + (0.382)L and x2 = b −

(0.382)L.

 Compute f (x1) and f (x2).

 If f(x1) < f(x2), b=x2, x2=x1, x1=a+(b-x2) else a=x1, x1=x2,

x2=b – (x1-a)

 Is |L=b-a | < ϵ small? If no, set k = k + 1, go to Step 3;

Else Terminate.
 Comparison of elimination methods
• The efficiency of an elimination method can be measured in terms of
the ratio of the final and the initial intervals of uncertainty, Ln/L0

• Fibonacci method is the most efficient method, followed by the

golden section method, in reducing the interval.
 Interpolation Methods
 Requiring no derivatives
 Quadratic
 Requiring derivatives
 Cubic
 Direct root
o Newton
o Quasi-Newton
o Secant
Quadratic Interpolation Method
 Cubic Interpolation Method
• This method is similar to quadratic interpolation method.

• In stead of quadratic polynomial, cubic polynomial is

used to approximate the objective function.

• Since 4 coefficients are involved in a cubic polynomial, 4

points on the objective function is required to evaluate
the coefficients.

• If f(x) is the objective function, then four points on the

objective function considered are [x1, f(x1)], [x2, f(x2)], [x1,
g(x1)], [x2, g(x2)], where g(xi) = df(xi)/dx
Cubic Interpolation Method
 Cubic Interpolation Method
So, at (x1,f1) : a0 = c1 = f1
at (x2,f2) : a0 + a1(x2 – x1) =c2 = f2
at (x1, g1) : a1 + a2(x1 –x2) = g1
at (x2, g2) : a1 + a2 (x2 – x1) + a3 (x2 – x1)2 = g2

• The above system can be solved very easily in a recursive

manner to express the coefficients in terms of objective
function f and its derivative g at point x1 and x2.
• An estimate of stationary point of f can be obtained by
𝑑𝑐
putting 𝑑𝑥 = 0 and solving for estimated optimum point
𝑥.ҧ
Cubic Interpolation Method
Cubic Interpolation Method
Cubic Interpolation Method
 Optimization
Nonlinear programming:
One dimensional minimization methods
(contd.)
 Direct root methods
The necessary condition for f () to have a minimum of * is

𝑓 ′ (𝜆) = 0
Three root finding methods will be considered here:

• Newton method

• Quasi-Newton method

• Secant methods
 Newton method
Consider the quadratic approximation of the function f ()
at = i using the Taylor’s series expansion:

1 ″
𝑓(𝜆) = 𝑓(𝜆𝑖 ) + 𝑓 ′ (𝜆 𝑖 )(𝜆 − 𝜆𝑖 ) + 𝑓 (𝜆𝑖 )(𝜆 − 𝜆𝑖 )
2
2

By setting the derivative of this equation equal to zero for

the minimum of f (), we obtain:

𝑓 ′ (𝜆) = 𝑓 ′ (𝜆𝑖 ) + 𝑓 ″ (𝜆𝑖 )(𝜆 − 𝜆𝑖 ) = 0

If i denotes an approximation to the minimum of f (),

the above equation can be rearranged to obtain an
improved approximation as:
 Newton method
𝑓 ′ (𝜆𝑖 )
𝜆𝑖+1 = 𝜆𝑖 − ″
𝑓 (𝜆𝑖 )

Thus, the Newton method is equivalent to using a

quadratic approximation for the function f () and applying
the necessary conditions.

The iterative process given by the above equation can be

assumed to have converged when the derivative, f’(i+1) is
close to zero:
𝑓 ′ (𝜆𝑖+1 ) ≤ 𝜀
where 𝜀 is a small quantity
 Newton method

FIGURE 5.18a sayfa 318

 Newton method
• If the starting point for the iterative process is not close to the true
solution *, the Newton iterative process may diverge as illustrated:
 Example
Find the minimum of the function:

0.75 −1
1
𝑓(𝜆) = 0.65 − − 0.65𝜆 tan
1 + 𝜆2 𝜆

Using the Newton-Raphson method with the starting point

1=0.1. Use =0.01 in the equation
𝑓 ′ (𝜆𝑖+1 ) ≤ 𝜀
where 𝜀 is a small quantity

for checking the convergence.

 Example
Solution: The first and second derivatives of the function f
() are given by :
1.5𝜆 0.65𝜆 1
𝑓 ′ (𝜆)
= + − 0.65 tan −1
(1 + 𝜆2 )2 1 + 𝜆2 𝜆
″
1.5(1 − 3𝜆2 ) 0.65(1 − 𝜆2 ) 0.65 2.8 − 3.2𝜆2
𝑓 (𝜆) = + + =
(1 + 𝜆2 )3 (1 + 𝜆2 )2 1 + 𝜆2 (1 + 𝜆2 )3

Iteration 1
1=0.1, f (1) = -0.188197, f’ (1) = -0.744832, f’’
(1)=2.68659
𝑓 ′ (𝜆1 )
𝜆2 = 𝜆1 − ″ = 0.377241
𝑓 (𝜆1 )

Convergence check: | f (2)| =|-0.138230| > 

 Example
Iteration 2
f (2 ) = -0.303279, f’ (2) = -0.138230, f’’ (2) = 1.57296
𝑓 ′ (𝜆2 )
𝜆3 = 𝜆2 − ″ = 0.465119
𝑓 (𝜆2 )

Convergence check: | f’(3)| =|-0.0179078| > 

Iteration 3
f (3 ) = -0.309881, f’ (3) = -0.0179078, f’’ (3) = 1.17126
𝑓 ′ (𝜆3 )
𝜆4 = 𝜆3 − ″ = 0.480409
𝑓 (𝜆3 )

Convergence check: | f’(4)| =|-0.0005033| < 

Since the process has converged, the optimum solution is
taken as * 4=0.480409
 Quasi-Newton Method
If the function minimized f () is not available in closed form
or is difficult to differentiate, the derivatives f’ () and f’’ ()
in the equation
𝑓 ′ (𝜆𝑖 )
𝜆𝑖+1 = 𝜆𝑖 − ″
𝑓 (𝜆𝑖 )

can be approximated by the finite difference formula as:

′
𝑓(𝜆𝑖 + Δ𝜆) − 𝑓(𝜆𝑖 − Δ𝜆)
𝑓 (𝜆𝑖 ) =
2Δ𝜆
″
𝑓(𝜆𝑖 + Δ𝜆) − 2𝑓(𝜆𝑖 ) + 𝑓(𝜆𝑖 − Δ𝜆)
𝑓 (𝜆𝑖 ) =
Δ𝜆2

where  is a small step size.

 Quasi-Newton Method
Substitution of:

′
𝑓(𝜆𝑖 + Δ𝜆) − 𝑓(𝜆𝑖 − Δ𝜆)
𝑓 (𝜆𝑖 ) =
2Δ𝜆
″
𝑓(𝜆𝑖 + Δ𝜆) − 2𝑓(𝜆𝑖 ) + 𝑓(𝜆𝑖 − Δ𝜆)
𝑓 (𝜆𝑖 ) =
Δ𝜆2

𝑓 ′ (𝜆𝑖 )
into 𝜆𝑖+1 = 𝜆𝑖 − ″
𝑓 (𝜆𝑖 )
leads to
Δ𝜆[𝑓(𝜆𝑖 + Δ𝜆) − 𝑓(𝜆𝑖 − Δ𝜆)]
𝜆𝑖+1 = 𝜆𝑖 −
2[𝑓(𝜆𝑖 + Δ𝜆) − 2𝑓(𝜆𝑖 ) + 𝑓(𝜆𝑖 − Δ𝜆)]
 Quasi-Newton Method
This iterative process is known as the quasi-Newton method.
To test the convergence of the iterative process, the following
criterion can be used:
′
𝑓(𝜆𝑖+1 + Δ𝜆) − 𝑓(𝜆𝑖+1 − Δ𝜆)
𝑓 (𝜆𝑖+1 ) = <𝜀
2Δ𝜆

where a central difference formula has been used for

evaluating the derivative of f and  is a small quantity.

Remarks:
Δ𝜆[𝑓(𝜆𝑖 + Δ𝜆) − 𝑓(𝜆𝑖 − Δ𝜆)]
The equation 𝜆𝑖+1 = 𝜆𝑖 −
2[𝑓(𝜆𝑖 + Δ𝜆) − 2𝑓(𝜆𝑖 ) + 𝑓(𝜆𝑖 − Δ𝜆)]

requires the evaluation of the function at the points i+

and i - in addition to i in each iteration.
 Example
Find the minimum of the function

0.75 −1
1
𝑓(𝜆) = 0.65 − − 0.65𝜆 tan
1 + 𝜆2 𝜆

using the quasi-Newton method with the starting point 1=0.1 and the
step size =0.01 in central difference formulas. Use =0.01 in equation

𝑓(𝜆𝑖+1 + Δ𝜆) − 𝑓(𝜆𝑖+1 − Δ𝜆)

𝑓 ′ (𝜆 𝑖+1 ) = <𝜀
2Δ𝜆

for checking the convergence.

 Example
Solution: Iteration I

𝜆 = 0.1, Δ𝜆 = 0.01, 𝑓1 = 𝑓(𝜆1 ) = −0.188197

𝑓1+ = 𝑓(𝜆1 + Δ𝜆) = −0.195512, 𝑓1− = 𝑓(𝜆1 − Δ𝜆)
= −0.180615
Δ𝜆(𝑓1+ − 𝑓1− )
𝜆2 = 𝜆1 − + − = 0.377882
2(𝑓1 − 2𝑓1 + 𝑓1 )
𝐶𝑜𝑛𝑣𝑒𝑟𝑔𝑒𝑛𝑐𝑒 𝑐ℎ𝑒𝑐𝑘 :
′
𝑓2+ − 𝑓2−
𝑓 (𝜆2 ) = = 0.137300 > 𝜀
2Δ𝜆
 Example
Solution: Iteration 2

𝑓2 = 𝑓(𝜆2 ) = −0.303368, 𝑓2+ = 𝑓(𝜆2 + Δ𝜆)

= −0.304662
𝑓2− = 𝑓(𝜆2 − Δ𝜆) = −0.301916,
Δ𝜆(𝑓2+ − 𝑓2− )
𝜆3 = 𝜆2 − + − = 0.465390
2(𝑓2 − 2𝑓2 + 𝑓2 )
𝐶𝑜𝑛𝑣𝑒𝑟𝑔𝑒𝑛𝑐𝑒 𝑐ℎ𝑒𝑐𝑘 :
′
𝑓3+ − 𝑓3−
𝑓 (𝜆3 ) = = 0.017700 > 𝜀
2Δ𝜆
 Example
Solution: Iteration 3

𝑓3 = 𝑓(𝜆3 ) = −0.309885, 𝑓3+ = 𝑓(𝜆3 + Δ𝜆) = −0.310004

𝑓3− = 𝑓(𝜆3 − Δ𝜆) = −0.309650,
Δ𝜆(𝑓3+ − 𝑓3− )
𝜆4 = 𝜆3 − + − = 0.480600
2(𝑓3 − 2𝑓3 + 𝑓3 )
𝐶𝑜𝑛𝑣𝑒𝑟𝑔𝑒𝑛𝑐𝑒 𝑐ℎ𝑒𝑐𝑘 :
+ −
𝑓4 − 𝑓4
𝑓 ′ (𝜆4 ) = = 0.000350 < 𝜀
2Δ𝜆
Since the process has converged, we take the optimum solution
as 𝜆 ∗≈ 𝜆4 = 0.480600
 Secant method
• The secant method uses an equation similar to
equation:
𝑓 ′ (𝜆) = 𝑓 ′ (𝜆𝑖 ) + 𝑓 ″ (𝜆𝑖 )(𝜆 − 𝜆𝑖 ) = 0
as:
𝑓 ′ (𝜆) = 𝑓 ′ (𝜆𝑖 ) + 𝑠(𝜆 − 𝜆𝑖 ) = 0

where s is the slope of the line connecting the two

points (A, f ’(A)) and (B, f ’(B)), where A and B denote
two different approximations to the correct solution, *.
The slope s can be expressed as:
𝑓 ′ (𝐵) − 𝑓 ′ (𝐴)
𝑠=
𝐵−𝐴
 Secant method
The equation:
𝑓 ′ (𝜆) = 𝑓 ′ (𝜆𝑖 ) + 𝑠(𝜆 − 𝜆𝑖 ) = 0

approximates the function f’() between A and B as a linear

equation (secant), and hence the solution of the above equation
gives the new approximation to the root of the f’() as:

𝑓 ′ (𝜆𝑖 ) 𝑓 ′ (𝐴)(𝐵 − 𝐴)
𝜆𝑖+1 = 𝜆𝑖 − =𝐴− ′
𝑠 𝑓 (𝐵) − 𝑓 ′ (𝐴)

The iterative process given by the above equation is known as the

secant method. Since the secant approaches the second derivative
of f () at A as B approaches A, the secant method can also be
considered as a quasi-Newton method.
Secant method
 Secant method
• It can also be considered as a form of elimination technique since
part of the interval, (A,İ+1) in the figure is eliminated in every
iteration.

The iterative process can be implemented by using the following

step-by-step procedure:

1. Set 1=A=0 and evaluate f’(A). The value of f’(A) will be negative.
Assume an initial trial step length t0.
2. Evaluate f’(t0).
3. If f’(t0)<0, set A= i=t0, f’(A)= f’(t0), new t0=2t0, and go to step 2.
4. If f’(t0)≥0, set B= t0, f’(B)= f’(t0), and go to step 5.
5. Find the new approximate solution of the problem as:
𝑓 ′ (𝐴)(𝐵 − 𝐴)
𝜆𝑖+1 =𝐴− ′
𝑓 (𝐵) − 𝑓 ′ (𝐴)
 Secant method
6. Test for convergence:
𝑓 ′ (𝜆𝑖+1 ) ≤ 𝜀

where  is a small quantity. If the above equation is satisfied, take

* i+1 and stop the procedure. Otherwise, go to step 7.

7. If f’(İ+1) ≥ 0, set new B= İ+1, f’(B) =f’(İ+1), i=i+1, and go to step 5.

8. If f’(İ+1) < 0, set new A= İ+1, f’(A) =f’(İ+1), i=i+1, and go to step 5.
 Secant method
Remarks:

1. The secant method is identical to assuming a linear equation for

f’(). This implies that the original function, f(), is approximated
by a quadratic equation.

2. In some cases, we may encounter a situation where the function

f’() varies very slowly with . This situation can be identified by
noticing that the point B remains unaltered for several consecutive
refits. Once such a situation is suspected, the convergence
process can be improved by taking the next value of i+1 as
(A+B)/2 instead of finding its value from
𝑓 ′ (𝐴)(𝐵 − 𝐴)
𝜆𝑖+1 =𝐴− ′
𝑓 (𝐵) − 𝑓 ′ (𝐴)
 Secant method
Example
0.75 −1
1
Find the minimum of the function 𝑓(𝜆) = 0.65 − − 0.65𝜆 tan
1 + 𝜆2 𝜆
using the secant method with an initial step size of t0=0.1, 1=0.0, and
=0.01.
Solution
1=A=0.0, t0=0.1, f’(A)=-1.02102, B=A+t0=0.1, f’(B)=-0.744832
Since f’(B)<0, we set new A=0.1, f’(A)=-0.744832, t0=2(0.1)=0.2,
B= 1+t0=0.2, and compute f’(B)=-0.490343. Since f’(B)<0, we set new
A=0.2, f’(A)=-0.490343, t0=2(0.2)=0.4, B= 1+t0=0.4, and compute f’(B)=-
0.103652. Since f’(B)<0, we set new A=0.4, f’(A)=-0.103652
t0=2(0.4)=0.8, B= 1+t0=0.8, and compute f’(B)=-0.180800. Since, f’(B)>0,
we proceed to find 2
 Secant method
Iteration 1
Since A=1=0.4, f’(A)=-0.103652, B=0.8, f’(B)= 0.180800, we compute
𝑓 ′ (𝐴)(𝐵 − 𝐴)
𝜆2 = 𝐴 − ′ = 0.545757
𝑓 (𝐵) − 𝑓 ′ (𝐴)
Convergence check:
𝑓 ′ (𝜆2 ) = +0.0105789 > 𝜀
Iteration 2
Since f’(2)=+0.0105789 > 0, we set new A=0.4,f’(A)=-0.103652, B=
2=0.545757, f’(B)=f’(2)=0.0105789, and compute
𝑓 ′ (𝐴)(𝐵 − 𝐴)
𝜆3 = 𝐴 − ′ = 0.490632
𝑓 (𝐵) − 𝑓 ′ (𝐴)
Convergence check:
f (3 )   0.00151235  
Since the process has converged, the optimum solution is given by
*3=0.490632
 Practical Considerations
Sometimes, the Direct Root Methods such as the
Newton, Quasi-Newton and the Secant method or the
interpolation methods such as the quadratic and the
cubic interpolation methods may be:
• very slow to converge,
• may diverge
• may predict the minimum of the function f() outside the
initial interval of uncertainty, especially when the
interpolating polynomial is not representative of the
variation of the function being minimized.
In such cases, we can use the Fibonacci or the golden
section method to find the minimum.
 Practical Considerations
In some problems, it might prove to be more
efficient to combine several techniques. For
example:
• The bounding phase search with an accelerated
step size can be used to bracket the minimum
and then the Fibonacci or the golden section
method can be used to find the optimum point.
• In some cases, the Fibonacci or the golden
section method can be used in conjunction with
an interpolation method.
 Comparison of methods
• The Fibonacci method is the most efficient elimination technique in finding
the minimum of a function if the initial interval of uncertainty is known.

• In the absence of the initial interval of uncertainty, the quadratic

interpolation method or the quasi-Newton method is expected to be more
efficient when the derivatives of the function are not available.

• When the first derivatives of the function being minimized are available,
the cubic interpolation method or the secant method are expected to be
very efficient.

• On the other hand, if both the first and the second derivatives of the
function are available, the Newton method will be the most efficient one in
finding the optimal step length, *.