METHOD OF QUADRATIC INTERPOLATION

KELLER VANDEBOGERT

1. Introduction

Interpolation methods are a common approach to the more general

area of line search for optimization. In the case of quadratic inter-
polation, the function’s critical value is bracketed, and a quadratic
interpolant is fitted to the arc contained in the interval. Then, the
interpolant is minimized, and the new interval is determined based on
the relation of the minimizer to the original endpoints of the interval.
Put more formally, let x* maximize (or minimize) f (x). If x* is
not easily found through analytic methods, then it is significantly eas-
ier to bracket the interval over which this critical point occurs. Let
q(x) denote the quadratic interpolant of f (x). Minimizing a quadratic
function is trivial, and so the critical point of q is easily obtained. We
then form a new bracketing interval by throwing away the ”worst”
point, which for our purposes would be the point that is the largest or
smallest, depending on whether we want to approximate a maximum
or minimum. The process is then iterated until a desired precision is
reached.

Date: September 3, 2017.

1
2 KELLER VANDEBOGERT

2. Methods of Interpolation

In practice there are 3 methods of interpolation. There are 2 types

of 2-point interpolation methods, and a 3-point interpolation method.
The 2-point methods require knowledge of the derivative of the func-
tion f in which we are interested in optimizing. The 3-point method
does not require any derivatives, but of course requires an extra point.
Intuitively, knowing f 0 gives a better sense of the function’s behavior,
and will hence provide a faster rate of convergence.

2.1. Method 1. Let q(x) denote the quadratic interpolant of f (x).

Then, by definition:

q(x) = ax2 + bx + c

For a, b, c ∈ R. Then, we can find our constants by bracketing the

critical point of f , whose endpoints are x1 and x2 . We have:

(2.1) f (x1 ) = ax21 + bx1 + c = q(x1 )

f (x2 ) = ax22 + bx2 + c = q(x2 )

f 0 (x1 ) = 2ax1 + b = q 0 (x1 )

This is a system of 3 equations with 3 unknowns, which are our

constants. Let fi = f (xi ), and fi0 = f 0 (xi ). Then we can solve (2.1) for
a and b.

f1 − f2 − f10 (x1 − x2 )
(2.2) a=
−(x1 − x2 )2
f1 − f2 − f10 (x1 − x2 )
b = f10 + 2 x1
(x1 − x2 )2
 METHOD OF QUADRATIC INTERPOLATION 3

The minimizer of q is easily found to be −b/2a by setting q 0 (x) = 0.

From (2.2), our minimizer xmin can be found:

−b 1 (x1 − x2 )f10
(2.3) xmin = = x1 −
2a 2 f10 − xf1 −f 2
1 −x2

This of course readily yields an explicit iteration formula by letting

xmin = x3 . We have from (2.3):

0
1 (xk−1 − xk )fk−1
(2.4) xk+1 = xk−1 −
2 fk−1
0
− fk−1 −fk
xk−1 −xk

With (2.4), we generate xk+1 and compare it with the previous two
points to find our new bracketing interval, and then continue the iter-
ation.

2.2. Method 2. This method is very similar to our first method but
instead generates q(x) by solving a different set of equations. We have:

(2.5) f (x1 ) = ax21 + bx1 + c = q(x1 )

f 0 (x1 ) = 2ax1 + b = q 0 (x1 )

f 0 (x2 ) = 2ax2 + b = q 0 (x2 )

We can of course solve these for a and b in the same way as in Section
2.1. We find the explicit form of the minimizer of q(x) to be:

x1 − x2 0
(2.6) xmin = x1 − f
f10 − f20 1
In an identical manner to (2.4), we generate a recursion by defining
xmin = x3 , and we have:
4 KELLER VANDEBOGERT

xk−1 − xk 0
(2.7) xk+1 = xk−1 − f
0
fk−1 − fk0 k−1
Which is commonly called the secant formula.

Remark 2.1. Letting xk−1 → xk in (2.7), and assuming that f 00 (xk )

exists, (2.7) becomes:

fk0
xk+1 = xk − 00
fk
But this is precisely the iteration defined by Newton’s method. This
motivates calling (2.7) the secant method, because it is just Newton’s
method with the secant approximation of f 00 (xk ) instead.

2.3. Method 3. Our third method is the 3 point method. Choose

3 points, 2 endpoints to bracket our critical point, and then a point
within the interval as well.
Using the Lagrange Interpolation formula, we can easily find our
interpolant q(x). We have:

(2.8)
(x − x2 )(x − x3 ) (x − x1 )(x − x3 ) (x − x1 )(x − x2 )
q(x) = f1 + f2 + f3
(x1 − x2 )(x1 − x3 ) (x2 − x1 )(x2 − x3 ) (x3 − x1 )(x3 − x2 )
To find the minimum, we of course take the derivative of (2.8) and
set it to 0. By doing this, we obtain:

1 1 (f1 − f2 )(f2 − f3 )(f3 − f1 )

(2.9) xmin = (x1 + x2 ) +
2 2 (x2 − x3 )f1 + (x3 − x1 )f2 + (x1 − x2 )f3
And the iteration scheme is easily deduced:
 METHOD OF QUADRATIC INTERPOLATION 5

(2.10)
1 1 (fk−1 − fk )(fk − fk+1 )(fk+1 − fk−1 )
xk+2 = (xk−1 +xk )+
2 2 (xk − xk+1 )fk−1 + (xk+1 − xk−1 )fk + (xk−1 − xk )fk+1
This method differs slightly from the previous two methods, because
it is not as simple to determine the new bracketing interval. If xmin
lies between x1 and x3 , then we want to compare the distance between
xmin and x2 . If

|xmin − x2 | < 

Where  is the prescribed tolerance, then we are done. Otherwise

we create the new interval by checking which point is the largest or
smallest, depending on the nature of our critical point.
If xmin lies outside of our bracketing interval [x1 , x3 ], then we im-
mediately create a new bracketing interval in the same way as we just
described.

3. Convergence Rates

In the beginning of Section 2, we made a comment on the convergence

rates of the 2-point versus 3-point method. In this section, we intend
to make this comment precise.

Definition 3.1. Let xn → x∗ . Then, the sequence {xn } is said to

converge to x* with order α if

||xn − x∗ || ≤ M ||xn−1 − x∗ ||α

For some constant M ∈ R.

6 KELLER VANDEBOGERT

We will also need the following result, which we shall state without
proof:

Lemma 3.2. Let f ∈ C n+1 [a, b], and let p be the polynomial of degree
≤ n that interpolates f at n + 1 distinct points x0 , x1 , ..., xn ∈ [a, b].
Then, to each x ∈ [a, b] there exists a ξx ∈ (a, b) such that

k
f (n+1) (ξx ) Y
f (x) − p(x) = (x − xi )
(n + 1)! i=0
Where f (n) (x) denotes the nth derivative of f . This remainder term
will often be denoted Rn+1 .

Theorem 3.3. Let f : R → R be 3 times continuously differentiable.

Let x* be such that f 0 (x∗ ) = 0 and f 00 (x∗ ) 6= 0. Then the sequence {xn }
√
generated by (2.7) converges to x∗ with order 1+ 5
2
.

Proof. We first want to prove that (2.7) does indeed converge to our
minimizer, x∗ .
Let L(x) be the 2 point interpolant for f 0 (x). Then, with Lemma
3.2, we have:

f 000 (ξ)
f 0 (x) − L(x) = (x − xk−1 )(x − xk )
2
Since xk+1 is generated by maximizing q(x), we have that L(xk+1 ) =
0. Thus,

f 000 (ξ)
f 0 (xk+1 ) = (xk+1 − xk−1 )(xk+1 − xk )
2
If we substitute the recursion for xk+1 given by (2.7) into the above
equation, we can simplify this into:
 METHOD OF QUADRATIC INTERPOLATION 7

1 (xk − xk−1 )2
(3.1) 0
f 0 (xk+1 ) = f 000 (ξ)fk0 fk−1
2 (fk0 − fk−1
0
)2
We now want to take advantage of the Mean Value Theorem. We
have:

fk0 − fk−1
0
= f 00 ξ0
xk − xk−1
Where ξ0 ∈ (xk , xk−1 ). Also note that since f 0 (x∗ ) = 0, we have:

(3.2) fi0 = f 0 (xi ) − f 0 (x∗ ) = (xi − x∗ )f 00 (ξi )

For some ξi ∈ (xi , x∗ ), i = k − 1, k, k + 1. Using (3.2) and the

previous line, we can rewrite (3.1) as so:

1 f 000 (ξ)f 00 (ξk )f 00 (ξk−1 )

(3.3) xk+1 − x∗ = (xk − x∗ )(xk−1 − x∗ )
2 f 00 (ξk+1 )[f (ξ0 )]2
Now, let ei = |xi − x∗ |, i = k − 1, k, k + 1. Find constants m1 , m2 ,
M1 , M2 such that

0 < m1 ≤ |f 000 (x)| ≤ M1

0 < m2 ≤ |f 00 (x)| ≤ M2

where x is any value in the bracketing interval considered. Then, we

can bound (3.3):

m1 m22 M1 M22
(3.4) e e
k k−1 ≤ e k+1 ≤ ek ek−1
2M23 2m32
However, using (3.3):
8 KELLER VANDEBOGERT

ek+1 1 f 000 (ξ)f 00 (ξk )f 00 (ξk−1 )

=
ek ek−1 2 f 00 (ξk+1 )[f (ξ0 )]2
Now if we let each xi → x∗ , each ξi will tend toward x∗ as well since
they are contained within their respective bracketing intervals and f 00 ,
f 000 are continuous. Thus we see:

ek+1 f 000 (x∗ )

(3.5) → 00 ∗
ek ek−1 2f (x )
This is well-defined since f 00 (x∗ ) 6= 0 by assumption. Define M =
 000 ∗ 
 f (x ) 
 2f 00 (x∗ ) , and using (3.4) and (3.5), we have that

(3.6) ek+1 ≤ M ek ek−1

In which case if δ = max{ek , ek−1 },

ek+1 ≤ M δ 2 → 0

So our sequence converges to x∗ when x0 6= x1 . Now we seek to

determine the rate of convergence. To do this, we define yk = log(M ek ).
We then have, using (3.6):

yk+1 = yk + yk−1
√
1+ 5
This is of course a simple a recurrence relation. Let φ = 2
and
√
1− 5
φ̄ = 2
. We easily find the closed form of yk to be:

(3.7) yk = αφk + β φ̄k

Where α, β are to be determined from our initial conditions. Since

φ̄ < 1, for k sufficiently large, yk ≈ αφk . We then have, as k → ∞:
 METHOD OF QUADRATIC INTERPOLATION 9

M ek+1 exp(αφk+1 )
≈ →1
M φ eφk exp(αφkφ )
With this:

|xk+1 − x∗ |
→ M φ−1
|xk − x∗ |φ
And with Definition 3.1, this implies that xk → x∗ with order φ =
√
1+ 5
2
.


Remark 3.4. We see that the secant method converges at a superlinear

rate, whereas Newton’s method converges quadratically under suitable
conditions.

We now seek to find the rate of convergence of the 3-point method.

The following result answers this question:

Theorem 3.5. Let f ∈ C 4 . Suppose that x∗ satisfies f 0 (x∗ ) = 0 and

f 00 (x∗ ) 6= 0. Then the sequence {xk } generated by (2.10) converges to
x∗ of order 1.32.

Proof. Similar to Theorem 3.3, we begin by writing our function f in

the following form:

(3.8) f (x) = L(x) + R3 (x)

Where R3 in the third degree remainder term for the Lagrange inter-
polant. Now, since x∗ is a critical point, we clearly have that f 0 (x∗ ) = 0.
With this and (3.8):
10 KELLER VANDEBOGERT

L0 (x∗ ) + R30 (x∗ ) = 0

L0 (x∗ ) can be easily computed, we have:

(3.9)
2x∗ − (x2 + x3 ) 2x∗ − (x1 + x3 ) 2x∗ − (x1 + x2 )
f1 +f2 +f3 +R30 (x∗ ) = 0
(x1 − x2 )(x1 − x3 ) (x2 − x1 )(x2 − x3 ) (x3 − x1 )(x3 − x2 )
We now use (3.9) to solve for x∗ :

 
∗ f1 f2 f3
2x + + +R30 (x∗ )
(x1 − x2 )(x1 − x3 ) (x2 − x1 )(x2 − x3 ) (x3 − x1 )(x3 − x1 )
f1 (x2 + x3 ) f2 (x1 + x3 ) f3 (x1 + x2 )
= + +
(x1 − x2 )(x1 − x3 ) (x2 − x1 )(x2 − x3 ) (x3 − x1 )(x3 − x2 )
Which then gives:

1 R30 (x∗ )
(3.10) x∗ = − f1 f2 f3
2 (x1 −x2 )(x1 −x3 )
+ (x2 −x1 )(x2 −x3 )
+ (x3 −x1 )(x3 −x1 )
f1 (x2 +x3 ) f2 (x1 +x3 ) f3 (x1 +x2 )
+ +
 
1 (x1 −x2 )(x1 −x3 ) (x2 −x1 )(x2 −x3 ) (x3 −x1 )(x3 −x2 )
+ f1 f2 f3
2 (x1 −x2 )(x1 −x3 )
+ (x2 −x1 )(x2 −x3 )
+ (x3 −x1 )(x3 −x1 )

However, it is easy to see that we can use (2.10) and rewrite it in an

awfully convenient form:

f1 (x2 +x3 ) f2 (x1 +x3 ) f3 (x1 +x2 )

1 (x1 −x2 )(x1 −x3 )
+ (x2 −x1 )(x2 −x3 )
+ (x3 −x1 )(x3 −x2 )
x4 = f1 f2 f3
2 (x1 −x2 )(x1 −x3 )
+ (x2 −x1 )(x2 −x3 )
+ (x3 −x1 )(x3 −x1 )

But this is precisely the rightmost term of (3.10), so we easily deduce:

1 R30 (x∗ )
(3.11) x∗ − x4 = − f1 f2 f3
2 (x1 −x2 )(x1 −x3 )
+ (x2 −x1 )(x2 −x3 )
+ (x3 −x1 )(x3 −x1 )
 METHOD OF QUADRATIC INTERPOLATION 11

Now, similar to the proof of Theorem 3.3, let ei = x∗ − xi , where

i = 1, 2, 3, 4. With elementary manipulations of (3.11), we find the
following form:

(3.12)
1
e4 f1 (e2 −e3 )+f2 (e3 −e1 )+f3 (e1 −e2 ) = − R30 (x∗ )(e1 −e2 )(e2 −e3 )(e3 −e1 )


2
By means of the Taylor expansion about x∗ , it is clear that

1
fi = f (x∗ ) + e2i f 00 (x∗ ) + O(e3i )
2
Where we’ve used the fact that f 0 (x∗ ) = 0. Now, for ei sufficiently
small, we have neglect the third order term of the Taylor expansion.
Substituting each respective Taylor expansion into (3.12), we get the
following:

 1 1 1
f (x∗ )+ e21 f 00 (x∗ ) (e2 −e3 )+ f (x∗ )+ e22 f 00 (x∗ ) (e3 −e1 )+ f (x∗ )+ e23 f 00 (x∗ ) (e1 −e2 )





e4
2 2 2
1
= − R30 (x∗ )(e1 − e2 )(e2 − e3 )(e3 − e1 )
2
Now examine the coefficient of e4 in the above expression. We find:

(3.13)
1 1 1
f (x∗ )+ e21 f 00 (x∗ ) (e2 −e3 )+ f (x∗ )+ e22 f 00 (x∗ ) (e3 −e1 )+ f (x∗ )+ e23 f 00 (x∗ ) (e1 −e2 )





2 2 2
1 00 ∗  2 
= f (x ) e1 (e2 − e3 ) + e22 (e3 − e1 ) + e23 (e1 − e2 )
2
1  
= f 00 (x∗ ) (e22 − e21 )e3 + (e23 − e21 )e2 + (e23 − e22 )e1
2
1
= f 00 (x∗ )(e1 − e2 )(e2 − e3 )(e3 − e1 )
2
12 KELLER VANDEBOGERT

Using (3.12) and (3.13), we see:

1
(3.14) e4 = − R0 (x∗ )
f 00 (x∗ ) 3
We now seek an explicit form of the derivative of our remainder term.
From the form given by Lemma 3.2, it can be found that

1 1
(3.15) R30 (x∗ ) = − f 000 (ξx∗ )(e1 e2 + e2 e3 + e3 e1 ) + f (4) (η)e1 e2 e3
6 24
Where η is some constant. We now want to to neglect the fourth
order term, because the fourth derivative is bounded and the product
e1 e2 e3 will become arbitrarily small much faster than the first term.
Using (3.15) and (3.14):

f 000 (ξx∗
(3.16) e4 = (e1 e2 + e2 e3 + e3 e1 )
6f 00 (x∗ )
Which easily generalizes to:

(3.17) ek+2 = M (ek−1 ek + ek ek+1 + ek+1 ek−1 )

f (ξx∗ 000
Where we define M = − 6f 00 (x∗ ) . It is also clear that for sufficiently

small ek , ek+1 = O(ek ) = O(ek−1 ). Thus, it is possible to find some

constant, M̄ , such that:

|ek+2 | ≤ M̄ |ek−1 ||ek |

In a similar fashion to the proof for Theorem 3.3, we can easily define
δi = log(M̄ |ei |). By doing so, we easily see that:
 METHOD OF QUADRATIC INTERPOLATION 13

(3.18) δk+2 ≤ δk + δk−1

Now tend k sufficiently large. Then, it is clear that δk+2 ≈ δk + δk−1 .

With this and (3.18) we spawn the following recurrence:

δk+2 = δk + δk−1

This recurrence has a solution of the form:

(3.19) δk = ατ1k + βτ2k + γτ3k

Where each τi satisfies the characteristic equation τ 3 − τ − 1 = 0, i

= 1, 2, 3, and our constants are to be determined by initial conditions.
Let τ1 denote the real root of this characteristic equation. Then, by
examining the roots of this equation, we see that τ2 = τ¯3 , complex
conjugates, and that |τ2 | = |τ3 | < 1. Thus for k sufficiently large,
qk ≈ ατ1k .
Now examine δk − τ1 δk−1 . By (3.19), we can clearly see that this
must tend to 0. However, using our definition for δk :

M̄ |ek+1 |
δk+1 − τ1 δk → 0 =⇒ →1
M̄ τ1 |ek |τ1
But then we have that

|xk − x∗ |
(3.20) → M̄ τ1 −1
|xk−1 − x∗ |τ1
But of course by Definition 3.1, this implies xk → x∗ with order τ1 .
Numerically, we see that τ1 ≈ 1.3247, and we are done.

14 KELLER VANDEBOGERT

From these two results we see that having knowledge of the derivative
of f does indeed allow for a more accurate interpolant and thus a faster
rate of convergence to the minimizer x∗ .

4. Numerical Tests

A selection of interesting and nonelementary functions were chosen

to optimize. The material below gives the function, along with the
estimated critical points solved in MATLAB. The rightmost results
are found with Method 2 (secant method), and the leftmost results
are found with Method 3 (3-point). The appendix will contain the
MATLAB code used for each method, and some discussion on possible
improvements. Our allowed tolerance was 10−32 , or machine precision.
 METHOD OF QUADRATIC INTERPOLATION 15

(1) f (x) = e−2x + x2

√ √
(2) f (x) = −2e− x
( x + 1) + cos(x)

 
1
(3) f (x) = 720
x6 − 36x5 + 450x4 − 2400x3 + 5400x2 − 4320x + 720
(The 6th Laguerre Polynomial)

1
(4) f (x) = Γ(x)
(Reciprocal of the Gamma Function)
16 KELLER VANDEBOGERT

(5) f (x) = 64x7 − 112x5 + 56x3 − 7x (The 7th Chebyshev Polyno-

mial)

 
(6) f (x) = x log(x) − 1 − sin(x)

(7) f (x) = −x + e−x + x log(x)

(8) f (x) = −li(x) + x log(log(x)) + cos(x) (li(x) denotes the Loga-

rithmic Integral)
 METHOD OF QUADRATIC INTERPOLATION 17

1√ x3
(9) f (x) = 2
πerf(x) − 3
(erf(x) denotes the Error Function)

1√
(10) f (x) = 2
πerf(x) − sin(x)

It is interesting to note some of the discrepancies in iteration. In-

deed the 3-point method took almost twice as many iterations as the
secant method on average. However, this could be due to many fac-
tors. Firstly, the use of nonelementary functions perhaps made the
3-point interpolant less accurate than the secant method, which took
advantage of the function’s derivative.
It would be expected that the 3-point interpolant could be as good
as or possibly better than the secant method for polynomials. For
function (3), this holds, however, when looking at (5), we see that
it took significantly more iterations for just a simple polynomial. We
also note the largest discrepancy in iterations occurred for function (2).
This is an interesting case because we are only using transcendental
functions, as opposed to some of the other nonelementary functions.
18 KELLER VANDEBOGERT

5. Conclusion

We explored the method of quadratic interpolation for optimization

and some of the different approaches to actually find our quadratic
interpolant. Analytically, it was shown that the secant method should
in general converge more quickly than the 3-point method. This is
intuitively explained by the fact that the derivative of a function gives
information on that function’s curvature. After this, numerical tests
were shown on a wide variety of functions and on average the secant
method did significantly better than was theoretically predicted.
 METHOD OF QUADRATIC INTERPOLATION 19

Appendix

Two codes were written in MATLAB for Section 4 on numerical

testing. In this appendix, the code will be provided and some dicussion
on possible improvements as well.

Secant Method.

Figure 1. The Secant Method

This code is actually pretty compact and runs quickly. It would be

useful to input some kind of searching mechanism which can actually
determine the bracketing interval for the user, whereas this code re-
quires the user to find their own bracketing interval. Also, in many
cases, if your critical point is not contained within the initial interval,
20 KELLER VANDEBOGERT

this programs will still find the optimizer, but it takes more iterations
than normal.
 METHOD OF QUADRATIC INTERPOLATION 21

3-Point Method.

Figure 2. 3-Point Method

This code is not a perfect recreation of the theoretical 3-Point Method,

because it does not search both ends of the bracketing interval to find
the new bracketing interval. This is one major reason why it is possible
that there was a larger than theoretically expected discrepancy between
the amount of iterations in Section 4. For a code that compares both
ends of the interval, we would find that each consecutive interval is
more accurate. Another point of this is that the points chosen for the
interval were always uniformly spaced. An interesting consideration is
to find a bracketing interval, and then interpolate the Chebyshev nodes
as opposed to just the uniformly spaced nodes. This special set of
nodes will provide the best possible polynomial interpolant over any
22 KELLER VANDEBOGERT

given interval. Also, another possible reason for the discrepancy in it-
erations is the tolerance values. Choosing the tolerance too small in
some cases would cause the bracketing interval to become too small
and it was difficult for MATLAB to distinguish between points when
the tolerance was right at machine precision.

References
[1] Optimization Theory and Methods: Nonlinear Programming. Wenyu Sun, Ya-
Xiang Yuan. 89-98.