FMN050–Spring 2014.

Claus Führer and Alexandros Sopasakis page 4

n xn f (xn ) Error = |xn − root|

1 0.5 −0.375 0.1527
2 0.25 0.2656 0.097296
−2
3 0.375 −7.23 × 10 0.027704
4 0.3125 9.30 × 10−2 3.4796 × 10−2
5 0.34375 9.37 × 10−3 0.3546 × 10−2

20 0.3472967 −9.54 × 10−7 3.6162 × 10−7

Therefore the root of f (x) = x3 − 3x + 1 is approximated to be at x = .3472967 with an error of
approximately .000000362.
We can obtain the following result on the rate of convergence of the bisection method:

Theorem 1.1.1. Suppose f (x) is continuous on [a, b] and f (a)f (b) ≤ 0. Then after n steps the
computed root will insure a maximum possible error of (b − a)/2n+1 .

Note that the theorem implies that at the very worst case the bisection method converges to the
root at the same rate as the sequence {2−n } converges to zero. Depending on the given problem the
method may converge even faster.
The bisection method will work under most circumstances but its convergence rate may be slow.
As a result when programming the method, and any other method which you might program for
that matter, you should allow a maximum number of iterates after which the code should terminate
anyway even though the required tolerance has not been reached yet. This will prevent the algorithm
from getting into infinite loops as well.
Let us now provide a quick overview of advantages and possible disadvantages of the bisection
method. Note that the list below and other similar lists thoughout the text are not intended to be
complete and in some cases may require further arguing.
Advantages:

• Always converges to the root (this is not the case for some other methods which we will
encounter)!

• Simple and easy to implement.

Disadvantages:

• It may converge slowly to the root (especially if we use complicated functions or large starting
intervals).

• An early iteration which is actually quite close to the root may be easily discarded.

1.2 Newton-Raphson Method

This is probably the most well known (also known as Newton’s) method for finding function roots.
The method can produce faster convergence by cleverly implementing some information about the
function f . Unlike the bisection method, Newton’s method requires only one starting value and does
not need to satisfy any other serious conditions (except maybe one!). This is a one-point method.
FMN050–Spring 2014. Claus Führer and Alexandros Sopasakis page 5

Let us start by presenting one of the ways that we can actually obtain such a numerical root
finding scheme. We start by writing the Taylor polynomial for f (x) of first order for x near x∗ :

0 (x − x∗ )2 00
f (x) = f (x∗ ) + (x − x∗ )f (x∗ ) + f (ξ(x))
2
where as usual ξ(x) lies between x and x∗ . Let us denote the root we are trying to find by r. Thus
f (r) = 0. Therefore,
0 (r − x∗ )2 00
0 = f (x∗ ) + (r − x∗ )f (x∗ ) + f (ξ(r)) (1.2)
2
Assuming now that we take x∗ close enough to the true root r then the term (r − x∗ ) is very small
and further (r − x∗ )2 is almost zero! Thus we can disregard the last term in (1.2). As a result, under
this assumption, we obtain the following scheme,
0
0 = f (x∗ ) + (r − x∗ )f (x∗ ).

Therefore rewriting the above and solving for the root r we obtain the famous Newton-Raphson
method,
f (x∗ )
r = x∗ − 0 ∗
f (x )
This gives the idea of how we can iterate an initial guess in order to obtain the root of f :

f (xn−1 )
xn = xn−1 −
f 0 (xn−1 )

The following theorem, without proof, gives a little bit more insight on the conditions under which
we expect the method to converge to the root of the function f (x).
0
Theorem 1.2.1. Suppose f ∈ C 2 ([a, b]). If the root r is in [a, b] and f (r) ≤ 0 then there exists
δ > 0 such that the Newton-Raphson method produces a sequence of iterates {xn }∞
n=1 which converges
to r for any initial approximation x0 which is near the root [r − δ, r + δ].

That is if we choose the initial guess x0 close enough to the root of the function f (x) we are
essentially guaranteed that we will find the true root!
In short the method is implemented as follows:
0
1. Obtain a starting guess x0 and find a formula for the derivative f of f .

2. Produce the next iterate xn through the following equation:

f (xn−1 )
xn = xn−1 −
f 0 (xn−1 )

3. Stop if any of the following occurs: required tolerance is reached, maximum iterates exceeded,
0
f (xn−1 ) = 0.

Advantages:

• Fast convergence!
FMN050–Spring 2014. Claus Führer and Alexandros Sopasakis page 6

Disadvantages:
0
• Calculating the required derivative itself f (xn−1 ) for every iteration may be a costly task for
some functions f .

• May not produce a root unless the starting value x0 is close to the actual root of the function.
0
• May not produce a root if for instance the iterations get to a point xn−1 such that f (xn−1 ) = 0.
Then the method fails!

1.3 Regula Falsi Method

This is another two-point encompassing method. However it differs from the bisection method since
it uses some information from the function f (x) in order to faster predict its root much like Newton’s
method. In essence this is a clever remake of the bisection and Newton’s method we just encountered.
As you might recall for instance the bisection method always divides in half the given interval in
order to obtain the next iterate closer to the root. Instead regular falsi tries to cleverly divide the
interval closer to the root!
Let’s take an example in order to outline the thinking behind this new method: We let for
instance x0 = 2 and x1 = 10 for the function f (x) = x2 − 6 and we wish to approximate its root in
that interval [2, 10]. Then we have that f (2) = −2 while f (10) = 94 and note that the condition
f (x0 )f (x1 ) < 0 is satisfied. Here the bisection method, which does not use any information from
the function f , would produce the next iterate at x2 = 6. Whereas just looking at the two values
for f we would expect the root to be much closer to x0 = 2. The regula falsi method instead takes
into account precisely that information about the function at the end points in order to make an
educated guess as to where it is most likely that the root ought to be closer to. Regula falsi weighs
that information in the calculation and produces the next iterate much closer to x0 = 2. In gives
that x2 = 2.16. Note the difference between the two methods in just one iteration:

• Bisection method x2 = 6,

• Regula Falsi method x2 = 2.16,

• True root x = 2.4496

As you can see regula falsi got a lot closer to the true solution in just one iteration.
Let’s us then describe in detail how the method is implemented. In essence the algorithm which
we provide below is the same as the bisection method except for the main formula. The pseudo-code
follows:

1. Obtain two starting guesses, xi and xj such that f (xi )f (xj ) ≤ 0.

2. Produce the next iterate xk from the following formula

(xj − xi )
xk = xi − f (xi )
f (xj ) − f (xi )

3. If required tolerance or maximum iterates have been reached then stop.

FMN050–Spring 2014. Claus Führer and Alexandros Sopasakis page 7

4. Otherwise continue as follows:

If f (xk ) has the same sign as f (xi ) redo step 2 above by assigning xi = xk .
If f (xk ) has the same sign as f (xj ) redo step 2 above by assigning xj = xk .

Although this method is more clever than the bisection method it also have its drawbacks. In fact
it converges very slowly for certain functions which display high curvature. Therefore there exists a
modification which allows a small speed up for such type of functions. We will present this method
in the next section.
Advantages:

• No need to calculate a complicated derivative (as in Newton’s method).

Disadvantages:

• May converge slowly for functions with big curvatures.

• Newton-Raphson may be still faster if we can apply it.

1.4 Tolerance criteria

Let us also discuss briefly how we should measure tolerance limits. Measuring when a given iterate
is “close enough” to the true root is important in its own right since it influences significantly the
number of times a given method must be performed.
There are a number of different ways to measure how close an iterate xn is to a root r for a given
function f (x). We may wish for instance to just measure how close f (xn ) is to the true root f (r).
Note that since f (r) = 0 then we can simply measure

|f (xn ) − f (r)| = |f (xn )| ≤ TOL

where TOL denotes a given predefined tolerance. This is the typical way to measure absolute error.
Similarly we might instead wish to measure whether the iterates themselves are converging suffi-
ciently. This is another form of absolute error,

|xn − xn−1 | ≤ TOL

Another possibility is to have a bit more impartial way of measuring differences by using something
called the relative error
|xn − xn−1 |
≤ TOL
xn
Thus, in this later formulation, we try to disengage the size of the values of the iterates by “normal-
izing” them. In essence therefore the above fraction should be less than one when the iteration is
converging.
In all cases, as also mentioned before, we should not simply rely on the tolerance computation
provided above in order to terminate an iteration. We should always include a secondary mechanism
in our iterations which terminates the method once a maximum number of iterations has been
performed.
FMN050–Spring 2014. Claus Führer and Alexandros Sopasakis page 8

1.5 Modified Regula Falsi Method

As mentioned above the problem of slow convergence rates appears when the function in question f
has high concavity. In this case the usual regula falsi method produces iterates only on one side of
the actual root at a slow rate. So to speed things up the modified regula falsi method tries to force
the scheme to produce iterates on the other side of the root. This is achieved by changing the value
of f (x) to half: f (x)/2. In more detail the method is applied as follows:

1. Obtain two starting guesses, xi and xj such that f (xi )f (xj ) ≤ 0.

2. Produce the next iterate xk from the following formula

(xj − xi )
xk = xi − f (xi )
f (xj ) − f (xi )

and let fi = f (xi ), fj = f (xj ) and fk = f (xk ). Let us also denotes the previous iterate at xk−1
as fk−1 .

3. If required tolerance or maximum iterates have been reached then stop.

4. If fi fk > 0 then go to step 6

5. (Divide fi in half):
Let xj = xk and fj = fk .
If fk−1 · fk > 0 then xk−1 = xk , fk−1 = fk and fi = fi /2. Go to step 2.

6. (Divide fj in half):
Let xi = xk and fi = fk .
If fk−1 · fk > 0 then xk−1 = xk , fk−1 = fk and fj = fj /2. Go to step 2.

Let us now give an example of how the usual and the modified regula falsi method fairs in prac-
tice:
Example:
Suppose f (x) = x2 − 6 and we wish to obtain the root of the function in the interval [2, 3] with an
error not to exceed 10−5 .
Solution:
We start as usual by setting x0 = 2 and x1 = 3. We first implement, for comparison purposes the
usual Regula Falsi method:

Regula Falsi

n a b c f (c) Rel. Error

1 2.000000 3.000000 2.400000 −0.240000 0.166667
2 2.400000 3.000000 2.444444 −0.024691 0.018182
3 2.444444 3.000000 2.448980 −0.002499 0.001852
4 2.448980 3.000000 2.449438 −0.000252 0.000187
5 2.449438 3.000000 2.449485 −0.000026 0.000019
6 2.449485 3.000000 2.449489 −0.000003 0.000002
FMN050–Spring 2014. Claus Führer and Alexandros Sopasakis page 9

Now we present the implementation of the modified version of the above for the exact same
problem:
Modified Regula Falsi

n a b c f (c) Rel. Error

1 2.000000 3.000000 2.400000 −0.240000 0.166667
2 2.400000 3.000000 2.482759 0.164090 0.033333
3 2.400000 2.482759 2.449153 −0.001652 0.013722
4 2.449153 2.482759 2.449487 −0.000011 0.000137
5 2.449487 2.482759 2.449492 0.000011 0.000002
Note that in fact the modified regular falsi did slightly better. Also note how the values of a and
b changed in the modified regula falsi whereas b did not change throughout in the usual regula falsi.
Advantages:

• No need to calculate a complicated derivative.

• Increased convergence when compared to the usual regula falsi method.

Disadvantages:

• Not many, but Newton-Raphson may be still faster if we can apply it.

1.6 Secant Method

The Secant method is another two point scheme which avoids the problem of calculating the derivative
of the function f at each iteration (as Newton’s method does) but still obtains improved convergence
to the root.
The method uses the same formulation as Regula Falsi or the Modified Regula Falsi to obtain
the next iterate xn ,
xn−1 − xn−2
xn = xn−1 − f (xn−1 )
f (xn−1 ) − f (xn−1 )
Thus it has a much improved convergence over the bisection method since it uses cleverly information
about the function f (x). However this is not an encompassing scheme. As such it does not require
that the iterates xn−1 and xn−2 contain the solution.
In short the pseudocode for the Secant method goes as follows:

1. Obtain two starting guesses, xn−1 and xn−2 .

2. Produce the next iterate xn through the formula,

xn−1 − xn−2
xn = xn−1 − f (xn−1 )
f (xn−1 ) − f (xn−1 )

3. If required tolerance has been reached or maximum number of iterates exceeded then stop with
appropriate message.

4. Otherwise continue from step 2 above.

FMN050–Spring 2014. Claus Führer and Alexandros Sopasakis page 10

As we also showed earlier we expect that in general the convergence of the secant method will be
good but probably not as good as Newton’s method is. In general the Secant method is used in
conjunction with some other method (Bisection) in order to refine the solution and the convergence.
In other words we start our code with the Bisection scheme and then once a certain first tolerance
has been satisfied we pass the result to the Secant (or some other method) for fast and more accurate
resolution of the root.
Example:
Find the root of the function f (x) = x2 − 6 (as usual) in the interval [2, 10] with an accuracy of 10−6 .
Solution:
Lets suppose that we take x0 = 2 and x1 = 10 so that we can compare with results from previous
examples (for that reason we denote a = x0 = 2 and b = x1 = 10).

n a b c f (c) Rel. Error

1 2.000000 10.000000 2.166667 −1.305556 3.615385
2 10.000000 2.166667 2.273973 −0.829049 0.047189
3 2.166667 2.273973 2.460668 0.054889 0.075872
4 2.166667 2.460668 2.449075 −0.002030 0.004734
5 2.460668 2.449075 2.449489 −0.000005 0.000169
6 2.449075 2.449489 2.449490 −0.000000 0.000000
Check these results with the equivalent ones from the regula falsi and the modified regula falsi
methods. What do you think?
Advantages:

• No need to calculate a complicated derivative.

• Increased convergence when compared to the usual Bisection method.

• Not encompassing, thus less things to worry about, easier to program.

Disadvantages:

• Not many, but Newton-Raphson may be still faster.

FMN050–Spring 2014. Claus Führer and Alexandros Sopasakis page 11

1.7 Fixed Point Method

This method is altogether very different in form from the usual root finding schemes we have discussed
thus far. In particular the fixed point method does not solve directly f (x) = 0 but instead it produces
the solution for the problem
g(x) = x
The solution x is called the “fixed point” of g(x). It is not hard to see the reason this problem is
considered as equivalent to our usual f (x) = 0. Simply let f (x) = g(x) − x and use any of the
methods we discussed so far in order to find the root of f . Once you solve the problem f (x) = 0
then you automatically have also found the solution to g(x) = x. Take a minute to convince yourself
about that fact. Some examples later below will further convince you of that.
It is important to note that not all continuous functions g(x) have fixed points! In that respect
the following theorem gives some very simple guidelines for functions with fixed points,

Theorem 1.7.1. If g ∈ C[a, b] and g(x) ∈ [a, b] for all x ∈ [a, b] then g has a fixed point in [a, b].
0
Furthermore assuming g (x) exists on (a, b) and that there exists a positive constant c < 1 such that,
0
|g (x)| ≤ c < 1 for all x ∈ (a, b)

then the fixed point is unique.

Proof: Lets start by first showing that a fixed point does exist for g as specified in the theorem.
Naturally if g(a) = a or g(b) = b then we are done. Then by contradiction we suppose this is not the
case: i.e. instead we must have that g(a) > a and g(b) < b. Let us now define

h(x) = g(x) − x.

Note that since g(x) and x are both continuous functions then h(x) must also be continuous on [a, b]
and the following must hold:

h(a) = g(a) − a > 0 and h(b) = g(b) − b < 0

Therefore by the intermediate value theorem there must exist a r ∈ (a, b) such that h(r) = 0. Thus,
g(r) − r = 0 or g(r) = r in (a, b). That shows existence.
For uniqueness we assume again by contradiction that there exist two different fixed points, say
p and q in [a, b] such that g(p) = p and g(q) = q but p 6= q. Then by the mean value theorem there
must exist a number ξ ∈ (p, q) such that

0 g(p) − g(q)
g (ξ) = (1.3)
p−q

Taking absolute value of (1.3) and rewriting we get,

0
|g(p) − g(q)| = |p − q||g (ξ)| (1.4)
0
Note however that in the statement of the theorem we are given that |g (x)| < c < 1 for all x ∈ (a, b).
Thus (1.4) becomes,
0
|g(p) − g(q)| = |p − q||g (ξ)| ≤ |p − q|c < |p − q|
FMN050–Spring 2014. Claus Führer and Alexandros Sopasakis page 12

The left hand side of the above however is also equal to |p − q|. Therefore we showed that |p − q| <
|p − q|! Not possible and therefore by contradiction we showed there must exists a single fixed point
for g.
It is very important before you go on solving any problem to first check that a fixed point exists
by applying the theorem above or some other type of logical argument. Now to find such a fixed point
numerically we only need to take a single initial guess x0 and generate iterations by substituting it
into the function g(x) itself. After several such iterations the solution (or an approximation to the
solution) should emerge. The following simple algorithm makes this clear:

1. Obtain a starting guess, x0 .

2. Produce the next iterate xn through the formula,

xn = g(xn−1 )

3. If required tolerance has been reached or maximum number of iterates exceeded then stop with
appropriate message. Otherwise continue from step 2 above.

How is it possible that this simple scheme works? Isn’t it surprising that you just plug in the previous
iterate into the function and there comes the next iterate which is closer to the true solution?
Let us give an example to further shed light into the idea. In order to make it easier to compare
we will take our usual example f (x) = x2 − 6 = 0 in the interval [2, 10] which we know has a unique
root and arrange it appropriately in order to become a fixed point problem. That is quite easy simply
(but not optimaly) done by adding x on both sides

x2 − 6 + x = x

So now we have a fixed point problem to solve for which we are guaranteed (why?) to have a solution
in the interval [2, 10]:
Example:
Find the fixed point of the following function g(x) = x2 − 6 + x in the interval [2, 10].
Solution:
The solution can now be easily found by applying our pseudocode from above:

n 0 1 2 3 4 ... 11
xn 2 0 −6 24 594 ... ∞
Error 0 −6 24 594 353424 . . . ∞

What happened? We were supposed to obtain a solution to our fixed point problem near 2.4 not ∞!
The problem is the derivative of g. The fixed point iteration method we outlined above failed in this
case. Why? Because in this problem
0
|g (x)| > 1 in the interval [2, 10].

The next theorem gives some guidelines for convergence or divergence of the fixed point iteration
scheme.
FMN050–Spring 2014. Claus Führer and Alexandros Sopasakis page 13

Theorem 1.7.2. Let, as usual, g ∈ C[a, b] and suppose that g(x) ∈ [a, b] for all x ∈ [a, b]. Further
suppose that the derivative exist on (a, b) and
0
g (x) ≤ c < 1 for all x ∈ (a, b)

If we take any initial value x0 ∈ (a, b) then the sequence arising from

xn = g(xn−1 )

converges to the unique fixed point r ∈ [a, b].

Proof: We already know that there must exist a unique fixed point in [a, b] from our previous
theorem. Note now that from the assumption that g(x) ∈ [a, b] for all x ∈ [a, b] we must also have
that all the xn ’s are also in [a, b]. Using (1.3) and the mean value theorem we then must have,
0
|xn − r| = |g(xn−1 ) − g(r)| = |g (ξ)||xn−1 − r| ≤ c|xn−1 − r|

for some ξ ∈ (a, b). We now keep reapplying this inequality on the above and obtain,

|xn − r| ≤ c|xn−1 − r| ≤ c2 |xn−2 − r| < · · · ≤ cn−1 |x1 − r| ≤ cn |x0 − r| (1.5)

However, c < 1. Thus,

lim |xn − r| ≤ lim cn |x0 − r| = 0
n→∞ n→∞

That is the sequence of {xn }∞

n=1converges to the true solution r.
We can now use this theorem to obtain some very useful results about the error of our approximations
regarding the fixed point algorithm,

Corollary 1.7.3. Suppose g as in Theorem (1.7.2) above. Then the error of our approximating the
fixed point r of g can be estimated by either of the following two formulas:

|xn − r| ≤ cn max{x0 − a, b − x0 } (1.6)

cn
|xn − r| ≤ |x0 − r| (1.7)
1−c
for all n ≥ 1.

Proof: Since r is in [a, b] the first formula (1.6) is clear from inequality (1.5).
To prove the second formula (1.7) we need some more work. Note that, as in Theorem (1.7.2),
the following is true,

|xn+1 − xn | = |g(xn ) − g(xn−1 )| ≤ c|xn − xn−1 | < · · · ≤ cn |x1 − x0 |

Therefore for any m > n ≥ 1 by adding and subtracting similar terms the following must be true,

|xm − xn | = |xm − xm−1 + xm−1 − · · · − xn+1 + xn+1 − xn |

But the right hand side of this is clearly less than,

≤ |xm − xm−1 | + |xm−1 − xm−2 | + · · · + |xn+1 − xn |