ICS 226: Numerical Analysis

Oliver Mhlanga

HARARE INSTITUTE OF TECHNOLOGY

[email protected] or [email protected]; 0712531415

March 23, 2020

Solution of Nonlinear Equations

Solution of Nonlinear Equations

Our goal is to develop simple numerical methods for the

approximate solution of the equation

f (x) = 0 (1)

where f is a real-valued function, defined and continuous on a

bounded and closed interval of the real line.
We will restrict our discussion to the case where x ∈ <, but x ∈ C
can also be of significant interest in practice (especially if f is a
polynomial). Solutions of nonlinear equations are called roots or
zeroes.
Various solution methods will be considered such the bisection
method, fixed-point method, Secant Method, and the
Newton-Raphson method.
Solution of Nonlinear Equations

These methods compute a solution of a nonlinear equation starting

from one initial data, then adopting some iterative method that,
under favorable conditions, will converge to a zero of the function
f.
The number of iterations needed to achieve a solution with a given
accuracy is considered. Iterative procedures can break down (i.e.,
fail) in various ways. The sequence generated by a procedure may
diverge, oscillate, or display chaotic behavior ( a.k.a. ’wandering
behavior’).
Theorem
2.1. Let f be a real-valued function, defined and continuous on a
bounded closed interval [a, b] of the real line. Assume, further,
that f (a)f (b) < 0;then, there exists r in [a, b] such that f (r ) = 0.
Bisection Method.

The bisection method is a procedure that repeatedly ’halves’ the

interval in which a root r has been located. This ’halving’ process
is reiterated until the desired accuracy is reached.
Suppose that we are given an interval [a, b] satisfying (theorem
2.1) and an error tolerance  > 0. The bisection method consists
of the following steps:
1. Define c = (a + b)/2.
2. If b − c ≤ , then accept c as the root and stop.
3. If f (b) × f (c) ≤ 0, then set a = c. Otherwise, set b = c.
Return to step 1.
The interval [a, b] is halved with each loop through steps 1 to 3.
Step 2 will be satisfied eventually, and with it the condition
|r − c| ≤  will be satisfied.
Bisection Method.
Example 2.1. Find the largest root of

f (x) = x 6 − x − 1 = 0 (2)

accurate to within  = 0.001.

With a graph, it is easy to check that 1 < r < 2

We choose a = 1, b = 2; then f (a) = −1, f (b) = 61, and

theorem (2.1) is satisfied.
Bisection Method.

n a b c b-c f(c)
1 1.0000 2.0000 1.5000 0.5000 8.8906
2 1.0000 1.5000 1.2500 0.2500 1.5647
3 1.0000 1.2500 1.1250 0.1250 -0.0977
4 1.1250 1.2500 1.1875 0.0625 0.6167
5 1.1250 1.1875 1.1562 0.0312 0.2333
6 1.1250 1.1562 1.1406 0.0156 0.0616
7 1.1250 1.1406 1.1328 0.0078 -0.0196
8 1.1328 1.1406 1.1367 0.0039 0.0206
9 1.1328 1.1367 1.1348 0.0020 0.0004
10 1.1328 1.1348 1.1338 0.00098 -0.0096

Table 1: Bisection Method for (??)

Bisection Method.
Convergence analysis
Let an , bn and cn denote the nth computed values of a, b and c,
respectively. Then we easily get
1
bn+1 − an+1 = (bn − an ), n ≥ 1,
2
and it is straightforward to deduce that
1
bn − an = (b − a), n≥1 (3)
2n−1
where b − a denotes the length of the original interval with which
we started. Since the root r ∈ [an , cn ] or r ∈ [cn , bn ], we know that

1
|r − cn | ≤ cn − an = bn − cn = (bn − an ) (4)
2
Bisection Method.
This is the error bound for cn that is used in step 2 of the earlier
algorithm. Combining it with (??), we obtain the further bound
1
|r − cn | ≤ (b − a)
2n
This shows that the iterates cn → r as n → ∞.
To see how many iterations will be necessary, suppose we want to
have
|r − cn | ≤ 
This will be satisfied if
1
(b − a) ≤ 
2n
Solving for n we get

ln( b−a
 )
n≥ (5)
ln 2
Bisection Method.

For the previous example (??),we need

1
ln( 0.001 )
n≥ = 9.97
ln 2
i.e., we need n = 10 iterates, exactly the number computed.
Advantages of the Bisection Method
I It is guaranteed to converge.
I The error bound (??) is guaranteed to decrease by one-half
with each iteration
Many other numerical methods have variable rates of decrease for
the error, and these may be worse than the bisection method for
some equations.
Bisection Method.

Disadvantage of the Bisection Method

I generally converges more slowly than most other methods.
For functions f (x) that have a continuous derivative, other
methods are usually faster. These methods may not always
converge; when they do converge, however, they are almost always
much faster than the bisection method.
Bisection Method.

The Matlab code for bisection is

k = 0;
while abs(b − a) > eps ∗ abs(b)
x = (a + b)/2;
if sign(f(x)) == sign(f(b))
b = x;
else
a = x;
end
k = k + 1;
end
Newton-Raphson Method.
We are now going to establish general methods for computing a
root of the equation f (x) = 0, where f (x) can be an algebraic or
transcendental function.

Figure: The schematic for Newtons method

Newton-Raphson Method.
If we represent the function y = f (x) graphically, the problem can
be so formulated that we are looking for the intersection between
the curve and the x-axis.
Let x0 be a good estimate of the true root α and let α = x0 + h.
Since the true root is α, the number h measures how far the
estimate x0 is from the truth.
The equation of the tangent line to the graph y = f (x) at the
point (x0 , f (x0 )) is

y − f (x0 ) y − f (x0 )
= f 0 (x0 ) ⇒ x = x0 +
x − x0 f 0 (x0 )

The tangent line intersects the x-axis when y = 0 & x = x1 so that

f (x0 )
x1 = x0 −
f 0 (x0 )
Newton-Raphson Method.

Continue in this way. If xn is the current estimate, then the next

estimate xn+1 is given by

f (xn )
xn+1 = xn − , n = 0, 1, 2, .... (6)
f 0 (xn )

(??) is referred to as the Newton’s method, or

Newton-Raphson, for solving f (x) = 0.
Newton-Raphson Method.

It is guaranteed to converge if the initial guess x0 is close enough,

but it is hard to make a clear statement about what we mean by
close enough because this is highly problem specific. A sketch of
the graph of f(x) can help us decide on an appropriate initial guess
x0 for a particular problem.
Example 2.2: From example 2.1, Use Newtons method to solve
(??).
Here f (x) = x 6 − x − 1, f 0 (x) = 6x 5 − 1
and the iteration is given by

xn6 − xn − 1
xn+1 = xn − (7)
6xn5 − 1
Newton-Raphson Method.

Newton’s Method for x 6 − x − 1 = 0

n xn f (xn ) xn − xn−1 α − xn−1
0 1.5 8.89E+1
1 1.30049088 2.54E+1 -2.00E-1 -3.65E-1
2 1.18148042 5.38E-1 -1.19E-1 -1.66E-1
3 1.13945559 4.92E-2 -4.20E-2 -4.68E-2
4 1.13477763 5.50E-4 -4.68E-3 -4.73E-3
5 1.13472415 7.11E-8 -5.35E-5 -5.35E-5
6 1.13472414 1.55E-15 -6.91E-9 -6.91E-9
7 1.134724138
The true root is α = 1.134724138, and x6 = α to nine significant
digits.
N.B: Newton’s method may converge slowly at first. However, as
the iterates come closer to the root, the speed of convergence
increases.
Newton-Raphson Method.

Convergence analysis
Let α be the root so that f (α) = 0.
f 0 (xk )ek −f (xk )
Define Error: ek+1 = xk+1 − α = xk − ff 0(x k)
(xk ) − α = f 0 (xk )
Using Taylor’s theorem
0 = f (α) = f (xk − ek ) = f (xk ) − f 0 (xk )ek + 12 f 00 (k )ek2 ,
where k is between xk and α,
f 0 (xk )ek − f (xk ) = 21 f 00 (k )ek2 .
Hence
|f 00 (k )| 2 |f 00 (α)| 2 2
|ek+1 | = 2|f 0 (x )| |ek | ≈ 2|f 0 (α)| |ek | ≡ C |ek |
k
This is called quadratic convergence.
Newton-Raphson Method.

Theorem
2.2 Suppose f 00 is continuous and α is a simple zero of f . Then
∃ α ∈ D & C ∈ <: when Newton’s method is applied with starting
point 0 ∈ D, the sequence xk generated converges to α and
satisfies

|xk+1 − α| ≤ C |xk − α| (8)

The Matlab code is

k = 0;
while abs(x − xprev ) > eps ∗ abs(x)
xprev = x;
x = x - f(x)/fprime(x)
k = k + 1;
end
The Secant Method.

One drawback of Newton’s method is the necessary availability of

the derivative. The secant method avoids this issue by using a
finite difference to approximate the derivative. As a result, f 0 (xn )
is approximated by a secant line through two points on the graph
of f , rather than a tangent line through one point on the graph.
The derivative of the function, f 0 (xn ) is approximated as

f (xn ) − f (xn−1 )
f 0 (xn ) = (9)
xn − xn−1

Substituting Equation (??) in Equation (??) gives

f (xn )(xn − xn−1 )

xn+1 = xn − , n = 1, 2, .... (10)
f (xn ) − f (xn−1 )

Equation (??) is called the secant method.

The Secant Method.

This method now requires two initial guesses, x0 and x1 , but unlike
the bisection method, the two initial guesses do not need to
bracket the root of the equation.
When secant method converges, it will typically converge faster
than the bisection method. However, since the derivative is
approximated as given by equation (??), it typically converges
slower than the Newton-Raphson method.
Example 2.3. Solve f (x) = x 6 − x − 1
The Secant Method.

Secant Method for x 6 − x − 1 = 0

n xn f (xn ) xn − xn−1 α − xn−1
0 2.0 61.0
1 1.0 -1.0 -1.0
2 1.01612903 -9.15E-1 1.61E-2 1.35E-1
3 1.19057777 6.57E-1 1.74E-1 1.19E-1
4 1.11765583 -1.68E-1 -7.29E-2 -5.59E-2
5 1.13253155 -2.24E-2 1.49E-2 1.71E-2
6 1.13481681 9.54E-4 2.29E-3 2.19E-3
7 1.13472365 -5.07E-6 -9.32E-5 -9.27E-5
8 1.13472414 -1.13E-9 4.29E-7 4.92E-7
The iterate x8 equals α rounded to nine significant digits.
Secant Method.

A bit on convergence analysis

Assume |ek+1 | ≈ η|ek |α ,
where η, α > 0 are constants, i.e.,
|ek+1 |
η|ek |α → 1 as k → ∞.
On can show that √
|ek+1 | = η|ek |α , α = 1+2 5 ≈ 1.62.
This is called superlinear convergence. (1 < α < 2)
It converges for f if x0 & x1 are close to the root r .
Secant Method.

The formulation in the following Matlab code is a little more

compact:
while abs(b − a) > eps ∗ abs(b)
c = a;
a = b;
b = b + (b - c)/(f(c)/f(b)-1);
k = k + 1;
end
Fixed point iteration.

Strictly speaking, Newton-Raphson’s methods could be considered

as iterative methods. We want to solve

f (x) = 0 (11)

If f : < → < is a function, then the fixed point can be viewed as

the intersection of the curve y = f (x) and the straight line y = x.
So that (??) can be transformed into the form

x = g (x) (12)

where |g 0 (x)| < 1 (locally convergent) then we choose x0 to

compute x1 = g (x0 ), x2 = g (x1 ), ... and in general

xn+1 = g (xn ), (n = 0, 1, 2, 3, ....) (13)

Fixed point iteration.
Convergence (b) and divergence (a) of the fixed point iteration
Fixed point iteration.
Example 2.4. Find a solution of f (x) = x 3 + x − 1 by iteration.
Solution: f (x) = x 3 + x − 1 = 0. The approximate root can be
found by use of a rough sketch

1
We can write the equation in the form x = 1+x 2
, thus
1
xn+1 =
1 + xn2
Fixed point iteration.

Let g (x) = 1
1+x 2
, |g 0 (x)| = 2x
(1+x 2 )2
, |g 0 (1)| = 1
2 <1

n xn
0 1.00000
1 0.50000
2 0.80000
3 0.61975
4 0.72897
5 0.65300
- -
- -
28 0.82329

The exact root is 0.682328 to six decimal places.

Fixed point iteration.

Convergence Analysis
Our iteration is xk+1 = g (xk ). Let r be the exact root, s.t.
r = g (r ).
Define Error: ek = |xk − r |.

ek+1 = |xk+1 − r | = |g (xk ) − r | = |g (xk ) − g (r )|

= |g 0 ()||xk − r |, ( ∈ (xk , r ), since g is continuous)
⇒ ek+1 = |g 0 ()|ek

Observation
-if |g 0 ()| < 1, then ek+1 < ek , error decreases, linear convergence.
-if |g 0 ()| > 1, then ek+1 > ek , error increases, iteration diverges.
Fixed point iteration.

Pseudo Code
r=fixedpoint(’g’,x,tol,nmax)
r=g(x); % first iteration
nit=1; while (abs(r-g(r))>tol and nit<nmax) do
r=g(r);
nit=nit+1;
end
end
Fixed point iteration.
Fixed point iteration.

Corollary
2.1 Suppose f satisfies the conditions of the preivious theorem and
the sequence xk k≥0 is produced by the function iteration
xk+1 = g (xk ) with and arbitrary initial point x0 . Then
|ek | = |xk − r | ≤ M k maxx0 − a, b − x0
and
Mk Mk
|ek | = |xk − r | ≤ 1−M |x0 − x1 | = 1−M |e0 |

Proof
Exercise
Fixed point iteration.
Fixed point iteration.