2022T1 MATH2089 – Numerical Methods Lectures Frances Kuo

§4 Nonlinear Equations
Motivation.
Many engineering problems are nonlinear with no analytic solution
Analytic solution
√
2 −b ± b2 − 4 a c
a x + b x + c = 0 =⇒ x =
2a
No analytic solution
eα x = 2 sin(π x) =⇒ x = ?
1 2
3

−1

−2
0 0.5 1 1.5 2 2.5 3

Relevant questions:
Does a solution exist? (existence)
Is the solution unique? (uniqueness)
What information do you have? data, function values, derivatives, . . .

Key concepts.
Iterative methods
Initial guess or starting point x1
Generate a sequence of iterates xk , k = 2, 3, . . ., based on a simple ap-
proximation of nonlinear problem
Convergence of xk to a/the true solution x∗
Order of convergence
Bisection – linear convergence
Fixed point iteration – linear convergence (Ts & Cs apply)
Newton’s method – quadratic convergence (Ts & Cs apply)
Secant method – superlinear convergence (Ts & Cs apply)

References. Rao 2.1 – 2.6, 2.8, 2.14

MATLAB M-files. exnle1.m exnle2.m exnthroot.m exnlog2n.m

1
 Single nonlinear equation of one variable in standard form

f (x) = 0, x∈R

Want to solve for x (find the zeros or roots of f )

Rearrange if necessary to get in the standard form
Standard form assumed by most software package (Matlab fzero)

Example. To solve sin(x) = 0 near x = 3 using Matlab, we use the command

x = fzero(@sin,3)

Example. To solve x 2x = 10 using Matlab, we write f (x) = x 2x − 10, and

observe that a solution is near x = 2 (or between 2 and 3). We use the commands
myf = @(x) x.*(2.^x) - 10
x = fzero(myf,2)
% x = fzero(myf,[2 3])

Example. (Intersection of two functions)

Find the points of intersection of f1 (x) = eα x with α = ln(4)/5 and f2 (x) =
2 sin(π x) for x > 0.
Answer. We solve the equation f (x) = eα x − 2 sin(π x) = 0 for x > 0.
αx
f1(x) = e and f2(x) = 2 sin(π x)
3

−1

−2
0 0.5 1 1.5 2 2.5 3
t

αx
f(x) = e − 2 sin(π x)
4

−1
0 0.5 1 1.5 2 2.5 3
x

See Matlab exnle1.m.

2
Example. (Inverse function)
Given y ∈ [−1, 1], find x = sin−1 (y) in [− π2 , π2 ].
Answer. Since
x = sin−1 (y) ⇐⇒ sin(x) = y ⇐⇒ sin(x) − y = 0,
we solve the equation f (x) = sin(x) − y = 0 for x ∈ [− π2 , π2 ].

sine function f(x) = sin(x) − 0.6 = 0

1 0.4

0.8 0.2

0.6 0

0.4 −0.2

0.2 −0.4

0 −0.6

−0.2 −0.8

−0.4 −1

−0.6 −1.2

−0.8 −1.4

−1 −1.6

−1.5 −1 −0.5 0 0.5 1 1.5 −1.5 −1 −0.5 0 0.5 1 1.5

x x

Example. (nth root)

Given a > 1 and an integer n > 1, find the nth root of a.
Answer. Since
x = a1/n ⇐⇒ xn = a ⇐⇒ xn − a = 0,
we solve the equation f (x) = xn − a = 0 for x > 0.
2
f(x) = x − 8.00
60
f(x)
Solution to f(x) = 0

50

40

30

20

10

−10
0 1 2 3 4 5 6 7 8
x

3
Example. (Optimization)
2
Find the minimum of F (x) = e−x sin(2πx) on [0, 2].
Answer. Since the minimum of F (x) occurs at the stationary points
2 2
F ′ (x) = −2xe−x sin(2πx) + 2πe−x cos(2πx) = 0,
2 2
we solve the equation f (x) = −2xe−x sin(2πx) + 2πe−x cos(2πx) = 0.
2 2
−x −x
Function F(x) = e sin(2π x) Derivative F’(x) = 2 e (−x sin(2π x) + π cos(2π x))
1 8

0.8
6

0.6
4

0.4
2

0.2

0
0

−2
−0.2

−4
−0.4

−0.6 −6
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
x x

See Matlab exnle2.m.

4
 Simple and multiple roots (zeros)
f (x∗ ) = 0 and f ′ (x∗ ) 6= 0 ⇐⇒ x∗ is a simple root of f
f (x∗ ) = f ′ (x∗ ) = · · · = f (k−1) (x∗ ) = 0 and f k (x∗ ) 6= 0 for k ≥ 2 ⇐⇒
x∗ is a multiple root of multiplicity k
Notation for continuity and differentiability
f ∈ C([a, b]) ⇐⇒ f is continuous on the interval [a, b]
f ∈ C 1 ([a, b]) ⇐⇒ f is continuous on [a, b] and differentiable on (a, b)
f ∈ C 2 ([a, b]) ⇐⇒ f is continuous on [a, b] and twice differentiable on
(a, b)
f ∈ C n ([a, b]) ⇐⇒ f is continuous on [a, b] and n times differentiable on
(a, b)
Existence and uniqueness
If f ∈ C([a, b]) and f (a)f (b) < 0, then there exists at least one zero of f
on (a, b).
(We say that the interval [a, b] brackets a root.)
If f is strictly monotone (either strictly increasing or strictly decreasing) on
the interval [a, b], then there exists at most one zero of f on [a, b].
f ′ (x) > 0 for all x ∈ (a, b) =⇒ f is strictly increasing on [a, b]
f ′ (x) < 0 for all x ∈ (a, b) =⇒ f is strictly decreasing on [a, b]
If f is continuous and strictly monotone on the interval [a, b] and f (a)f (b) <
0, then f has a unique root on (a, b).

Exercise. For a > 1 and integer n > 1, show that f (x) = xn − a has a simple root.

Exercise. For each of the following functions, show that it has a unique root on
the given interval, or explain why the theory does not apply.
(i) f (x) = x5 − 10 on [0, 2]

Answer. f is continuous on [0, 2], f (0) = −10 < 0, f (2) = 22 > 0. Thus f
has at least one zero. Moreover, f ′ (x) = 5x4 > 0 for all x ∈ (0, 2), implying
that f is strictly increasing on [0, 2]. Thus f has at most one zero. Hence f
has a unique root on [0, 2].

5
 1
(ii) f (x) = on [0, 2]
x−1

4
(iii) f (x) = x + 3 − on [−3, 1]
1 + x2

(iv) f (x) = eα x − 2 sin(π x), α = ln(4)/5, on [ 12 , 1]

(v) f (x) = sin(x) − y for fixed y ∈ (−1, 1), on [− π2 , π2 ]

6
Iterative methods for solving nonlinear equation f (x) = 0
True solution x∗ : f (x∗ ) = 0
Initial guess or starting point x1
Generate a sequence of iterates xk , k = 2, 3, . . ., based on a simple approx-
imation of the nonlinear equation
Compare four methods below (bisection, fixed point iteration, Newton’s
method, secant method)
Convergence: lim xk = x∗
k→∞

If f is continuous then limk→∞ f (xk ) = 0.

Practical convergence: given tolerance τ
|xk − x∗ | < τ (but we don’t know x∗ )
|xk+1 − xk | < τ (convergence =⇒ |xk+1 − xk | → 0)
|f (xk )| < τ
Maximum number of iterations
Error:
ek := |xk − x∗ | (or in practice ek := |xk − xk−1 | when x∗ is unknown),
and we have convergence if ek → 0 as k → ∞.

The order of convergence of an iterative method is the largest ν such that

ek+1
lim = β,
k→∞ eν k

where 0 < β < ∞ is the asymptotic constant.

ν = 1 and 0 < β < 1: linear convergence or first order convergence
For linear convergence, the asymptotic constant β is critical.
If β is close to 1, e.g., β = 0.99, then the convergence is very slow.
If β = 0.1, then the error is reduced by a factor of 10 on each iteration.
ek+1 ek+1
To check: → β ∈ (0, 1) and 2 → ∞
ek ek
1 < ν < 2: superlinear convergence
ek+1 ek+1
To check: → 0 and 2 → ∞
ek ek
ν = 2: quadratic convergence
ek+1 ek+1 ek+1
To check: → 0 and 2 → β ∈ (0, ∞) and 3 → ∞
ek ek ek

7
Exercise. For each of the tables of errors ek = |xk −x∗ | below, answer the following
questions.
(i) Is the method converging?
(ii) What is the order of convergence (linear, superlinear, quadratic)?
(iii) Can you trust the last few values?

Method A
k e(k) e(k+1)/e(k) e(k+1)/e(k)^2 e(k+1)/e(k)^3
1 7.41e-02 1.15e-01 1.56e+00 2.10e+01
2 8.53e-03 8.41e-03 9.86e-01 1.16e+02
3 7.18e-05 7.41e-05 1.03e+00 1.44e+04
4 5.32e-09 7.34e-09 1.38e+00 2.60e+08
5 3.91e-17

Method B
k e(k) e(k+1)/e(k) e(k+1)/e(k)^2 e(k+1)/e(k)^3
1 1.76e-01 1.84e+00 1.05e+01 5.95e+01
2 3.24e-01 2.91e-01 8.96e-01 2.77e+00
3 9.41e-02 1.41e+00 1.50e+01 1.59e+02
4 1.33e-01 8.86e-02 6.67e-01 5.03e+00
5 1.18e-02 9.82e-02 8.35e+00 7.11e+02
6 1.15e-03 1.26e-02 1.09e+01 9.43e+03
7 1.45e-05 1.20e-03 8.24e+01 5.68e+06
8 1.74e-08 1.50e-05 8.64e+02 4.98e+10
9 2.60e-13 1.50e-04 5.77e+08 2.22e+21
10 3.91e-17

8
Method C
k e(k) e(k+1)/e(k) e(k+1)/e(k)^2 e(k+1)/e(k)^3
1 1.00e+00 6.67e-01 6.67e-01 6.67e-01
2 6.67e-01 6.67e-01 1.00e-00 1.50e+00
3 4.44e-01 6.67e-01 1.50e+00 3.38e+00
4 2.96e-01 6.67e-01 2.25e+00 7.59e+00
5 1.98e-01 6.67e-01 3.38e+00 1.71e+01
6 1.32e-01 6.67e-01 5.06e+00 3.84e+01
7 8.78e-02 6.67e-01 7.59e+00 8.65e+01
8 5.85e-02 6.67e-01 1.14e+01 1.95e+02
9 3.90e-02 6.67e-01 1.71e+01 4.38e+02
10 2.60e-02

Method D
k e(k) e(k+1)/e(k) e(k+1)/e(k)^2 e(k+1)/e(k)^3
1 1.00e-01 1.49e-01 1.49e+00 1.49e+01
2 1.49e-02 8.24e+00 5.52e+02 3.69e+04
3 1.23e-01 1.31e-01 1.07e+00 8.66e+00
4 1.61e-02 2.26e+01 1.40e+03 8.68e+04
5 3.65e-01 4.29e-01 1.18e+00 3.23e+00
6 1.57e-01 3.05e-01 1.95e+00 1.24e+01
7 4.78e-02 6.31e+00 1.32e+02 2.76e+03
8 3.01e-01 3.42e-01 1.14e+00 3.77e+00
9 1.03e-01 5.07e-02 4.92e-01 4.78e+00
10 5.23e-03 4.21e+01 8.04e+03 1.54e+06
11 2.20e-01 4.12e-01 1.88e+00 8.53e+00
12 9.07e-02 3.53e-01 3.89e+00 4.29e+01
13 3.20e-02 2.63e+00 8.21e+01 2.57e+03
14 8.40e-02 5.58e-01 6.65e+00 7.91e+01
15 4.69e-02 1.29e+00 2.76e+01 5.88e+02
16 6.07e-02 2.34e+00 3.85e+01 6.33e+02
17 1.42e-01 1.54e-01 1.09e+00 7.67e+00
18 2.19e-02 4.78e+00 2.18e+02 9.95e+03
19 1.05e-01 7.19e-02 6.87e-01 6.56e+00
20 7.53e-03

9
Example. Given the initial error e1 = |x1 − x∗ | < 1, estimate the number of
iterations required to satisfy an error tolerance τ , depending on the order of con-
vergence.
linear convergence with asymptotic constant β < 1:

ek+1 ≈ β ek ≈ β 2 ek−1 ≈ β 3 ek−2 ≈ · · · ≈ β k e1 < τ ⇐⇒ k log(β) < log( eτ1 )

log( eτ1 )
⇐⇒ k > inequality reversed because β < 1
log(β)

order ν > 1 convergence:

2 3 k
ek+1 ≈ eνk ≈ eνk−1 ≈ eνk−2 ≈ · · · ≈ eν1 < τ ⇐⇒ ν k log(e1 ) < log(τ )
log(τ )
log( log(e 1)
)
⇐⇒ k > inequality reversed because e1 < 1
log(ν)

Example. If e1 = |x1 − x∗ | ≈ 0.1, estimate how many iterations it will take to get
ek+1 = |xk+1 − x∗ | < 10−14 using a method which is
(i) linearly convergent with asymptotic constant β = 0.99
(ii) superlinearly convergent with order ν = 1.6
(iii) quadratically convergent

Answer.
(i) ek+1 ≈ 0.99 ek ≈ 0.99k e1 ≈ 0.99k × 0.1 < 10−14

log( 100.1 )
−14

k> = 2978.3 · · · so k = 2979

log(0.99)

2 3 k k
(ii) ek+1 ≈ e1.6 1.6 1.6 1.6
k ≈ ek−1 ≈ ek−2 ≈ · · · ≈ e1 = 0.11.6 < 10−14

log(10−14 )

log log(0.1)
k> = 5.6 · · · so k=6
log(1.6)

2 3 k k
(iii) ek+1 ≈ e2k ≈ e2k−1 ≈ e2k−2 ≈ · · · ≈ e21 = 0.12 < 10−14

log(10−14 )

log log(0.1)
k> = 3.8 · · · so k=4
log(2)

10
Exercise. Consider interactive methods for solving nonlinear equations.
(i) If e1 ≈ 0.1, estimate e5 using a method which is linearly convergent with
asymptotic constant β = 0.3.

(ii) If e2 ≈ 0.6, estimate e9 using a method which is superlinearly convergent

with order ν = 1.4.

(iii) If e3 ≈ 0.4, estimate the number of further iterations required to satisfy

an error tolerance τ = 10−12 using a method which converges linearly with
asymptotic constant β = 0.5.

(iv) If e4 ≈ 0.7, estimate the number of further iterations required to reach ma-
chine precision eps ≈ 2.2 × 10−16 using a method which converges quadrat-
ically.

11
 Bisection
Suppose f ∈ C([a, b]) and f (a)f (b) < 0, i.e., [a, b] brackets a root.
a+b b−a
Take the midpoint xmid = as the starting point x1 =⇒ e1 ≤ .
2 2
Choose a new bracket:
If f (a)f (xmid ) < 0 then [a, xmid ] brackets a root.
If f (xmid )f (b) < 0 then [xmid , b] brackets a root.
Take the new midpoint as the next iterate and repeat the process.
Each bisection step reduces the bracket length by a half
=⇒ linear convergence with asymptotic constant β = 12 .
Fixed point iteration
A fixed point x∗ of a function g(x) satisfies x∗ = g(x∗ ).
Given a starting point x1 , the fixed point iteration computes

xk+1 := g(xk ) for k = 1, 2, . . . .

Theorem. If g ∈ C([a, b]) and g(x) ∈ [a, b] for all x ∈ [a, b], then g has a
fixed point in [a, b].
Theorem. If g ∈ C 1 ([a, b]) and there exists K ∈ (0, 1) such that |g ′ (x)| ≤
K for all x ∈ (a, b), then the fixed point iteration converges linearly with
asymptotic constant β ≤ K, for any x1 ∈ [a, b].

Illustrations.

12
 Newton’s method
Approximate f by the tangent at (xk , f (xk ))

f (x) ≈ f (xk ) + (x − xk )f ′ (xk ),

and choose x = xk+1 to make the right-hand side zero:

f (xk )
xk+1 := xk − , assuming f ′ (xk ) 6= 0.
f ′ (xk )

Theorem. If f ∈ C 2 ([a, b]) and x1 is sufficiently close to a simple root

x∗ ∈ (a, b) (i.e., f (x∗ ) = 0 and f ′ (x∗ ) 6= 0), then Newton’s method is
well-defined and converges quadratically to x∗ .
Difficulties: need to calculate derivative f ′ (xk ) as well as f (xk ), possibly
dividing by zero if f ′ (xk ) = 0 for some k.
Secant method
Approximate f by a line through (xk , f (xk )) and (xk−1 , f (xk−1 ))

f (xk ) − f (xk−1 )
f (x) ≈ f (xk ) + (x − xk ) ,
xk − xk−1

and choose x = xk+1 to make the right-hand side zero:

f (xk ) (xk − xk−1 ) f (xk ) xk−1 − f (xk−1 ) xk

xk+1 := xk − = .
f (xk ) − f (xk−1 ) f (xk ) − f (xk−1 )

Requires two starting values x1 and x2

Only requires function values f (xk ) and f (xk−1 ), no derivatives
Theorem. If f ∈ C 2 ([a, b]) and x1 , x2 are sufficiently close to a simple
root x∗ ∈ (a, b) (i.e., f (x∗ ) = 0 and f ′ (x∗√) 6= 0), then the secant method
converges superlinearly with order ν = 1+2 5 ≈ 1.6.

Illustrations.

13
Example. Formulate Newton’s method for finding the nth root of a > 1 for integer
n > 1.
Answer. We want to find x = a1/n =⇒ xn − a = 0. Let f (x) = xn − a. Then
f ′ (x) = n xn−1 . So Newton’s iteration is

f (xk ) xnk − a
xk+1 = xk − ′ = xk − .
f (xk ) n xn−1
k

The Matlab commands for computing the iterates from Newton’s method are as
follows:
n = 2;
a = 8;
f = @(x) x.^n - a;
df = @(x) n * x.^(n-1);

x(1) = a; % starting value

N = 10; % number of iterations
for k = 1:N
x(k+1) = x(k) - f(x(k))/df(x(k)); % Newton’s iteration
end
x
See Matlab exnthroot.m.

Exercise. Formulate Newton’s method for finding the solution to the nonlinear
equation x 2x = 10.

14
Example. Formulate Newton’s method and a fixed point iteration for solving
x log2 (x) = c for some constant c > 0.
Answer. We can write
x ln(x) c ln(2)
x log2 (x) = c ⇐⇒ = c ⇐⇒ x = .
ln(2) ln(x)

Matlab exnlog2n.m compares the order of convergence for the two methods.
x ln(x) ln(x) + 1
Let f (x) = − c. Then f ′ (x) = . So Newton’s iteration is
ln(2) ln(2)
xk ln(xk )
f (xk ) ln(2) − c xk ln(xk ) − c ln(2)
xk+1 = xk − ′ = xk − ln(xk )+1
= xk − .
f (xk ) ln(xk ) + 1
ln(2)

A fixed point iteration is given by

c ln(2)
xk+1 = g(xk ), with g(x) = .
ln(x)

The Matlab commands for computing the iterates from this fixed point
iteration are as follows:
c = 100;
g = @(x) c*log(2)/log(x);

x(1) = c/2; % starting value

N = 30; % number of iterations
for k = 1:N
x(k+1) = g(x(k)); % fixed point iteration
end
x
See Matlab exnlog2n.m for a more sophisticated while loop that checks for
the required error tolerance.
Note that we can also write
 
c ln(2) c ln(2) c ln(2)
x= ⇐⇒ ln(x) = ⇐⇒ x = exp .
ln(x) x x

But a fixed point iteration based on this formulation does NOT converge.

15
16