Math 164: Week 3

Spring 2025
Chapter 3. Root-finding methods
3.1 Roots of a function
Given a function f (x), a number r that satisfies
f (r ) = 0
is called a root of f (x).
• Example of finding a root:
Find a steady state solution of a differential equation describing
the activity of a population of neurons:
dx
= −x + ϕ(x + I )
dt
ϕ(x) = 1/(e x + 1)
We want to find the steady state solution, i.e., x that satisfies
dx/dt = 0 or 0 = −x + ϕ(x + I ).
Then, the problem boils down to finding the root of
f (x) = −x + ϕ(x + I ).
Condition number of finding a root
How sensitively does r (i.e., a root of f (x)) change in response to
a change in the function f (x)?
To answer this question, let’s formulate the problem of finding a
root as a map F from a function f to its root r :
F :f →r where f (r ) = 0.
Next, consider a perturbation of f (x):
f˜(x) = f (x) + ϵh(x).
Then, find a root r˜ of f˜(x):
F : f˜ → r˜ where f˜(˜
r ) = 0.

How much does the root r˜ of f˜(x) deviate from the root r of f (x)?
In other words, how large is δr in
r˜ = r + δr ?
Estimate δr by finding an approximation of its value.

0 = f˜(˜
r ) = f (r + δr ) + ϵh(r + δr )
f ′′ (r )
= f (r ) + f ′ (r )δr + (δr )2 + ...
2
+ ϵ[h(r ) + h′ (r )δr + ...]
≈ 0 + f ′ (r )δr + ϵh(r )

Then, the first order approximation of δr is:

ϵh(r )
δr = − .
f ′ (r )

The absolute condition number of F (i.e., the problem of finding a

root) is:

r˜ − r δr 1
Ka (r ) = = = ′ .
˜
f −f ϵh(r ) |f (r )|
Example: condition number of finding a root
Consider a function

f (x) = e kx − e k with k>0

f ′ (x) = ke kx

The root of f is

f (r ) = e kr − e k = 0 → r = 1.

Let’s show that the condition number of finding a root of f (x)

depends on the parameter k.
The absolute condition number is
1 1
Ka (r ) = = k.
|f ′ (r )| ke

Here, ke k increases monotonically as k increases, so Ka (r )

decreases monotonically with k. This means that for a large k the
root r is not sensitive to changes in f , but for a small k it is.
Python example: condition number of root finding
3.2 Bisection method
Theorem (Intermediate Value Theorem)
If f(x) is continuous on [a,b], then, between a and b, f(x) assumes
all possible values between f(a) and f(b).
In particular, if f(a) and f(b) have opposite signs, then there is at
least one root of f(x) between a and b.
Algorithm:
1. Choose a and b, such that f (a)f (b) < 0, i.e., f (a) and f (b)
have opposite signs.
▶ By the IVT, f (x) has a root r ∈ [a, b].
▶ The worst case error is |a − b|.
▶ Improve the error by dividing the interval [a, b] by half.
2. Define the mid-point: m = (a + b)/2.
f (a), f (m) opposite signs → b = m and update to [a, b = m]
f (a), f (m) same signs → a = m and update to [a = m, b]
The worst case error is |a − b|/2.

3. Repeat Step 2 until error tolerance is met.

At the end of Step 2, we are always left with an interval that
contains a root of f (x).
The length of the interval is |a − b|/2k , if repeated k times.

error ≤ |a − b|/2k
Python example of Bisection Method
3.4 Newton’s method

The bisection method uses only the function f itself. Newton’s

method uses the derivative f ′ to get closer to the root.
Derivation of Newton’s method
Start at an initial point x0 . Find the line tangent to f at x0 .

y = f (x0 ) + f ′ (x0 )(x − x0 ) (1)

The tangent line intersects with x-axis at

f (x0 )
x = x0 − (2)
f ′ (x0 )

Denote this point as x1 and repeat the procedure.

x1 = x0 − f (x0 )/f ′ (x0 ) (3)

′
x2 = x1 − f (x1 )/f (x1 ) (4)
′
x3 = x2 − f (x2 )/f (x2 ) (5)

So, Newton’s algorithm is

xn+1 = xn − f (xn )/f ′ (xn ) (6)

Python example of Newton’s Method
Convergence of Newton’s Method: Proof
The sequence x0 , x1 , ... from the Newton’s method is NOT
guaranteed to converge (we will see such examples). However, IF
the sequence of points converges, then it converges to a root.
Assume that limn→∞ xn = r (i.e., the sequence converges) and
f ′ (r ) ̸= 0. We don’t know yet whether r is a root or not. Let’s
show that in fact r is a root.
Take the limit of Newton’s algorithm:

f (xn )
lim xn+1 = lim xn − lim
n→∞ n→∞ n→∞ f ′ (xn )
f (r )
r =r−
f ′ (r )
0 = f (r )

In other words, r is a root of f .

Convergence of Newton’s Method: Convergence rate
How quickly does the Newton’s algorithm converge to a root?
Define the error en = xn − r . We assume the error en is small. If
not, the convergence rate shown below is NOT always valid.
Now, let’s estimate its convergence rate:

f (xn )
xn+1 − r = xn − r −
f ′ (xn )
f (xn )
en+1 = en − ′
f (xn )

This expression can be simplified by expanding f around r :

f ′′ (s)
f (xn ) = f (r + xn − r ) ≈ f (r ) + f ′ (r )(xn − r ) + (xn − r )2
2
f ′′ (s) 2 f (xn ) f ′′ (s)
f (xn ) ≈ f ′ (r )en + e → en − ′ ≈ − ′ en2
2 n f (r ) 2f (r )
Then convergence rate of the error is exponential:

f ′′ (s) 2
en+1 ≈ − e
2f ′ (r ) n
en+1 = O(en2 )
n+1
en+1 = O(e02 )
Problems with Newton’s Method I

The root found by the Newton’s Method depends on the starting

point x0 . Roots far away from the starting point may not be found.
Example: f (x) = x 2 − 2.
Its Newton’s algorithm is:

xn2 − 2 x2 + 2
xn+1 = xn − = n (7)
2xn 2xn
√
If x0 < 0, then xn < 0 for all n, so xn converges to −
√ 2.
If x0 > 0, then xn > 0 for all n, so xn converges to 2.
Problems with Newton’s Method II

If the derivative near the root becomes infinite, then the sequence
does not converge.
p
Example; f (x) = sign(x) |x|p
Its derivative is f ′ (x) = 1/(2 |x|)
Its Newton’s algorithm is
p
sign(xn ) |xn |
xn+1 = xn − p = −xn
1/(2 |xn |)

Then, the sequence alternates between x0 and −x0 and does not
converge.
3.5 Secant Method (Modified Newton’s Method)
The derivative in f ′ in Newton’s Method can be replaced by the
forward difference approximation:
f (xn ) − f (xn−1 )
(8)
xn − xn−1
Then we get the Secant Method;
f (xn )
xn+1 = xn − f (x )−f (x ) (9)
n n−1
xn −xn−1
Python example of Secant Method