Root Finding Methods
Root Finding Methods
This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/ or send a letter
to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
Contents
Contents iii
List of Algorithms iv
1 Fixed-Point Iteration 1
1.1 Basic Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Termination Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Iterative Approaches in Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5 Root Finding Using Fixed-Point Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
References 15
List of Algorithms
The sequence for the refined fixed point estimates (the ck ’s) can be iteratively generated using the function, f (x),
whose fixed point we are trying to find.
ck+1 = f (ck ) (1.2)
To show that this is valid, from Eqs. (1.1) and (1.2), we have [1, p. 60]
c = lim ck = lim f (ck−1 ) = f lim ck−1 = f (c)
k→∞ k→∞ k→∞
unless otherwise specified. Note that a relative error, εr , is also often used in iterative algorithms such as fixed point
iteration.
|xk+1 − xk |
εr =
|xk |
εr does a better job in comparing iterates with respect to each other, since depending on the problem, we could consider
a difference of xk+1 − xk = 100 to be really small, while for other problems, that difference is very large. However,
using εr can lead to issues when |xk | is 0 (or numerically very small) since it appears in the denominator. Additionally,
when manually setting the tolerance, it is more intuitive to set an absolute tolerance rather than a relative tolerance; an
absolute tolerance gives a direct measure of the numerical precision of the result, while a relative tolerance provides
an estimate of the number of correct digits.
If we predetermine that, at most, we can tolerate an error of TOL, then we will keep iterating Eq. (1.2) until
ε < TOL. In some cases, the error may never decrease below TOL, or take too long to decrease below TOL.
To account for this, we also define the maximum number of iterations, kmax , so that the algorithm does not keep
iterating forever, or for too long of a time [1, pp. 49–50]. For all the algorithms discussed in this document, we assume
a maximum number of iterations of
kmax = 200
unless otherwise specified.
1.3 Algorithm
Algorithm 1 is adapted from [1, pp. 60–61].
Algorithm 1: fixed_point_iteration
Fixed-point iteration for finding the fixed point of a univariate, scalar-
valued function.
Inputs:
• f (x) - univariate, scalar-valued function (f : R → R)
• x0 ∈ R - initial guess for fixed point
• TOL ∈ R - (OPTIONAL) tolerance
• kmax ∈ Z - (OPTIONAL) maximum number of iterations
Procedure:
1. Default the tolerance to TOL = 10−12 if not specified.
2. Default the maximum number of iterations to kmax = 200 if not specified.
3. Return the initial guess if it is a fixed point of f (x).
if f (x0 ) = x0
return c = x0
end
xcurr = x0
5. Initialize xnext .
xnext = 0
for k = 1 to kmax
(a) Update the fixed point estimate.
xnext = f (xcurr )
(c) Store the updated fixed point estimate for the next iteration.
xcurr = xnext
end
return c = xnext
Outputs:
• c ∈ R - fixed point of f (x)
y = f (x) (1.4)
x = g(y) (1.5)
In one function, you input x and get y, while in the other function, you input y and get x. Since f (x) and g(y) are
nonlinear (as previously mentioned), we cannot obtain closed-form solutions for x and y.
Let’s say we’re primarily interested in the variable x, where the variable y mainly serves to place a constraint on
x. We want to find the value x = c such that both equations are satisfied simultaneously (i.e. y = f (c) and c = g(y)).
Then we can define a new function h(x) as the function composition h = g ◦ f by substituting Eq. (1.4) into Eq. (1.5).
1 This example is adapted from my personal solutions to Problem 8.96 in [4, p. 476] using the Haaland equation [4, p. 431]. However, this fluid
mechanics text, in general, does not take a computational approach to such problems. Rather, it performs a “trial and error” procedure (including
hand calculations and reading values off of a chart), which essentially follows the same process as fixed-point iteration – an example of this can
be found in Example 8.7 [4, p. 444].
4 Chapter 1 Fixed-Point Iteration
f (x) = 0
Similarly, to find a fixed-point of f (x), we know we need to find a value of x that satisfies
f (x) = x
Finding the root of f (x) is equivalent to finding the fixed-point of g(x), where g(x) is
defined as [1, p. 56]
g(x) = x − f (x)
2
Univariate Root Finding
Equation (2.1) essentially approximates f (x) as the tangent line to the curve y = f (x) at the point x = x0 . The
x-intercept of this tangent line (i.e. its root or zero, x1 , can be found by setting the left hand side of Eq. (2.1) equal to
0.
f (x0 )
0 = f 0 (x0 )(x1 − x0 ) + f (x0 ) → x1 = x0 − 0
f (x0 )
x1 represents an updated estimate of the root of f (x), given an initial guess x0 . To keep refining our estimate, we can
keep iterating through this procedure. Eq. (2.2) below finds the (k + 1)th iterate given the kth iterate.
f (xk )
xk+1 = xk − (2.2)
f 0 (xk )
The iteration can be termination in the same manner as described in Section 1.2 [1, pp. 67–68].
All root-finding methods have cases where they can fail to converge, but Newton’s method has a particular
vulnerability; if at any point f 0 (xk ) = 0, Newton’s method will result in an Inf solution (due to division by 0). To
safeguard against this failure, we can perturb xk by (100)(TOL) |xk |. However, when xk = 0, this results in no
perturbation; in this case, we just perturb xk by (100)(TOL).
1 Often, a function f (x) will have multiple roots. Therefore, Newton’s method typically finds the root closest to the initial guess x0 . However,
this is not always the case; the algorithm depends heavily on the derivative of f (x), which, depending on its form, may cause it to converge on
a root further from x0 .
6 Chapter 2 Univariate Root Finding
Algorithm 2: newtons_method
Newton's method for finding the root of a differentiable, univariate,
scalar-valued function.
Inputs:
• f (x) - differentiable, univariate, scalar-valued function (f : R → R)
• f 0 (x) - derivative of f (x) (f 0 : R → R)
• x0 ∈ R - initial guess for root
• TOL ∈ R - (OPTIONAL) tolerance (defaults to 10−10 )
• kmax ∈ Z - (OPTIONAL) maximum number of iterations (defaults to 200)
Procedure:
1. Default the tolerance to TOL = 10−10 if not input.
2. Default the maximum number of iterations to kmax = 200 if not input.
3. Return the initial guess if it is a root of f (x).
if f (x0 ) = 0
return x∗ = x0
end
xcurr = x0
5. Initialize xnext .
xnext = 0
for k = 1 to kmax
2.2 Secant Method 7
0
if fcurr =0
if xcurr 6= 0
xcurr = xcurr (1 + (100)(TOL) |xcurr |)
else
xcurr = (100)(TOL)
end
end
f (xcurr )
xnext = xcurr − 0
fcurr
(e) Store the updated root estimate for the next iteration.
xcurr = xnext
end
7. Converged root.
return x∗ = xnext
Outputs:
• x∗ ∈ R - root of f (x)
f (xk )
xk+1 = xk − (2.3)
f 0 (xk )
If we don’t know f 0 (x) (for example, it could be very complicated and tedious to derive by hand), then we can instead
approximate it using some numerical method. Specifically, for the secant method, we use the backward approximation
of a derivative, given by Eq. (2.4) below.
f (xk ) − f (xk−1 )
f 0 (xk ) ≈ (2.4)
xk − xk−1
8 Chapter 2 Univariate Root Finding
Algorithm 3: secant_method
Secant method for finding the root of a univariate, scalar-valued
function.
Inputs:
• f (x) - univariate, scalar-valued function (f : R → R)
• x0 ∈ R - initial guess for root
• TOL ∈ R - (OPTIONAL) tolerance (defaults to 10−10 )
• kmax ∈ Z - (OPTIONAL) maximum number of iterations (defaults to 200)
Procedure:
1. Default the tolerance to TOL = 10−10 if not input.
2. Default the maximum number of iterations to kmax = 200 if not input.
3. Return the initial guess if it is a root of f (x).
if f (x0 ) = 0
return x∗ = x0
end
xprev = x0
if x0 6= 0
xcurr = x0 (1 + (100)(TOL) |x0 |)
else
xcurr = (100)(TOL)
end
2.2 Secant Method 9
fprev = f (x0 )
6. Initialize xnext .
xnext = 0
for k = 2 to kmax
(a) Function evaluation at the current iteration.
fcurr = f (xcurr )
if fcurr = fprev
if xcurr 6= 0
xcurr = xcurr (1 + (100)(TOL) |xcurr |)
else
xcurr = (100)(TOL)
end fcurr = f (xcurr )
end
(e) Store the next and current root estimates for the next iteration.
xprev = xcurr
xcurr = xnext
(f) Store the current function evaluation for the next iteration.
fprev = fcurr
end
8. Converged root.
return x∗ = xnext
Outputs:
• x∗ ∈ R - root of f (x)
10 Chapter 2 Univariate Root Finding
1. Make an initial guess for the interval [a, b] containing the root.
2. Assume the root, c, is the midpoint of this interval: c = (a + b)/2.
3. Evaluate f (a) and f (c).
(a) If f (a) < 0 and f (c) > 0 (i.e. they have different sign), we know the true root, x∗ , is contained in the
interval [a, c]. Therefore, we update our interval so that a remains the same, but b is updated to be c.
(b) If f (a) and f (c) have the same sign (either both negative or both positive), we know the true root, x∗ , must
be contained in the interval [c, b]. Therefore, we update our interval so that b remains the same, but a is
updated to be c.
4. Repeating steps 2 and 3, the interval [a, b] will keep shrinking. Once the difference (b − a) is small enough, we
say that the estimate of the root has converged to the true root, x∗ , within some tolerance, TOL.
In some cases, the difference (b − a) may never decrease below TOL, or take too long to decrease to below TOL.
Therefore, like with the previous methods, we also use the maximum number of iterations, kmax , as another termination
condition [1, pp. 48–49].
Algorithm 4:
Bisection method for finding the root of a univariate, scalar-valued
function.
Inputs:
• f (x) - univariate, scalar-valued function (f : R → R)
• a∈R - lower bound of interval containing root
• b∈R - upper bound of interval containing root
• TOL ∈ R - (OPTIONAL) tolerance (defaults to 10−10 )
• kmax ∈ Z - (OPTIONAL) maximum number of iterations (defaults to 200)
Procedure:
1. Default the tolerance to TOL = 10−10 if not input.
2. Default the maximum number of iterations to kmax = 200 if not input.
3. Root estimate at the first iteration.
a+b
c=
2
4. Return the root estimate at the first iteration if it is a root of f (x).
if f (c) = 0
return x∗ = c
end
for k = 1 to kmax
(a) Update interval.
if fc = 0
break
else if fa fc > 0
a=c
fa = fc
else
b=c
end
a+b
c=
2
(c) Terminate if converged.
if (b − a) < TOL
break
end
fc = f (c)
end
7. Converged root.
return x∗ = c
Outputs:
• x∗ ∈ R - root of f (x)
3
Multivariate Root Finding
f (x) = 0 (3.1)
In Section 2.1, we introduced Newton’s method as an algorithm for finding the root of a scalar-valued, univariate
function, f : R → R. Finding the root of f (x) is, by definition, solving the equation
f (x) = 0
for x. Note the similarity of this equation to Eq. (3.1). We can extend Newton’s method to the case of a multivariate,
vector-valued function whose input and output dimensions are the same (i.e. same number of equations and unknowns).
For the univariate case, we used the update equation
f (xk )
xk+1 = xk −
f 0 (xk )
By analogy, in the multivariate, vector-valued case, this becomes
xk+1 = xk − J(xk )−1 f (xk )
However, in its implementation, we avoid computing the inverse of the Jacobian matrix. Instead, we solve the rear-
ranged equation
J(xk )(xk+1 − xk ) = −f (xk )
for the unknown xk+1 − xk , and then find xk+1 accordingly. In two steps, this can be written as
For the multivariate case, we define the absolute error for the termination condition using the 2-norm:
ε = kxk+1 − xk k
However, from Eq. (3.2), we also know that yk = xk+1 − xk . Thus, we can rewrite the absolute error as [1, pp.
638-641]
ε = yk (3.3)
3.2 Algorithm 13
J(xk ) ≈ ijacobian(f , xk )
J(xk ) ≈ cjacobian(f , xk )
J(xk ) ≈ fjacobian(f , xk )
The ijacobian function is the most accurate, providing a numerical approximation that is typically accurate to within
double precision. However, there are a few functions that special care must be taken with in order to be compatible
with the complex-step approximation; see Sections 3.3 and 3.4 of [2].
3.2 Algorithm
Algorithm 5: newtons_method_n
Newton's method for finding the root of a differentiable, multivariate,
vector-valued function.
Inputs:
• f (x) - multivariate, vector-valued function (f : Rn → Rn )
• J(x) - Jacobian of f (x)
• x0 ∈ Rn - initial guess for solution
• TOL ∈ R - (OPTIONAL) tolerance (defaults to 10−10 )
• kmax ∈ Z - (OPTIONAL) maximum number of iterations (defaults to 200)
Procedure:
1. Default the tolerance to TOL = 10−10 if not input.
2. Default the maximum number of iterations to kmax = 200 if not input.
3. Return the initial guess if it is a root of f (x).
if f (x0 ) = 0n
return x∗ = x0
end
xcurr = x0
5. Initialize xnext .
xnext = 0
for k = 1 to kmax
14 Chapter 3 Multivariate Root Finding
J(xcurr )y = −f (xcurr )
xnext = xcurr + y
ε = kxnext − xcurr k
(e) Store the updated root estimate for the next iteration.
xcurr = xnext
end
7. Converged root.
return x∗ = xnext
Outputs:
• x∗ ∈ Rn - root of f (x)
f (x) = 0
[1] Richard L. Burden and J. Douglas Faires. Numerical Analysis. 9th ed. Boston, MA: Brooks/Cole, Cengage Learn-
ing, 2011.
[2] Tamas Kis. Numerical Differentiation. 2021. URL: https://tamaskis.github.io/files/Numerical_
Differentiation.pdf.
[3] Tamas Kis. Numerical Differentiation Toolbox. 2021. URL: https://github.com/tamaskis/Numerical_
Differentiation_Toolbox-MATLAB.
[4] Bruce R. Munson et al. Fundamentals of Fluid Mechanics. 7th . Hoboken, NJ: John Wiley & Sons, 2013.