Numerical Analysis using Matlab and Maple

Lectures on YouTube:
https://www.youtube.com/channel/UCmRbK4vlGDht-joOQ5g0J2Q

Seongjai Kim

Department of Mathematics and Statistics

Mississippi State University
Mississippi State, MS 39762 USA
Email: [email protected]

Updated: June 26, 2021

Seongjai Kim, Professor of Mathematics, Department of Mathematics and Statistics, Mississippi
State University, Mississippi State, MS 39762 USA. Email: [email protected].
Prologue

Currently the lecture note is not fully grown up; other useful techniques and interesting examples
would be soon incorporated. Any questions, suggestions, comments will be deeply appreciated.

Seongjai Kim
June 26, 2021

iii
iv
Contents

Title ii

Prologue iii

Table of Contents viii

1 Mathematical Preliminaries 1
1.1. Review of Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1. Continuity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2. Differentiability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.3. Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.1.4. Taylor’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2. Review of Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.2.1. Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.2.2. System of linear equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.2.3. Invertible (nonsingular) matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.2.4. Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.2.5. Eigenvectors and eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.2.6. Vector and matrix norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.3. Computer Arithmetic and Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.3.1. Computational algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.3.2. Big O and little o notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.4. Programming with Matlab/Octave . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Exercises for Chapter 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

2 Solutions of Equations in One Variable 39

2.1. The Bisection Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.1.1. Implementation of the bisection method . . . . . . . . . . . . . . . . . . . . . . 40
2.1.2. Error analysis for the bisection method . . . . . . . . . . . . . . . . . . . . . . 43
2.2. Fixed-Point Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

v
vi Contents

2.2.1. Existence and uniqueness of fixed points . . . . . . . . . . . . . . . . . . . . . 48

2.2.2. Fixed-point iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.3. Newton’s Method and Its Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.3.1. The Newton’s method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.3.2. Systems of nonlinear equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
2.3.3. The secant method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
2.3.4. The method of false position . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
2.4. Zeros of Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
2.4.1. Horner’s method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
2.4.2. Complex zeros: Finding quadratic factors . . . . . . . . . . . . . . . . . . . . . 70
2.4.3. Bairstow’s method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Exercises for Chapter 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

3 Interpolation and Polynomial Approximation 79

3.1. Polynomial Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.1.1. Newton form of the interpolating polynomials . . . . . . . . . . . . . . . . . . 82
3.1.2. Lagrange Form of Interpolating Polynomials . . . . . . . . . . . . . . . . . . . 88
3.1.3. Polynomial interpolation error . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.1.4. Chebyshev polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.2. Divided Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
3.3. Data Approximation and Neville’s Method . . . . . . . . . . . . . . . . . . . . . . . . . 104
3.4. Hermite Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
3.5. Spline Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
3.5.1. Runge’s phenomenon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
3.5.2. Linear splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
3.5.3. Quadratic (Second Degree) Splines . . . . . . . . . . . . . . . . . . . . . . . . . 115
3.5.4. Cubic splines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
3.6. Parametric Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Exercises for Chapter 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

4 Numerical Differentiation and Integration 131

4.1. Numerical Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
4.2. Richardson Extrapolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
4.3. Numerical Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
4.3.1. The trapezoid rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
4.3.2. Simpson’s rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
4.3.3. Simpson’s three-eights rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
4.4. Romberg Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Contents vii

4.4.1. Recursive Trapezoid rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

4.4.2. The Romberg algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
4.5. Gaussian Quadrature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
4.5.1. The method of undetermined coefficients . . . . . . . . . . . . . . . . . . . . . 156
4.5.2. Legendre polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
4.5.3. Gauss integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
4.5.4. Gauss-Lobatto integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Exercises for Chapter 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

5 Numerical Solution of Ordinary Differential Equations 169

5.1. Elementary Theory of Initial-Value Problems . . . . . . . . . . . . . . . . . . . . . . . 170
5.2. Taylor-Series Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
5.2.1. The Euler method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
5.2.2. Higher-order Taylor methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
5.3. Runge-Kutta Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
5.3.1. Second-order Runge-Kutta method . . . . . . . . . . . . . . . . . . . . . . . . . 182
5.3.2. Fourth-order Runge-Kutta method . . . . . . . . . . . . . . . . . . . . . . . . . 184
5.3.3. Adaptive methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
5.4. One-Step Methods: Accuracy Comparison . . . . . . . . . . . . . . . . . . . . . . . . . 187
5.5. Multi-step Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
5.6. High-Order Equations & Systems of Differential Equations . . . . . . . . . . . . . . 194
Exercises for Chapter 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

6 Gauss Elimination and Its Variants 203

6.1. Systems of Linear Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
6.1.1. Nonsingular matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
6.1.2. Numerical solutions of differential equations . . . . . . . . . . . . . . . . . . . 206
6.2. Triangular Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
6.2.1. Lower-triangular systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
6.2.2. Upper-triangular systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
6.3. Gauss Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
6.3.1. The LU factorization/decomposition . . . . . . . . . . . . . . . . . . . . . . . . . 213
6.3.2. Solving linear systems by LU factorization . . . . . . . . . . . . . . . . . . . . 218
6.3.3. Gauss elimination with pivoting . . . . . . . . . . . . . . . . . . . . . . . . . . 220
6.3.4. Calculating A −1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
Exercises for Chapter 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

Bibliography 227
viii Contents

Index 229
Chapter 1

Mathematical Preliminaries

In this chapter, after briefly reviewing calculus and linear algebra, we

study about computer arithmetic and convergence. The last section of the
chapter presents a brief introduction on programming with Matlab/Octave.

1
2 Chapter 1. Mathematical Preliminaries

1.1. Review of Calculus

1.1.1. Continuity

Definition 1.1. A function f is continuous at x0 if

lim f (x) = f (x0 ). (1.1)

x →x0

In other words, if for every ε > 0, there exists a δ > 0 such that

|f (x) − f (x0 )| < ε for all x such that |x − x0 | < δ . (1.2)

Example 1.2. Examples and Discontinuities

Solution.

Answer: Jump discontinuity, infinite discontinuity, and removable discontinuity

Definition 1.3. Let {xn }∞ n=1 be an infinite sequence of real numbers.

This sequence has the limit x (converges to x ), if for every ε > 0 there
exists a positive integer Nε such that |xn − x | < ε whenever n > Nε . The
notation
lim xn = x or xn → x as n → ∞
n→∞

means that the sequence {xn }∞

n=1 converges to x .

Theorem 1.4. If f is a function defined on a set X of real numbers and

x ∈ X , then the following are equivalent:

• f is continuous at x
• If {xn }∞
n=1 is any sequence in X converging to x , then

lim f (xn ) = f (x).

n→∞
1.1. Review of Calculus 3

1.1.2. Differentiability

Definition 1.5. Let f be a function defined on an open interval con-

taining x0 . The function is differentiable at x0 , if
f (x) − f (x0 )
f 0 (x0 ) := lim (1.3)
x →x0 x − x0
exists. The number f 0 (x0 ) is called the derivative of f at x0 .

Important theorems for continuous/differentiable functions

Theorem 1.6. If the function f is differentiable at x0 , then f is

continuous at x0 .

Note: The converse is not true.

Example 1.7. Consider f (x) = |x |.
Solution.

Theorem 1.8. (Intermediate Value Theorem; IVT): Suppose f ∈

C[a, b] and K is a number between f (a) and f (b). Then, there exists a
number c ∈ (a, b) for which f (c) = K .

Example 1.9. Show that x 5 − 2x 3 + 3x 2 = 1 has a solution in the interval

[0, 1].
Solution. Define f (x) = x 5 − 2x 3 + 3x 2 − 1. Then check values f (0) and f (1)
for the IVT.

Theorem 1.10. (Rolle’s Theorem): Suppose f ∈ C[a, b] and f is dif-

ferentiable on (a, b). If f (a) = f (b), then there exists a number c ∈ (a, b)
such that f 0 (c) = 0.
4 Chapter 1. Mathematical Preliminaries

Mean Value Theorem (MVT)

Theorem 1.11. Suppose f ∈ C[a, b] and f is differentiable on (a, b).
Then there exists a number c ∈ (a, b) such that
f (b) − f (a)
f 0 (c) = , (1.4)
b −a
which can be equivalently written as

f (b) = f (a) + f 0 (c)(b − a). (1.5)

Example 1.12. Let f (x) = x +sin x be defined on [0,2]. Find c which assigns
the average slope.

Figure 1.1: A maple implementation.

1.1. Review of Calculus 5

Figure 1.2: The resulting figure, from the implementation in Figure 1.1.

Theorem 1.13. (Extreme Value Theorem): If f ∈ C[a, b], then there

exist c1 , c2 ∈ [a, b] for f (c1 ) ≤ f (x) ≤ f (c2 ) for all x ∈ [a, b]. In addition, if
f is differentiable on (a, b), then the numbers c1 and c2 occur either at
the endpoints of [a, b] or where f 0 is zero.

Example 1.14. Find the absolute minimum and absolute maximum val-
ues of f (x) = 5 ∗ cos(2 ∗ x) − 2 ∗ x ∗ sin(2 ∗ x) on the interval [1, 2].
Maple-code
1 a := 1: b := 2:
2 f := x -> 5*cos(2*x) - 2*x*sin(2*x):
3 fa := f(a);
4 = 5 cos(2) - 2 sin(2)
5 fb := f(b);
6 = 5 cos(4) - 4 sin(4)
7

8 #Now, find the derivative of "f"

9 fp := x -> diff(f(x), x):
10 fp(x);
11 = -12 sin(2 x) - 4 x cos(2 x)
12

13 fsolve(fp(x), x, a..b);
14 1.358229874
15 fc := f(%);
16 -5.675301338
17 Maximum := evalf(max(fa, fb, fc));
18 = -0.241008123
19 Minimum := evalf(min(fa, fb, fc));
20 = -5.675301338
6 Chapter 1. Mathematical Preliminaries

21

22 with(plots);
23 plot([f(x), fp(x)], x = a..b, thickness = [2, 2],
24 linestyle = [solid, dash], color = black,
25 legend = ["f(x)", "f'(x)"],
26 legendstyle = [font = ["HELVETICA", 10], location = right]);

Figure 1.3: The figure from the Maple-code.

The following theorem can be derived by applying Rolle’s Theorem suc-

cessively to f , f 0 , · · · and finally to f (n−1) .

Theorem 1.15. (Generalized Rolle’s Theorem): Suppose f ∈ C[a, b]

is n times differentiable on (a, b). If f (x) = 0 at the (n + 1) distinct points
a ≤ x0 < x1 < · · · < xn ≤ b , then there exists a number c ∈ (x0 , xn ) such
that f (n) (c) = 0.

1.1.3. Integration

Definition 1.16. The Riemann integral of a function f on the inter-

val [a, b] is the following limit, provided it exists:
ˆ b X n
f (x)dx = lim f (xi∗ )∆x i , (1.6)
a max ∆x i →0
i=1

where a = x0 < x1 < · · · < xn = b , with ∆x i = xi − xi −1 and xi∗ arbitrarily

chosen in the subinterval [xi −1 , xi ].
1.1. Review of Calculus 7

Note: Continuous functions are Riemann integrable, which allows us

to choose, for computational convenience, the points xi to be equally
b −a
spaced in [a, b] and choose xi∗ = xi , where xi = a + i∆x , ∆x = . In this
n
case,
ˆ b Xn
f (x)dx = lim f (xi )∆x. (1.7)
a n→∞
i=1

Theorem 1.17. (Fundamental Theorem of Calculus; FTC): Let f

be continuous on [a, b]. Then,
ˆ x
d
Part I: f (t)dt = f (x).
dx a
ˆ b
Part II: f (x)dx = F(b) − F(a), where F is an antiderivative of f ,
a
i.e. F 0 = f .

Weighted Mean Value Theorem on Integral (WMVT)

Theorem 1.18. Suppose f ∈ C[a, b], the Riemann integral of g exists
on [a, b], and g(x) does not change sign on [a, b]. Then, there exists
a number c ∈ (a, b) such that
ˆ b ˆ b
f (x)g(x)dx = f (c) g(x)dx. (1.8)
a a

Remark 1.19. When g(x) ≡ 1, the WMVT becomes the usual Mean
Value Theorem on Integral, which gives the average value of f ∈
C[a, b] over the interval [a, b]:
ˆ b
1
f (c) = f (x)dx. (1.9)
b −a a
8 Chapter 1. Mathematical Preliminaries

1.1.4. Taylor’s Theorem

Theorem 1.20. (Taylor’s Theorem with Lagrange Remainder):

Suppose f ∈ C n [a, b], f (n+1) exists on (a, b), and x0 ∈ [a, b]. Then, for every
x ∈ [a, b],
n
X f (k ) (x0 )
f (x) = (x − x0 )k + Rn (x), (1.10)
k!
k =0

where, for some ξ between x and x0 ,

f (n+1) (ξ )
Rn (x) = (x − x0 )n+1 .
(n + 1)!

Example 1.21. Let f (x) = cos(x) and x0 = 0. Determine the second and
third Taylor polynomials for f about x0 .
Maple-code
1 f := x -> cos(x):
2 fp := x -> -sin(x):
3 fpp := x -> -cos(x):
4 fp3 := x -> sin(x):
5 fp4 := x -> cos(x):
6

7 p2 := x -> f(0) + fp(0)*x/1! + fpp(0)*x^2/2!:

8 p2(x);
9 = 1 - 1/2 x^2
10 R2 := fp3(xi)*x^3/3!;
11 = 1/6 sin(xi) x^3
12 p3 := x -> f(0) + fp(0)*x/1! + fpp(0)*x^2/2! + fp3(0)*x^3/3!:
13 p3(x);
14 = 1 - 1/2 x^2
15 R3 := fp4(xi)*x^4/4!;
16 = 1/24 cos(xi) x^4
17

18 # On the other hand, you can find the Taylor polynomials easily
19 # using built-in functions in Maple:
20 s3 := taylor(f(x), x = 0, 4);
21 = 1 - 1/2 x^2 + O(x^4)
22 convert(s3, polynom);
23 = 1 - 1/2 x^2
1.1. Review of Calculus 9

1 plot([f(x), p3(x)], x = -2 .. 2, thickness = [2, 2],

2 linestyle = [solid, dash], color = black,
3 legend = ["f(x)", "p3(x)"],
4 legendstyle = [font = ["HELVETICA", 10], location = right])

Figure 1.4: f (x) = cos x and its third Taylor polynomial P3 (x).

Note: When n = 0, x = b , and x0 = a , the Taylor’s Theorem reads

f (b) = f (a) + R0 (b) = f (a) + f 0 (ξ ) · (b − a), (1.11)

which is the Mean Value Theorem.

Theorem 1.22. (Taylor’s Theorem with Integral Remainder):

Suppose f ∈ C n [a, b] and x0 ∈ [a, b]. Then, for every x ∈ [a, b],
n
X f (k ) (x0 )
f (x) = (x − x0 )k + En (x), (1.12)
k!
k =0

where ˆ x
1
En (x) = f (n+1) (t) · (x − t)n dt.
n! x0
10 Chapter 1. Mathematical Preliminaries

Alternative Form of Taylor’s Theorem

Remark 1.23. Suppose f ∈ C n [a, b], f (n+1) exists on (a, b). Then, for
every x, x + h ∈ [a, b],
n
X f (k ) (x)
f (x + h) = h k + Rn (h), (1.13)
k!
k =0

where, for some ξ between x and x + h ,

f (n+1) (ξ ) n+1
Rn (h) = h .
(n + 1)!
In detail,

0 f 00 (x) 2 f 000 (x) 2 f (n) (x) n

f (x + h) = f (x) + f (x)h + h + 3 + ··· + h + Rn (h). (1.14)
2! 3! n!

Theorem 1.24. (Taylor’s Theorem for Two Variables): Let f ∈

C (n+1) ([a, b] × [c, d]). If (x, y) and (x +h, y +k ) are points in [a, b] × [c, d] ⊂ R2 ,
then
n
X 1 ∂ ∂ i
f (x + h, y + k ) = h +k f (x, y) + Rn (h, k ), (1.15)
i! ∂ x ∂y
i=1

where
1  ∂ ∂ n+1
Rn (h, k ) = h +k f (x + θh, y + θk ),
(n + 1)! ∂ x ∂y
in which θ ∈ [0, 1].

Note: For n = 1, the Taylor’s theorem for two variables reads

f (x + h, y + k ) = f (x, y) + h fx (x, y) + k fy (x, y) + R1 (h, k ), (1.16)

where
R1 (h, k ) = O(h 2 + k 2 ).
Equation (1.16), as a linear approximation or tangent plane ap-
proximation, will be used for various applications.
1.1. Review of Calculus 11

Example 1.25. Find the tangent plane approximation of

2x + 3
f (x, y) = at (0, 0).
4y + 1

Maple-code
1 f := (2*x + 3)/(4*y + 1):
2 f0 := eval(%, {x = 0, y = 0});
3 = 3
4 fx := diff(f, x);
5 = 2/(4*y + 1)
6 fx0 := eval(%, {x = 0, y = 0});
7 = 2
8 fy := diff(f, y);
9 = 4*(2*x + 3)/(4*y + 1)^2
10 fy0 := eval(%, {x = 0, y = 0});
11 = -12
12

13 # Thus the tangent plane approximation $L(x, y)$ at $(0, 0)$ is

14 L(x, y) = 3 + 2*x - 12*y
12 Chapter 1. Mathematical Preliminaries

1.2. Review of Linear Algebra

1.2.1. Vectors
Definition 1.26. Let u = [u1 , u2 , · · · , un ]T and v = [v1 , v2 , · · · , vn ]T are
vectors in Rn . Then, the inner product (or dot product) of u and v is
given by  
v1
v 
 2
u•v = uT v = [u1 u2 · · · un ] . 
. .
(1.17)
vn
n
X
= u1 v1 + u2 v2 + · · · + un vn = uk vk .
k =1

Definition 1.27. The length (Euclidean norm) of v is nonnegative

scalar kvk defined by
√ q
kvk = v•v = v12 + v22 + · · · + vn2 and kvk2 = v•v. (1.18)

Definition 1.28. For u, v ∈ Rn , the distance between u and v is

dist(u, v) = ku − vk, (1.19)

the length of the vector u − v.

   
1 3
Example 1.29. Let u = −2 and v =  2. Find u•v, kuk, and dist(u, v).
   
2 −4
Solution.
1.2. Review of Linear Algebra 13

Definition 1.30. Two vectors u and v in Rn are orthogonal if u•v = 0.

Theorem 1.31. Pythagorean Theorem: Two vectors u and v are

orthogonal if and only if

ku + vk2 = kuk2 + kvk2 . (1.20)

Note: The inner product can be defined as

u•v = kuk kvk cos θ, (1.21)

where θ is the angle between u and v.

" # " #
1 −1/2
Example 1.32. Let u = √ and v = √ . Use (1.21) to find the
3 3/2
angle between u and v.
Solution.
14 Chapter 1. Mathematical Preliminaries

1.2.2. System of linear equations

Linear systems of m equations of n unknowns can be expressed as the
algebraic system:
A x = b, (1.22)
where b ∈ Rm is the source (input), x ∈ Rn is the solution (output), and
 
a11 a12 · · · a1n
 21 a22 · · · a2n 
a 
A = .
. .
. . . .
.  ∈ R m ×n .
 . . . . 
am1 am2 · · · amn

The above algebraic system can be solved by the elementary row operations
applied to the augmented matrix, augmented system:
 
a11 a12 · · · a1n b1
 21 a22 · · · a2n b2 
a 
[A b] =  . .. . . . .. ..  , (1.23)
 .. . . . 
am1 am2 · · · amn bm

and by transforming it to the reduced echelon form.

Tools 1.33. Three Elementary Row Operations (ERO):

• Replacement: Replace one row by the sum of itself and a multiple
of another row
Ri ← Ri + k · Rj , j 6= i
• Interchange: Interchange two rows
Ri ↔ Rj , j =6 i
• Scaling: Multiply all entries in a row by a nonzero constant
Ri ← k · Ri , k = 6 0
Every elementary row operation can be expressed as a matrix to be left-
multiplied. Such a matrix is called an elementary matrix.
1.2. Review of Linear Algebra 15

Example 1.34. Solve the following system of linear equations, using the
3 EROs. Then, determine if the system is consistent.

4x2 + 2x3 = 6
x1 − 4x2 + 2x3 = −1
4x1 − 8x2 + 12x3 = 8

Solution.

Example 1.35. Find the parabola y = a0 + a1 x + a2 x 2 that passes through

(1, 1), (2, 2), and (3, 5).
Solution.

Answer: y = 2 − 2x + x 2
16 Chapter 1. Mathematical Preliminaries

1.2.3. Invertible (nonsingular) matrices

Definition 1.36. An n × n matrix A is said to be invertible (nonsin-

gular) if there is an n × n matrix B such that AB = In = BA , where In is
the identity matrix.

Note: In this case, B is the unique inverse of A denoted by A −1 .

(Thus AA −1 = In = A −1 A .)
Theorem 1.37. (Inverse of an n × n matrix, n ≥ 2) An n × n matrix
A is invertible if and only if A is row equivalent to In and in this case
any sequence of elementary row operations that reduces A into In will
also reduce In to A −1 .

Algorithm 1.38. Algorithm to find A −1 :

1) Row reduce the augmented matrix [A : In ]
2) If A is row equivalent to In , then [A : In ] is row equivalent to [In : A −1 ].
Otherwise A does not have any inverse.
 
0 1 0
Example 1.39. Find the inverse of A = 1 0 3, if it exists.
 
4 −3 8
Solution.
1.2. Review of Linear Algebra 17

Theorem 1.40.
" #
a b
a. (Inverse of a 2 × 2 matrix) Let A = . If ad − bc 6= 0, then
c d
A is invertible and
" #
1 d −b
A −1 = (1.24)
ad − bc −c a

b. If A is an invertible matrix, then A −1 is also invertible; (A −1 )−1 = A .

c. If A and B are n × n invertible matrices then AB is also invertible
and (AB)−1 = B −1 A −1 .
d. If A is invertible, then A T is also invertible and (A T )−1 = (A −1 )T .
e. If A is an n × n invertible matrix, then for each b ∈ Rn , the equation
A x = b has a unique solution x = A −1 b.

Theorem 1.41. (Invertible Matrix Theorem) Let A be an n × n

matrix. Then the following are equivalent.
a. A is an invertible matrix.
b. A is row equivalent to the n × n identity matrix.
c. A has n pivot positions.
d. The columns of A are linearly independent.
e. The equation A x = 0 has only the trivial solution x = 0.
f. The equation A x = b has unique solution for each b ∈ Rn .
g. The linear transformation x 7→ A x is one-to-one.
h. The linear transformation x 7→ A x maps Rn onto Rn .
i. There is a matrix C ∈ Rn×n such that CA = I
j. There is a matrix D ∈ Rn×n such that AD = I
k. A T is invertible and (A T )−1 = (A −1 )T .
l. The number 0 is not an eigenvalue of A .
m. det A 6= 0.
18 Chapter 1. Mathematical Preliminaries

1.2.4. Determinants
Definition 1.42. Let A be an n × n square matrix. Then determinant
is a scalar value denoted by det A or |A |.
1) Let A = [a] ∈ R1 × 1 . Then det A = a .
" #
a b
2) Let A = ∈ R2 × 2 . Then det A = ad − bc .
c d
" #
2 1
Example 1.43. Let A = . Consider a linear transformation T : R2 → R2
0 3
defined by T (x) = A x.

a. Find the determinant of A .

b. Determine the image of a rectangle R = [0, 2] × [0, 1] under T .
c. Find the area of the image.
d. Figure out how det A , the area of the rectangle (= 2), and the area of
the image are related.

Solution.

Answer: c. 12
Note: The determinant can be viewed as the volume scaling factor.
1.2. Review of Linear Algebra 19

Definition 1.44. Let Aij be the submatrix of A obtained by deleting

row i and column j of A . Then the (i, j)-cofactor of A = [aij ] is the scalar
Cij , given by
Cij = (−1)i+j det Aij . (1.25)

Definition 1.45. For n ≥ 2, the determinant of an n × n matrix A =

[aij ] is given by the following formulas:

1. The cofactor expansion across the first row:

det A = a11 C11 + a12 C12 + · · · + a1n C1n (1.26)

2. The cofactor expansion across the row i :

det A = ai1 Ci1 + ai2 Ci2 + · · · + ain Cin (1.27)

3. The cofactor expansion down the column j :

det A = a1j C1j + a2j C2j + · · · + anj Cnj (1.28)

 
1 5 0
Example 1.46. Find the determinant of A = 2 4 −1, by expanding
 
0 −2 0
across the first row and down column 3.
Solution.

Answer: −2
20 Chapter 1. Mathematical Preliminaries

1.2.5. Eigenvectors and eigenvalues

Definition 1.47. Let A be an n × n matrix. An eigenvector of A is

a nonzero vector x such that A x = λx for some scalar λ. In this case, a
scalar λ is an eigenvalue and x is the corresponding eigenvector.

Definition 1.48. The scalar equation det (A − λI) = 0 is called the

characteristic equation of A ; the polynomial p(λ) = det (A − λI)
is called the characteristic polynomial of A . The solutions of
det (A − λI) = 0 are the eigenvalues of A .

Example 1.49. Find the characteristic polynomial and all eigenvalues of

 
1 1 0
A = 6 0 5
 
0 0 2
Solution.

Remark 1.50. Let A be an n × n matrix. Then the characteristic equa-

tion of A is of the form
p(λ) = det (A − λI) = (−1)n (λn + cn−1 λn−1 + · · · + c1 λ + c0 )
n
n
Y (1.29)
= (−1) (λ − λi ),
i=1

where some of eigenvalues λi can be complex-valued numbers. Thus

n
Y n
Y
n
det A = p(0) = (−1) (0 − λi ) = λi . (1.30)
i=1 i=1

That is, det A is the product of all eigenvalues of A .

1.2. Review of Linear Algebra 21

Theorem 1.51. If v1 , v2 , · · · , vr are eigenvectors that correspond

to distinct eigenvalues λ1 , λ2 , · · · , λr of n × n matrix A , then the set
{v1 , v2 , · · · , vr } is linearly independent.

Proof.

• Assume that {v1 , v2 , · · · , vr } is linearly dependent.

• One of the vectors in the set is a linear combination of the preceding
vectors.
• {v1 , v2 , · · · , vp } is linearly independent; vp+1 is a linear combination of
the preceding vectors.
• Then, there exist scalars c1 , c2 , · · · , cp such that

c1 v1 + c2 v2 + · · · + cp vp = vp+1 (1.31)

• Multiplying both sides of (1.31) by A , we obtain

c1 A v1 + c2 A v2 + · · · + cp A vp = A vp+1

and therefore, using the fact A vk = λk vk :

c1 λ1 v1 + c2 λ2 v2 + · · · + cp λp vp = λp+1 vp+1 (1.32)

• Multiplying both sides of (1.31) by λp+1 and subtracting the result from
(1.32), we have

c1 (λ1 − λp+1 )v1 + c2 (λ2 − λp+1 )v2 + · · · + cp (λp − λp+1 )vp = 0. (1.33)

• Since {v1 , v2 , · · · , vp } is linearly independent,

c1 (λ1 − λp+1 ) = 0, c2 (λ2 − λp+1 ) = 0, · · · , cp (λp − λp+1 ) = 0.

• Since λ1 , λ2 , · · · , λr are distinct,

c1 = c2 = · · · = cp = 0 ⇒ vp+1 = 0,

which is a contradiction.
22 Chapter 1. Mathematical Preliminaries

1.2.6. Vector and matrix norms

Definition 1.52. A norm (or, vector norm) on Rn is a function that
assigns to each x ∈ Rn a nonnegative real number kx k such that the
following three properties are satisfied: for all x, y ∈ Rn and λ ∈ R,

kxk > 0 if x 6= 0 (positive definiteness)

kλxk = |λ| kxk (homogeneity) (1.34)
kx + yk ≤ kxk + kyk (triangle inequality)

Example 1.53. The most common norms are

X 1/p
p
kx kp = |xi | , 1 ≤ p < ∞, (1.35)
i

which we call the p -norms, and

kx k∞ = max |xi |, (1.36)

i

which is called the infinity-norm or maximum-norm.

Note: Two of frequently used p -norms are
X X 1/2
2
kx k1 = |xi |, kx k2 = |xi | (1.37)
i i

The 2-norm is also called the Euclidean norm, often denoted by k · k.

Example 1.54. One may consider the infinity-norm as the limit of p -

norms, as p → ∞.
Solution.
1.2. Review of Linear Algebra 23

Definition 1.55. A matrix norm on m × n matrices is a vector norm

on the mn-dimensional space, satisfying

kA k ≥ 0, and kA k = 0 ⇔ A = 0 (positive definiteness)

kλ A k = |λ| kA k (homogeneity) (1.38)
kA + B k ≤ kA k + kB k (triangle inequality)
X 1/2
2
Example 1.56. kA kF ≡ |aij | is called the Frobenius norm.
i,j

Definition 1.57. Once a vector norm || · || has been specified, the in-
duced matrix norm is defined by
kA xk
kA k = max . (1.39)
x6=0 kxk
It is also called an operator norm or subordinate norm.

Theorem 1.58.
a. For all operator norms and the Frobenius norm,

kAx k ≤ kA k kx k, kA B k ≤ kA k kB k. (1.40)

kA xk1 X
b. kA k1 ≡ max = max |aij |
x6=0 kxk1 j
i
kA xk∞ X
c. kA k∞ ≡ max = max |aij |
x6=0 kxk∞ i
j

kA xk2 p
d. kA k2 ≡ max = λmax (A T A ),
x6=0 kxk2
where λmax denotes the largest eigenvalue.
e. kA k2 = kA T k2 .
f. kA k2 = max |λi (A )|, when A T A = AA T (normal matrix).
i
24 Chapter 1. Mathematical Preliminaries

Definition 1.59. Let A ∈ Rn × n . Then

κ(A ) ≡ kA k kA −1 k

is called the condition number of A , associated to the matrix norm.

 
1 2 −2
Example 1.60. Let A =  0 4 1 . Then, we have
 
1 −2 2
   
10 0 10 2 0 0
1 
A −1 = 1 4 −1  and A T A =  0 24 −4  .
  
20

−4 4 4 0 −4 9

a. Find kA k1 , kA k∞ , and kA k2 .
b. Compute the `1 -condition number κ1 (A ).

Solution.
1.3. Computer Arithmetic and Convergence 25

1.3. Computer Arithmetic and Convergence

Errors in Machine Numbers and Computational Results

• Numbers are saved with an approximation by either rounding or
chopping.
– integer: in 4 bites (32 bits)
– float: in 4 bites
– double: in 8 bites (64 bits)
• Computations can be carried out only for finite sizes of data points.
Maple-code
1 Pi;
2 = Pi
3 evalf(Pi);
4 = 3.141592654
5 evalf[8](Pi);
6 = 3.1415927
7 evalf[16](Pi);
8 = 3.141592653589793
9 evalf(Pi)*(evalf[8](Pi) - evalf[16](Pi));
10 = 1.445132621*E-07
11 #On the other hand,
12 Pi*(Pi - Pi);
13 = 0

Definition 1.61. Suppose that p ∗ is an approximation to p . Then

• The absolute error is |p − p ∗ |, and

|p − p ∗ |
• the relative error is , provided that p 6= 0.
|p |

Definition 1.62. The number p ∗ is said to approximate p to t -

significant digits (or figures) if t is the largest nonnegative integer
for which
|p − p ∗ |
≤ 5 × 10−t .
|p |
26 Chapter 1. Mathematical Preliminaries

1.3.1. Computational algorithms

Definition 1.63. An algorithm is a procedure that describes, in an

unambiguous manner, a finite sequence of steps to be carried out in a
specific order.

Algorithms consist of various steps for inputs, outputs, and functional op-
erations, which can be described effectively by a so-called pseudocode.

Definition 1.64. An algorithm is called stable, if small changes in the

initial data produce correspondingly small changes in the final results.
Otherwise, it is called unstable. Some algorithms are stable only for
certain choices of data/parameters, and are called conditionally sta-
ble.

Notation 1.65. (Growth rates of the error): Suppose that E0 > 0

denotes an error introduced at some stage in the computation and En
represents the magnitude of the error after n subsequent operations.
• If En = C × n E0 , where C is a constant independent of n, then the
growth of error is said to be linear, for which the algorithm is stable.
• If En = C n E0 , for some C > 1, then the growth of error is exponential,
which turns out unstable.
1.3. Computer Arithmetic and Convergence 27

Rates (Orders) of Convergence

Definition 1.66. Let {xn } be a sequence of real numbers tending to a
limit x ∗ .
• The rate of convergence is at least linear if there are a constant c1 <
1 and an integer N such that
|xn+1 − x ∗ | ≤ c1 |xn − x ∗ |, ∀ n ≥ N. (1.41)

• We say that the rate of convergence is at least superlinear if there

exist a sequence εn tending to 0 and an integer N such that
|xn+1 − x ∗ | ≤ εn |xn − x ∗ |, ∀ n ≥ N. (1.42)

• The rate of convergence is at least quadratic if there exist a constant

C (not necessarily less than 1) and an integer N such that
|xn+1 − x ∗ | ≤ C |xn − x ∗ |2 , ∀ n ≥ N. (1.43)

• In general, we say that the rate of convergence is of α at least if

there exist a constant C (not necessarily less than 1 for α > 1) and an
integer N such that
|xn+1 − x ∗ | ≤ C |xn − x ∗ |α , ∀ n ≥ N. (1.44)

Example 1.67. Consider a sequence defined recursively as

xn 1
x1 = 2, xn+1 = + . (1.45)
2 xn
(a) Find the limit of the sequence; (b) show that the convergence is quadratic.
√ 2
Hint : You may first prove that xn > 2 for all n ≥ 1 ( ∵ xn+1 − 2 > 0). Then
1 1
you can see that xn+1 < xn ( ∵ xn − xn+1 = xn ( − ) > 0).
2 xn2
Solution.
28 Chapter 1. Mathematical Preliminaries

1.3.2. Big O and little o notation

Definition 1.68.

• A sequence {αn }∞ ∞
n=1 is said to be in O (big Oh) of {βn }n=1 if a positive
number K exists for which
 |αn | 
|αn | ≤ K |βn |, for large n or equivalently, ≤K . (1.46)
|βn |
In this case, we say “αn is in O (βn )" and denote αn ∈ O (βn ) or αn =
O (βn ).
• A sequence {αn } is said to be in o (little oh) of {βn } if there exists a
sequence εn tending to 0 such that
 |αn | 
|αn | ≤ εn |βn |, for large n or equivalently, lim = 0 . (1.47)
n→∞ |βn |

In this case, we say “αn is in o(βn )" and denote αn ∈ o(βn ) or αn =

o(βn ).
n+1 1
Example 1.69. Show that αn = 2 = O and
n n
n+3
f (n) = 3 2
∈ O(n−2 ) ∩ o(n−1 ).
n + 20n
Solution.
1.3. Computer Arithmetic and Convergence 29

Definition 1.70. Suppose lim G(h) = 0. A quantity F(h) is said to be in

h →0
O (big Oh) of G(h) if a positive number K exists for which
|F(h)|
≤ K , for h sufficiently small. (1.48)
|G(h)|
In this case, we say F(h) is in O(G(h)), and denote F(h) ∈ O(G(h)). Little
oh of G(h) can be defined the same way as for sequences.

Example 1.71. Taylor’s series expansion for cos(x) is given as

1 2 1 4 1 6
cos(x) = 1 − x + x − x + ···
2! 4! 6!
1 1 4 1 6
= 1 − x2 + x − x + ··· .
2 24 720
If you use a computer algebra software (e.g. Maple), you will obtain
1 2
taylor(cos(x), x = 0, 4) = 1 − x + O(x 4 )
2!
which implies that
1 4 1 6
x − x + · · · = O(x4 ). (1.49)
|24 720
{z }
=: F(x)
Indeed,
|F(x)| 1 1 2 1
= − x + · · · ≤ , for sufficiently small x. (1.50)
|x 4 | 24 720 24

Thus F(x) ∈ O(x 4 ).

Example 1.72. Choose the correct assertions (in each, n → ∞)

a. (n2 + 1)/n3 ∈ o(1/n)

√
b. (n + 1)/ n ∈ o(1)
c. 1/ ln n ∈ O(1/n)
d. 1/(n ln n) ∈ o(1/n)
e. e n /n5 ∈ O(1/n)
30 Chapter 1. Mathematical Preliminaries

Example 1.73. Determine the best integer value of k in the following

equation
arctan(x) = x + O(x k ), as x → 0.

Solution.

Answer: k = 3.
1
Self-study 1.74. Let f (h) = (1 + h − e h ). What are the limit and the rate
h
of convergence of f (h) as h → 0?
Solution.
1.3. Computer Arithmetic and Convergence 31

Self-study 1.75. Show that these assertions are not true.

a. e x − 1 = O(x 2 ), as x → 0
b. x = O(tan−1 x), as x → 0
c. sin x cos x = o(1), as x → 0

Solution.

Example 1.76. Let {an } → 0 and λ > 1. Show that

n
X
ak λk = o(λn ), as n → ∞.
k =0
Pn k
| k =0 ak λ |
Hint : = |an + an−1 λ−1 + · · · + a0 λ−n | =: εn . Then, we have to show
|λn |
1
εn → 0 as n → ∞. For this, you can first observe εn+1 = an+1 + εn , which
λ
implies that εn is bounded and converges to ε. Now, can you see ε = 0?
Solution.
32 Chapter 1. Mathematical Preliminaries

1.4. Programming with Matlab/Octave

Note: In computer programming, important things are

• How to deal with objects (variables, arrays, functions)
• How to deal with repetition effectively
• How to make the program reusable

Vectors and matrices

The most basic thing you will need Vectors and Matrices
to do is to enter vectors and matri- 1 >> u = [1; 2; 3] % column vector
2 u=
ces. You would enter commands to 3 1
Matlab or Octave at a prompt that 4 2
looks like >>. 5 3
6 >> v = [4; 5; 6];
7 >> u + 2*v
• Rows are separated by semi- 8 ans =
colons (;) or Enter . 9 9
10 12
• Entries in a row are separated 11 15
by commas (,) or space Space . 12 >> w = [5, 6, 7, 8] % row vector
13 w=
For example, 14 5 6 7 8
15 >> A = [2 1; 1 2]; % matrix
16 >> B = [-2, 5
17 1, 2]
18 B=
19 -2 5
20 1 2
21 >> C = A*B % matrix multiplication
22 C=
23 -3 12
24 0 9
1.4. Programming with Matlab/Octave 33

You can save the commands in a file to run and get the same results.
tutorial1_vectors.m
1 u =
[1; 2; 3]
2 v =
[4; 5; 6];
3 u +
2*v
4 w =
[5, 6, 7, 8]
5 A =
[2 1; 1 2];
6 B =
[-2, 5
7 1, 2]
8 C = A*B

Solving equations
   
1 −4 2 3
Let A = 0 3 5 and b = −7 . Then A x = b can be numerically
   
2 8 −4 −3
solved by implementing a code as follows.

tutorial2_solve.m Result
1 A = [1 -4 2; 0 3 5; 2 8 -4]; 1 x =
2 b = [3; -7; -3]; 2 0.75000
3 x = A\b 3 -0.97115
4 -0.81731

Graphics with Matlab

In Matlab, the most popular graphic command is plot, which creates a 2D
line plot of the data in Y versus the corresponding values in X. A general
syntax for the command is
plot(X1,Y1,LineSpec1,...,Xn,Yn,LineSpecn)
34 Chapter 1. Mathematical Preliminaries

tutorial3_plot.m
1 close all
2

3 %% a curve
4 X1 = linspace(0,2*pi,10); % n=10
5 Y1 = cos(X1);
6

7 %% another curve
8 X2=linspace(0,2*pi,20); Y2=sin(X2);
9

10 %% plot together
11 plot(X1,Y1,'-or',X2,Y2,'--b','linewidth',3);
12 legend({'y=cos(x)','y=sin(x)'},'location','best',...
13 'FontSize',16,'textcolor','blue')
14 print -dpng 'fig_cos_sin.png'

Figure 1.5: fig_cos_sin.png: plot of y = cos x and y = sin x .

Above tutorial3_plot.m is a typical M-file for figuring with plot.

• Line 1: It closes all figures currently open.

• Lines 3, 4, 7, and 10 (comments): When the percent sign (%) appears,
the rest of the line will be ignored by Matlab.
• Lines 4 and 8: The command linspace(x1,x2,n) returns a row vector
of n evenly spaced points between x1 and x2.
• Line 11: Its result is a figure shown in Figure 1.5.
• Line 14: it saves the figure into a png format, named fig_cos_sin.png.
1.4. Programming with Matlab/Octave 35

Repetition: iteration loops

Note: In scientific computation, one of most frequently occurring events

is repetition. Each repetition of the process is also called an iteration.
It is the act of repeating a process, to generate a (possibly unbounded)
sequence of outcomes, with the aim of approaching a desired goal, tar-
get or result. Thus,
• iteration must start with an initialization (starting point) and
• perform a step-by-step marching in which the results of one itera-
tion are used as the starting point for the next iteration.

In the context of mathematics or computer science, iteration (along with

the related technique of recursion) is a very basic building block in pro-
gramming. Matlab provides various types of loops: while loops, for loops,
and nested loops.

while loop
The syntax of a while loop in Matlab is as follows.
while <expression>
<statements>
end
An expression is true when the result is nonempty and contains all nonzero
elements, logical or real numeric; otherwise the expression is false. Here
is an example for the while loop.
n1=11; n2=20;
sum=n1;
while n1<n2
n1 = n1+1; sum = sum+n1;
end
fprintf('while loop: sum=%d\n',sum);
When the code above is executed, the result will be:
while loop: sum=155
36 Chapter 1. Mathematical Preliminaries

for loop
A for loop is a repetition control structure that allows you to efficiently
write a loop that needs to execute a specific number of times. The syntax
of a for loop in Matlab is as following:
for index = values
<program statements>
end
Here is an example for the for loop.
n1=11; n2=20;
sum=0;
for i=n1:n2
sum = sum+i;
end
fprintf('for loop: sum=%d\n',sum);
When the code above is executed, the result will be:
for loop: sum=155

Functions: Enhancing reusability

Program scripts can be saved to reuse later conveniently. For example,
the script for the summation of integers from n1 to n2 can be saved as a
form of function.
mysum.m
1 function s = mysum(n1,n2)
2 % sum of integers from n1 to n2
3

4 s=0;
5 for i=n1:n2
6 s = s+i;
7 end

Now, you can call it with e.g. mysum(11,20).

Then the result reads ans = 155.
1.4. Programming with Matlab/Octave 37

Exercises for Chapter 1

1.1. Prove that the following equations have at least one solution in the given intervals.

(a) x − (ln x)3 = 0, [5, 7]

(b) (x − 1)2 − 2 sin(π x /2) = 0, [0, 2]
(c) x − 3 − x 2 e −x = 0, [2, 4]
(d) 5x cos(π x) − 2x 2 + 3 = 0, [0, 2]

1.2. C 1 Let f (x) = 5x cos(3x) − (x − 1)2 and x0 = 0.

(a) Find the third Taylor polynomial of f about x = x0 , p3 (x), and use it to approxi-
mate f (0.2).
(b) Use the Taylor’s Theorem (Theorem 1.20) to find an upper bound for the error
|f (x) − p3 (x)| at x = 0.2. Compare it with the actual error.
(c) Find the fifth Taylor polynomial of f about x = x0 , p5 (x), and use it to approximate
f (0.2).
(d) Use the Taylor’s Theorem to find an upper bound for the error |f (x) − p5 (x)| at
x = 0.2. Compare it with the actual error.

1.3. For the fair (xn , αn ), is it true that xn = O(αn ) as n → ∞?

(a) xn = 3n2 − n4 + 1; αn = 3n2

1 √
(b) xn = n − √ + 1; αn = n
n
√
(c) xn = n − 10; αn = 1
(d) xn = −n2 + 1; αn = n 3

1.4. Let a sequence xn be defined recursively by xn+1 = g(xn ), where g is continuously

differentiable. Suppose that xn → x ∗ as n → ∞ and g 0 (x ∗ ) = 0. Show that

xn+2 − xn+1 = o(xn+1 − xn ). (1.51)

Hint : Begin with

xn+2 − xn+1 g(xn+1 ) − g(xn )
= ,
xn+1 − xn xn+1 − xn
and use the Mean Value Theorem (on page 4) and the fact that is continuously dif-
ferentiable, to show that the quotient converges to zero as n → ∞.

1
The mark C indicates that you should solve the problem via computer programming. Attach hard
copies of your code and results. For other problems, if you like and doable, you may try to solve them with
computer programming.
38 Chapter 1. Mathematical Preliminaries

1.5. A square matrix A ∈ Rn×n is said to be skew-symmetric if A T = −A . Prove that if

A is skew-symmetric, then xT A x = 0 for all x ∈ Rn .
Hint : The quantity xT A x is scalar so that (xT A x)T = xT A x.
1.6. Suppose that A , B , and C are square matrices and that ABC is invertible. Show that
each of A , B , and C is invertible.
1.7. Find the determinant and eigenvalues of the following matrices, if it exists. Com-
pare the determinant with the product of eigenvalues, i.e. check if (1.30) is true.
   
2 3 6 0 5
(a) P =
7 6 (c) R = 0 4 0
1 −5 2
 
1 2 4 8
 
1 2 0 3 0 0
 
(d) S = 
(b) Q = −3 0 0 5 0

1
0 1 0 0 5 7

1.8. Show that the `2 -norm kxk2 , defined as

q
kxk2 = x12 + x22 + · · · + xn2 ,

satisfies the three conditions in (1.34), page 22.

Hint : For the last condition, you may begin with

kx + yk22 = (x + y)•(x + y) = kxk22 + 2 x•y + kyk22 .

Now, compare this with (kxk2 + kyk2 )2 .

1.9. Show that kxk∞ ≤ kxk2 ≤ kxk1 for all x ∈ Rn .
Chapter 2

Solutions of Equations in One

Variable

Through the chapter, the objective is to find solutions for equations of the
form
f (x) = 0. (2.1)
Various numerical methods will be considered for the solutions of (2.1).
Although the methods will be derived for a simple form of equations, they
will be applicable for various general problems.

39
40 Chapter 2. Solutions of Equations in One Variable

2.1. The Bisection Method

It is also called the binary-search method or interval-halving method.
Note: The objective is to find solutions for

f (x) = 0. (2.2)

2.1.1. Implementation of the bisection method

Assumption. For the bisection method, we assume that

1) f is continuous in [a, b].
2) f (a) · f (b) < 0 (so that there must be a solution by the IVP).
3) There is a single solution in [a, b].

Pseudocode 2.1. The Bisection Method:

• Given [a1 , b1 ] = [a, b], p1 = (a1 + b1 )/2;
• For n = 1, 2, · · · , itmax
if ( f (pn ) = 0 ) then
stop;
elseif ( f (an ) · f (pn ) < 0 ) then
an+1 = an ; bn+1 = pn ;
else
an+1 = pn ; bn+1 = bn ;
endif
pn+1 = (an+1 + bn+1 )/2
2.1. The Bisection Method 41

Example 2.2. Find the solution of the equation x 3 + 4x 2 − 10 = 0 in [1, 2].

Bisection
1 with(Student[NumericalAnalysis]):
2 f := x -> x^3 + 4*x^2 - 10:
3

4 Bisection(f(x), x = [1,2], tolerance = 0.1,

5 stoppingcriterion = absolute, output = sequence);
6 [1., 2.],
7 [1., 1.500000000],
8 [1.250000000, 1.500000000],
9 [1.250000000, 1.375000000],
10 1.312500000
11

12 Bisection(f(x), x = [1, 2], tolerance = 0.1,

13 stoppingcriterion = absolute, output = plot)

Figure 2.1: The bisection method.

42 Chapter 2. Solutions of Equations in One Variable

Figure 2.2: A Maple code for the bisection method

Result
1 > bisection2(1, 2, 0.01, 20);
2 k= 1: a= 1.000000 b= 2.000000 p= 1.500000 f(p)= 2.375000
3 k= 2: a= 1.000000 b= 1.500000 p= 1.250000 f(p)=-1.796875
4 k= 3: a= 1.250000 b= 1.500000 p= 1.375000 f(p)= 0.162109
5 k= 4: a= 1.250000 b= 1.375000 p= 1.312500 f(p)=-0.848389
6 k= 5: a= 1.312500 b= 1.375000 p= 1.343750 f(p)=-0.350983
7 k= 6: a= 1.343750 b= 1.375000 p= 1.359375 f(p)=-0.096409
8 k= 7: a= 1.359375 b= 1.375000 p= 1.367188 f(p)= 0.032356
9 3 2
10 "f(x)=", x + 4 x - 10
11 p_7 = 1.367187500
12 dp = +- 0.007812 = (b0-a0)/2^k= 0.007812
13 f(p) = 0.032355785
14 175
15 ---
16 128
2.1. The Bisection Method 43

2.1.2. Error analysis for the bisection method

Theorem 2.3. Suppose that f ∈ C[a, b] and f (a) · f (b) < 0. Then, the
Bisection method generates a sequence pn approximating a zero p of f
with
b −a
|p − pn | ≤ n
, n ≥ 1. (2.3)
2

Proof. For n ≥ 1,
1
bn − an = (b − a) and p ∈ (an , bn ). (2.4)
2 n −1
It follows from pn = (an + bn )/2 that
1 1
|p − pn | ≤ (bn − an ) = n (b − a), (2.5)
2 2
which completes the proof.
Note: The right-side of (2.3) is the upper bound of the error in the n-th
iteration.

Example 2.4. Determine the number of iterations necessary to solve

x 3 + 4x 2 − 10 = 0 with accuracy 10−3 using [a1 , b1 ] = [1, 2].
Solution. We have to find the iteration count n such that the error upper-
bound is not larger than 10−3 . That is, incorporating (2.3),
b −a
|p − pn | ≤ n
≤ 10−3 . (2.6)
2
Since b − a = 1, it follows from the last inequality that 2n ≥ 103 , which
implies that
3 ln(10)
n≥ ≈ 9.965784285.
ln(2)
Answer: n = 10
44 Chapter 2. Solutions of Equations in One Variable

Remark 2.5. The zero p is unknown so that the quantity |p − pn | is a

theoretical value; it is not useful in computation.
Note that pn is the midpoint of [an , bn ] and pn+1 is the midpoint of
either [an , pn ] or [pn , bn ]. So,
1 1
|pn+1 − pn | = (bn − an ) = (b − a). (2.7)
4 2n+1
In other words,
1
|pn − pn−1 | = (b − a), (2.8)
2n
which implies that
1
|p − pn | ≤ (b − a) = |pn − pn−1 |. (2.9)
2n
The approximate solution, carried out with the absolute difference
|pn − pn−1 | being used for the stopping criterion, guarantees the actual
error not greater than the given tolerance.

Example 2.6. Suppose that the bisection method begins with the interval
[45, 60]. How many steps should be taken to compute a root with a relative
error not larger than 10−8 ?
Solution.

Answer: n ≥ ln(108 /3)/ ln(2). Thus n = 25

2.1. The Bisection Method 45

bisect: a Matlab code

bisect.m
1 function [c,err,fc]=bisect(f,a,b,TOL)
2 %Input - f is the function input as a string 'f'
3 % - a and b are the left and right endpoints
4 % - TOL is the tolerance
5 %Output - c is the zero
6 % - err is the error estimate for c
7 % - fc= f(c)
8

9 fa=feval(f,a);
10 fb=feval(f,b);
11 if fa*fb > 0,return,end
12 max1=1+round((log(b-a)-log(TOL))/log(2));
13

14 for k=1:max1
15 c=(a+b)/2;
16 fc=feval(f,c);
17 if fc==0
18 a=c; b=c;
19 elseif fa*fc<0
20 b=c; fb=fc;
21 else
22 a=c; fa=fc;
23 end
24 if b-a < TOL, break,end
25 end
26

27 c=(a+b)/2; err=(b-a)/2; fc=feval(f,c);

Example 2.7. You can call the above algorithm with varying function, by

>> f = @(x) x.^3+4*x.^2-10;

>> [c,err,fc]=bisect(f,1,2,0.005)
c =
1.3652
err =
0.0020
fc =
7.2025e-005
46 Chapter 2. Solutions of Equations in One Variable

|p − pn+1 |
Example 2.8. In the bisection method, does lim exist?
n→∞ |p − pn |

Solution.

Answer: no
2.2. Fixed-Point Iteration 47

2.2. Fixed-Point Iteration

Definition 2.9. A number p is a fixed point for a given function g if
g(p) = p .

Note: A point p is a fixed point of g , when the point remains unaltered

under the action of g .

Remark 2.10. Given a root-finding problem f (p) = 0, let

g(x) = x − h(x) · f (x), (2.10)

for some h(x). Then, since g(p) = p − h(p) · f (p) = p − 0 = p , Equation (2.10)
defines a fixed-point problem.

Example 2.11. Find fixed points of g(x) = x 2 − 2.

Solution.
Maple-code
1 g := x -> x^2 - 2:
2 g(p) = p;
3 2
4 p - 2 = p
5 factor(g(p) - p);
6 (p + 1) (p - 2)
7 solve(g(p) = p, p);
8 2, -1

Figure 2.3: Fixed points of g(x) = x 2 − 2.

48 Chapter 2. Solutions of Equations in One Variable

2.2.1. Existence and uniqueness of fixed points

Theorem 2.12. (Existence and Uniqueness).

• If g ∈ C[a, b] and g(x) ∈ [a, b] for all x ∈ [a, b], then g has at least
one fixed point in [a, b].
• If, in addition, g is differentiable in (a, b) and there exists a positive
constant K < 1 such that

|g0 (x)| ≤ K < 1 for all x ∈ (a, b), (2.11)

then there is a unique fixed point in [a, b].

Proof.

Figure 2.4: Illustration of the existence-and-uniqueness theorem.

Example 2.13. Show that g(x) = (x 2 − 2)/3 has a unique fixed point on
[−1, 1].
Solution.
2.2. Fixed-Point Iteration 49

2.2.2. Fixed-point iteration

Definition 2.14. A fixed-point iteration is an iterative procedure of

the form: For a given p0 ,

pn = g(pn−1 ) for n ≥ 1. (2.12)

If the sequence pn converges to p , since g is continuous, we have

p = lim pn = lim g(pn−1 ) = g( lim pn−1 ) = g(p).

n→∞ n→∞ n→∞

This implies that the limit p is a fixed point of g , i.e., the iteration
converges to a fixed point.

Example 2.15. The equation x 3 + 4x 2 − 10 = 0 has a unique root in [1, 2].

There are many ways to transform the equation to the fixed-point form
x = g(x):

(1) x = g1 (x) = x − (x 3 + 4x 2 − 10)

1  10 2
 10
(2) x = g2 (x) = −x ⇐ x 2 + 4x − =0
4 x x
 10 1/2
(3) x = g3 (x) = − 4x
x
1
(4) x = g4 (x) = (−x 3 + 10)1/2 ⇐ 4x 2 = −x 3 + 10
2
 10 1/2
(5) x = g5 (x) = ⇐ x 2 (x + 4) − 10 = 0
x +4
x 3 + 4x 2 − 10
(6) x = g6 (x) = x −
3x 2 + 8x
The associated fixed-point iteration may not converge for some choices of
g . Let’s check it.
50 Chapter 2. Solutions of Equations in One Variable

Evaluation of max |gk0 (x)| for the fixed-point iteration

x ∈[1,2]

The real root of x 3 + 4x 2 − 10 = 0 is p = 1.3652300134142.

Maple-code
1 with(Student[NumericalAnalysis]);
2

3 g1 := x -> x - x^3 - 4*x^2 + 10:

4 maximize(abs(diff(g1(x), x)), x = 1..2);
5 27
6 FixedPointIteration(x-g1(x), x=1.5, tolerance=10^-3, output=sequence);
7 1.5, -0.875, 6.732421875, -469.7200120, 1.027545552 10 ,
8 24 72 216
9 -1.084933871 10 , 1.277055593 10 , -2.082712916 10 ,
10 648 1946 5840
11 9.034169425 10 , -7.373347340 10 , 4.008612522 10
12

13 g2 := x -> 5/2*1/x - 1/4*x^2:

14 maximize(abs(diff(g2(x), x)), x = 1..2);
15 3
16 FixedPointIteration(x-g2(x), x=1.5, tolerance=10^-3, output=sequence);
17 1.5, 1.104166667, 1.959354936, 0.3161621898, 7.882344262,
18 5
19 -15.21567323, -58.04348221, -842.3045280, -1.773692325 10 ,
20 9 19
21 -7.864961160 10 , -1.546440351 10
22

23 g3 := x -> (10/x - 4*x)^(1/2):

24 maximize(abs(diff(g3(x), x)), x = 1..2);
25 infinity
26 FixedPointIteration(x-g3(x), x=1.5, tolerance=10^-3, output=sequence);
27 1.5, 0.8164965811
28

29 g4 := x -> 1/2*(10 - x^3)^(1/2):

30 evalf(maximize(abs(diff(g4(x), x)), x = 1..2));
31 2.121320343
32 FixedPointIteration(x-g4(x), x=1.5, tolerance=10^-3, output=sequence);
33 1.5, 1.286953768, 1.402540804, 1.345458374, 1.375170253,
34 1.360094192, 1.367846968, 1.363887004, 1.365916734, 1.364878217
35

36 g5 := x -> 10^(1/2)*(1/(x + 4))^(1/2):

37 evalf(maximize(abs(diff(g5(x), x)), x = 1..2));
38 0.1414213562
39 FixedPointIteration(x-g5(x), x=1.5, tolerance=10^-3, output=sequence);
40 1.5, 1.348399725, 1.367376372, 1.364957015, 1.365264748
41
2.2. Fixed-Point Iteration 51

42 g6 := x -> x - (x^3 + 4*x^2 - 10)/(3*x^2 + 8*x):

43 maximize(diff(g6(x), x), x = 1..2);
44 5
45 --
46 14
47 maximize(-diff(g6(x), x), x = 1..2);
48 70
49 ---
50 121
51 FixedPointIteration(x-g6(x), x=1.5, tolerance=10^-3, output=sequence);
52 1.5, 1.373333333, 1.365262015, 1.365230014

Theorem 2.16. (Fixed-Point Theorem):

Let g ∈ C[a, b] and g(x) ∈ [a, b] for all x ∈ [a, b]. Suppose that g is
differentiable in (a, b) and there is a positive constant K < 1 such that

|g0 (x)| ≤ K < 1 for all x ∈ (a, b). (2.13)

Then, for any number p0 ∈ [a, b], the sequence defined by

pn = g(pn−1 ), n ≥ 1, (2.14)

converges to the unique fixed point p ∈ [a, b].

Proof.

• It follows from Theorem 2.12 that there exists a unique fixed point p ∈
[a, b], i.e., p = g(p) ∈ [a, b].
• Since g(x) ∈ [a, b] for all x ∈ [a, b], we have pn ∈ [a, b] for all n ≥ 1. It
follows from the MVT that

|p − pn | = |g(p) − g(pn−1 )| = |g 0 (ξn )(p − pn−1 )| ≤ K |p − pn−1 |,

for some ξn ∈ (a, b). Therefore

|p − pn | ≤ K |p − pn−1 | ≤ K 2 |p − pn−2 | ≤ · · · ≤ K n |p − p0 |, (2.15)

which converges to 0 as n → ∞.
52 Chapter 2. Solutions of Equations in One Variable

Remark 2.17. The Fixed-Point Theorem deserves some remarks.

• From (2.15), we can see
|p − pn | ≤ K n max{p0 − a, b − p0 }. (2.16)

• For m > n ≥ 1,
|pm − pn | = |pm − pm−1 + pm−1 − · · · − pn+1 + pn+1 − pn |
≤ |pm − pm−1 | + |pm−1 − pm−2 | + · · · + |pn+1 − pn |
≤ K m−1 |p1 − p0 | + K m−2 |p1 − p0 | + · · · + K n |p1 − p0 |
= K n |p1 − p0 |(1 + K + K 2 + · · · + K m−1−n ).

(Here we have used the MVT, for the last inequality.) Thus,
∞
n
X
i Kn
|p − pn | = lim |pm − pn | ≤ K |p1 − p0 | K = |p1 − p0 |.
m→∞ 1−K
i=0

That is,
Kn
|p − pn | ≤ |p1 − p0 |. (2.17)
1−K
• The Fixed-Point Theorem holds for any contractive mapping g de-
fined on any closed subset C ⊂ R. By a contractive mapping, we
mean a function g that satisfies for some 0 < K < 1,
|g(x) − g(y)| ≤ K |x − y | for all x, y ∈ C. (2.18)

Note: If a contractive mapping g is differentiable, then (2.18) implies

that
|g 0 (x)| ≤ K for all x ∈ C.

Practice 2.18. In practice, p is unknown. Consider the following:

|pn+1 − pn | ≥ |pn − p | − |pn+1 − p |
≥ |pn − p | − K |pn − p | = (1 − K )|pn − p |

and therefore
1 K
|p − pn | ≤ |pn+1 − pn | ≤ |pn − pn−1 |, (2.19)
1−K 1−K

which is useful for stopping of the iteration.

2.2. Fixed-Point Iteration 53

Example 2.19. For each of the following equations, (1) determine an

interval [a, b] on which the fixed-point iteration will converge. (2) Estimate
the number of iterations necessary to obtain approximations accurate to
within 10−5 .

2 − ex + x 2 5
(a) x = (b) x = +2
3 x2

Solution. You may first try to visualize the functions.

(a) (b)

Figure 2.5: Visualization of the functions.

Answer: (a): (1) [0, 1], (2) K = 1/3 ⇒ n ≥ 5 ln(10)/ ln(3) ≈ 10.48.
Example 2.20. Prove that the sequence xn defined recursively as follows
is convergent. 
 x0 = −15
1
 xn+1 = 3 − |xn | (n ≥ 0)
2

Solution. Begin with setting g(x) = 3 − 21 |x |; then show g is a contractive

mapping on C = R.
54 Chapter 2. Solutions of Equations in One Variable

2.3. Newton’s Method and Its Variants

2.3.1. The Newton’s method
The Newton’s method is also called the Newton-Raphson method.
Recall: The objective is to find a zero p of f :

f (p) = 0. (2.20)

Strategy 2.21. Let p0 be an approximation of p . We will try to find

a correction term h such that (p0 + h) is a better approximation of p
than p0 ; ideally (p0 + h) = p .

• If f 00 exists and is continues, then by Taylor’s Theorem

0 (p − p0 )2 00
0 = f (p) = f (p0 + h) = f (p0 ) + (p − p0 )f (p0 ) + f (ξ ), (2.21)
2
where ξ lies between p and p0 .
• If |p − p0 | is small, it is reasonable to ignore the last term of (2.21) and
solve for h = p − p0 :
f (p0 )
h = p − p0 ≈ − . (2.22)
f 0 (p0 )
• Define
f (p0 )
p1 = p0 − ; (2.23)
f 0 (p0 )
then p1 may be a better approximation of p than p0 .
• The above can be repeated.

Algorithm 2.22. (Newton’s method for solving f(x) = 0). For p0

chosen close to a root p , compute {pn } repeatedly satisfying
f (pn−1 )
pn = pn−1 − , n ≥ 1. (2.24)
f 0 (pn−1 )
2.3. Newton’s Method and Its Variants 55

Graphical interpretation
• Let p0 be the initial approximation close to p . Then, the tangent line
at (p0 , f (p0 )) reads
L (x) = f 0 (p0 )(x − p0 ) + f (p0 ). (2.25)
• To find the x -intercept of y = L (x), let
0 = f 0 (p0 )(x − p0 ) + f (p0 ).

Solving the above equation for x becomes

f (p0 )
x = p0 − , (2.26)
f 0 (p0 )
of which the right-side is the same as in (2.23).

Figure 2.6: Graphical interpretation of the Newton’s method.

An Example of Divergence
1 f := arctan(x);
2 Newton(f, x = Pi/2, output = plot, maxiterations = 3);

Figure 2.7: An example of divergence.

56 Chapter 2. Solutions of Equations in One Variable

Remark 2.23.
• The Newton’s method may diverge, unless the initialization is ac-
curate.
• The Newton’s method can be interpreted as a fixed-point itera-
tion:
f (pn−1 )
pn = g(pn−1 ) := pn−1 − 0 . (2.27)
f (pn−1 )
• It cannot be continued if f 0 (pn−1 ) = 0 for some n. As a matter of fact,
the Newton’s method is most effective when f 0 (x) is bounded away
from zero near p .

Convergence analysis for the Newton’s method: Define the error in

the n-th iteration: en = pn − p . Then
f (pn−1 ) en−1 f 0 (pn−1 ) − f (pn−1 )
en = pn − p = pn−1 − 0 −p = . (2.28)
f (pn−1 ) f 0 (pn−1 )
On the other hand, it follows from the Taylor’s Theorem that
1
0 = f (p) = f (pn−1 − en−1 ) = f (pn−1 ) − en−1 f 0 (pn−1 ) + en2−1 f 00 (ξn−1 ), (2.29)
2
for some ξn−1 . Thus, from (2.28) and (2.29), we have
1 f 00 (ξn−1 ) 2
en = e . (2.30)
2 f 0 (pn−1 ) n−1

Theorem 2.24. (Convergence of Newton’s method): Let f ∈

C 2 [a, b] and p ∈ (a, b) is such that f (p) = 0 and f 0 (p) 6= 0. Then, there
is a neighborhood of p such that if the Newton’s method is started p0 in
that neighborhood, it generates a convergent sequence pn satisfying

|pn − p | ≤ C |pn−1 − p |2 , (2.31)

for a positive constant C .

2.3. Newton’s Method and Its Variants 57

Example 2.25. Apply the Newton’s method to solve f (x) = arctan(x) = 0,

with p0 = π/5.

1 Newton(arctan(x), x = Pi/5, output = sequence, maxiterations = 5)

2 0.6283185308, -0.1541304479, 0.0024295539, -9.562*10^(-9), 0., 0.

Since p = 0, en = pn and
|en | ≤ 0.67|en−1 |3 , (2.32)
which is an occasional super-convergence.

Theorem 2.26. (Newton’s Method for a Convex Function): Let

f ∈ C 2 (R) be increasing, convex, and of a zero. Then, the zero is unique
and the Newton iteration will converge to it from any starting point.

Example 2.27. Use the Newton’s method to find the square root of a
positive number Q .
√
Solution. Let x = Q . Then x is a root of x 2 − Q = 0. Define f (x) = x 2 − Q ;
set f 0 (x) = 2x . The Newton’s method reads
f (pn−1 ) pn2−1 − Q 1  Q 
pn = pn−1 − 0 = pn−1 − = pn−1 + . (2.33)
f (pn−1 ) 2pn−1 2 pn−1
(Compare the above with (1.45), p. 27.)
NR.mw
1 NR := proc(Q, p0, itmax)
2 local p, n;
3 p := p0;
4 for n to itmax do
5 p := (p+Q/p)/2;
6 print(n, evalf[14](p));
7 end do;
8 end proc:
58 Chapter 2. Solutions of Equations in One Variable

Q := 16: p0 := 1: itmax := 8: Q := 16: p0 := -1: itmax := 8:

NR(Q,p0,itmax); NR(Q,p0,itmax);
1, 8.5000000000000 1, -8.5000000000000
2, 5.1911764705882 2, -5.1911764705882
3, 4.1366647225462 3, -4.1366647225462
4, 4.0022575247985 4, -4.0022575247985
5, 4.0000006366929 5, -4.0000006366929
6, 4.0000000000000 6, -4.0000000000000
7, 4.0000000000000 7, -4.0000000000000
8, 4.0000000000000 8, -4.0000000000000

2.3.2. Systems of nonlinear equations

The Newton’s method for systems of nonlinear equations follows the
same strategy that was used for single equation.That is,
(a) we first linearize,
(b) solve for the correction vector, and
(c) update the solution,

repeating the steps as often as necessary.

An illustration:
• We begin with a pair of equations involving two variables:
(
f1 (x1 , x2 ) = 0
(2.34)
f2 (x1 , x2 ) = 0

• Suppose that (x1 , x2 ) is an approximate solution of the system. Let us

compute the correction vector (h1 , h2 ) so that (x1 + h1 , x2 + h2 ) is a better
approximate solution.

∂ f1 ∂ f1
 0 = f1 (x1 + h1 , x2 + h2 ) ≈ f1 (x1 , x2 ) + h1 + h2 ,


∂ x1 ∂ x2 (2.35)
∂ f2 ∂ f2
 0 = f2 (x1 + h1 , x2 + h2 ) ≈ f2 (x1 , x2 ) + h1 + h2 .


∂ x1 ∂ x2
2.3. Newton’s Method and Its Variants 59

• Define the Jacobian of (f1 , f2 ) at (x1 , x2 ):

" #
∂ f1 /∂ x1 ∂ f1 /∂ x2
J(x1 , x2 ) := (x1 , x2 ). (2.36)
∂ f2 /∂ x1 ∂ f2 /∂ x2

Then, the Newton’s method for two nonlinear equations in two variables
reads " # " # " #
x1n x1n−1 h1n−1
= n −1 + n −1 , (2.37)
x2n x2 h2

where the correction vector satisfies

" # " #
h1n−1 f1 (x1n−1 , x2n−1 )
J(x1n−1 , x2n−1 ) =− . (2.38)
h2n−1 f2 (x1n−1 , x2n−1 )

Summary 2.28. In general, the system of m nonlinear equations,

fi (x1 , x2 , · · · , xm ) = 0, 1 ≤ i ≤ m,

can be expressed as
F(X ) = 0, (2.39)
where X = (x1 , x2 , · · · , xm )T and F = (f1 , f2 , · · · , fm )T . Then

0 = F(X + H) ≈ F(X ) + J(X )H, (2.40)

 
T
∂ fi
where H = (h1 , h2 , · · · , hm ) , the correction vector, and J(X ) = (X ),
∂ xj
the Jacobian of F at X . Hence, Newton’s method for m nonlinear equa-
tions in m variables is given by

X n = X n−1 + H n−1 , (2.41)

where H n−1 is the solution of the linear system:

J(X n−1 )H n−1 = −F(X n−1 ). (2.42)

60 Chapter 2. Solutions of Equations in One Variable

Example 2.29. Starting with (1, 1, 1)T , carry out 6 iterations of the New-
ton’s method to find a root of the nonlinear system
xy = z2 + 1
xyz + y 2 = x 2 + 2
ex + z = ey + 3

Solution.
Procedure NewtonRaphsonSYS.mw
1 NewtonRaphsonSYS := proc(X, F, X0, TOL, itmax)
2 local Xn, H, FX, J, i, m, n, Err;
3 m := LinearAlgebra[Dimension](Vector(X));
4 Xn := Vector(m);
5 H := Vector(m);
6 FX := Vector(m);
7 J := Matrix(m, m);
8 Xn := X0;
9 for n to itmax do
10 FX := eval(F, [seq(X[i] = Xn[i], i = 1..m)]);
11 J := evalf[15](VectorCalculus[Jacobian](F, X=convert(Xn,list)));
12 H := -MatrixInverse(J).Vector(FX);
13 Xn := Xn + H;
14 printf(" %3d %.8f ", n, Xn[1]);
15 for i from 2 to m do; printf(" %.8f ", Xn[i]); end do;
16 for i to m do; printf(" %.3g ", H[i]); end do;
17 printf("\n");
18 if (LinearAlgebra[VectorNorm](H, 2) < TOL) then break endif:
19 end do;
20 end proc:

Result
1 F := [x*y-z^2-1, x*y*z-x^2+y^2-2, exp(x)+z-exp(y)-3]:
2 X := [x, y, z]:
3 X0 := <1, 1, 1>:
4 TOL := 10^-8: itmax := 10:
5 NewtonRaphsonSYS(X, F, X0, TOL, itmax):
6 1 2.18932610 1.59847516 1.39390063 1.19 0.598 0.394
7 2 1.85058965 1.44425142 1.27822400 -0.339 -0.154 -0.116
8 3 1.78016120 1.42443598 1.23929244 -0.0704 -0.0198 -0.0389
9 4 1.77767471 1.42396093 1.23747382 -0.00249 -0.000475 -0.00182
10 5 1.77767192 1.42396060 1.23747112 -2.79e-006 -3.28e-007 -2.7e-006
11 6 1.77767192 1.42396060 1.23747112 -3.14e-012 -4.22e-014 -4.41e-012
2.3. Newton’s Method and Its Variants 61

2.3.3. The secant method

Recall: The Newton’s method, defined as
f (pn−1 )
pn = pn−1 − , n ≥ 1, (2.43)
f 0 (pn−1 )
is a powerful technique. However it has a major drawback: the need to
know the value of derivative of f at each iteration. Frequently, f 0 (x) is far
more difficult to calculate than f (x).

Algorithm 2.30. (The Secant method). To overcome the disadvan-

tage of the Newton’s method, a number of methods have been proposed.
One of most popular variants is the secant method, which replaces
f 0 (pn−1 ) by a difference quotient:
f (pn−1 ) − f (pn−2 )
f 0 (pn−1 ) ≈ . (2.44)
pn−1 − pn−2
Thus, the resulting algorithm reads

n −1 − p n −2
h p i
pn = pn−1 − f (pn−1 ) , n ≥ 2. (2.45)
f (pn−1 ) − f (pn−2 )

Note:
• Two initial values (p0 , p1 ) must be given, which however is not a
drawback.
• It requires only one new evaluation of f per step.
• The graphical interpretation of the secant method is similar to that
of Newton’s method.
• Convergence:
f 00 (p) f 00 (p) 0.62 √
(1+ 5)/2
|en | ≈ en−1 en−2 ≈ |en−1 | . (2.46)
2f 0 (p) 2f 0 (p)
Here evalf((1+sqrt(5))/2) = 1.618033988.
62 Chapter 2. Solutions of Equations in One Variable

Graphical interpretation
with(Student[NumericalAnalysis]):
f := x b 3 - 1:
Secant(f, x = [1.5, 0.5], maxiterations = 3, output = sequence);
1.5, 0.5, 0.7692307692, 1.213510253, 0.9509757891
Secant(f, x = [1.5, 0.5], maxiterations = 3, output = plot);

Figure 2.8: Graphical interpretation of the secant method.

Example 2.31. Apply one iteration of the secant method to find p2 if

p0 = 1, p1 = 2, f (p0 ) = 2, f (p1 ) = 1.5.

Solution.

Answer: p2 = 5.0
2.3. Newton’s Method and Its Variants 63

2.3.4. The method of false position

It generates approximations in a similar manner as the secant method;
however, it includes a test to ensure that the root is always bracketed be-
tween successive iterations.
Algorithm 2.32. (Method of false position).
• Select p0 and p1 such that f (p0 ) · f (p1 ) < 0.
• Compute
p2 = the x -intercept of the line joining (p0 , f (p0 )) and (p1 , f (p1 )).

• If (f (p1 ) · f (p2 ) < 0), set (p1 and p2 bracket the root)
p3 = the x -intercept of the line joining (p2 , f (p2 )) and (p1 , f (p1 )).

else, set
p3 = the x -intercept of the line joining (p2 , f (p2 )) and (p0 , f (p0 )).

endif
Graphical interpretation
with(Student[NumericalAnalysis]):
f := x b 3 - 1:
FalsePosition(f,x=[1.5,0.5], maxiterations=3, output=plot);

Figure 2.9: Graphical interpretation of the false position method.

64 Chapter 2. Solutions of Equations in One Variable

Convergence Speed:
Find a root for x = cos x , starting with π/4 or [0.5, π/4].
Comparison
1 with(Student[NumericalAnalysis]):
2 f := cos(x) - x:
3

4 N := Newton(f, x=Pi/4, tolerance=10^-8, maxiterations=10,

5 output=sequence);
6 0.7853981635, 0.7395361335, 0.7390851781, 0.7390851332, 0.7390851332
7

8 S := Secant(f,x=[0.5,Pi/4],tolerance=10^-8,maxiterations=10,
9 output=sequence);
10 0.5, 0.7853981635, 0.7363841388, 0.7390581392, 0.7390851493,
11 0.7390851332, 0.7390851332
12

13 F := FalsePosition(f,x=[0.5,Pi/4],tolerance=10^-8,maxiterations=10,
14 output=sequence);
15 [0.5, 0.7853981635], [0.7363841388, 0.7853981635],
16 [0.7390581392, 0.7853981635], [0.7390848638, 0.7853981635],
17 [0.7390851305, 0.7853981635], [0.7390851332, 0.7853981635],
18 [0.7390851332, 0.7853981635], [0.7390851332, 0.7853981635],
19 [0.7390851332, 0.7853981635], [0.7390851332, 0.7853981635],
20 [0.7390851332, 0.7853981635]

# print out
n Newton Secant False Position
0 0.7853981635 0.5000000000 0.5000000000
1 0.7395361335 0.7853981635 0.7363841388
2 0.7390851781 0.7363841388 0.7390581392
3 0.7390851332 0.7390581392 0.7390848638
4 0.7390851332 0.7390851493 0.7390851305
5 0.7390851332 0.7390851332 0.7390851332
6 0.7390851332 0.7390851332 0.7390851332
7 0.7390851332 0.7390851332 0.7390851332
8 0.7390851332 0.7390851332 0.7390851332
2.4. Zeros of Polynomials 65

2.4. Zeros of Polynomials

Definition 2.33. A polynomial of degree n has a form

P(x) = an x n + an−1 x n−1 + · · · + a1 x + a0 , (2.47)

where an 6= 0 and ai ’s are called the coefficients of P .

Theorem 2.34. (Theorem on Polynomials).

• Fundamental Theorem of Algebra: Every nonconstant polyno-
mial has at least one root (possibly, in the complex field).
• Complex Roots of Polynomials: A polynomial of degree n has
exactly n roots in the complex plane, being agreed that each root
shall be counted a number of times equal to its multiplicity. That
is, there are unique (complex) constants x1 , x2 , · · · , xk and unique in-
tegers m1 , m2 , · · · , mk such that
k
X
m1 m2 mk
P(x) = an (x − x1 ) (x − x2 ) · · · (x − xk ) , mi = n. (2.48)
i=1

• Localization of Roots: All roots of the polynomial P lie in the open

disk centered at the origin and of radius of
1
ρ=1+ max |ai |. (2.49)
|an | 0≤i <n

• Uniqueness of Polynomials: Let P(x) and Q(x) be polynomials of

degree n. If x1 , x2 , · · · , xr , with r > n, are distinct numbers with P(xi ) =
Q(xi ), for i = 1, 2, · · · , r , then P(x) = Q(x) for all x . For example, two
polynomials of degree n are the same if they agree at (n + 1) points.
66 Chapter 2. Solutions of Equations in One Variable

2.4.1. Horner’s method

Note: Known as nested multiplication and also as synthetic divi-
sion, Horner’s method can evaluate polynomials very efficiently. It
requires n multiplications and n additions to evaluate an arbitrary n-th
degree polynomial.

Algorithm 2.35. Let us try to evaluate P(x) at x = x0 .

• Utilizing the Remainder Theorem, we can rewrite the polynomial
as
P(x) = (x − x0 )Q(x) + r = (x − x0 )Q(x) + P(x0 ), (2.50)
where Q(x) is a polynomial of degree n − 1, say

Q(x) = bn x n−1 + · · · + b2 x + b1 . (2.51)

• Substituting the above into (2.50), utilizing (2.47), and setting equal
the coefficients of like powers of x on the two sides of the resulting
equation, we have
bn = an
b n −1 = an−1 + x0 bn
.. (2.52)
.
b1 = a1 + x0 b2
P(x0 ) = a0 + x0 b1

• Introducing b0 = P(x0 ), the above can be rewritten as

bn+1 = 0; bk = ak + x0 bk +1 , n ≥ k ≥ 0. (2.53)

• If the calculation of Horner’s algorithm is to be carried out with pencil

and paper, the following arrangement is often used (known as syn-
thetic division):
2.4. Zeros of Polynomials 67

Example 2.36. Use Horner’s algorithm to evaluate P(3), where

P(x) = x 4 − 4x 3 + 7x 2 − 5x − 2. (2.54)

Solution. For x0 = 3, we arrange the calculation as mentioned above:

Note that the 4-th degree polynomial in (2.54) is written as

P(x) = (x − 3)(x 3 − x 2 + 4x + 7) + 19.

Note: When the Newton’s method is applied for finding an approximate

zero of P(x), the iteration reads
P(xn−1 )
xn = xn−1 − . (2.55)
P 0 (xn−1 )
Thus both P(x) and P 0 (x) must be evaluated in each iteration.

How to evaluate P 0 (x): The derivative P 0 (x) can be evaluated by using the
Horner’s method with the same efficiency. Indeed, differentiating (2.50)
reads
P 0 (x) = Q(x) + (x − x0 )Q 0 (x). (2.56)
Thus
P 0 (x0 ) = Q(x0 ). (2.57)
That is, the evaluation of Q at x0 becomes the desired quantity P 0 (x0 ).
68 Chapter 2. Solutions of Equations in One Variable

Example 2.37. Evaluate P 0 (3) for P(x) considered in Example 2.36, the
previous example.
Solution. As in the previous example, we arrange the calculation and
carry out the synthetic division one more time:

Example 2.38. Implement the Horner’s algorithm to evaluate P(3) and

P 0 (3), for the polynomial in (2.54): P(x) = x 4 − 4x 3 + 7x 2 − 5x − 2.
Solution.
horner.m
1 function [p,d] = horner(A,x0)
2 % function [px0,dpx0] = horner(A,x0)
3 % input: A = [a_0,a_1,...,a_n]
4 % output: p=P(x0), d=P'(x0)
5

6 n = size(A(:),1);
7 p = A(n); d=0;
8

9 for i = n-1:-1:1
10 d = p + x0*d;
11 p = A(i) +x0*p;
12 end

Call_horner.m
1 a = [-2 -5 7 -4 1];
2 x0=3;
3 [p,d] = horner(a,x0);
4 fprintf(" P(%g)=%g; P'(%g)=%g\n",x0,p,x0,d)
5 P(3)=19; P'(3)=37
2.4. Zeros of Polynomials 69

Example 2.39. Let P(x) = x 4 − 4x 3 + 7x 2 − 5x − 2, as in (2.54). Use the

Newton’s method and the Horner’s method to implement a code and find
an approximate zero of P near 3.
Solution.
newton_horner.m
1 function [x,it] = newton_horner(A,x0,tol,itmax)
2 % function x = newton_horner(A,x0)
3 % input: A = [a_0,a_1,...,a_n]; x0: initial for P(x)=0
4 % outpue: x: P(x)=0
5

6 x = x0;
7 for it=1:itmax
8 [p,d] = horner(A,x);
9 h = -p/d;
10 x = x + h;
11 if(abs(h)<tol), break; end
12 end
Call_newton_horner.m
1 a = [-2 -5 7 -4 1];
2 x0=3;
3 tol = 10^-12; itmax=1000;
4 [x,it] = newton_horner(a,x0,tol,itmax);
5 fprintf(" newton_horner: x0=%g; x=%g, in %d iterations\n",x0,x,it)
6 newton_horner: x0=3; x=2, in 7 iterations

Figure 2.10: Polynomial P(x) = x 4 − 4x 3 + 7x 2 − 5x − 2. Its two zeros are −0.275682 and 2.
70 Chapter 2. Solutions of Equations in One Variable

2.4.2. Complex zeros: Finding quadratic factors

Note: (Quadratic Factors of Real-coefficient Polynomials).

As mentioned in (2.47), a polynomial of degree n has a form

P(x) = an x n + an−1 x n−1 + · · · + a1 x + a0 . (2.58)

• Theorem on Real Quadratic Factor: If P is a polynomial whose

coefficients are all real, and if z is a nonreal root of P , then z is also a
root and
(x − z)(x − z)
is a real quadratic factor of P .
• Polynomial Factorization: If P is a nonconstant polynomial of
real coefficients, then it can be factorized as a multiple of linear and
quadratic polynomials of which coefficients are all real.
• Theorem on Quotient and Remainder: If the polynomial is di-
vided by the quadratic polynomial (x 2 − ux − v), then we can formally
write the quotient and remainder as

Q(x) = bn x n−2 + bn−1 x n−3 + · · · + b3 x + b2

(2.59)
r(x) = b1 (x − u) + b0 ,

with which P(x) = (x 2 − ux − v)Q(x) + r(x). As in Algorithm 2.35, the

coefficients bk can be computed recursively as follows.

bn+1 = bn+2 = 0
(2.60)
bk = ak + ubk +1 + vbk +2 , n ≥ k ≥ 0.
2.4. Zeros of Polynomials 71

2.4.3. Bairstow’s method

Bairstow’s method seeks a real quadratic factor of P of the form (x 2 −
ux − v). For simplicity, all the coefficients ai ’s are real so that both u and v
will be real.

Observation 2.40. In order for the quadratic polynomial to be a factor

of P , the remainder r(x) must be zero. That is, the process seeks a
quadratic factor (x 2 − ux − v) of P such that

b0 (u, v) = 0, b1 (u, v) = 0. (2.61)

The quantities b0 and b1 must be functions of (u, v), which is clear from
(2.59) and (2.60).

Key Idea 2.41. An outline of the process is as follows:

• Starting values are assigned to (u, v). We seek corrections (δ u, δ v) so
that
b0 (u + δ u, v + δ v) = b1 (u + δ u, v + δ v) = 0 (2.62)

• Linearization of these equations reads

∂ b0 ∂ b0
0 ≈ b0 (u, v) + δu + δv
∂u ∂v (2.63)
∂ b1 ∂ b1
0 ≈ b1 (u, v) + δu + δv
∂u ∂v
• Thus, the corrections can be found by solving the linear system
" # " #
δu b0 (u, v) ∂ (b0 , b1 )
J =− , where J = . (2.64)
δv b1 (u, v) ∂ (u, v)

Here J is the Jacobian matrix.

72 Chapter 2. Solutions of Equations in One Variable

Question: How to compute the Jacobian matrix

Bairstow’s method
Algorithm 2.42.
• As first appeared in the appendix of the 1920 book “Applied Aero-
dynamics" by Leonard Bairstow, we consider the partial derivatives

∂ bk ∂ bk −1
ck = , dk = (0 ≤ k ≤ n). (2.65)
∂u ∂v
• Differentiating the recurrence relation, (2.60), results in the fol-
lowing pair of additional recurrences:

ck = bk +1 + uck +1 + vck +2 (cn+1 = cn+2 = 0)

(2.66)
dk = bk +1 + udk +1 + vdk +2 (dn+1 = dn+2 = 0)

Note that these recurrence relations obviously generate the same two
sequences (ck = dk ); we need only the first.
• The Jacobian explicitly reads
" #
∂ (b0 , b1 ) c0 c1
J= = , (2.67)
∂ (u, v) c1 c2

and therefore
" # " # " #
δu b0 1 b1 c1 − b0 c2
= −J −1 = 2
. (2.68)
δv b1 c0 c2 − c1 b0 c1 − b1 c0

We summarize the above procedure as in the following code:

2.4. Zeros of Polynomials 73

Figure 2.11: Bairstow’s method.

74 Chapter 2. Solutions of Equations in One Variable

Run Bairstow
1 P := x -> x^4 - 4*x^3 + 7*x^2 - 5*x - 2:
2 n := degree(P(x)):
3 a := Array(0..n):
4 for i from 0 to n do
5 a[i] := coeff(P(x), x, i);
6 end do:
7 itmax := 10: TOL := 10^-10:
8

9 u := 3:
10 v := -4:
11 Bairstow(n, a, u, v, itmax, TOL);
12 1 2.2000000 -2.7000000 -0.8 1.3
13 2 2.2727075 -3.9509822 0.07271 -1.251
14 3 2.2720737 -3.6475280 -0.0006338 0.3035
15 4 2.2756100 -3.6274260 0.003536 0.0201
16 5 2.2756822 -3.6273651 7.215e-05 6.090e-05
17 6 2.2756822 -3.6273651 6.316e-09 -9.138e-09
18 7 2.2756822 -3.6273651 -1.083e-17 -5.260e-17
19 Q(x) = (1)x^2 + (-1.72432)x^1 + (-0.551364)
20 Remainder: -2.66446e-18 (x - (2.27568)) + (-2.47514e-16)
21 Quadratic Factor: x^2 - (2.27568)x - (-3.62737)
22 Zeros: 1.137841102 +- (1.527312251) i
2.4. Zeros of Polynomials 75

Deflation
• Given a polynomial of degree n, P(x), if the Newton’s method finds a
zero (say, b
x1 ), it will be written as

P(x) ≈ (x − b
x1 )Q1 (x). (2.69)

• Then, we can find a second approximate zero b

x2 (or, a quadratic factor)
of P by applying Newton’s method to the reduced polynomial Q1 (x):

Q1 (x) ≈ (x − b
x2 )Q2 (x). (2.70)

• The computation continues up to the point that P is factorized by

linear and quadratic factors. The procedure is called deflation.

Remark 2.43.
• The deflation process introduces an accuracy issue, due to the fact
that when we obtain the approximate zeros of P(x), the Newton’s
method is applied to the reduced polynomials Qk (x).
• An approximate zero b xk +1 of Qk (x) will generally not approximate a
root of P(x) = 0; inaccuracy increases as k increases.
• One way to overcome the difficulty is to improve the approximate
zeros; starting with these zeros, apply the Newton’s method with the
original polynomial P(x).
76 Chapter 2. Solutions of Equations in One Variable

Exercises for Chapter 2

2.1. Let the bisection method be applied to a continuous function, resulting in intervals
[a1 , b1 ], [a2 , b2 ], · · · . Let pn = (an + bn )/2 and p = lim pn . Which of these statements can
n→∞
be false?

(a) a1 ≤ a2 ≤ · · ·
b1 − a1
(b) |p − pn | ≤ , n≥1
2n
(c) |p − pn+1 | ≤ |p − pn |, n ≥ 1
(d) [an+1 , bn+1 ] ⊂ [an , bn ]
1
(e) |p − pn | = O n as n → ∞
2

2.2. C Modify the Matlab code used in Example 2.7 for the bisection method to incorpo-
rate 
Inputs : f, a, b, TOL, itmax
Stopping criterion : Relative error ≤ TOL or k ≤ itmax
Consider the following equations defined on the given intervals:

I. 3x − e x = 0, [0, 1]
II. 2x cos(2x) − (x + 1)2 = 0, [−1, 0]

For each of the above equations,

(a) Use Maple or Matlab (or something else) to find a very accurate solution in the
interval.
(b) Find the approximate root by using your Matlab with TOL=10−6 and itmax=10.
(c) Report pn , |p − pn |, and |p − pn−1 |, for n ≥ 1, in a table format.

2.3. C Let us try to find 51/3 by the fixed-point method. Use the fact that the result must
be the positive solution of f (x) = x 3 − 5 = 0 to solve the following:

(a) Introduce two different fixed-point forms which are convergent for x ∈ [1, 2].
(b) Perform five iterations for each of the iterations with p0 = 1.5, and measure
|p − p5 |.
(c) Rank the associated iterations based on their apparent speed of convergence
with p0 = 1.5. Discuss why one is better than the other.

2.4. Kepler’s equation in astronomy reads

y = x − ε sin(x), with 0 < ε < 1. (2.71)

(a) Show that for each y ∈ [0, π ], there exists an x satisfying the equation.
(b) Interpret this as a fixed-point problem.
2.4. Zeros of Polynomials 77

(c) C Find x ’s for y = 1, π/2, 2, using the fixed-point iteration. Set ε = 1/2.

Hint : For (a), you may have to use the IVT for x − ε ∗ sin(x) defined on [0, π ], while
for (b) you should rearrange the equation in the form of x = g(x). For (c), you may
use any source of program which utilizes the fixed-point iteration.
2.5. Consider a variation of Newton’s method in which only one derivative is needed;
that is,
f (pn−1 )
pn = pn−1 − , n ≥ 1. (2.72)
f 0 (p0 )
Find C and s such that
en ≈ Cens−1 . (2.73)
1
Hint : You may have to use f (pn−1 ) = en−1 f 0 (pn−1 ) − en2−1 f 00 (ξn−1 ).
2
2.6. (Note: Do not use programming for this problem.) Starting with x0 = (0, 1)T , carry
out two iterations of the Newton’s method on the system:
 2
4x − y 2 = 0
4xy 2 − x = 1

Hint : Define"f1 (x, y) = 4x 2 −#

y 2 , f2 (x, y) = 4xy 2 − x − 1. Then try
" to use# (2.37)-(2.38),
" p.59.
# "Note#
8x −2y 0 −2 f1 (x0 , y0 ) −1
that J(x, y) = . Thus, for example, J(x0 , y0 ) = and = .
4y 2 − 1 8xy 3 0 f2 (x0 , y0 ) −1
Now you can find the correction vector to update iterate and get x1 . Do it once more for x2 .

2.7. C Consider the polynomial

P(x) = 3x 5 − 7x 4 − 5x 3 + x 2 − 8x + 2.

(a) Use the Horner’s algorithm to find P(4).

(b) Use the Newton’s method to find a real-valued root, starting with x0 = 4. and
applying the Horner’s algorithm for the evaluation of P(xk ) and P 0 (xk ).
(c) Apply the Bairstow’s method, with the initial point (u, v) = (0, −1), to find a pair
of complex-valued zeros.
(d) Find a disk centered at the origin that contains all the roots.
78 Chapter 2. Solutions of Equations in One Variable
Chapter 3

Interpolation and Polynomial

Approximation

This chapter introduces the following.

Topics Applications/Properties
Polynomial interpolation The first step toward approximation theory
Newton form
Lagrange form Basis functions for various applications in-
cluding visualization and FEMs
Chebyshev polynomial Optimized interpolation
Divided differences
Neville’s method Evaluation of interpolating polynomials
Hermite interpolation It incorporates f (xi ) and f 0 (xi )
Spline interpolation Less oscillatory interpolation
B-splines
Parametric curves Curves in the plane or the space
Rational interpolation Interpolation of rough data with minimum
oscillation

79
80 Chapter 3. Interpolation and Polynomial Approximation

3.1. Polynomial Interpolation

Each continuous function can be approximated (arbitrarily close) by a poly-
nomial, and polynomials of degree n interpolating values at (n + 1) distinct
points are all the same polynomial, as shown in the following theorems.

Theorem 3.1. (Weierstrass approximation theorem): Suppose f ∈

C[a, b]. Then, for each ε > 0, there exists a polynomial P(x) such that

|f (x) − P(x)| < ε, for all x ∈ [a, b]. (3.1)

Example 3.2. Let f (x) = e x . Then

Figure 3.1: Polynomial approximations for f (x) = e x .

3.1. Polynomial Interpolation 81

Theorem 3.3. (Polynomial Interpolation Theorem):

If x0 , x1 , x2 , · · · , xn are (n + 1) distinct real numbers, then for arbitrary
values y0 , y1 , y2 , · · · , yn , there is a unique polynomial pn of degree at
most n such that
pn (xi ) = yi (0 ≤ i ≤ n). (3.2)

Proof. (Uniqueness). Suppose there were two such polynomials, pn and qn .

Then pn − qn would have the property

(pn − qn )(xi ) = 0, for 0 ≤ i ≤ n. (3.3)

Since the degree of pn − qn is at most n, the polynomial can have at most n

zeros unless it is a zero polynomial. Since xi are distinct, pn − qn has n + 1
zeros and therefore it must be 0. Hence,

pn ≡ qn .

(Existence). For the existence part, we proceed inductively through con-

struction.
• For n = 0, the existence is obvious since we may choose the constant
function
p0 (x) = y0 . (3.4)

• Now suppose that we have obtained a polynomial pk −1 of degree ≤ k − 1

with
pk −1 (xi ) = yi , for 0 ≤ i ≤ k − 1. (3.5)

• We try to construct pk in the form

pk (x) = pk −1 (x) + ck (x − x0 )(x − x1 ) · · · (x − xk −1 ) (3.6)

for some ck .

(a) Note that (3.6) is unquestionably a polynomial of degree ≤ k .

(b) Furthermore, pk interpolates the data that pk −1 interpolates:

pk (xi ) = pk −1 (xi ) = yi , 0 ≤ i ≤ k − 1. (3.7)

82 Chapter 3. Interpolation and Polynomial Approximation

• Now we determine the constant ck to satisfy the condition

pk (xk ) = yk , (3.8)
which leads to
pk (xk ) = pk −1 (xk ) + ck (xk − x0 )(xk − x1 ) · · · (xk − xk −1 ) = yk . (3.9)
This equation can certainly be solved for ck :
yk − pk −1 (xk )
ck = , (3.10)
(xk − x0 )(xk − x1 ) · · · (xk − xk −1 )
because the denominator is not zero. (Why?)

3.1.1. Newton form of the interpolating polynomials

As in the proof of the previous theorem, each pk (k ≥ 1) is obtained by
adding a single term to pk −1 . Thus, at the end of the process, pn will be a
sum of terms and p0 , p1 , · · · , pn−1 will be easily visible in the expression of
pn . Each pk has the form

pk (x) = c0 + c1 (x − x0 ) + · · · + ck (x − x0 )(x − x1 ) · · · (x − xk −1 ). (3.11)

The compact form of this reads
k
X i −1
Y
pk (x) = ci (x − xj ). (3.12)
i=0 j=0

m
Y
(Here the convention has been adopted that (x − xj ) = 1 when m < 0.)
j=0
The first few cases of (3.12) are
p0 (x) = c0 ,
p1 (x) = c0 + c1 (x − x0 ), (3.13)
p2 (x) = c0 + c1 (x − x0 ) + c2 (x − x0 )(x − x1 ).
These polynomials are called the interpolating polynomials in Newton
form, or Newton form of interpolating polynomials.
3.1. Polynomial Interpolation 83

Illustration of Newton’s interpolating polynomials

Example 3.4. Let f (x) = sin(x · (x − 1)) + 1. Let [0, 0.5, 1.0, 2.0, 1.5] be
a collection of distinct points. Find Newton’s interpolating polynomials
which pass {(xi , f (xi )) | i ≤ k } for each k .
A Maple implementation
1 restart:
2 with(Student[NumericalAnalysis]):
3 f := x -> sin(x*(x - 1)) + 1:
4

5 xk := 0:
6 xy := [[xk, f(xk)]]:
7 P0 := PolynomialInterpolation(xy, independentvar = x,
8 method = newton, function = f);
9 p0 := x -> Interpolant(P0):
10 p0(x)
11 1
12

13 xk := 0.5:
14 P1 := AddPoint(P0, [xk, f(xk)]):
15 p1 := x -> Interpolant(P1):
16 p1(x)
17 1. - 0.4948079186 x
18

19 xk := 1.0:
20 P2 := AddPoint(P1, [xk, f(xk)]):
21 p2 := x -> Interpolant(P2):
22 p2(x)
23 1. - 0.4948079186 x + 0.9896158372 x (x - 0.5)
24

25 xk := 2.0:
26 P3 := AddPoint(P2, [xk, f(xk)]):
27 p3 := x -> Interpolant(P3):
28 p3(x)
29 1. - 0.4948079186 x + 0.9896158372 x (x - 0.5)
30 - 0.3566447492 x (x - 0.5) (x - 1.0)
31
84 Chapter 3. Interpolation and Polynomial Approximation

32 xk := 1.5:
33 P4 := AddPoint(P3, [xk, f(xk)]):
34 p4 := x -> Interpolant(P4):
35 p4(x)
36 1. - 0.4948079186 x + 0.9896158372 x (x - 0.5)
37 - 0.3566447492 x (x - 0.5) (x - 1.0)
38 - 0.5517611839 x (x - 0.5) (x - 1.0) (x - 2.0)

Figure 3.2: Illustration of Newton’s interpolating polynomials, with f (x) = sin(x · (x − 1)) + 1,
at [0, 0.5, 1.0, 2.0, 1.5].

Evaluation of pk (x), assuming that c0 , c1 , · · · , ck are known:

We may use an efficient method called nested multiplication or
Horner’s method. This can be explained most easily for an arbitrary
expression of the form
Xk i −1
Y
u= ci dj . (3.14)
i=0 j=0

The idea begins with rewriting it in the form

u = c0 + c1 d0 + c2 d0 d1 + · · · + ck −1 d0 d1 · · · dk −2 + ck d0 d1 · · · dk −1
= ck d0 d1 · · · dk −1 + ck −1 d0 d1 · · · dk −2 + · · · + c2 d0 d1 + c1 d0 + c0
= (ck d1 · · · dk −1 + ck −1 d1 · · · dk −2 + · · · + c2 d1 + c1 )d0 + c0
(3.15)
= ((ck d2 · · · dk −1 + ck −1 d2 · · · dk −2 + · · · + c2 )d1 + c1 )d0 + c0
...
= (· · · (((ck )dk −1 + ck −1 )dk −2 + ck −2 )dk −3 + · · · + c1 )d0 + c0
3.1. Polynomial Interpolation 85

Algorithm 3.5. (Nested Multiplication). Thus the algorithm for the

evaluation of u in (3.14) can be written as
u := c[k];
for i from k-1 by -1 to 0 do
u := u*d[i] + c[i];
end do
The computation of ck , using Horner’s algorithm
Algorithm 3.6. The Horner’s algorithm for the computation of coeffi-
cients ck in Equation (3.12) gives
c[0] := y[0];
for k to n do
d := x[k] - x[k-1];
u := c[k-1];
for i from k-2 by -1 to 0 do
u := u*(x[k] - x[i]) + c[i];
d := d*(x[k] - x[i]);
end do;
c[k] := (y[k] - u)/d;
end do
A more efficient procedure exists that achieves the same result. The alter-
native method uses divided differences to compute the coefficients ck .
The method will be presented later.
86 Chapter 3. Interpolation and Polynomial Approximation

Example 3.7. Let

f (x) = 4x 3 + 35x 2 − 84x − 954.

Four values of this function are given as

xi 5 −7 −6 0
yi 1 −23 −54 −954

Construct the Newton form of the polynomial from the data.

Solution.
Maple-code
1 with(Student[NumericalAnalysis]):
2 f := 4*x^3 + 35*x^2 - 84*x - 954:
3 xy := [[5, 1], [-7, -23], [-6, -54], [0, -954]]:
4 N := PolynomialInterpolation(xy, independentvar = x,
5 method = newton, function = f):
6 Interpolant(N)
7 -9 + 2 x + 3 (x - 5) (x + 7) + 4 (x - 5) (x + 7) (x + 6)
8 # Since "-9 + 2*x = 1 + 2*(x - 5)", the coefficients are
9 # "c[0] = 1, c[1] = 2, c[2] = 3, c[3] = 4"
10 expand(Interpolant(N));
11 3 2
12 4 x + 35 x - 84 x - 954
13 # which is the same as f
14 RemainderTerm(N);
15 0 &where {-7 <= xi_var and xi_var <= 5}
16 Draw(N);
3.1. Polynomial Interpolation 87

DividedDifferenceTable(N);
 
1 0 0 0
 −23 2 0 0


 −54 −31
 
3 0
−954 −150 −17 4

Example 3.8. Find the Newton form of the interpolating polynomial of

the data.
xi 2 −1 1
yi 1 4 −2

Solution.

Answer: p2 (x) = 1 − (x − 2) + 2(x − 2)(x + 1)

88 Chapter 3. Interpolation and Polynomial Approximation

3.1.2. Lagrange Form of Interpolating Polynomials

Let data points (xk , yk ), 0 ≤ k ≤ n be given, where n + 1 abscissas xi are
distinct. The interpolating polynomial will be sought in the form
n
X
pn (x) = y0 Ln,0 (x) + y1 Ln,1 (x) + · · · + yn Ln,n (x) = yk Ln,k (x), (3.16)
k =0

where Ln,k (x) are polynomials that depend on the nodes x0 , x1 , · · · , xn , but
not on the ordinates y0 , y1 , · · · , yn .
How to determine the basis {Ln,k (x)}
Observation 3.9. Let all the ordinates be 0 except for a 1 occupying
i -th position, that is, yi = 1 and other ordinates are all zero.

• Then,
n
X
pn (xj ) = yk Ln,k (xj ) = Ln,i (xj ). (3.17)
k =0

• On the other hand, the polynomial pn interpolating the data must

satisfy pn (xj ) = δij , where δij is the Kronecker delta
(
1 if i = j,
δij =
6 j.
0 if i =

• Thus all the basis polynomials must satisfy

Ln,i (xj ) = δij , for all 0 ≤ i, j ≤ n. (3.18)

Polynomials satisfying such a property are known as the cardinal

functions.
3.1. Polynomial Interpolation 89

Example 3.10. Construction of Ln,0 (x): It is to be an nth-degree

polynomial that takes the value 0 at x1 , x2 , · · · , xn and the value 1 at x0 .
Clearly, it must be of the form
n
Y
Ln,0 (x) = c(x − x1 )(x − x2 ) · · · (x − xn ) = c (x − xj ), (3.19)
j=1

where c is determined for which Ln,0 (x0 ) = 1. That is,

1 = Ln,0 (x0 ) = c(x0 − x1 )(x0 − x2 ) · · · (x0 − xn ) (3.20)

and therefore
1
c= . (3.21)
(x0 − x1 )(x0 − x2 ) · · · (x0 − xn )
Hence, we have
n
Y (x − xj )
(x − x1 )(x − x2 ) · · · (x − xn )
Ln,0 (x) = = . (3.22)
(x0 − x1 )(x0 − x2 ) · · · (x0 − xn ) (x0 − xj )
j=1

Summary 3.11. Each cardinal function is obtained by similar reason-

ing; the general formula is then
n
Y (x − xj )
Ln,i (x) = , i = 0, 1, · · · , n. (3.23)
(xi − xj )
j=0, j 6=i

Example 3.12. Find the Lagrange form of interpolating polynomial for

the two-point table
x x0 x1
y y0 y1

Solution.
90 Chapter 3. Interpolation and Polynomial Approximation

Example 3.13. Determine the Lagrange interpolating polynomial that

passes through (2, 4) and (5, 1).
Solution.

Example 3.14. Let x0 = 2, x1 = 4, x2 = 5

(a) Use the points to find the second Lagrange interpolating polynomial
p2 for f (x) = 1/x .
(b) Use p2 to approximate f (3) = 1/3.

Solution.
3.1. Polynomial Interpolation 91

Maple-code
1 with(Student[NumericalAnalysis]);
2 f := x -> 1/x:
3 unassign('xy'):
4 xy := [[2, 1/2], [4, 1/4], [5, 1/5]]:
5

6 L2 := PolynomialInterpolation(xy, independentvar = x,
7 method = lagrange, function = f(x)):
8 Interpolant(L2);
9 1 1 1
10 -- (x - 4) (x - 5) - - (x - 2) (x - 5) + -- (x - 2) (x - 4)
11 12 8 15
12 RemainderTerm(L2);
13 / (x - 2) (x - 4) (x - 5)\
14 |- -----------------------| &where {2 <= xi_var and xi_var <= 5}
15 | 4 |
16 \ xi_var /
17 p2 := x -> expand(Interpolant(L2));
18 1 2 11 19
19 -- x - -- x + --
20 40 40 20
21 evalf(p2(3));
22 0.3500000000

3.1.3. Polynomial interpolation error

Theorem 3.15. (Polynomial Interpolation Error Theorem). Let

f ∈ C n+1 [a, b], and let Pn be the polynomial of degree ≤ n that interpo-
lates f at n + 1 distinct points x0 , x1 , · · · , xn in the interval [a, b]. Then,
for each x ∈ (a, b), there exists a number ξx between x0 , x1 , · · · , xn , hence
in the interval [a, b], such that
n
f (n+1) (ξx ) Y
f (x) − Pn (x) = (x − xi ) =: Rn (x). (3.24)
(n + 1)!
i=0
92 Chapter 3. Interpolation and Polynomial Approximation

Recall: Theorem 1.20. (Taylor’s Theorem with Lagrange Re-

mainder), page 8. Suppose f ∈ C n [a, b], f (n+1) exists on (a, b), and
x0 ∈ [a, b]. Then, for every x ∈ [a, b],
n
X f (k ) (x0 )
f (x) = (x − x0 )k + Rn (x), (3.25)
k!
k =0

where, for some ξ between x and x0 ,

f (n+1) (ξ )
Rn (x) = (x − x0 )n+1 .
(n + 1)!

Example 3.16. For Example 3.14, determine the error bound in [2, 5].
Solution.
Maple-code
1 p2 := x -> interp([2, 4, 5], [1/2, 1/4, 1/5], x):
2 p2(x):
3 f := x -> 1/x:
4 fd := x -> diff(f(x), x, x, x):
5 fd(xi)
6 6
7 - ---
8 4
9 xi
10 fdmax := maximize(abs(fd(x)), x = 2..5)
11 3
12 -
13 8
14 r := x -> (x - 2)*(x - 4)*(x - 5):
15 rmax := maximize(abs(r(x)), x = 2..5);
16 /5 1 (1/2)\ /4 1 (1/2)\ /1 1 (1/2)\
17 |- - - 7 | |- + - 7 | |- + - 7 |
18 \3 3 / \3 3 / \3 3 /
19 #Thus, "|f(x)-p2(x)|<=(max)|R[2](x)|="
20 evalf(fdmax*rmax/3!)
21 0.1320382370
3.1. Polynomial Interpolation 93

Example 3.17. If the function f (x) = sin(x) is approximated by a poly-

nomial of degree 5 that interpolates f at six equally distributed points in
[−1, 1] including end points, how large is the error on this interval?
Solution. The nodes xi are −1, −0.6, −0.2, 0.2, 0.6, and 1. It is easy to see
that
|f (6) (ξ )| = | − sin(ξ )| ≤ sin(1).

g := x -> (x+1)*(x+0.6)*(x+0.2)*(x-0.2)*(x-0.6)*(x-1):
gmax := maximize(abs(g(x)), x = -1..1)
0.06922606316
Thus,
5
f (6) (ξ ) Y sin(1)
| sin(x) − P5 (x)| = (x − xi ) ≤ gmax
6! 6! (3.26)
i=0
= 0.00008090517158

Theorem 3.18. (Polynomial Interpolation Error Theorem for

Equally Spaced Nodes): Let f ∈ C n+1 [a, b], and let Pn be the polyno-
mial of degree ≤ n that interpolates f at
b −a
xi = a + ih, h= , i = 0, 1, · · · , n.
n
Then, for each x ∈ (a, b),

h n+1
|f (x) − Pn (x)| ≤ M, (3.27)
4(n + 1)

where
M = max |f (n+1) (ξ )|.
ξ∈[a,b]

Proof. Recall the interpolation error Rn (x) given in (3.24). We consider

bounding
Yn
max |x − xi |.
x ∈[a,b]
j=1

Start by picking an x . We can assume that x is not one of the nodes, be-
94 Chapter 3. Interpolation and Polynomial Approximation

cause otherwise the product in question is zero. Let x ∈ (xj , xj+1 ), for some
j . Then we have
h2
|x − xj | · |x − xj+1 | ≤ . (3.28)
4
Now note that
(
(j + 1 − i)h for i < j
|x − xi | ≤ (3.29)
(i − j)h for j + 1 < i.

Thus
n
Y h2
|x − xi | ≤ [(j + 1)! h j ] [(n − j)! h n−j −1 ]. (3.30)
4
j=1

Since (j + 1)!(n − j)! ≤ n!, we can reach the following bound

n
Y 1
|x − xi | ≤ h n+1 n!. (3.31)
4
j=1

The result of the theorem follows from the above bound.

Example 3.19. How many equally spaced nodes are required to interpo-
late f (x) = cos x + sin x to within 10−8 on the interval [−1, 1]?
h n+1
Solution. Recall the formula: |f (x) − Pn (x)| ≤ M . Then, for n, solve
4(n + 1)

(2/n)n+1 √
2 ≤ 10−8 .
4(n + 1)

Answer: n = 10
3.1. Polynomial Interpolation 95

3.1.4. Chebyshev polynomials

In the Polynomial Interpolation Error Theorem, there is a term that can
be optimized by choosing the nodes in a special way. An analysis of
this problem was first given by a great mathematician Chebyshev (1821-
1894). The optimization process leads naturally to a system of polynomials
called Chebyshev polynomials.
Definition 3.20. The Chebyshev polynomials (of the first kind) are
defined recursively as follows:
(
T0 (x) = 1, T1 (x) = x
(3.32)
Tn+1 (x) = 2x Tn (x) − Tn−1 (x), n ≥ 1.

The explicit forms of the next few Tn are readily calculated:

Chebyshev-polynomials
1 T2 := x -> simplify(ChebyshevT(2, x)): T2(x)
2 2
3 2 x - 1
4

5 T3 := x -> simplify(ChebyshevT(3, x)): T3(x)

6 3
7 4 x - 3 x
8

9 T4 := x -> simplify(ChebyshevT(4, x)): T4(x)

10 4 2
11 8 x - 8 x + 1
12

13 T5 := x -> simplify(ChebyshevT(5, x)): T5(x)

14 5 3
15 16 x - 20 x + 5 x
16

17 T6 := x -> simplify(ChebyshevT(6, x)): T6(x)

18 6 4 2
19 32 x - 48 x + 18 x - 1
96 Chapter 3. Interpolation and Polynomial Approximation

Figure 3.3: Chebyshev polynomials.

Theorem 3.21. (Properties of Chebyshev polynomials):

(a) For x ∈ [−1, 1], the Chebyshev polynomials have this closed-form
expression:
Tn (x) = cos n cos−1 (x) , n ≥ 0.


(3.33)

(b) It has been verified that if the nodes x0 , x1 , · · · , xn ∈ [−1, 1], then
n
Y
max (x − xi ) ≥ 2−n , n ≥ 0, (3.34)
|x |≤1
i=0

and its minimum value will be attained if

n
Y
(x − xi ) = 2−n Tn+1 (x). (3.35)
i=0

(c) The nodes then must be the roots of Tn+1 , which are
 (2i + 1)π 
xi = cos , i = 0, 1, · · · , n. (3.36)
2n + 2
3.1. Polynomial Interpolation 97

Theorem 3.22. (Interpolation Error Theorem, Chebyshev

nodes): If the nodes are the roots of the Chebyshev polynomial Tn+1 ,
as in (3.36), then the error bound for the nth-degree interpolating poly-
nomial Pn reads
1
|f (x) − Pn (x)| ≤ n
max |f (n+1) (t)|. (3.37)
2 (n + 1)! |t |≤1

Example 3.23. (A variant of Example 3.17): If the function f (x) = sin(x)

is approximated by a polynomial of degree 5 that interpolates f at at roots
of the Chebyshev polynomial T6 in [−1, 1], how large is the error on this
interval?
Solution. From Example 3.17, we know that

|f (6) (ξ )| = | − sin(ξ )| ≤ sin(1).

Thus
sin(1)
|f (x) − P5 (x)| ≤ = 0.00003652217816. (3.38)
2n (n + 1)!
It is an optimal upper bound of the error and smaller than the one in
Equation (3.26), 0.00008090517158.
Accuracy comparison between uniform nodes and Chebyshev nodes:
Maple-code
1 with(Student[NumericalAnalysis]):
2 n := 5:
3 f := x -> sin(2*x*Pi):
4 xd := Array(0..n):
5

6 for i from 0 to n do
7 xd[i] := evalf[15](-1 + (2*i)/n);
8 end do:
9 xyU := [[xd[0],f(xd[0])], [xd[1],f(xd[1])], [xd[2],f(xd[2])],
10 [xd[3],f(xd[3])], [xd[4],f(xd[4])], [xd[5],f(xd[5])]]:
11 U := PolynomialInterpolation(xyU, independentvar = x,
12 method = lagrange, function = f(x)):
13 pU := x -> Interpolant(U):
98 Chapter 3. Interpolation and Polynomial Approximation

14

15 for i from 0 to n do
16 xd[i] := evalf[15](cos((2*i + 1)*Pi/(2*n + 2)));
17 end do:
18 xyC := [[xd[0],f(xd[0])], [xd[1],f(xd[1])], [xd[2],f(xd[2])],
19 [xd[3],f(xd[3])], [xd[4],f(xd[4])], [xd[5],f(xd[5])]]:
20 C := PolynomialInterpolation(xyC, independentvar = x,
21 method = lagrange, function = f(x)):
22 pC := x -> Interpolant(C):
23

24 plot([pU(x), pC(x)], x = -1..1, thickness = [2,2],

25 linestyle = [solid, dash], color = [red, blue],
26 legend = ["Uniform nodes", "Chebyshev nodes"],
27 legendstyle = [font = ["HELVETICA", 13], location = bottom])

Figure 3.4: Accuracy comparison between uniform nodes and Chebyshev nodes.
3.2. Divided Differences 99

3.2. Divided Differences

It turns out that the coefficients ck for the interpolating polynomials in
Newton’s form can be calculated relatively easily by using divided differ-
ences.
Remark 3.24. For {(xk , yk )}, 0 ≤ k ≤ n, the k th-degree Newton inter-
polating polynomials are of the form

pk (x) = c0 + c1 (x − x0 ) + · · · + ck (x − x0 )(x − x1 ) · · · (x − xk −1 ), (3.39)

for which pk (xk ) = yk . The first few cases are

p0 (x) = c0 = y0 ,
p1 (x) = c0 + c1 (x − x0 ), (3.40)
p2 (x) = c0 + c1 (x − x0 ) + c2 (x − x0 )(x − x1 ).

(a) The coefficient c1 is determined to satisfy

y1 = p1 (x1 ) = c0 + c1 (x1 − x0 ). (3.41)

Note c0 = y0 . Thus, we have

y1 − y0 = c1 (x1 − x0 ) (3.42)

and therefore
y1 − y0
c1 = . (3.43)
x1 − x0
(b) Now, since

y2 = p2 (x2 ) = c0 + c1 (x2 − x0 ) + c2 (x2 − x0 )(x2 − x1 ),

it follows from the above, (3.42), and (3.43) that

y2 − y0 − c1 (x2 − x0 ) (y2 − y1 ) + (y1 − y0 ) − c1 (x2 − x0 )
c2 = =
(x2 − x0 )(x2 − x1 ) (x2 − x0 )(x2 − x1 )
(y2 − y1 ) + c1 (x1 − x0 ) − c1 (x2 − x0 ) (y2 − y1 )/(x2 − x1 ) − c1
= = .
(x2 − x0 )(x2 − x1 ) x2 − x0
(3.44)
100 Chapter 3. Interpolation and Polynomial Approximation

Definition 3.25. (Divided differences):

• The zeroth divided difference of the function f with respect to xi ,

denoted f [xi ], is the value of at xi :

f [xi ] = f (xi ) (3.45)

• The remaining divided differences are defined recursively. The first

divided difference of f with respect to xi , xi+1 is defined as
f [xi+1 ] − f [xi ]
f [xi , xi+1 ] = . (3.46)
xi+1 − xi

• The second divided difference relative to xi , xi+1 , xi+2 is defined as

f [xi+1 , xi+2 ] − f [xi , xi+1 ]
f [xi , xi+1 , xi+2 ] = . (3.47)
xi+2 − xi

• In general, the k th divided difference relative to xi , xi+1 , · · · , xi+k is

defined as
f [xi+1 , · · · , xi+k ] − f [xi , · · · , xi+k −1 ]
f [xi , xi+1 , · · · , xi+k ] = . (3.48)
xi+k − xi

Note: It follows from Remark 3.24 that the coefficients of the Newton
interpolating polynomials read

c0 = f [x0 ], c1 = f [x0 , x1 ], c2 = f [x0 , x1 , x2 ]. (3.49)

In general,
ck = f [x0 , x1 , · · · , xk ]. (3.50)
3.2. Divided Differences 101

Newton’s Divided Difference Table

x f[x] DD1 (f[, ]) DD2 (f[, , ]) DD3 (f[, , , ])
x0 f [x0 ]
x1 f [x1 ] f [x0 , x1 ]
f [x1 ]−f [x0 ]
= x1 −x0
(3.51)
x2 f [x2 ] f [x1 , x2 ] f [x0 , x1 , x2 ]
f [x2 ]−f [x1 ] f [x1 ,x2 ]−f [x0 ,x1 ]
= x2 −x1
= x2 −x0
x3 f [x3 ] f [x2 , x3 ] f [x1 , x2 , x3 ] f [x0 , x1 , x2 , x3 ]
f [x3 ]−f [x2 ] f [x2 ,x3 ]−f [x1 ,x2 ] f [x1 ,x2 ,x3 ]−f [x0 ,x1 ,x2 ]
= x3 −x2
= x3 −x1
= x3 −x0

Pseudocode 3.26. (Newton’s Divided Difference Formula):

Input: (xi , yi ), i = 0, 1, · · · , n, saved as Fi,0 = yi
Output: Fi,i , i = 0, 1, · · · , n
Step 1: For i = 1, 2, · · · , n
For j = 1, 2, · · · , i
Fi,j −1 − Fi −1,j −1
Fi,j =
xi − xi −j
Step 2: Return (F0,0 , F1,1 , · · · , Fn,n )

Example 3.27. Determine the Newton interpolating polynomial for the

data:
x 0 1 2 5
y 1 −1 3 −189

Solution.

Answer: f (x) = 1 − 2x + 3x(x − 1) − 4x(x − 1)(x − 2)

102 Chapter 3. Interpolation and Polynomial Approximation

Theorem 3.28. (Properties of Divided Differences):

• If f is a polynomial of degree k , then
f [x0 , x1 , · · · , xn ] = 0, for all n > k . (3.52)

• Permutations in Divided Differences: The divided difference is

a symmetric function of its arguments. That is, if z0 , z1 , · · · , zn is a
permutation of x0 , x1 , · · · , xn , then
f [z0 , z1 , · · · , zn ] = f [x0 , x1 , · · · , xn ]. (3.53)

• Error in Newton Interpolation: Let P be the polynomial of degree

≤ n that interpolates f at n + 1 distinct nodes, x0 , x1 , · · · , xn . If t is a
point different from the nodes, then
n
Y
f (t) − P(t) = f [x0 , x1 , · · · , xn , t] (t − xi ). (3.54)
i=0

Proof: Let Q be the polynomial of degree at most (n + 1) that interpolates f at

nodes, x0 , x1 , · · · , xn , t . Then, we know that Q is obtained from P by adding one
more term. Indeed,
n
Y
Q(x) = P(x) + f [x0 , x1 , · · · , xn , t] (x − xi ). (3.55)
i=0

Since f (t) = Q(t), the result follows.

• Derivatives and Divided Differences: If f ∈ C n [a, b] and if
x0 , x1 , · · · , xn are distinct points in [a, b], then there exists a point
ξ ∈ (a, b) such that
1 (n)
f [x0 , x1 , · · · , xn ] = f (ξ ). (3.56)
n!
Proof: Let pn−1 be the polynomial of degree at most n − 1 that interpolates f at
x0 , x1 , · · · , xn−1 . By the Polynomial Interpolation Error Theorem, there exists
a point ξ ∈ (a, b) such that
n−1
1 Y
f (xn ) − pn−1 (xn ) = f (n) (ξ ) (xn − xi ). (3.57)
n!
i=1

On the other hand, by the previous theorem, we have

n −1
Y
f (xn ) − pn−1 (xn ) = f [x0 , x1 , · · · , xn ] (xn − xi ). (3.58)
i=1

The theorem follows from the comparison of above two equations.

3.2. Divided Differences 103

Self-study 3.29. Prove that for h > 0,

f (x) − 2f (x + h) + f (x + 2h) = h 2 f 00 (ξ ), (3.59)

for some ξ ∈ (x, x + 2h).

Hint : Use the last theorem; employ the divided difference formula to find
f [x, x + h, x + 2h].
Solution.
104 Chapter 3. Interpolation and Polynomial Approximation

3.3. Data Approximation and Neville’s Method

Remark 3.30.
• We have studied how to construct interpolating polynomials. A fre-
quent use of these polynomials involves the interpolation of tabulated
data.
• However, in in many applications, an explicit representation of the
polynomial is not needed, but only the values of the polynomial at
specified points.
• In this situation, the function underlying the data might be unknown
so the explicit form of the error cannot be used to assure the accuracy
of the interpolation.
• Neville’s Method provides an adaptive mechanism for the evalua-
tion of accurate interpolating values.

Definition 3.31. (Interpolating polynomial at xm1 , xm2 , · · · , xmk ):

Let f be defined at x0 , x1 , · · · , xn , and suppose that m1 , m2 , · · · , mk are
k distinct integers with 0 ≤ mi ≤ n for each i . The polynomial that
agrees with at the points xm1 , xm2 , · · · , xmk is denoted by Pm1 , m2 , · · · , mk .

Example 3.32. Suppose that x0 = 1, x1 = 2, x2 = 3, x3 = 4, x4 = 6 and

f (x) = e x . Determine the interpolating polynomial P1,2,4 (x) and use this
polynomial to approximate f (5).
Solution. It can be the Lagrange polynomial that agrees with f (x) at
x1 = 2, x2 = 3, x4 = 6:
(x − 3)(x − 6) 2 (x − 2)(x − 6) 3 (x − 2)(x − 3) 6
P1,2,4 (x) = e + e + e .
(2 − 3)(2 − 6) (3 − 2)(3 − 6) (6 − 2)(6 − 3)
Thus
1 2 1
e + e 3 + e 6 ≈ 218.1054057.
P1,2,4 (5) =
2 2
On the other hand, f (5) = e 5 ≈ 148.4131591.
3.3. Data Approximation and Neville’s Method 105

Theorem 3.33. Let f be defined at (n + 1) distinct points, x0 , x1 , · · · , xn .

Then for each 0 ≤ i < j ≤ n,
(x − xi )Pi+1,i+2,··· ,j (x) − (x − xj )Pi,i+1,··· ,j −1 (x)
Pi,i+1,··· ,j (x) = , (3.60)
xj − xi

which is the polynomial interpolating f at xi , xi+1 , · · · , xj .

Note: The above theorem implies that the interpolating polynomial can
be generated recursively. For example,
(x − x0 )P1 (x) − (x − x1 )P0 (x)
P0,1 (x) =
x1 − x0
(x − x1 )P2 (x) − (x − x2 )P1 (x)
P1,2 (x) = (3.61)
x2 − x1
(x − x0 )P1,2 (x) − (x − x2 )P0,1 (x)
P0,1,2 (x) =
x2 − x0
and so on. They are generated in the manner shown in the following ta-
ble, where each row is completed before the succeeding rows are begun.
x0 y0 = P0
x1 y1 = P1 P0,1
(3.62)
x2 y2 = P2 P1,2 P0,1,2
x3 y3 = P3 P2,3 P1,2,3 P0,1,2,3

For simplicity in computation, we may try to avoid multiple subscripts by

defining the new variable

Qi,j = Pi −j,i −j+1,··· ,i

Then the above table can be expressed as

x0 P0 = Q0,0
x1 P1 = Q1,0 P0,1 = Q1,1
(3.63)
x2 P2 = Q2,0 P1,2 = Q2,1 P0,1,2 = Q2,2
x3 P3 = Q3,0 P2,3 = Q3,1 P1,2,3 = Q3,2 P0,1,2,3 = Q3,3
106 Chapter 3. Interpolation and Polynomial Approximation

Example 3.34. Let x0 = 2.0, x1 = 2.2, x2 = 2.3, x3 = 1.9, x4 = 2.15. Use

Neville’s method to approximate f (2.1) = ln(2.1) in a four-digit accuracy.
Solution.
Maple-code
1 with(Student[NumericalAnalysis]):
2 x0 := 2.0:
3 x1 := 2.2:
4 x2 := 2.3:
5 x3 := 1.9:
6 x4 := 2.15:
7 xy := [[x0,ln(x0)], [x1,ln(x1)], [x2,ln(x2)], [x3,ln(x3)], [x4,ln(x4)]]:
8 P := PolynomialInterpolation(xy, method = neville):
9

10 Q := NevilleTable(P, 2.1)
11 [[0.6931471806, 0, 0, 0, 0 ],
12 [0.7884573604, 0.7408022680, 0, 0, 0 ],
13 [0.8329091229, 0.7440056025, 0.7418700461, 0, 0 ],
14 [0.6418538862, 0.7373815030, 0.7417975693, 0.7419425227, 0 ],
15 [0.7654678421, 0.7407450500, 0.7418662324, 0.7419348958, 0.7419374382]]

Note that
|Q3,3 − Q2,2 | = |0.7419425227 − 0.7418700461| = 0.0000724766
|Q4,4 − Q3,3 | = |0.7419374382 − 0.7419425227| = 0.0000050845
Thus Q3,3 = 0.7419425227 is already in a four-digit accuracy.
Check: The real value is ln(2.1) = 0.7419373447. The absolute error:
| ln(2.1) − Q3,3 | == 0.0000051780.

Pseudocode 3.35.
(
the nodes x0 , x1 , · · · , xn ; the evaluation point x; the tolerance ε;
Input:
and values y0 , y1 , · · · , yn in the 1st column of Q ∈ R(n+1)×(n+1)
Output: Q
Step 1: For i = 1, 2, · · · , n
For j = 1, 2, · · · , i
(x − xi −j )Qi,j −1 − (x − xi )Qi −1,j −1
Qi,j =
xi − xi −j
if (|Qi,i − Qi −1,i −1 | < ε) {i0 = i; break; }
Step 2: Return (Q, i0 )
3.3. Data Approximation and Neville’s Method 107

Example 3.36. Neville’s method is used to approximate f (0.3), giving the

following table.

x0 =0 Q0,0 = 1
x1 = 0.25 Q1,0 = 2 Q1,1 = 2.2
(3.64)
x2 = 0.5 Q2,0 Q2,1 Q2,2
x3 = 0.75 Q3,0 = 5 Q3,1 Q3,2 = 2.12 Q3,3 = 2.168

Determine Q2,0 = f (x2 ).

Solution.

Answer: Q2,2 = 2.2; Q2,1 = 2.2; Q2,0 = 3

108 Chapter 3. Interpolation and Polynomial Approximation

3.4. Hermite Interpolation

The Hermite interpolation refers to the interpolation of a function and
some of its derivatives at a set of nodes. When a distinction is being made
between this type of interpolation and its simpler type (in which no deriva-
tives are interpolated), the latter is often called Lagrange interpolation.

Key Idea 3.37. (Basic Concepts of Hermite Interpolation):

• For example, we require a polynomial of least degree that interpo-
lates a function f and its derivative f 0 at two distinct points, say x0
and x1 .
• Then the polynomial p sought will satisfy these four conditions:

p(xi ) = f (xi ), p 0 (xi ) = f 0 (xi ); i = 0, 1. (3.65)

• Since there are four conditions, it seems reasonable to look for a so-
lution in P3 , the space of all polynomials of degree at most 3. Rather
than writing p(x) in terms of 1, x, x 2 , x 3 , let us write it as

p(x) = a + b(x − x0 ) + c(x − x0 )2 + d(x − x0 )2 (x − x1 ), (3.66)

because this will simplify the work. This leads to

p 0 (x) = b + 2c(x − x0 ) + 2d(x − x0 )(x − x1 ) + d(x − x0 )2 . (3.67)

• The four conditions on p , in (3.65), can now be written in the form

f (x0 ) = a
f 0 (x0 ) = b
(3.68)
f (x1 ) = a + bh + ch 2 (h = x1 − x0 )
f 0 (x1 ) = b + 2ch + dh 2

Thus, the coefficients a, b, c, d can be obtained easily.

3.4. Hermite Interpolation 109

Theorem 3.38. (Hermite Interpolation Theorem): If f ∈ C 1 [a, b]

and x0 , x1 , · · · , xn ∈ [a, b] are distinct, then the unique polynomial of
least degree agreeing with f and f 0 at the (n + 1) points is the Hermite
polynomial of degree at most (2n + 1) given by
n
X n
X
H2n+1 (x) = f (xi )Hn,i (x) + f (xi )H
b n,i (x), (3.69)
i=0 i=0

where 0 2
Hn,i (x) = [1 − 2(x − xi )Ln,i (xi )]Ln,i (x),
2
b n,i (x) = (x − xi )Ln,i (x).
H
Here Ln,i (x) is the i th Lagrange polynomial of degree n. Moreover, if
f ∈ C 2n+2 [a, b], then
n
f (2n+2) (ξ ) Y
f (x) − H2n+1 (x) = (x − xi )2 . (3.70)
(2n + 2)!
i=0

Construction of Hermite Polynomials using Divided Differences

Recall: The polynomial Pn that interpolates f at x0 , x1 , · · · , xn is given
n
X
Pn (x) = f [x0 ] + f [x0 , x1 , · · · , xk ](x − x0 ) · · · (x − xk −1 ). (3.71)
k =1

Strategy 3.39. (Construction of Hermite Polynomials):

• Define a new sequence by z0 , z1 , · · · , z2n+1 by
z2i = z2i+1 = xi , i = 0, 1, · · · , n. (3.72)

• Then the Newton form of the Hermite polynomial is given by

2n+1
X
H2n+1 (x) = f [z0 ] + f [z0 , z1 , · · · , zk ](x − z0 ) · · · (x − zk −1 ), (3.73)
k =1

with
f [xi ] − f [xi ]
f [z2i , z2i+1 ] = f [xi , xi ] = replaced by f 0 (xi ). (3.74)
xi − xi
110 Chapter 3. Interpolation and Polynomial Approximation

Note: For each i = 0, 1, · · · , n,

f (x) − f (xi )
lim = f 0 (xi ). (3.75)
x →xi x − xi

The extended Newton divided difference table: Consider the Her-

mite polynomial that interpolates f and f 0 at three points, x0 , x1 , x2 .

z f(z) DD1 Higher DDs

z0 = x0 f [z0 ] = f (x0 )
z1 = x0 f [z1 ] = f (x0 ) f [z0 , z1 ] = f0 (x0 )
f [z2 ] − f [z1 ]
z2 = x1 f [z2 ] = f (x1 ) f [z1 , z2 ] = (3.76)
z2 − z1
0
z3 = x1 f [z3 ] = f (x1 ) f [z2 , z3 ] = f (x1 ) as usual
f [z4 ] − f [z3 ]
z4 = x2 f [z4 ] = f (x2 ) f [z3 , z4 ] =
z4 − z3
0
z5 = x2 f [z5 ] = f (x2 ) f [z4 , z5 ] = f (x2 )

Example 3.40. Use the extended Newton divided difference method to

obtain a cubic polynomial that takes these values:

x f (x) f 0 (x)
0 2 −9
1 −4 4
3.4. Hermite Interpolation 111

Example 3.41. (Continuation): Find a quartic polynomial p4 that takes

values given in the preceding example and, in addition, satisfies p4 (2) = 44.
112 Chapter 3. Interpolation and Polynomial Approximation

3.5. Spline Interpolation

3.5.1. Runge’s phenomenon

Recall: (Weierstrass approximation theorem): Suppose f ∈ C[a, b].

Then, for each ε > 0, there exists a polynomial P(x) such that

|f (x) − P(x)| < ε, for all x ∈ [a, b]. (3.77)

Interpolation at equidistant points is a natural and common approach

to construct approximating polynomials. Runge’s phenomenon demon-
strates, however, that interpolation can easily result in divergent approxi-
mations.
Example 3.42. (Runge’s phenomenon): Consider the function
1
f (x) = , x ∈ [−1, 1]. (3.78)
1 + 25x 2
Runge found that if this function was interpolated at equidistant points
2
xi = −1 + i , i = 0, 1, · · · , n,
n
the resulting interpolation pn oscillated toward the end of the interval,
i.e. close to -1 and 1. It can even be proven that the interpolation error
tends toward infinity when the degree of the polynomial increases:
 
lim max |f (x) − pn (x)| = ∞. (3.79)
n→∞ −1≤x ≤1

Figure 3.5: Runge’s phenomenon.

3.5. Spline Interpolation 113

Mitigation to the problem

• Change of interpolation points: e.g., Chebyshev nodes
• Constrained minimization: e.g., Hermite-like higher-order poly-
nomial interpolation, whose first (or second) derivative has minimal
norm.
• Use of piecewise polynomials: e.g., Spline interpolation

Definition 3.43. A partition of the interval [a, b] is an ordered se-

quence {xi }ni=0 such that

a = x0 < x1 < x2 < · · · < xn = b.

The numbers xi are known as knots or nodes.

Definition 3.44. A function S is a spline of degree k on [a, b] if

1) The domain of S is [a, b].

2) There exits a partition {xi }ni=0 of [a, b] such that on each subinterval
[xi −1 , xi ], S ∈ Pk .
3) S, S 0 , · · · , S (k −1) are continuous on (a, b).

3.5.2. Linear splines

A linear spline is a continuous function which is linear on each subin-

terval. Thus it is defined entirely by its values at the nodes. That is,
given
x x0 x1 · · · xn
y y0 y1 · · · yn
the linear polynomial on each subinterval is defined as
y i − y i −1
Li (x) = yi −1 + (x − xi −1 ), x ∈ [xi −1 , xi ]. (3.80)
x i − x i −1
114 Chapter 3. Interpolation and Polynomial Approximation

Example 3.45. Find the linear spline for

x 0.0 0.2 0.5 0.8 1.0

y 1.3 3.0 2.0 2.1 2.5

Solution.

The linear spline can be easily com-

puted as


 1.3 + 8.5x, x < 0.2

11 10x


−


 , x < 0.5
L (x) = 3 3
13 x
+ , x < 0.8





 6 3

0.5 + 2.0, x otherwise

(3.81) Figure 3.6: Linear spline.

First-Degree Spline Accuracy

Theorem 3.46. To find the error bound, we will consider the error on
a single subinterval of the partition, and apply a little calculus. Let p(x)
be the linear polynomial interpolating f (x) at the endpoints of [xi −1 , xi ].
Then,
f 00 (ξ )
f (x) − p(x) = (x − xi −1 )(x − xi ), (3.82)
2!
for some ξ ∈ (xi −1 , xi ). Thus
M2
|f (x) − p(x)| ≤ max (xi − xi −1 )2 , x ∈ [a, b], (3.83)
8 1≤i ≤n

where
M2 = max |f 00 (x)|.
x ∈(a,b)
3.5. Spline Interpolation 115

3.5.3. Quadratic (Second Degree) Splines

Remark 3.47. A quadratic spline is a piecewise quadratic function,

of which the derivative is continuous on (a, b).
• Typically, a quadratic spline Q is defined by its piecewise polynomi-
als: Let Qi = Q |[xi−1 ,xi ] . Then

Qi (x) = ai x 2 + bi x + ci , x ∈ [xi −1 , xi ], i = 1, 2, · · · , n. (3.84)

Thus there are 3n parameters to define Q(x).

• For each of the n subintervals, the data (xi , yi ), i = 1, 2, · · · , n, gives
two equations regarding Qi (x):

Qi (xi −1 ) = yi −1 and Qi (xi ) = yi , i = 1, 2, · · · , n. (3.85)

This is 2n equations. The continuity condition on Q 0 gives a single

equation for each of the (n − 1) internal nodes:

Qi0 (xi ) = Qi+1

0
(xi ), i = 1, 2, · · · , n − 1. (3.86)

This totals (3n − 1) equations, but 3n unknowns.

• Thus an additional user-chosen condition is required, e.g.,

Q 0 (a) = f 0 (a), Q 0 (a) = 0, or Q 00 (a) = 0. (3.87)

Alternatively, the additional condition can be given at x = b .

116 Chapter 3. Interpolation and Polynomial Approximation

Algorithm 3.48. (Construction of quadratic splines):

(0) Define
zi = Q 0 (xi ), i = 0, 1, · · · , n; (3.88)
suppose that the additional condition is given by specifying z0 .
(1) Because Qi0 = Q 0 |[xi−1 ,xi ] is a linear function satisfying

Qi0 (xi −1 ) = zi −1 and Qi0 (xi ) = zi , (continuity of Q 0 ) (3.89)

we have
zi − zi −1
Qi0 (x) = zi −1 + (x − xi −1 ), x ∈ [xi −1 , xi ]. (3.90)
xi − xi −1

(2) By integrating it and using Qi (xi −1 ) = yi −1 (left edge value)

z i − z i −1
Qi (x) = (x − xi −1 )2 + zi −1 (x − xi −1 ) + yi −1 . (3.91)
2(xi − xi −1 )

(3) In order to determine zi , 1 ≤ i ≤ n, we use the above at xi

(right edge value):
zi − zi −1
yi = Qi (xi ) = (xi − xi −1 )2 + zi −1 (xi − xi −1 ) + yi −1 , (3.92)
2(xi − xi −1 )
which implies
1
yi − yi −1 = (zi − zi −1 )(xi − xi −1 ) + zi −1 (xi − xi −1 )
2
(zi + zi −1 )
= (xi − xi −1 ) .
2
Thus we have
yi − yi −1
zi = 2 − zi −1 , i = 1, 2, · · · , n. (3.93)
xi − xi −1
3.5. Spline Interpolation 117

Note:
a. You should first decide zi using (3.93) and then finalize Qi from
(3.91).
b. When zn is specified, Equation (3.93) can be replaced by
y i − y i −1
zi −1 = 2 − zi , i = n, n − 1, · · · , 1. (3.94)
xi − xi −1

Example 3.49. Find the quadratic spline for the same dataset used in
Example 3.45, p. 114:

x 0.0 0.2 0.5 0.8 1.0

y 1.3 3.0 2.0 2.1 2.5

Solution. zi = Qi0 (xi ) are computed as

z[0]=0
z[1]=17
z[2]=-23.6667
z[3]=24.3333
z[4]=-20.3333

Figure 3.7: The graph of Q(x) is superposed over the graph of the linear spline L (x).
118 Chapter 3. Interpolation and Polynomial Approximation

3.5.4. Cubic splines

Recall: (Definition 3.44): A function S is a cubic spline on [a, b] if

1) The domain of S is [a, b].
2) S ∈ P3 on each subinterval [xi −1 , xi ].
3) S, S 0 , S 00 are continuous on (a, b).

Remark 3.50. By definition, a cubic spline is a continuous piecewise

cubic polynomial whose first and second derivatives are continuous.
• On each subinterval [xi −1 , xi ], 1 ≤ i ≤ n, we have to determine coeffi-
cients of a cubic polynomial of the form

Si (x) = ai x 3 + bi x 2 + ci x + di , i = 1, 2, · · · , n. (3.95)

Thus there are 4n unknowns to define S(x).

• On the other hand, equations we can get are

left an right values of Si : 2n

continuity of S 0 : n−1 (3.96)
continuity of S 00 : n−1

Thus there are (4n − 2) equations.

• Two degrees of freedom remain, and there have been various
ways of choosing them to advantage.
3.5. Spline Interpolation 119

Algorithm 3.51. (Construction of cubic splines):

(0) Similarly as for quadratic splines, we define

zi = S 00 (xi ), i = 0, 1, · · · , n. (3.97)

(1) Because Si00 = S 00 |[xi−1 ,xi ] is a linear function satisfying

Si00 (xi −1 ) = zi −1 and Si00 (xi ) = zi , (continuity of S 00 ) (3.98)

and therefore is given by the straight line between zi −1 and zi :

zi −1 (xi − x) zi (x − xi −1 )
Si00 (x) = + , hi = xi − xi −1 , x ∈ [xi −1 , xi ]. (3.99)
hi hi

(2) If (3.99) is integrated twice, the result reads

zi −1 (xi − x)3 zi (x − xi −1 )3
Si (x) = + + C(x − xi −1 ) + D(xi − x). (3.100)
6hi 6hi
In order to determine C and D , we use Si (xi −1 ) = yi −1 and Si (xi ) = yi
(left and right edge values):
zi −1 2 zi 2
Si (xi −1 ) = h + Dhi = yi −1 , Si (xi ) = h + Chi = yi . (3.101)
6 i 6 i
Thus (3.100) becomes

zi −1 (xi − x)3 zi (x − xi −1 )3
Si (x) = +
y 6h i 6hi (3.102)
i 1  y
i −1 1 
+ − zi hi (x − xi −1 ) + − zi −1 hi (xi − x).
hi 6 hi 6

(3) The values z1 , z2 , · · · , zn−1 can be determined from the continuity of

S 0:
0 zi −1 (xi − x)2 zi (x − xi −1 )2
Si (x) = − +
2hi 2hi (3.103)
y 1   y i −1 1 
i
+ − zi hi − − zi −1 hi .
hi 6 hi 6
120 Chapter 3. Interpolation and Polynomial Approximation

Construction of cubic splines (continue):

Then substitution of x = xi and simplification lead to
0 hi hi yi − yi −1
Si (xi ) = zi −1 + zi + . (3.104)
6 3 hi
0
Analogously, after obtaining Si+1 , we have
0 hi+1 hi+1 yi+1 − yi
Si+1 (xi ) = − zi −1 − zi+1 + . (3.105)
3 6 hi+1
When the right sides of (3.104) and (3.105) are set equal to each
other, the result reads
6(yi+1 − yi ) 6(yi − yi −1 )
hi zi −1 + 2(hi + hi+1 ) zi + hi+1 zi+1 = − , (3.106)
hi+1 hi

for i = 1, 2, · · · , n − 1.
(4) Two additional user-chosen conditions are required to deter-
mine (n + 1) unknowns, z0 , z1 , · · · , zn . There are two popular ap-
proaches for the choice of the two additional conditions.

Natural Cubic Spline : z0 = 0, zn = 0

Clamped Cubic Spline : S 0 (a) = f 0 (a), S 0 (b) = f 0 (b)

Natural Cubic Splines: Let z0 = zn = 0. Then the system of linear equa-

tions in (3.106) can be written as
   
z1 b2 − b1
 z   b −b 
 2   3 2 
A . = . , (3.107)
 ..   .. 
zn−1 b n − b n −1

where 
2(h1 + h2 ) h2
 h2 2(h2 + h3 ) h3 

... ... ...
 6
A =  and bi = (yi − yi −1 ).
 
  hi
 hn−2 2(hn−2 + hn−1 ) hn−1 
hn − 1 2(hn−1 + hn )
3.5. Spline Interpolation 121

Clamped Cubic Splines: Let f 0 (a) and f 0 (b) be prescribed. Then the two
extra conditions read

S 0 (a) = f 0 (a), S 0 (b) = f 0 (b). (3.108)

Since a = x0 and b = xn , utilizing Equation (3.103), the conditions read

6
2h1 z0 + h1 z1 = (y1 − y0 ) − 6f 0 (x0 )
h1 (3.109)
6
hn zn−1 + 2hn zn = 6f 0 (xn ) − (yn − yn−1 )
hn

Equation (3.106) and the above two equations clearly make (n + 1) condi-
tions for (n + 1) unknowns, z0 , z1 , · · · , zn . It is a good exercise to compose an
algebraic system for the computation of clamped cubic splines.

Example 3.52. Find the natural cubic spline for the same dataset

x 0.0 0.2 0.5 0.8 1.0

y 1.3 3.0 2.0 2.1 2.5

Solution.

Figure 3.8: The graph of S(x) is superposed over the graphs of the linear spline L (x) and
the quadratic spline Q(x).
122 Chapter 3. Interpolation and Polynomial Approximation

Example 3.53. Find the natural cubic spline that interpolates the data

x 0 1 3
y 4 2 7

Solution.
Maple-code
1 with(CurveFitting):
2 xy := [[0, 4], [1, 2], [3, 7]]:
3 n := 2:
4 L := x -> Spline(xy, x, degree = 1, endpoints = 'natural'):
5 Q := x -> Spline(xy, x, degree = 2, endpoints = 'natural'):
6 S := x -> Spline(xy, x, degree = 3, endpoints = 'natural'):
7 S(x)
8 / 11 3 3 41 49 27 2 3 3\
9 piecewise|x < 1, 4 - -- x + - x , otherwise, -- - -- x + -- x - - x |
10 \ 4 4 8 8 8 8 /

Figure 3.9
3.5. Spline Interpolation 123

Optimality Theorem for Natural Cubic Splines:

We now present a theorem to the effect that the natural cubic spline pro-
duces the smoothest interpolating function. The word smooth is given a
technical meaning in the theorem.

Theorem 3.54. Let f 00 be continuous in [a, b] and a = x0 < x1 < · · · <

xn = b . If S is the natural cubic spline interpolating f at the nodes xi
for 0 ≤ i ≤ n, then
ˆ b ˆ b
00 2
[S (x)] dx ≤ [f 00 (x)]2 dx. (3.110)
a a
124 Chapter 3. Interpolation and Polynomial Approximation

3.6. Parametric Curves

Consider the data of the form:
xy := [[-1, 0], [0, 1], [1, 0.5], [0, 0], [1, -1]]
of which the point plot is given

• None of the interpolation methods we have learnt so far can be used

to generate an interpolating curve for this data, because the curve
cannot be expressed as a function of one coordinate variable to the
other.
• In this section we will see how to represent general curves by using a
parameter to express both the x - and y -coordinate variables.

Example 3.55. Construct a pair of interpolating polynomials, as a func-

tion of t , for the data:
i 0 1 2 3 4
t 0 0.25 0.5 0.75 1
x −1 0 1 0 1
y 0 1 0.5 0 −1
Solution.
Maple-code
1 with(CurveFitting):
2 unassign('t'):
3 tx := [[0, -1], [0.25, 0], [0.5, 1], [0.75, 0], [1, 1]]:
4 ty := [[0, 0], [0.25, 1], [0.5, 0.5], [0.75, 0], [1, -1]]:
5 x := t -> PolynomialInterpolation(tx, t, form = Lagrange):
6 y := t -> PolynomialInterpolation(ty, t, form = Lagrange):
7 plot([x(t), y(t), t = 0..1], color = blue, thickness = 2)
3.6. Parametric Curves 125

Figure 3.10

Remark 3.56. (Applications in Computer Graphics):

• Required: Rapid generation of smooth curves that can be quickly
and easily modified.
• Preferred: Change of one portion of a curve should have little or no
effect on other portions of the curve.
⇒ The choice of curve is a form of the piecewise cubic Hermite poly-
nomial.

Example 3.57. For data {(xi , f (xi ), f 0 (xi )}, i = 0, 1, · · · , n, the piecewise cu-
bic Hermite polynomial can be generated independently in each portion
[xi −1 , xi ]. Why?
Solution.
126 Chapter 3. Interpolation and Polynomial Approximation

Piecewise cubic Hermite polynomial for General Curve Fitting

Algorithm 3.58. Let us focus on the first portion of the piecewise

cubic Hermite polynomial interpolating between
(x0 , y0 ) and (x1 , y1 ).

• For the first portion, the given data are

dy
x(0) = x0 , y(0) = y0 , (t = 0);
dx (3.111)
dy
x(1) = x1 , y(1) = y1 , (t = 1);
dx
Only six conditions are specified, while the cubic polynomials x(t)
and y(t) each have four parameters, for a total of eight.
• This provides flexibility in choosing the pair of cubic polynomials
to specify the conditions.
• Notice that the natural form for determining x(t) and y(t) requires
to specify
x(0), x(1), x 0 (0), x 0 (1);
0 0
(3.112)
y(0), y(1), y (0), y (1);

• On the other hand, the slopes at the endpoints can be expressed

using the so-called guidepoints which are to be chosen from the
desired tangent line:

(x0 + α0 , y0 + β0 ) : guidepoint for (x0 , y0 )

(3.113)
(x1 − α1 , y1 − β1 ) : guidepoint for (x1 , y1 )

Thus
dy (y0 + β0 ) − y0 β0 y 0 (0)
(t = 0) = = = 0 ,
dx (x0 + α0 ) − x0 α0 x (0)
(3.114)
dy y1 − (y1 − β1 ) β1 y 0 (1)
(t = 1) = = = 0 .
dx x1 − (x1 − α1 ) α1 x (1)

• Therefore, we may specify

x 0 (0) = α0 , y 0 (0) = β0 ; x 0 (1) = α1 , y 0 (1) = β1 . (3.115)

3.6. Parametric Curves 127

Formula. (The cubic Hermite polynomial (x(t), y(t)) on [0, 1]):

• The unique cubic Hermite polynomial x(t) satisfying
x(0) = x0 , x 0 (0) = α0 ; x(1) = x1 , x 0 (1) = α1

can be constructed as
x(t) = [2(x0 − x1 ) + (α0 + α1 )] t 3 + [3(x1 − x0 ) − (2α0 + α1 )] t 2
(3.116)
+α0 t + x0 .

• Similarly, the unique cubic Hermite polynomial y(t) satisfying

y(0) = y0 , y 0 (0) = β0 ; y(1) = y1 , y 0 (1) = β1

can be constructed as
y(t) = [2(y0 − y1 ) + (β0 + β1 )] t 3 + [3(y1 − y0 ) − (2β0 + β1 )] t 2
(3.117)
+β0 t + y0 .

Example 3.59. Determine the parametric curve when

dy dy
(x0 , y0 ) = (0, 0), (t = 0) = 1; (x1 , y1 ) = (1, 0), (t = 1) = −1.
dx dx

Solution.

• Let α0 = 1, β0 = 1 and α1 = 1, β1 = −1.

– The cubic Hermite polynomial x(t) satisfying

x0 := 0: a0 := 1: x1 := 1: a1 := 1:
is
x:=t->(2*(x0-x1)+a0+a1)*t^3+(3*(x1-x0)-a1-2*a0)*t^2+a0*t+x0
⇒ x(t) = t
– The cubic Hermite polynomial y(t) satisfying
y0 := 0: b0 := 1: y1 := 0: b1 := -1:
is
y:=t->(2*(y0-y1)+b0+b1)*t^3+(3*(y1-y0)-b1-2*b0)*t^2+b0*t+y0
⇒ y(t) = -t^2 + t
128 Chapter 3. Interpolation and Polynomial Approximation

– H1 := plot([x(t),y(t),t=0..1], coordinateview=[0..1, 0..1],

thickness=2, linestye=solid)

• Let α0 = 0.5, β0 = 0.5 and α1 = 0.5, β1 = −0.5.

– a0 := 0.5: b0 := 0.5: a1 := 0.5: b1 := -0.5:

x:=t->(2*(x0-x1)+a0+a1)*t^3+(3*(x1-x0)-a1-2*a0)*t^2+a0*t+x0
⇒ x(t) = -1.0*t^3 + 1.5*t^2 + 0.5*t
y:=t->(2*(y0-y1)+b0+b1)*t^3+(3*(y1-y0)-b1-2*b0)*t^2+b0*t+y0
⇒ y(t) = -0.5*t^2 + 0.5*t
– H2 := plot([x(t),y(t),t=0..1], coordinateview=[0..1, 0..1],
thickness=2, linestye=dash)

• Tan :=plot([t,-t+1],t =0..1, thickness=[2,2],

linestyle=dot, color = blue)
display(H1, H2, Tan)

Figure 3.11: The parametric curves: H1 (t) and H2 (t).

3.6. Parametric Curves 129

Exercises for Chapter 3

3.1. C For the given functions f (x), let x0 = 0, x1 = 0.5, x2 = 1. Construct interpolation
polynomials of degree at most one and at most two to approximate f (0.4), and find
the absolute error.

(a) f (x) = cos x

(b) f (x) = ln(1 + x)

3.2. Use the Polynomial Interpolation Error Theorem to find an error bound for the
approximations in Problem 1 above.
3.3. The polynomial p(x) = 1 − x + x(x + 1) − 2x(x + 1)(x − 1) interpolates the first four points
in the table:
x −1 0 1 2 3
y 2 1 2 −7 10
By adding one additional term to p , find a polynomial that interpolates the whole
table. (Do not try to find the polynomial from the scratch.)
3.4. Determine the Newton interpolating polynomial for the data:

x 4 2 0 3
y 63 11 7 28

3.5. Neville’s method is used to approximate f (0.4), giving the following table.

x0 = 0 Q0,0
x1 = 0.5 Q1,0 = 1.5 Q1,1 = 1.4
x2 = 0.8 Q2,0 Q2,1 Q2,2 = 1.2

Fill out the whole table.

3.6. Use the extended Newton divided difference method to obtain a quintic polynomial
that takes these values:
x f (x) f 0 (x)
0 2 −9
1 −4 4
2 44
3 2

3.7. Find a natural cubic spline for the data.

x −1 0 1
f (x) 5 7 9

(Do not use computer programming for this problem.)

130 Chapter 3. Interpolation and Polynomial Approximation

3.8. C Consider the data

x 0 1 3
f (x) 4 2 7
with f 0 (0) = −1/4 and f 0 (3) = 5/2. (The points are used in Example 3.53, p.122.)

(a) Find the quadratic spline that interpolates the data (with z0 = f 0 (0)).
(b) Find the clamped cubic spline that interpolates the data.
(c) Plot the splines and display them superposed.

3.9. Construct the piecewise cubic Hermite interpolating polynomial for

x f (x) f 0 (x)
0 2 −9
1 −4 4
2 4 12

3.10. C Let C be the unit circle of radius 1: x 2 + y 2 = 1. Find a piecewise cubic para-
metric curve that interpolates the circle at (1, 0), (0, 1), (−1, 0), (1, 0). Try to make the
parametric curve as circular as possible.

Hint : For the first portion, you may set

x0 := 1: x1 := 0: a0 := 0: a1 := -1:
x := t->(2*x0-2*x1+a0+a1)*t^3+(3*x1-3*x0-a1-2*a0)*t^2+a0*t+x0:
x(t)
3 2
t - 2 t + 1
y0 := 0: y1 := 1: b0 := 1: b1 := 0:
y := t->(2*y0-2*y1+b0+b1)*t^3+(3*y1-3*y0-b1-2*b0)*t^2+b0*t+y0:
y(t)
3 2
-t + t + t
plot([x(t),y(t),t=0..1], coordinateview = [0..1, 0..1], thickness = 2)

Now, (1) you can make it better, (2) you should find parametric curves for the other two
portions, and (3) combine them for a piece.
Chapter 4

Numerical Differentiation and

Integration

In this chapter:

Topics Applications/Properties
Numerical Differentiation f (x) ≈ Pn (x) locally
⇒ f 0 (x) ≈ Pn0 (x)
Three-point rules
Five-point rules
Richardson extrapolation Combination of low-order differences,
to get higher-order accuracy
Numerical Integration f (x) ≈ Pn (x) piecewisely
ˆ b ˆ b
⇒ f (x) ≈ Pn (x)
a a
Trapezoid rule
Simpson’s rule Newton-Cotes formulas
Simpson’s Three-Eights rule
Romberg integration
Gaussian Quadrature Method of undetermined coefficients &
orthogonal polynomials
Legendre polynomials

131
132 Chapter 4. Numerical Differentiation and Integration

4.1. Numerical Differentiation

Note: The derivative of f at x0 is defined as
f (x0 + h) − f (x0 )
f 0 (x0 ) = lim . (4.1)
h →0 h
This formula gives an obvious way to generate an approximation of
f 0 (x0 ):
f (x0 + h) − f (x0 )
f 0 (x0 ) ≈ . (4.2)
h

Formula. (Two-Point Difference Formulas): Let x1 = x0 + h and P0,1

be the first Lagrange polynomial interpolating f and x0 , x1 . Then
(x − x0 )(x − x1 ) 00
f (x) = P0,1 (x) + f (ξ )
2! (4.3)
x − x1 x − x0 (x − x0 )(x − x1 ) 00
= f (x0 ) + f (x1 ) + f (ξ ).
−h h 2!
Differentiating it, we obtain
f (x1 ) − f (x0 ) 2x − x0 − x1 00 (x − x0 )(x − x1 ) d 00
f 0 (x) = + f (ξ ) + f (ξ ). (4.4)
h 2 2! dx
Thus
f (x1 ) − f (x0 ) h 00
f 0 (x0 ) = − f (ξ (x0 ))
h 2 (4.5)
f (x 1 ) − f (x0 ) h
f 0 (x1 ) = + f 00 (ξ (x1 ))
h 2

Definition 4.1. For h > 0,

f (xi + h) − f (xi )
f 0 (xi ) ≈ Dx+ f (xi ) = , (forward-difference)
h (4.6)
f (xi ) − f (xi − h)
f 0 (xi ) ≈ Dx− f (xi ) = . (backward-difference)
h
4.1. Numerical Differentiation 133

Example 4.2. Use the forward-difference formula to approximate f (x) =

x 3 at x0 = 1 using h = 0.1, 0.05, 0.025.
Solution. Note that f 0 (1) = 3.
Maple-code
1 f := x -> x^3: x0 := 1:
2

3 h := 0.1:
4 (f(x0 + h) - f(x0))/h
5 3.310000000
6 h := 0.05:
7 (f(x0 + h) - f(x0))/h
8 3.152500000
9 h := 0.025:
10 (f(x0 + h) - f(x0))/h
11 3.075625000

The error becomes half, as h halves?

Formula. (In general): Let {x0 , x1 , · · · , xn } be (n + 1) distinct points in
some interval I and f ∈ C n+1 (I). Then the Interpolation Error Theorem
reads
n n
X f (n+1) (ξ ) Y
f (x) = f (xk )Ln,k (x) + (x − xk ). (4.7)
(n + 1)!
k =0 k =0

Its derivative gives

n n
X
0 d  f (n+1) (ξ )  Y
f 0 (x) = f (xk )Ln,k (x) + (x − xk )
dx (n + 1)!
k =0 k =0
(n+1) n (4.8)
f (ξ ) d  Y 
+ (x − xk ) .
(n + 1)! dx
k =0

Hence,
n n
0
X
0 f (n+1) (ξ ) Y
f (xi ) = f (xk )Ln,k (xi ) + (xi − xk ). (4.9)
(n + 1)!
k =0 k =0,k 6=i
134 Chapter 4. Numerical Differentiation and Integration

Definition 4.3. An (n + 1)-point difference formula to approximate

f 0 (xi ) is
n
X
0 0
f (xi ) ≈ f (xk )Ln,k (xi ) (4.10)
k =0

Formula. (Three-Point Difference Formulas (n = 2)): For conve-

nience, let
x0, x1 = x0 + h, x2 = x0 + 2h, h > 0.
Recall
(x − x1 )(x − x2 ) (x − x0 )(x − x2 )
L2,0 (x) = , L2,1 (x) = ,
(x0 − x1 )(x0 − x2 ) (x1 − x0 )(x1 − x2 )
(x − x0 )(x − x1 )
L2,2 (x) = .
(x2 − x0 )(x2 − x1 )
Thus, the three-point endpoint formulas and the three-point mid-
point formula read
2
0 0 0 0 f (3) (ξ (x0 )) Y
f (x0 ) = f (x0 )L2,0 (x0 ) + f (x1 )L2,1 (x0 ) + f (x2 )L2,2 (x0 ) + (x0 − xk )
3!
k =0,k 6=0
2
−3f(x0 ) + 4f(x1 ) − f(x2 ) h (3)
= + f (ξ0 ),
2h 3
f(x2 ) − f(x0 ) h 2 (3)
f 0 (x1 ) = − f (ξ1 ),
2h 6
0 f(x0 ) − 4f(x1 ) + 3f(x2 ) h 2 (3)
f (x2 ) = + f (ξ2 ).
2h 3
(4.11)

Formula. (Five-Point Difference Formulas): Let fi = f (x0 + i h), h >

0, −∞ < i < ∞.
f−2 − 8f−1 + 8f1 − f2 h 4 (5)
f 0 (x0 ) = + f (ξ ),
12h 30 (4.12)
0 −25f0 + 48f1 − 36f2 + 16f3 − 3f4 h 4 (5)
f (x0 ) = + f (ξ ).
12h 5
4.1. Numerical Differentiation 135

Summary 4.4. (Numerical Differentiation, the (n + 1)-point dif-

ference formulas):
1. f (x) = Pn (x) + Rn (x), Pn (x) ∈ Pn
2. f 0 (x) = Pn0 (x) + O(h n ),
f 00 (x) = Pn00 (x) + O(h n−1 ), and so on.

Second-Derivative Midpoint Formula

Example 4.5. We can see from the above summary that when the three-
point (n = 2) difference formula is applied for the approximation of f 00 , the
accuracy reads O(h). Use the Taylor expansion to derive the formula
f−1 − 2f0 + f1 h 2 (4)
f 00 (x0 ) = − f (ξ ). (4.13)
h2 12

Solution.
136 Chapter 4. Numerical Differentiation and Integration

Example 4.6. Use the second-derivative midpoint formula to approxi-

mate f 00 (1) for f (x) = x 5 − 3x 2 , using h = 0.2, 0.1, 0.05.
Solution.
Maple-code
1 f := x -> x^5 - 3*x^2:
2 x0 := 1:
3

4 eval(diff(f(x), x, x), x = x0)

5 14
6 h := 0.2:
7 (f(x0 - h) - 2*f(x0) + f(x0 + h))/h^2
8 14.40000000
9 h := 0.1:
10 (f(x0 - h) - 2*f(x0) + f(x0 + h))/h^2
11 14.10000000
12 h := 0.05:
13 (f(x0 - h) - 2*f(x0) + f(x0 + h))/h^2
14 14.02500000
4.2. Richardson Extrapolation 137

4.2. Richardson Extrapolation

Richardson extrapolation is used to generate high-accuracy difference
results while using low-order formulas.
Example 4.7. Derive the three-point midpoint formulas:

0 f (x + h) − f (x − h) h h 2 (3) h 4 (5) h 6 (7) i

f (x) = − f (x) + f (x) + f (x) + · · · ,
2h 3! 5! 7! (4.14)
2
00 f (x − h) − 2f (x) + f (x + h) h h (4) h 4 (6) i
f (x) = − 2 f (x) + 2 f (x) + · · · .
h2 4! 6!

Solution.

Observation 4.8. The equations can be written as

M = N(h) + K2 h 2 + K4 h 4 + K6 h 6 + · · · , (4.15)

where M is the desired (unknown) quantity, N(h) s an approximation of

M using the parameter h , and Ki are values independent of h .
138 Chapter 4. Numerical Differentiation and Integration

How can we take advantage of the observation?

Strategy 4.9. (Richardson extrapolation):

1. Let’s first write out (4.15) with h replaced by h /2:

M = N(h /2) + K2 h 2 /4 + K4 h 4 /16 + K6 h 6 /64 + · · · . (4.16)

Then the leading term in the error series, K2 h 2 , can be eliminated as

follows:
M = N(h) +K2 h 2 + K4 h 4 + K6 h 6 + · · ·
4M = 4N(h /2) +K2 h 2 + K4 h 4 /4 + K6 h 6 /16 + · · · (4.17)
3 15
3M = 4N(h /2) − N(h) − K4 h 4 − K6 h 6 − · · ·
4 16
Thus we have
1 1 5
M = [4N(h /2) − N(h)] − K4 h 4 − K6 h 6 − · · · . (4.18)
3 4 16
The above equation embodies the first step in Richardson extrapola-
tion. It show that a simple combination of two second-order approx-
imations, N(h) and N(h /2), furnishes an estimate of M with accuracy
O(h 4 ).
2. For simplicity, we rewrite (4.18) as
1 5
M = N2 (h) − K4 h 4 − K6 h 6 − · · · . (4.19)
4 16
Then, similarly,
1 5
M = N2 (h /2) − K4 h 4 − 10 K6 h 6 − · · · . (4.20)
64 2
Subtract (4.18) from 16 times (4.20) to produce a new O(h 6 ) formula:

1 1
M = [16N2 (h /2) − N2 (h)] + K6 h 6 + · · · . (4.21)
15 64
4.2. Richardson Extrapolation 139

Algorithm 4.10. (Richardson extrapolation): The above idea can

be applied recursively. The complete algorithm of Richardson extrapo-
lation algorithm is formulated as:
1. Select a convenient h and compute

D(i, 0) = N(h /2i ), i = 0, 1, · · · , n. (4.22)

2. Compute additional quantities using the formula

for i = 1, 2, · · · , n do
for j = 1, 2, · · · , i do
1  j 
D(i, j) = j 4 · D(i, j − 1) − D(i − 1, j − 1) (4.23)
4 −1
end do
end do

Note:
(a) One can prove that

D(i, j) = M + O(h 2(j+1) ). (4.24)

(b) The second step in the algorithm can be rewritten for a column-wise
computation:

for j = 1, 2, · · · , i do
for i = j, j + 1, · · · , n do
1  j 
D(i, j) = j 4 · D(i, j − 1) − D(i − 1, j − 1) (4.25)
4 −1
end do
end do
140 Chapter 4. Numerical Differentiation and Integration

Example 4.11. Let f (x) = ln x . Use the Richardson extrapolation to esti-

mate f 0 (1) = 1 using h = 0.2, 0.1, 0.05.
Solution.
Maple-code
1 f := x -> ln(x):
2 h := 0.2:
3 D00 := (f(1 + h) - f(1 - h))/(2*h);
4 1.013662770
5 h := 0.1:
6 D10 := (f(1 + h) - f(1 - h))/(2*h);
7 1.003353478
8 h := 0.05:
9 D20 := (f(1 + h) - f(1 - h))/(2*h);
10 1.000834586
11

12 D11 := (4*D10 - D00)/3;

13 0.9999170470
14 D21 := (4*D20 - D10)/3;
15 0.9999949557
16 D22 := (16*D21 - D11)/15;
17 1.000000150
18 #Error Convergence:
19 abs(1 - D11);
20 0.0000829530
21 abs(1 - D21);
22 0.0000050443
23 #The Ratio:
24 abs(1 - D11)/abs(1 - D21);
25 16.44489820

h j = 0 : O(h 2 ) j = 1 : O(h 4 ) j = 2 : O(h 6 )

0.2 D0,0 = 1.013662770
0.1 D1,0 = 1.003353478 D1,1 = 0.9999170470
0.05 D2,0 = 1.000834586 D2,1 = 0.9999949557 D2,2 = 1.000000150
4.2. Richardson Extrapolation 141

Example 4.12. Let f (x) = ln x , as in the previous example, Example 4.11.

Produce a Richardson extrapolation table for the approximation of f 00 (1) =
−1, using h = 0.2, 0.1, 0.05.
Solution.
Maple-code
1 f := x -> ln(x):
2 h := 0.2:
3 D00 := (f(1 - h) - 2*f(1) + f(1 + h))/h^2;
4 -1.020549862
5 h := 0.1:
6 D10 := (f(1 - h) - 2*f(1) + f(1 + h))/h^2;
7 -1.005033590
8 h := 0.05:
9 D20 := (f(1 - h) - 2*f(1) + f(1 + h))/h^2;
10 -1.001252088
11

12 D11 := (4*D10 - D00)/3;

13 -0.9998614997
14 D21 := (4*D20 - D10)/3;
15 -0.9999915873
16 D22 := (16*D21 - D11)/15;
17 -1.000000260
18 #Error Convergence:
19 abs(-1 - D11);
20 0.0001385003
21 abs(-1 - D21);
22 0.0000084127
23 #The Ratio:
24 abs(-1 - D11)/abs(-1 - D21);
25 16.46324010

h j = 0 : O(h 2 ) j = 1 : O(h 4 ) j = 2 : O(h 6 )

0.2 D0,0 = −1.020549862
0.1 D1,0 = −1.005033590 D1,1 = −0.9998614997
0.05 D2,0 = −1.001252088 D2,1 = −0.9999915873 D2,2 = −1.000000260
142 Chapter 4. Numerical Differentiation and Integration

4.3. Numerical Integration

Note: Numerical integration can be performed by

(1) approximating the function f by an nth-degree polynomial Pn , and
(2) integrating the polynomial over the prescribed interval.
What a simple task it is!

Let {x0 , x1 , · · · , xn } be distinct points (nodes) in [a, b]. Then the Lagrange
interpolating polynomial reads
n
X
Pn (x) = f (xi )Ln,i (x), (4.26)
i=0

which interpolates the function f . Then, as just mentioned, we simply

approximate
ˆ b ˆ b n
X ˆ b
f (x) dx ≈ Pn (x) dx = f (xi ) Ln,i (x) dx. (4.27)
a a i=0 a

Definition 4.13. In this way, we obtain a formula which is a weighted

sum of the function values:
ˆ b n
X
f (x) dx ≈ Ai f (xi ), (4.28)
a i=0

where ˆ b
Ai = Ln,i (x) dx. (4.29)
a
The formula of the form in (4.28) is called a Newton-Cotes formula
when the nodes are equally spaced.
4.3. Numerical Integration 143

4.3.1. The trapezoid rule

The simplest case results if n = 1 and the nodes are x0 = a and x1 = b . In
this case,
1
X f 00 (ξx )
f (x) = f (xi )L1,i (x) + (x − x0 )(x − x1 ), (4.30)
2!
i=0

and its integration reads

ˆ x1 1 ˆ x1 ˆ x1
X f 00 (ξx )
f (x) dx = f (xi ) L1,i (x) dx + (x − x0 )(x − x1 ) dx. (4.31)
x0 x0 x0 2!
i=0

Derivation 4.14. Terms in the right-side of (4.31) must be verified to

get a formula and its error bound. Note that
ˆ x1 ˆ x1
x − x1 1
A0 = L1,0 (x) dx = dx = (x1 − x0 ),
x − x1 2
ˆx0x1 ˆx0x1 0 (4.32)
x − x0 1
A1 = L1,1 (x) dx = dx = (x1 − x0 ),
x0 x0 x 1 − x0 2

and
ˆ x1 ˆ
f 00 (ξx ) f 00 (ξ ) x1
(x − x0 )(x − x1 ) dx = (x − x0 )(x − x1 ) dx
x0 2! 2! x0
(4.33)
f 00 (ξ )
= − (x1 − x0 )3 .
12
(Here we could use the Weighted Mean Value Theorem on Integral be-
cause (x − x0 )(x − x1 ) ≤ 0 does not change the sign over [x0 , x1 ].)

Definition 4.15. The corresponding quadrature formula is

ˆ x1
h h 3 00
f (x) dx = [f (x0 ) + f (x1 )] − f (ξ ), (Trapezoid) (4.34)
x0 2 12

which is known as the trapezoid rule.

144 Chapter 4. Numerical Differentiation and Integration

Graphical interpretation:
with(Student[Calculus1]):
f := x^3 + 2 + sin(2*Pi*x):
ApproximateInt(f, 0..1, output = animation, partition = 1,
method = trapezoid, refinement = halve,
boxoptions = [filled = [color=pink,transparency=0.5]]);

Figure 4.1: Trapazoid rule.

Composite trapezoid rule: Let the interval [a, b] be partitioned as

a = x0 < x1 < · · · < xn = b.

Then the trapezoid rule can be applied to each subinterval. Here the nodes
are not necessarily uniformly spaced. Thus, we obtain the composite
trapezoid rule reads
ˆ b n ˆ xi n
X X hi
f (x) dx = f (x) dx ≈ (f (xi −1 ) + f (xi )) , hi = xi − xi −1 . (4.35)
a x i −1 2
i=1 i=1

With a uniform spacing:

b −a
xi = a + i h, h= ,
n
4.3. Numerical Integration 145

the composite trapezoid rule takes the form

ˆ b
" n −1
#
f (a) + f (b) X
f (x) dx ≈ h · + f (xi ) , (4.36)
a 2
i=1

for which the composite error becomes

n  n
X h 3 00  00
X h3
00 h3 00 (b − a)h 2
− f (ξi ) = −f (ξ ) = −f (ξ ) · n = −f (ξ ) , (4.37)
12 12 12 12
i=1 i=1

b −a
 b − a
where we have used h = ⇒n= .
n h
Example 4.16.
with(Student[Calculus1]):
f := x^3 + 2 + sin(2*Pi*x):
ApproximateInt(f, 0..1, output = animation, partition = 8,
method = trapezoid, refinement = halve,
boxoptions = [filled = [color=pink,transparency=0.5]]);

Figure 4.2: Composite trapazoid rule.

146 Chapter 4. Numerical Differentiation and Integration

4.3.2. Simpson’s rule

Simpson’s rule results from integrating over [a, b] the second Lagrange
polynomial with three equally spaced nodes:  b − a
x0 = a, x1 = a + h, x2 = a + 2h = b. h=
2

Definition 4.17. The elementary Simpson’s rule reads

ˆ b ˆ x2 ˆ x2 X
2
f (x) dx = f (x) dx ≈ f (xi )L2,i (x), (4.38)
a x0 x0 i=0

which is reduced to
ˆ x2
2h  
f (x) dx ≈ f (x0 ) + 4f (x1 ) + f (x2 ) . (4.39)
x0 6

Graphical interpretation:
with(Student[Calculus1]):
f := x^3 + 2 + sin(2*Pi*x):
ApproximateInt(f, 0..1, output = animation, partition = 1,
method = simpson, refinement = halve,
boxoptions = [filled = [color=pink,transparency=0.5]]);

Figure 4.3: The elementary Simpson’s rule, which is exact for the given problem.
4.3. Numerical Integration 147

Error analysis for the elementary Simpson’s rule:

• The error for the elementary Simpson’s rule can be analyzed from
ˆ x2 000
f (ξ )
(x − x0 )(x − x1 )(x − x2 ) dx, (4.40)
x0 3!
which must be in O (h 4 ).
• However, by approximating the problem in another way, one can show
the error is in O (h 5 ). (See Example 4.5, p.135.)
• It follows from the Taylor’s Theorem that for each x ∈ [x0 , x2 ], there is a
number ξ ∈ (x0 , x2 ) such that
f 00 (x1 ) f 000 (x1 ) f (4) (ξ )
f (x) = f (x1 )+f 0 (x1 )(x −x1 )+(x −x1 )2 + (x −x1 )3 + (x −x1 )4 . (4.41)
2! 3! 4!
By integrating the terms over [x0 , x2 ], we have
ˆ x2
f 0 (x1 ) 2 f 00 (x1 )
f (x) dx = f (x1 )(x − x1 ) + (x − x1 ) + (x − x1 )3
2 3!
x0
000 ˆ x (4) (4.42)
f (x1 ) x2 2
f (ξ )
+ (x − x1 )4 + (x − x1 )4 dx.
4! x0 x0 4!
The last term can be easily computed by using the Weighted Mean Value
Theorem on Integral:
ˆ x2 (4) ˆ
f (ξ ) 4 f (4) (ξ1 ) x2 4 f (4) (ξ1 ) 5
(x − x1 ) dx = (x − x1 ) dx = h . (4.43)
x0 4! 4! x0 60
Thus, Equation (4.42) reads
ˆ x2
h 3 00 f (4) (ξ1 ) 5
f (x) dx = 2h f (x1 ) + f (x1 ) + h . (4.44)
x0 3 60
See (4.14), p.137, to recall that
00 f (x0 ) − 2f (x1 ) + f (x2 ) h 2 (4)
f (x1 ) = − f (ξ2 ). (4.45)
h2 12
Plugging this to (4.44) reads
ˆ x2 3
h f (x0 ) − 2f (x1 ) + f (x2 )  h 3  h 2 (4)  f (4) (ξ )
1
f (x) dx = 2h f (x1 ) + 2
− f (ξ2 ) + h 5,
x0 3 h 3 12 60
(4.46)
148 Chapter 4. Numerical Differentiation and Integration

and therefore
ˆ x2
2h   h 5 (4)
f (x) dx = f (x0 ) + 4f (x1 ) + f (x2 ) − f (ξ3 ). (4.47)
x0 6 90

Composite Simpson’s rule: A composite Simpson’s rule, using an

even number of subintervals, is often adopted. Let n be even, and set
b −a
xi = a + i h, h= . (0 ≤ i ≤ n)
n
Then
ˆ b n/2 ˆ x2i n/2
X 2h X  
f (x) dx = f (x) dx ≈ f (x2i −2 ) + 4f (x2i −1 ) + f (x2i ) . (4.48)
a x2i −2 6
i=1 i=1

Example 4.18. Show that the error term for the composite Simpson’s
rule becomes
(b − a)h 4 (4)
− f (ξ ). (4.49)
180

Solution.
4.3. Numerical Integration 149

4.3.3. Simpson’s three-eights rule

We have developed quadrature rules when the function f is approximated
by piecewise Lagrange polynomials of degrees 1 and 2. Such integration
formulas are called the closed Newton-Cotes formulas, and the idea
can be extended for any degrees. The word closed is used, because the
formulas include endpoints of the interval [a, b] as nodes.

Theorem 4.19. When three equal subintervals are combined, the re-
sulting integration formula is called the Simpson’s three-eights rule:
ˆ x3
3h   3h 5 (4)
f (x) dx = f (x0 ) + 3f (x1 ) + 3f (x2 ) + f (x3 ) − f (ξ ). (4.50)
x0 8 80

Example 4.20. Let n be a multiple of 3. For the nodes,

b −a
xi = a + i h, h= , (0 ≤ i ≤ n)
n
derive the error term for the composite Simpson’s three-eights rule.
Solution.

Note: When n is not even, you may approximate the integration by a

combination of the Simpson’s rule and the Simpson’s three-eights rule.
For example, let n = 13 . Then you may apply the Simpson’s rule for
[x0 , x10 ] and the Simpson’s three-eights rule for the last three subinter-
vals [x10 , x13 ].
150 Chapter 4. Numerical Differentiation and Integration

Self-study 4.21. Consider

ˆ 2
x
dx,
0 x2 + 1
ln 5
of which the true value is ≈ 0.80471895621705018730. Use 6 equally
2
spaced points (n = 5) to approximate the integral using

(a) the trapezoid rule, and

(b) a combination of the Simpson’s rule and the Simpson’s three-eights
rule.

Solution.

(a) 0.7895495781.
(so, the error = 0.0151693779)
´ 0.8 ´2
(b) 0 f (x)dx + 0.8 f (x)dx ≈ 0.2474264468 + 0.5567293981 = 0.8041558449.
(so, the error = 0.0005631111)
4.4. Romberg Integration 151

4.4. Romberg Integration

In the previous section, we have found that the Composite Trapezoid rule
has a truncation error of order O(h 2 ). Specifically, we showed that for
b −a
xk = a + k h, h= , (0 ≤ k ≤ n)
n
we have
ˆ b
00 (b − a)h 2
f (x) dx = T (n) − f (ξ ) , (Composite Trapezoid) (4.51)
a 12
where T (n) is the Trapezoid quadrature obtained over n equal subintervals:

h1 n −1
X 1 i
T (n) = h f (a) + f (xk ) + f (b) . (4.52)
2 2
k =1

4.4.1. Recursive Trapezoid rule

Let us begin with an effective computation technique for the Composite
Trapezoid rule when n = 2i .
Example 4.22. What is the explicit formula for T (1), T (2), T (4), and T (8)
in the case in which the interval is [0, 1]?
Solution. Using Equation (4.52), we have
h1 1 i
T (1) = 1 · f (0) + f (1)
2 2
1 1 h 1 1 i
T (2) = · f (0) + f + f (1)
2 2 2 2
1 1 h  1  1 3 1 i
T (4) = · f (0) + f +f +f + f (1) (4.53)
4 2 4 2 4 2
1 1 h  1   1   3  1
T (8) = · f (0) + f +f +f +f
8 2 8 4 8 2
5 3 7 1 i
+f +f +f + f (1)
8 4 8 2
152 Chapter 4. Numerical Differentiation and Integration

Remark 4.23. It is clear that if T (2n) is to be computed, then we can

take advantage of the work already done in the computation of T (n).
For example, from the preceding example, we see that
1 1 h  1 i
T (2) = T (1) + · f
2 2 2
1 1 h  1   3 i
T (4) = T (2) + · f +f (4.54)
2 4 4 4
1 1 h  1   3   5   7 i
T (8) = T (4) + · f +f +f +f
2 8 8 8 8 8

With h = (b − a)/(2n), the general formula pertaining to any interval

[a, b] is as follows:
1
T (2n) = T (n) + h [f (a + h) + f (a + 3h) + · · · + f (a + (2n − 1)h)], (4.55)
2
or
n
1 X 
T (2n) = T (n) + h f (x2k −1 ) . (4.56)
2
k =1

Now, if there are 2i uniform subintervals, Equation (4.55) provides a

recursive Trapezoid rule:

2i −1
i 1 i −1
X 
T (2 ) = T (2 ) + hi f (a + (2k − 1)hi ) , (4.57)
2
k =1

where
1
h0 = b − a, hi = h i −1 , i ≥ 1.
2
4.4. Romberg Integration 153

4.4.2. The Romberg algorithm

By an alternative method (Taylor series method), it can be shown that if
f ∈ C ∞ [a, b], the Composite Trapezoid rule (4.52) can also be written with
an error term in the form
ˆ b
f (x) dx = T (n) + K2 h 2 + K4 h 4 + K6 h 6 + · · · , (4.58)
a

where Ki are constants independent of h . Since

ˆ b
f (x) dx = T (2n) + K2 h 2 /4 + K4 h 4 /16 + K6 h 6 /64 + · · · , (4.59)
a

as for Richardson extrapolation, we have

ˆ b
1  3 15
f (x) dx = 4T (2n) − T (n) − K4 h 4 − K6 h 6 − · · · . (4.60)
a 3 4 16

Algorithm 4.24. (Romberg algorithm): The above idea can be ap-

plied recursively. The complete algorithm of Romberg integration is
formulated as:
1. The computation of R(i, 0) which is the trapezoid estimate with 2i
subintervals obtained using the formula (4.57):
b −a
R(0, 0) = [f (a) + f (b)],
2
2i −1 (4.61)
1 X 
R(i, 0) = R(i − 1, 0) + hi f (a + (2k − 1)hi ) .
2
k =1

2. Then, evaluate higher-order approximations recursively using

for i = 1, 2, · · · , n do
for j = 1, 2, · · · , i do
1  j 
R(i, j) = j 4 · R(i, j − 1) − R(i − 1, j − 1) (4.62)
4 −1
end do
end do
154 Chapter 4. Numerical Differentiation and Integration

Example 4.25. Use the Composite Trapezoid rule to find approximations

ˆ π
to sin x dx , with n = 1, 2, 4, and 8. Then perform Romberg extrapolation
0
on the results.
Solution.
Romberg-extrapolation
1 a := 0: b := Pi:
2 f := x -> sin(x):
3 n := 3:
4 R := Array(0..n, 0..n):
5

6 # Trapezoid estimates
7 #----------------------------------
8 R[0, 0] := (b - a)/2*(f(a) + f(b));
9 0
10 for i to n do
11 hi := (b-a)/2^i;
12 R[i,0] := R[i-1,0]/2 +hi*add(f(a+(2*k-1)*hi), k=1..2^(i-1));
13 end do:
14

15 # Now, perform Romberg Extrapolation:

16 # -------------------------------------------------
17 for i to n do
18 for j to i do
19 R[i, j] := (4^j*R[i,j-1] - R[i-1, j-1])/(4^j-1);
20 end do
21 end do

j = 0 : O(h 2 ) j = 1 : O(h 4 ) j = 2 : O(h 6 ) j = 3 : O(h 8 )

R0,0 =0
R1,0 = 1.570796327 R1,1 = 2.094395103 (4.63)
R2,0 = 1.896118898 R2,1 = 2.004559755 R2,2 = 1.998570731
R3,0 = 1.974231602 R3,1 = 2.000269171 R3,2 = 1.999983131 R3,3 = 2.000005551
4.4. Romberg Integration 155
ˆ π
Self-study 4.26. The true value for the integral: sin x dx = 2. Use the
0
table in (4.63) to verify that the error is in O(h 4 ) for j = 1 and O(h 6 ) for j = 2.
Hint : For example, for j = 1, you should measure |R1,1 − 2|/|R2,1 − 2| and |R2,1 − 2|/|R3,1 − 2| and
interpret them.
156 Chapter 4. Numerical Differentiation and Integration

4.5. Gaussian Quadrature

Recall:
• In Section 4.3, we saw how to create quadrature formulas of the type
ˆ b n
X
f (x) dx ≈ wi f (xi ), (4.64)
a i=0

that are exact for polynomials of degree ≤ n, which is the case if and
only if
ˆ b ˆ b Y
n
x − xj
wi = Ln,i (x) dx = dx. (4.65)
a a xi − xj
j=0, j 6=i

• In those formulas, the choice of nodes x0 , x1 , x2 , · · · , xn were made a

priori. Once the nodes were fixed, the coefficients were determined
uniquely form the requirement that Formula (4.64) must be an equal-
ity for f ∈ Pn .

4.5.1. The method of undetermined coefficients

Example 4.27. Find w0 , w1 , w2 with which the following formula is exact
for all polynomials of degree ≤ 2:
ˆ 1
f (x) dx ≈ w0 f (0) + w1 f (1/2) + w2 f (1). (4.66)
0

Solution. Formula (4.66) must be exact for some low-order polynomials.

Consider trial functions f (x) = 1, x, x 2 . Then, for each of them,
ˆ 1
1 = 1 dx = w0 · 1 + w1 · 1 + w2 · 1 = w0 + w1 + w2
ˆ 1
0
1 1 1
= x dx = w0 · 0 + w1 · + w2 · 1 = w1 + w2 (4.67)
2 2 2
ˆ0 1
1 2
 1 2 1
= x dx = w0 · 0 + w1 · + w2 · 1 = w1 + w2
3 0 2 4
4.5. Gaussian Quadrature 157

The solution of this system of three simultaneous equations is

1 2 1
(w0 , w1 , w2 ) = , , . (4.68)
6 3 6
Thus the formula can be written as
ˆ 1
1 
f (x) dx ≈ f (0) + 4f (1/2) + f (1) , (4.69)
0 6
which will produce exact values of integrals for any quadratic polynomial,
f (x) = a0 + a1 x + a2 x 2 .
Note: It must be noticed that Formula (4.69) is the elementary Simp-
son’s rule with h = 1/2.

Key Idea 4.28. Gaussian quadrature chooses the points for evalu-
ation in an optimal way, rather than equally-spaced points. The nodes
x1 , x2 , · · · , xn in the interval [a, b] and the weights w1 , w2 , · · · , wn are
chosen to minimize the expected error obtained in the approximation
ˆ b n
X
f (x) dx ≈ wi f (xi ). (4.70)
a i=1

• To measure this accuracy, we assume that the best choice of these

values produces the exact result for the largest class of polynomials,
that is, the choice that gives the greatest degree of precision.
• The above formula gives 2n parameters (x1 , x2 , · · · , xn and
w1 , w2 , · · · , wn ) to choose. Since the class of polynomials of degree
at most (2n − 1) is 2n-dimensional (containing 2n parameters), one
may try to decide the parameters with which the quadrature formula
is exact for all polynomials in P2n−1 .
158 Chapter 4. Numerical Differentiation and Integration

Example 4.29. Determine x1 , x2 and w1 , w2 so that the integration for-

mula ˆ 1
f (x) dx ≈ w1 f (x1 ) + w2 f (x2 ) (4.71)
−1
gives the exact result whenever f ∈ P3 .
Solution. As in the previous example, we may apply the method of unde-
termined coefficients. This time, use f (x) = 1, x, x 2 , x 3 as trial functions.

Note: The method of undetermined coefficients can be used to deter-

mine the nodes and weights for formulas that give exact results for
high-order polynomials, but an alternative method obtained them mote
easily. The alternative is related to Legendre orthogonal polynomials.
4.5. Gaussian Quadrature 159

4.5.2. Legendre polynomials

Definition 4.30. Let {P0 (x), P1 (x), · · · , Pk (x), · · · } be a collection of poly-

nomials with Pk ∈ Pk . It is called orthogonal polynomials when it
satisfies ˆ 1
Q(x)Pk (x) dx = 0, ∀ Q(x) ∈ Pk −1 . (4.72)
−1
Such orthogonal polynomials can be formulated by a certain three-term
recurrence relation.

Definition 4.31. The Legendre polynomials obey the three-term

recurrence relation, known as Bonnet’s recursion formula:

(k + 1)Pk +1 (x) = (2k + 1)xPk (x) − kPk −1 (x), (4.73)

beginning with P0 (x) = 1, P1 (x) = x . A few first Legendre polynomials

are
P0 (x) = 1
P1 (x) = x
3 1
P2 (x) = (x 2 − )
2 3
5 3
P3 (x) = (x 3 − x) (4.74)
2 5
35 4 6 2 3
P4 (x) = (x − x + )
8 7 35
63 5 10 3 5
P5 (x) = (x − x + x)
8 9 21

Theorem 4.32. The Legendre polynomials satisfy

|Pk (x)| ≤ 1, ∀ x ∈ [−1, 1],

P (±1) = (±1)k ,
ˆ k1 ˆ 1
(4.75)
1
6 k;
Pj (x)Pk (x) dx = 0, j = Pk (x)2 dx = .
−1 −1 k + 1/2
160 Chapter 4. Numerical Differentiation and Integration

Note: The Chebyshev polynomials, defined in Definition 3.20, are also

orthogonal polynomials; see page 95. Frequently-cited classical or-
thogonal polynomials are: Jacobi polynomials, Laguerre polynomials,
Chebyshev polynomials, and Legendre polynomials.

4.5.3. Gauss integration

Theorem 4.33. (Gauss integration): Suppose that {x1 , x2 , · · · , xn }

are the roots of the nth Legendre polynomial Pn and {w1 , w2 , · · · , wn }
are obtained by
ˆ 1 Yn  ˆ 1
x − xj 
wi = dx. = Ln−1,i (x) dx (4.76)
−1 xi − xj −1
j=1, j 6=i

Then,
ˆ 1 n
X
f (x) dx ≈ wi f (xi ) is exact, ∀ f ∈ P2n−1 . (4.77)
−1 i=1

Note: Once the nodes are determined, the weights {w1 , w2 , · · · , wn } can
also be found by using the method of undetermined coefficients. That is,
the weights are the solution of the linear system
X n ˆ 1
(xj )i wj = x i dx, i = 0, 1, · · · , n − 1. (4.78)
j=1 −1
4.5. Gaussian Quadrature 161

Gauss-integration
1 with(LinearAlgebra):
2 with(Statistics):
3 WeightSystem := proc(n, A, b, X)
4 local i, j;
5 for j to n do
6 A[1, j] := 1;
7 for i from 2 to n do
8 A[i, j] := A[i - 1, j]*X[j];
9 end do:
10 end do:
11 for i to n do
12 b[i] := int(x^(i - 1), x = -1..1);
13 end do:
14 end proc
15

16 nmax := 5:
17 for k to nmax do
18 Legendre[k] := sort(orthopoly[P](k, x));
19 end do;
20 Legendre[1] := x
21 3 2 1
22 Legendre[2] := - x - -
23 2 2
24 5 3 3
25 Legendre[3] := - x - - x
26 2 2
27 35 4 15 2 3
28 Legendre[4] := -- x - -- x + -
29 8 4 8
30 63 5 35 3 15
31 Legendre[5] := -- x - -- x + -- x
32 8 4 8
33

34 Node := Array(0..nmax):
35 Weight := Array(0..nmax):
36

37 for k to nmax do
38 solve(Legendre[k] = 0, x):
39 Node[k] := Sort(Vector([%])):
40 n := Dimensions(Node[k]):
41 A := Matrix(n, n):
42 b := Vector(n):
43 WeightSystem(n, A, b, Node[k]):
162 Chapter 4. Numerical Differentiation and Integration

44 Weight[k] := A^(-1).b:
45 end do:
46

47 for k to nmax do
48 printf(" k=%d\n", k);
49 print(Nodek = evalf(Node[k]));
50 print(Weightk = evalf(Weight[k]));
51 end do;
52 k=1
53 Nodek = [0.]
54 Weightk = [2.]
55 k=2
56 [ -0.5773502693 ]
57 Nodek = [ ]
58 [ 0.5773502693 ]
59

60 [1.]
61 Weightk = [ ]
62 [1.]
63 k=3
64 [ -0.7745966692 ]
65 [ ]
66 Nodek = [ 0. ]
67 [ ]
68 [ 0.7745966692 ]
69

70 [0.5555555556]
71 [ ]
72 Weightk = [0.8888888889]
73 [ ]
74 [0.5555555556]
75 k=4
76 [ -0.8611363114 ]
77 [ ]
78 [ -0.3399810437 ]
79 Nodek = [ ]
80 [ 0.3399810437 ]
81 [ ]
82 [ 0.8611363114 ]
83

84 [0.3478548456]
85 [ ]
86 [0.6521451560]
87 Weightk = [ ]
88 [0.6521451563]
 4.5. Gaussian Quadrature 163

89 [ ]
90 [0.3478548450]
91 k=5
92 [ -0.9061798457 ]
93 [ ]
94 [ -0.5384693100 ]
95 [ ]
96 Nodek = [ 0. ]
97 [ ]
98 [ 0.5384693100 ]
99 [ ]
100 [ 0.9061798457 ]
101

102 [0.2427962711]
103 [ ]
104 [0.472491159 ]
105 [ ]
106 Weightk = [0.568220204 ]
107 [ ]
108 [0.478558682 ]
109 [ ]
110 [0.2369268937]

Remark 4.34. (Gaussian Quadrature on Arbitrary Intervals):

ˆ b
An integral f (x) dx over an interval [a, b] can be transformed into an
a
integral over [−1, 1] by using the change of variables:
b −a a+b
T : [−1, 1] → [a, b], x = T (t) = t+ . (4.79)
2 2
Using it, we have
ˆ b ˆ 1 b − a a + b b − a
f (x) dx = f t+ dt. (4.80)
a −1 2 2 2
164 Chapter 4. Numerical Differentiation and Integration
ˆ π
Example 4.35. Find the Gaussian Quadrature for sin(x) dx , with n =
0
2, 3, 4.
Solution.
Gaussian-Quadrature
1 a := 0:
2 b := Pi:
3 f := x -> sin(x):
4 nmax := 4:
5 T := t -> 1/2*(b - a)*t + 1/2*a + 1/2*b:
6 T(t)
7 1 1
8 - Pi t + - Pi
9 2 2
10 trueI := int(f(x), x = a..b)
11 2
12

13 g := t -> 1/2*f(T(t))*(b - a):

14 GI := Vector(nmax):
15 for k from 2 to nmax do
16 GI[k] := add(evalf(Weight[k][i]*g(Node[k][i])), i = 1..k);
17 end do:
18 for k from 2 to nmax do
19 printf(" n=%d GI=%g error=%g\n",k,GI[k],abs(trueI-GI[k]));
20 end do
21 n=2 GI=1.93582 error=0.0641804
22 n=3 GI=2.00139 error=0.00138891
23 n=4 GI=1.99998 error=1.5768e-05
4.5. Gaussian Quadrature 165

4.5.4. Gauss-Lobatto integration

Theorem 4.36. (Gauss-Lobatto integration): Let x0 = −1, xn =

1, and {x1 , x2 , · · · , xn−1 } are the roots of the first-derivative of the nth
Legendre polynomial, Pn0 (x). Let {w0 , w1 , w2 , · · · , wn } be obtained by
ˆ 1 Y n  ˆ 1
x − xj 
wi = dx. = Ln,i (x) dx (4.81)
−1 xi − xj −1
j=0, j 6=i

Then, ˆ n
1 X
f (x) dx ≈ wi f (xi ) is exact, ∀ f ∈ P2n−1 . (4.82)
−1 i=0

Recall: Theorem 4.33 (Gauss integration): Suppose that

{x1 , x2 , · · · , xn } are the roots of the nth Legendre polynomial Pn and
{w1 , w2 , · · · , wn } are obtained by
ˆ 1 Y n  ˆ 1
x − xj 
wi = dx. = Ln−1,i (x) dx (4.83)
−1 xi − xj −1
j=1, j 6=i

Then, ˆ n
1 X
f (x) dx ≈ wi f (xi ) is exact, ∀ f ∈ P2n−1 . (4.84)
−1 i=1

Remark 4.37.
• The Gauss-Lobatto integration is a closed formula for numerical
integrations, which is more popular in real-world applications than
open formulas such as the Gauss integration.
• Once the nodes are determined, the weights {w0 , w1 , w2 , · · · , wn } an
also be found by using the method of undetermined coefficients, as for
Gauss integration; the weights are the solution of the linear system
Xn ˆ 1
i
(xj ) wj = x i dx, i = 0, 1, · · · , n. (4.85)
j=0 −1
166 Chapter 4. Numerical Differentiation and Integration
ˆ π
Self-study 4.38. Find the Gauss-Lobatto Quadrature for sin(x) dx ,
0
with n = 2, 3, 4.
4.5. Gaussian Quadrature 167

Exercises for Chapter 4

4.1. Use the most accurate three-point formulas to determine the missing entries.

x f (x) f 0 (x) f 00 (x)

1.0 2.0000 6.00
1.2 1.7536
1.4 1.9616
1.6 2.8736
1.8 4.7776
2.0 8.0000
2.2 12.9056 52.08

4.2. Use your results in the above table to approximate f 0 (1.6) and f 00 (1.6) with O(h 4 )-
accuracy. Make a conclusion by comparing all results (obtained here and from Prob-
lem 1) with the exact values:

f 0 (1.6) = 6.784, f 00 (1.6) = 24.72.

4.3. Let a numerical process be described by

M = N(h) + K1 h + K2 h 2 + K3 h 3 + · · · (4.86)

Explain how Richardson extrapolation will work in this case. (Try to introduce a
formula described as in (4.23), page 139.)
ˆ 2
4.4. In order to approximate x ln(x 2 + 1) dx with h = 0.4, use
0

(a) the Trapezoid rule, and

(b) Simpson’s rules.

4.5. A car laps a race track in 65 seconds. The speed of the car at each 5 second interval
is determined by using a radar gun and is given from the beginning of the lap, in
feet/second, by the entries in the following table:

Time 0 5 10 15 20 25 30 35 40 25 50 55 60 65
Speed 0 90 124 142 156 147 133 121 109 99 95 78 89 98

How long is the track?

168 Chapter 4. Numerical Differentiation and Integration

4.6. C Consider integrals

ˆ 3 ˆ π
1
I. dx II. cos2 x dx
1 x 0

(a) For each of the integrals, use the Romberg extrapolation to find R[3, 3].
(b) Determine the number of subintervals required when the Composite Trapezoid
rule is used to find approximations within the same accuracy as R[3, 3].
ˆ π/2
4.7. C Find the Gaussian Quadrature for cos2 x dx , with n = 2, 3, 4.
0
Chapter 5

Numerical Solution of Ordinary

Differential Equations

In this chapter:

Topics Applications/Properties
Elementary Theory of IVPs Existence and uniqueness of so-
lution
Taylor-series Methods
Euler’s method
Higher-Order Taylor methods
Runge-Kutta (RK) Methods
Second-order RK (Heun’s method) Modified Euler’s method
Fourth-order RK
Runge-Kutta-Fehlberg method Variable step-size (adaptive
method)
Multi-step Methods
Adams-Bashforth-Moulton method
Higher-Order Equations &
Systems of Differential Equations

169
170 Chapter 5. Numerical Solution of Ordinary Differential Equations

5.1. Elementary Theory of Initial-Value Prob-

lems
Definition 5.1. The first-order initial value problem (IVP) is for-
mulated as follows: find {yi (x) : i = 1, 2, · · · , M } satisfying
dyi
= fi (x, y1 , y2 , · · · , yM ),
dx i = 1, 2, · · · , M, (5.1)
yi (x0 ) = yi0 ,

for a prescribed initial values {yi0 : i = 1, 2, · · · , M }.

• We assume that (5.1) admits a unique solution in a neighborhood of x0 .

• For simplicity, we consider the case M = 1:
dy
= f (x, y),
dx (5.2)
y(x0 ) = y0 .

Theorem 5.2. (Existence and Uniqueness of the Solution): Sup-

pose that R = {(x, y) | a ≤ x ≤ b, −∞ < y < ∞}, f is continuous on R ,
and x0 ∈ [a, b]. If f satisfies a Lipschitz condition on R in the variable y ,
i.e., there is a constant L > 0 such that

|f (x, y1 ) − f (x, y2 )| ≤ L |y1 − y2 |, ∀ y1 , y2 , (5.3)

then the IVP (5.2) has a unique solution y(x) in an interval I, where
x0 ∈ I ⊂ (a, b).

Theorem 5.3. Suppose that f (x, y) is defined on R ⊂ R2 . If a constant

L > 0 exists with
∂ f (x, y)
≤ L , ∀ (x, y) ∈ R, (5.4)
∂y
then f satisfies a Lipschitz condition on R in the variable y with the
same Lipschitz constant L .
5.1. Elementary Theory of Initial-Value Problems 171

Example 5.4. Prove that the initial-value problem

y 0 = (x + sin y)2 , y(0) = 3,

has a unique solution on the interval [−1, 2].

Solution.

Example 5.5. Show that each of the initial-value problems has a unique
solution and find the solution.

(a) y 0 = e x −y , 0 ≤ x ≤ 1, y(0) = 1
(b) y 0 = (1 + x 2 )y, 3 ≤ x ≤ 5, y(3) = 1

Solution. (Existence and uniqueness):

172 Chapter 5. Numerical Solution of Ordinary Differential Equations

(Find the solution): Here, we will find the solution for (b), using Maple.
Maple-code
1 DE := diff(y(x), x) = y(x)*(x^2 + 1);
2 d / 2 \
3 --- y(x) = y(x) \x + 1/
4 dx
5 IC := y(3) = 1;
6 y(3) = 1
7 dsolve({DE, IC}, y(x));
8 /1 / 2 \\
9 exp|- x \x + 3/|
10 \3 /
11 y(x) = -----------------
12 exp(12)

Strategy 5.6. (Numerical Solution): In the following, we describe

step-by-step methods for (5.2); that is, we start from y0 = y(x0 ) and
proceed stepwise.
• In the first step, we compute y1 which approximate the solution y
of (5.2) at x = x1 = x0 + h , where h is the step size.
• The second step computes an approximate value y2 of the solution
at x = x2 = x0 + 2h , etc..

Note: We first introduce the Taylor-series methods for (5.2), followed

by Runge-Kutta methods and multi-step methods. All of these methods
are applicable straightforwardly to (5.1).
5.2. Taylor-Series Methods 173

5.2. Taylor-Series Methods

Preliminary 5.7. Here we rewrite the initial value problem (IVP):

(
y0 = f (x, y),
(IVP) (5.5)
y(x0 ) = y0 .

For the problem, a continuous approximation to the solution y(x) will

not be obtained; instead, approximations to y will be generated at var-
ious points, called mesh points in the interval [x0 , T ], for some T > x0 .
Let
• h = (T − x0 )/Nt , for an integer Nt ≥ 1
• xn = x0 + nh , n = 0, 1, 2, · · · , Nt
• yn be the approximate solution of y at xn , i.e., yn ≈ y(xn ).

5.2.1. The Euler method

Step 1
• It is to find an approximation of y(x1 ), marching through the first
subinterval [x0 , x1 ] and using a Taylor-series involving only up to the
first-derivative of y .
• Consider the Taylor series

0 h 2 00
y(x + h) = y(x) + hy (x) + y (x) + · · · . (5.6)
2
• Letting x = x0 and utilizing y(x0 ) = y0 and y 0 (x0 ) = f (x0 , y0 ), the value
y(x1 ) can be approximated by

y1 = y0 + h f (x0 , y0 ), (5.7)

where the second- and higher-order terms of h are ignored.

174 Chapter 5. Numerical Solution of Ordinary Differential Equations

Such an idea can be applied recursively for the computation of solution on

later subintervals. Indeed, since
h 2 00
y(x2 ) = y(x1 ) + hy 0 (x1 ) + y (x1 ) + · · · ,
2
by replacing y(x1 ) and y 0 (x1 ) with y1 and f (x1 , y1 ), respectively, we obtain

y2 = y1 + h f (x1 , y1 ), (5.8)

which approximates the solution at x2 = x0 + 2h .

Algorithm 5.8. Summarizing the above, the Euler method solving

the first-order IVP (5.5) is formulated as

yn+1 = yn + h f (xn , yn ), n ≥ 0. (5.9)

Figure 5.1: The Euler method.

Geometrically it is an approximation of the curve {x, y(x)} by a polygon

of which the first segment is tangent to the curve at x0 , as shown in Fig-
ure 5.1. For example, y1 is determined by moving the point (x0 , y0 ) by the
length of h with the slope f (x0 , y0 ).
5.2. Taylor-Series Methods 175

Convergence of the Euler method

Theorem 5.9. Let f satisfy the Lipschitz condition in its second vari-
able, i.e., there is λ > 0 such that

|f (x, y1 ) − f (x, y2 )| ≤ λ|y1 − y2 |, ∀ y1 , y2 . (5.10)

Then, the Euler method is convergent; more precisely,

C
|yn − y(xn )| ≤ h[(1 + λh)n − 1], n = 0, 1, 2, · · · . (5.11)
λ

Proof. The true solution y satisfies

y(xn+1 ) = y(xn ) + hf (xn , y(xn )) + O(h 2 ). (5.12)

Thus it follows from (5.9) and (5.12) that

en+1 = en + h[f (xn , yn ) − f (xn , y(xn ))] + O(h 2 )
= en + h[f (xn , y(xn ) + en ) − f (xn , y(xn ))] + O(h 2 ),

where en = yn − y(xn ). Utilizing (5.10), we have

|en+1 | ≤ (1 + λh)|en | + Ch 2 . (5.13)

Here we will prove (5.11) by using (5.13) and induction. It holds trivially
when n = 0. Suppose it holds for n. Then,

|en+1 | ≤ (1 + λh)|en | + Ch 2
C
≤ (1 + λh) · h[(1 + λh)n − 1] + Ch 2
λ
C
= h[(1 + λh)n+1 − (1 + λh)] + Ch 2
λ
C
= h[(1 + λh)n+1 − 1],
λ
which completes the proof.
176 Chapter 5. Numerical Solution of Ordinary Differential Equations

Example 5.10. Consider

y0 = y − x 3 + x + 1, 0 ≤ x ≤ 3,
(5.14)
y(0) = 0.5.

1 1
As the step lengths become smaller, h = 1 → → , the numerical solu-
4 16
tions represent the exact solution better, as shown in the following figures:

1 1
Figure 5.2: The Euler method, with h = 1 → → .
4 16

Example 5.11. Implement a code for the Euler method to solve

1
y 0 = y − x 3 + x + 1, 0 ≤ x ≤ 3, y(0) = 0.5, with h = .
16
5.2. Taylor-Series Methods 177

Solution.
Euler.mw
1 Euler := proc(f, x0, b, nx, y0, Y)
2 local h, t, w, n:
3 h := (b - x0)/nx:
4 t := x0; w := y0:
5 Y[0] := w:
6 for n to nx do
7 w := w + h*eval(f, [x = t, y = w]);
8 Y[n] := w;
9 t := t + h;
10 end do:
11 end proc:
12

13 # Now, solve it using "Euler"

14 f := -x^3 + x + y + 1:
15 x0 := 0: b := 3: y0 := 0.5:
16

17 nx := 48:
18 YEuler := Array(0..nx):
19 Euler(f, x0, b, nx, y0, YEuler):
20

21 # Check the maximum error

22 DE := diff(y(x), x) = y(x) - x^3 + x + 1:
23 dsolve([DE, y(x0) = y0], y(x))
24 2 3 7
25 y(x) = 4 + 5 x + 3 x + x - - exp(x)
26 2
27 exacty := x -> 4 + 5*x + 3*x^2 + x^3 - 7/2*exp(x):
28 maxerr := 0:
29 h := (b - x0)/nx:
30 for n from 0 to nx do
31 maxerr := max(maxerr,abs(exacty(n*h)-YEuler[n]));
32 end do:
33 evalf(maxerr)
34 0.39169859
178 Chapter 5. Numerical Solution of Ordinary Differential Equations

5.2.2. Higher-order Taylor methods

Preliminary 5.12. Higher-order Taylor methods are based on

Taylor series expansion.
• If we expand the solution y(x), in terms of its mth-order Taylor poly-
nomial about xn and evaluated at xn+1 , we obtain
h 2 00 h m (m)
y(xn+1 ) = y(xn ) + hy 0 (xn ) + y (xn ) + · · · + y (xn )
2! m!
(5.15)
h m+1 (m+1)
+ y (ξn ).
(m + 1)!

• Successive differentiation of the solution, y(x), gives

y 0 (x) = f (x, y(x)), y 00 (x) = f 0 (x, y(x)), · · · ,

and generally,
y (k ) (x) = f (k −1) (x, y(x)). (5.16)

• Thus, we have
h2 0 h m (m−1)
y(xn+1 ) = y(xn ) + hf (xn , y(xn )) + f (xn , y(xn )) + · · · + f (xn , y(xn ))
2! m!
h m+1 (m)
+ f (ξn , y(ξn ))
(m + 1)!
(5.17)

Algorithm 5.13. The Taylor method of order m corresponding to

(5.17) is obtained by deleting the remainder term involving ξn :

yn+1 = yn + h Tm (xn , yn ), (5.18)

where
h 0 h m−1 (m−1)
Tm (xn , yn ) = f (xn , yn ) + f (xn , yn ) + · · · + f (xn , yn ). (5.19)
2! m!
5.2. Taylor-Series Methods 179

Remark 5.14.
• m = 1 ⇒ yn+1 = yn + hf (xn , yn )
which is the Euler method.
h h 0 i
• m = 2 ⇒ yn+1 = yn + h f (xn , yn ) + f (xn , yn )
2
• As m increases, the method achieves higher-order accuracy; how-
ever, it requires to compute derivatives of f (x, y(x)).

Example 5.15. Consider the initial-value problem

y 0 = y − x 3 + x + 1, y(0) = 0.5. (5.20)

(a) Find T3 (x, y).

(b) Perform two iterations to find y2 , with h = 1/2.

Solution. Part (a): Since y 0 = f (x, y) = y − x 3 + x + 1,

f 0 (x, y) = y 0 − 3x 2 + 1
= (y − x 3 + x + 1) − 3x 2 + 1
= y − x 3 − 3x 2 + x + 2

and
f 00 (x, y) = y 0 − 3x 2 − 6x + 1
= (y − x 3 + x + 1) − 3x 2 − 6x + 1
= y − x 3 − 3x 2 − 5x + 2
Thus
h 0 h 2 00
T3 (x, y) = f (x, y) + f (x, y) + f (x, y)
2 6
h
= y − x 3 + x + 1 + (y − x 3 − 3x 2 + x + 2) (5.21)
2
h2
+ (y − x 3 − 3x 2 − 5x + 2)
6
For m large, the computation of Tm is time-consuming and cumbersome.
180 Chapter 5. Numerical Solution of Ordinary Differential Equations

Part (b):
Maple-code
1 T3 := y - x^3 + x + 1 + 1/2*h*(-x^3 - 3*x^2 + x + y + 2)
2 + 1/6*h^2*(-x^3 - 3*x^2 - 5*x + y + 2):
3 h := 1/2:
4 x0 := 0: y0 := 1/2:
5 y1 := y0 + h*eval(T3, [x = x0, y = y0])
6 155
7 ---
8 96
9 y2 := y1 + h*eval(T3, [x = x0 + h, y = y1])
10 16217
11 -----
12 4608
13 evalf(%)
14 3.519314236
15

16 exacty := x -> 4 + 5*x + 3*x^2 + x^3 - 7/2*exp(x):

17 exacty(1)
18 7
19 13 - - exp(1)
20 2
21 evalf(%)
22 3.486013602
23 #The absolute error:
24 evalf(abs(exacty(1) - y2))
25 0.033300634
5.3. Runge-Kutta Methods 181

5.3. Runge-Kutta Methods

Note: What we are going to do is to solve the initial value problem

(IVP): (
y0 = f (x, y),
(IVP) (5.22)
y(x0 ) = y0 .

• The Taylor-series method of the preceding section has the drawback

of requiring the computation of derivatives of f (x, y). This is a
tedious and time-consuming procedure for most cases, which makes
the Taylor methods seldom used in practice.

Definition 5.16. Runge-Kutta methods

• have high-order local truncation error of the Taylor methods, but

• eliminate the need to compute the derivatives of f (x, y).
That is, the Runge-Kutta methods are formulated, incorporating a
weighted average of slopes, as follows:

yn+1 = yn + h (w1 K1 + w2 K2 + · · · + wm Km ) , (5.23)

where
(a) wj ≥ 0 and w1 + w2 + · · · + wm = 1
(b) Kj are recursive evaluations of the slope f (x, y)
(c) Need to determine wj and other parameters to satisfy

w1 K1 + w2 K2 + · · · + wm Km ≈ Tm (xn , yn ) + O(h m ) (5.24)

That is, Runge-Kutta methods evaluate an average slope of f (x, y)

on the interval [xn , xn+1 ] in the same order of accuracy as the mth-
order Taylor method.
182 Chapter 5. Numerical Solution of Ordinary Differential Equations

5.3.1. Second-order Runge-Kutta method

Formulation:
yn+1 = yn + h (w1 K1 + w2 K2 ) (5.25)
where
K1 = f (xn , yn )
K2 = f (xn + αh, yn + β hK1 )
Requirement: Determine w1 , w2 , α, β such that
h 0
w1 K1 + w2 K2 = T2 (xn , yn ) + O(h 2 ) = f (xn , yn ) + f (xn , yn ) + O(h 2 ). (5.26)
2

Derivation: For the left-hand side of (5.25), the Taylor series reads

0 h 2 00
y(x + h) = y(x) + h y (x) + y (x) + O(h 3 ).
2
Since y 0 = f and y 00 = fx + fy y 0 = fx + fy f ,

h2
y(x + h) = y(x) + h f + (fx + fy f ) + O(h 3 ). (5.27)
2

On the other hand, the right-side of (5.25) can be reformulated as

y + h(w1 K1 + w2 K2 )
= y + w1 h f (x, y) + w2 h f (x + αh, y + β h K1 )
= y + w1 h f + w2 h (f + αh fx + β h fy f ) + O(h 3 ),

which reads

y + h(w1 K1 + w2 K2 ) = y + (w1 + w2 )h f + h 2 (w2 αfx + w2 β fy f ) + O(h 3 ). (5.28)

The comparison of (5.27) and (5.28) drives the following result, for the
second-order Runge-Kutta methods.
Results:
1 1
w1 + w2 = 1, w2 α = , w2 β = . (5.29)
2 2
5.3. Runge-Kutta Methods 183

Common Choices:
Algorithm 5.17.
1
I. w1 = w2 = , α = β = 1:
2
Then, the algorithm (5.25) becomes
h
yn+1 = yn + (K1 + K2 ), (5.30)
2
where
K1 = f (xn , yn )
K2 = f (xn + h, yn + h K1 )

This algorithm is the Second-order Runge-Kutta (RK2) method,

which is also known as the Heun’s method.
1
II. w1 = 0, w2 = 1, α = β = :
2
For the choices, the algorithm (5.25) reads
 h h 
yn+1 = yn + h f xn + , yn + f (xn , yn ) (5.31)
2 2
which is also known as the Modified Euler method.

It follows from (5.27) and (5.28) that the local truncation error for the
above Runge-Kutta methods are O(h 3 ). Thus the global error becomes

O(h 2 ). (5.32)
184 Chapter 5. Numerical Solution of Ordinary Differential Equations

5.3.2. Fourth-order Runge-Kutta method

Formulation:

yn+1 = yn + h (w1 K1 + w2 K2 + w3 K3 + w4 K4 ) (5.33)

where
K1 = f (xn , yn )
K2 = f (xn + α1 h, yn + β1 h K1 )
K3 = f (xn + α2 h, yn + β2 h K1 + β3 h K2 )
K4 = f (xn + α3 h, yn + β4 h K1 + β5 h K2 + β6 h K3 )
Requirement: Determine wj , αj , βj such that

w1 K1 + w2 K2 + w3 K3 + w4 K4 = T4 (xn , yn ) + O(h 4 )

The most common choice:

Algorithm 5.18. Fourth-order Runge-Kutta method (RK4): The
most commonly used set of parameter values yields
h
yn+1 = yn + (K1 + 2K2 + 2K3 + K4 ) (5.34)
6
where
K1 = f (xn , yn )
1 1
K2 = f (xn + h, yn + hK1 )
2 2
1 1
K3 = f (xn + h, yn + hK2 )
2 2
K4 = f (xn + h, yn + hK3 )

The local truncation error for the above RK4 can be derived as
h 5 (5)
y (ξn ), (5.35)
5!
for some ξn ∈ [xn , xn+1 ]. Thus the global error reads, for some ξ ∈ [x0 , T ],
(T − x0 )h 4 (5)
y (ξ ). (5.36)
5!
5.3. Runge-Kutta Methods 185

Example 5.19. Use RK4 to solve the initial-value problem

y 0 = y − x 3 + x + 1, y(0) = 0.5. (5.37)

RK4.mw
1 f := proc(x,w)
2 w-x^3+x+1
3 end proc:
4 RK4 := proc(x0,xt,nt,y0,y)
5 local h,x,w,n,K1,K2,K3,K4;
6 h:=(xt-x0)/nt:
7 x:=x0: w:=y0:
8 y[0]:=w;
9 for n from 1 by 1 to nt do
10 K1:=f(x,w);
11 K2:=f(x+h/2,w+(h/2)*K1);
12 K3:=f(x+h/2,w+(h/2)*K2);
13 x:=x+h;
14 K4:=f(x,w+h*K3);
15 w:=w+(h/6.)*(K1+2*K2+2*K3+K4);
16 y[n]:=w;
17 end do
18 end proc:
19

20 x0 := 0: xt := 3: nt := 48: y0 := 0.5:
21 yRK4 := Array(0..nt);
22 RK4(x0,xt,nt,y0,yRK4):
23

24 exacty := x -> 4 + 5*x + 3*x^2 + x^3 - 7/2*exp(x):

25 h := (xt - x0)/nt:
26 maxerr := 0:
27 for n from 0 by 1 to nt do
28 maxerr:=max(maxerr,abs(exacty(n*h)-yRK4[n]));
29 end do:
30 evalf[16](maxerr)
31 0.00000184873274
186 Chapter 5. Numerical Solution of Ordinary Differential Equations

Convergence Test for RK4, with (5.37):

h Max-error Error-ratio

1 /4 4.61 · 10−4

−5 4.61 · 10−4
1 /8 2.93 · 10 = 15.73378840
2.93 · 10−5

−6 2.93 · 10−5
1/16 1.85 · 10 = 15.83783784
1.85 · 10−6

−7 1.85 · 10−6
1/32 1.01 · 10 = 18.31683168
1.01 · 10−7

Thus, the global truncation error of RK4 is in O(h 4 ).

5.3.3. Adaptive methods

Remark 5.20.
• Accuracy of numerical methods can be improved by decreasing
the step size.
• Decreasing the step size ≈ Increasing the computational cost
• There may be subintervals where a relatively large step size suffices
and other subintervals where a small step is necessary to keep the
truncation error within a desired limit.
• An adaptive method is a numerical method which uses a variable
step size.
• Example: Runge-Kutta-Fehlberg method (RKF45), which uses
RK5 to estimate local truncation error of RK4.
5.4. One-Step Methods: Accuracy Comparison 187

5.4. One-Step Methods: Accuracy Comparison

For an accuracy comparison among the one-step methods presented in the

previous sections, consider the motion of the spring-mass system:

κ F0
y 00 (t) + y= cos(µt),
m m (5.38)
y(0) = c0 , y 0 (0) = 0,

where m is the mass attached at the end of a spring of the spring constant
κ, the term F0 cos(µt) is a periodic driving force of frequency µ, and c0 is the
initial displacement from the equilibrium position.
• It is not difficult to find the analytic solution of (5.38):
F0
y(t) = A cos(ω t) + cos(µt), (5.39)
m(ω 2 − µ2 )
p
where ω = κ/m is the angular frequency and the coefficient A is
determined corresponding to c0 .
• Let y1 = y and y2 = −y10 /ω. Then, we can reformulate (5.38) as

y10 = −ω y2 , y0 (0) = c0 ,
F0 (5.40)
y20 = ω y1 − cos(µt), y2 (0) = 0.
mω
We will deal with details of High-Order Equations & Systems of Dif-
ferential Equations in § 5.6 on page 194.
• The motion is periodic only if µ/ω is a rational number. We choose

m = 1, F0 = 40, A = 1, ω = 4π , µ = 2π . (⇒ c0 ≈ 1.33774) (5.41)

Thus the fundamental period of the motion

2π q 2π p
T= = = 1.
ω µ
See Figure 5.3 for the trajectory of the mass satisfying (5.40)-(5.41).
188 Chapter 5. Numerical Solution of Ordinary Differential Equations

Figure 5.3: The trajectory of the mass satisfying (5.40)-(5.41).

Accuracy comparison
Table 5.1: The `2 -error at t = 1 for various time step sizes.

1/h Euler RK2 RK4

100 1.19 3.31E-2 2.61E-5
200 4.83E-1 (1.3) 8.27E-3 (2.0) 1.63E-6 (4.0)
400 2.18E-1 (1.1) 2.07E-3 (2.0) 1.02E-7 (4.0)
800 1.04E-1 (1.1) 5.17E-4 (2.0) 6.38E-9 (4.0)

• Table 5.1 presents the `2 -error at t = 1 for various time step sizes h ,
defined as
h

h
2  h 2 1/2
|yNt − y(1)| = y1,Nt − y1 (1) + y2,Nt − y2 (1) , (5.42)

where yhNt denotes the computed solution at the Nt -th time step with
h = 1 / Nt .
• The numbers in parenthesis indicate the order of convergence α,
defined as
ln(E(2h)/E(h))
α := , (5.43)
ln 2
where E(h) and E(2h) denote the errors obtained with the grid spacing
to be h and 2h , respectively.
5.4. One-Step Methods: Accuracy Comparison 189

• As one can see from the table, the one-step methods exhibit the ex-
pected accuracy.
• RK4 shows a much better accuracy than the lower-order methods,
which explains its popularity.

Definition 5.21. (Order of Convergence): Let’s assume that the

algorithm under consideration produces error in O(h α ). Then, we may
write
E(h) = C h α , (5.44)
where h is the grid size. When the grid size is ph , the error will become

E(ph) = C (ph)α . (5.45)

It follows from (5.44) and (5.45) that

E(ph) C (ph)α
= α
= pα. (5.46)
E(h) Ch
By taking a logarithm, one can solve the above equation for the order
of convergence α:
ln(E(ph)/E(h))
α= . (5.47)
ln p
When p = 2, the above becomes (5.43).
190 Chapter 5. Numerical Solution of Ordinary Differential Equations

5.5. Multi-step Methods

The problem: The first-order initial value problem (IVP)

(
y0 = f (x, y),
(IVP) (5.48)
y(x0 ) = y0 .

Numerical Methods:
• Single-step/Starting methods: Euler’s method, Modified Eu-
ler’s, Runge-Kutta methods
• Multi-step/Continuing methods: Adams-Bashforth-Moulton

Definition 5.22. An m-step method, m ≥ 2, for solving the IVP, is a

difference equation for finding the approximation yn+1 at x = xn+1 , given
by

yn+1 = a1 yn + a2 yn−1 + · · · + am yn+1−m

(5.49)
+h[b0 f (xn+1 , yn+1 ) + b1 f (xn , yn ) + · · · + bm f (xn+1−m , yn+1−m )].

The m-step method is said to be

(
explicit or open, if b0 = 0
implicit or closed, if b0 6= 0
5.5. Multi-step Methods 191

Algorithm 5.23. (Fourth-order multi-step methods):

Let yi0 = f (xi , yi ).
• Adams-Bashforth method (explicit)
h
yn+1 = yn + (55yn0 − 59yn0 −1 + 37yn0 −2 − 9yn0 −3 ) (5.50)
24

• Adams-Moulton method (implicit)

h 0
yn+1 = yn + (9yn+1 + 19yn0 − 5yn0 −1 + yn0 −2 ) (5.51)
24

• Adams-Bashforth-Moulton method (predictor-corrector)

h
∗
yn+1 = yn + (55yn0 − 59yn0 −1 + 37yn0 −2 − 9yn0 −3 )
24 (5.52)
h ∗
yn+1 = yn + (9y 0 n+1 + 19yn0 − 5yn0 −1 + yn0 −2 )
24
where y 0 ∗n+1 = f (xn+1 , yn+1
∗
)

Remark 5.24.
• y1 , y2 , y3 can be computed by RK4.
• Multi-step methods may save evaluations of f (x, y) such that in each
step, they require only one or two new evaluations of f (x, y) to fulfill
the step.
• RK methods are accurate enough and easy to implement, so that
multi-step methods are rarely applied in practice.
• ABM shows a strong stability for special cases, occasionally but not
often [1].
192 Chapter 5. Numerical Solution of Ordinary Differential Equations

ABM.mw
1 ## Maple code: Adams-Bashforth-Moulton (ABM) Method
2 ## Model Problem: y'= y- x^3 + x + 1, y(0) = 0.5, 0 <= x <= 3
3

4 f := proc(x,w)
5 w-x^3+x+1
6 end proc:
7 RK4 := proc(x0,xt,nt,y0,y)
8 local h,x,w,n,K1,K2,K3,K4;
9 h:=(xt-x0)/nt:
10 x:=x0: w:=y0:
11 y[0]:=w;
12 for n from 1 by 1 to nt do
13 K1:=f(x,w);
14 K2:=f(x+h/2,w+(h/2)*K1);
15 K3:=f(x+h/2,w+(h/2)*K2);
16 x:=x+h;
17 K4:=f(x,w+h*K3);
18 w:=w+(h/6.)*(K1+2*K2+2*K3+K4);
19 y[n]:=w;
20 end do
21 end proc:
22

23 ABM:= proc(x0,xt,nt,y0,y)
24 local h,x,w,n,ystar;
25 h:=(xt-x0)/nt:
26 ### Initialization with RK4
27 RK4(x0,x0+3*h,3,y0,y);
28 w:=y[3];
29 ### Now, ABM steps
30 for n from 4 by 1 to nt do
31 x:=x0+n*h;
32 ystar:=w +(h/24)*(55*f(x-h,y[n-1])-59*f(x-2*h,y[n-2])
33 +37*f(x-3*h,y[n-3])-9*f(x-4*h,y[n-4]));
34 w:=w +(h/24)*(9*f(x,ystar)+19*f(x-h,y[n-1])
35 -5*f(x-2*h,y[n-2])+f(x-3*h,y[n-3]));
5.5. Multi-step Methods 193

36 y[n]:=w;
37 end do;
38 end proc:
39

40 x0 := 0: xt := 3: nt := 48: y0 := 0.5:
41 yABM := Array(0..nt);
42 ABM(x0,xt,nt,y0,yABM):
43

44 exacty := x -> 4 + 5*x + 3*x^2 + x^3 - 7/2*exp(x):

45 h := (xt - x0)/nt:
46 maxerr := 0:
47 for n from 0 by 1 to nt do
48 maxerr:=max(maxerr,abs(exacty(n*h)-yABM[n]));
49 end do:
50 evalf[16](maxerr)
51 0.00005294884316

Note: The maximum error for RK4 = 1.85 · 10−6 .

194 Chapter 5. Numerical Solution of Ordinary Differential Equations

5.6. High-Order Equations & Systems of Dif-

ferential Equations

The problem: 2nd-order initial value problem (IVP)

(
y 00 = f (x, y, y 0 ), x ∈ [x0 , T ]
(5.53)
y(x0 ) = y0 , y 0 (x0 ) = u0 ,

An equivalent problem: Let u = y 0 . Then,

u0 = y 00 = f (x, y, y 0 ) = f (x, y, u)

Thus, the above 2nd-order IVP can be equivalently written as the fol-
lowing system of first-order DEs:
(
y 0 = u, y(x0 ) = y0 ,
x ∈ [x0 , T ] (5.54)
u0 = f (x, y, u), u(x0 ) = u0 ,

Remark 5.25.
• The right-side of the DEs involves no derivatives.
• The system (5.54) can be solved by one of the numerical methods (we
have studied), after modifying it for vector functions.
5.6. High-Order Equations & Systems of Differential Equations 195

Example 5.26. Write the following DEs as a system of first-order differ-

ential equations.
( (
y 00 + xy 0 + y = 0, x 00 − x 0 + 2y 00 = e t ,
(a) (b)
y(0) = 1, y 0 (0) = 2. −2x + y 00 + 2y = 3t 2 .

Hint : For (b), you should first rewrite the first equation as x 00 = F(t, x, x 0 ) and introduce x 0 = u
and y 0 = v .

Solution.
196 Chapter 5. Numerical Solution of Ordinary Differential Equations

RK4SYS.mw
1 ## Ex) IVP of 2 equations:
2 ## x' = 2x+4y, x(0)=-1
3 ## y' = -x+6y, y(0)= 6, 0 <= t <= 1
4

5 ef := proc(t,w,f)
6 f(1):=2*w(1)+4*w(2);
7 f(2):=-w(1)+6*w(2);
8 end proc:
9

10 RK4SYS := proc(t0,tt,nt,m,x0,x)
11 local h,t,w,n,j,K1,K2,K3,K4;
12 #### initial setting
13 w:=Vector(m):
14 K1:=Vector(m):
15 K2:=Vector(m):
16 K3:=Vector(m):
17 K4:=Vector(m):
18 h:=(tt-t0)/nt:
19 t:=t0;
20 w:=x0;
21 for j from 1 by 1 to m do
22 x[0,j]:=x0(j);
23 end do;
24 #### RK4 marching
25 for n from 1 by 1 to nt do
26 ef(t,w,K1);
27 ef(t+h/2,w+(h/2)*K1,K2);
28 ef(t+h/2,w+(h/2)*K2,K3);
29 ef(t+h,w+h*K3,K4);
30 w:=w+(h/6.)*(K1+2*K2+2*K3+K4);
31 for j from 1 by 1 to m do
32 x[n,j]:=w(j);
33 end do
34 end do
35 end proc:
5.6. High-Order Equations & Systems of Differential Equations 197

36

37 m := 2:
38 x0 := Vector(m):
39

40 t0 := 0: tt := 1.: nt := 40:
41 x0(1) := -1:
42 x0(2) := 6:
43

44 xRK4 := Array(0..nt, 1..m):

45 RK4SYS(t0,tt,nt,m,x0,xRK4):
46

47 # Compute the analytic solution

48 #--------------------------------
49 ODE := diff(x(t),t)=2*x(t)+4*y(t), diff(y(t),t)=-x(t)+6*y(t):
50 ivs := x(0) = -1, y(0) = 6;
51 dsolve([ODE, ivs]);
52 / 1 \
53 { x(t) = exp(4 t) (-1 + 26 t), y(t) = - exp(4 t) (24 + 52 t) }
54 \ 4 /
55 ex := t -> exp(4*t)*(-1 + 26*t):
56 ey := t -> 1/4*exp(4*t)*(24 + 52*t):
57

58 # Check error
59 #--------------------------------
60 h := (tt - t0)/nt:
61 printf(" n x(n) y(n) error(x) error(y)\n");
62 printf(" -----------------------------------------------\n");
63 for n from 0 by 2 to nt do
64 printf(" \t %5d %10.3f %10.3f %-10.3g %-10.3g\n",
65 n, xRK4[n,1], xRK4[n,2], abs(xRK4[n,1]-ex(n*h)),
66 abs(xRK4[n,2]-ey(n*h)) );
67 end do;
198 Chapter 5. Numerical Solution of Ordinary Differential Equations

Result
1 n x(n) y(n) error(x) error(y)
2 -----------------------------------------------------
3 0 -1.000 6.000 0 0
4 2 0.366 8.122 6.04e-06 4.24e-06
5 4 2.387 10.890 1.54e-05 1.07e-05
6 6 5.284 14.486 2.92e-05 2.01e-05
7 8 9.347 19.140 4.94e-05 3.35e-05
8 10 14.950 25.144 7.81e-05 5.26e-05
9 12 22.577 32.869 0.000118 7.91e-05
10 14 32.847 42.782 0.000174 0.000115
11 16 46.558 55.474 0.000251 0.000165
12 18 64.731 71.688 0.000356 0.000232
13 20 88.668 92.363 0.000498 0.000323
14 22 120.032 118.678 0.000689 0.000443
15 24 160.937 152.119 0.000944 0.000604
16 26 214.072 194.550 0.00128 0.000817
17 28 282.846 248.313 0.00174 0.0011
18 30 371.580 316.346 0.00233 0.00147
19 32 485.741 402.332 0.00312 0.00195
20 34 632.238 510.885 0.00414 0.00258
21 36 819.795 647.785 0.00549 0.0034
22 38 1059.411 820.262 0.00725 0.00447
23 40 1364.944 1037.359 0.00954 0.00586
5.6. High-Order Equations & Systems of Differential Equations 199

RK4SYSTEM.mw
1 ## Ex) y''-2*y'+2*y = exp(2*x)*sin(x), 0 <= x <= 1,
2 ## y(0) = -0.4, y'(0) = -0.6
3 Digits := 20:
4 RK4SYSTEM := proc(a,b,nt,X,F,x0,xn)
5 local h,hh,t,m,n,j,w,K1,K2,K3,K4;
6 #### initial setting
7 with(LinearAlgebra):
8 m := Dimension(Vector(F));
9 w :=Vector(m);
10 K1:=Vector(m);
11 K2:=Vector(m);
12 K3:=Vector(m);
13 K4:=Vector(m);
14 h:=(b-a)/nt; hh:=h/2;
15 t :=a;
16 w:=x0;
17 for j from 1 by 1 to m do
18 xn[0,j]:=x0[j];
19 end do;
20 #### RK4 marching
21 for n from 1 by 1 to nt do
22 K1:=Vector(eval(F,[x=t,seq(X[i+1]=xn[n-1,i], i = 1..m)]));
23 K2:=Vector(eval(F,[x=t+hh,seq(X[i+1]=xn[n-1,i]+hh*K1[i], i = 1..m)]));
24 K3:=Vector(eval(F,[x=t+hh,seq(X[i+1]=xn[n-1,i]+hh*K2[i], i = 1..m)]));
25 t:=t+h;
26 K4:=Vector(eval(F,[x=t,seq(X[i+1]=xn[n-1,i]+h*K3[i], i = 1..m)]));
27 w:=w+(h/6)*(K1+2*K2+2*K3+K4);
28 for j from 1 by 1 to m do
29 xn[n,j]:=evalf(w[j]);
30 end do
31 end do
32 end proc:
33

34 # Call RK4SYSTEM.mw
35 #--------------------------------
36 with(LinearAlgebra):
37 m := 2:
38 F := [yp, exp(2*x)*sin(x) - 2*y + 2*yp]:
39 X := [x, y, yp]:
40 X0 := <-0.4, -0.6>:
41 a := 0: b := 1: nt := 10:
42 Xn := Array(0..nt, 1..m):
43 RK4SYSTEM(a, b, nt, X, F, X0, Xn):
200 Chapter 5. Numerical Solution of Ordinary Differential Equations

44

45 # Compute the analytic solution

46 #--------------------------------
47 DE := diff(y(x), x, x) - 2*diff(y(x), x) + 2*y(x) = exp(2*x)*sin(x):
48 IC := y(0) = -0.4, D(y)(0) = -0.6:
49 dsolve({DE, IC}, y(x))
50 1
51 y(x) = - exp(2 x) (sin(x) - 2 cos(x))
52 5
53 ey := x -> 1/5*exp(2*x)*(sin(x) - 2*cos(x))
54 diff(ey(x), x)
55 2 1
56 - exp(2 x) (sin(x) - 2 cos(x)) + - exp(2 x) (2 sin(x) + cos(x))
57 5 5
58 eyp:=x->2/5*exp(2*x)*(sin(x)-2*cos(x))+1/5*exp(2*x)*(2*sin(x) + cos(x)):
59

60 # Check error
61 #--------------------------------
62 printf(" n y_n y(x_n) y'_n y'(x_n) err(y) err(y')\n");
63 printf("--------------------------------------------------------\n");
64 for n from 0 to nt do
65 xp := h*n + a;
66 printf(" %2d %12.8f %12.8f %12.8f %12.8f %.3g %.3g\n",
67 n, Xn[n, 1], ey(xp), Xn[n, 2], eyp(xp),
68 abs(Xn[n, 1] - ey(xp)), abs(Xn[n, 2] - eyp(xp)));
69 end do:

Result
1 n y_n y(x_n) y'_n y'(x_n) err(y) err(y')
2 ---------------------------------------------------------------------------------------
3 0 -0.40000000 -0.40000000 -0.60000000 -0.60000000 0 0
4 1 -0.46173334 -0.46173297 -0.63163124 -0.63163105 3.72e-07 1.91e-07
5 2 -0.52555988 -0.52555905 -0.64014895 -0.64014866 8.36e-07 2.84e-07
6 3 -0.58860144 -0.58860005 -0.61366381 -0.61366361 1.39e-06 1.99e-07
7 4 -0.64661231 -0.64661028 -0.53658203 -0.53658220 2.02e-06 1.68e-07
8 5 -0.69356666 -0.69356395 -0.38873810 -0.38873905 2.71e-06 9.58e-07
9 6 -0.72115190 -0.72114849 -0.14438087 -0.14438322 3.41e-06 2.35e-06
10 7 -0.71815295 -0.71814890 0.22899702 0.22899243 4.06e-06 4.59e-06
11 8 -0.66971133 -0.66970677 0.77199180 0.77198383 4.55e-06 7.97e-06
12 9 -0.55644290 -0.55643814 1.53478148 1.53476862 4.77e-06 1.29e-05
13 10 -0.35339886 -0.35339436 2.57876634 2.57874662 4.50e-06 1.97e-05
5.6. High-Order Equations & Systems of Differential Equations 201

Exercises for Chapter 5

5.1. Show that the initial-value problem

x 0 (t) = tan(x), x(0) = 0

has a unique solution in the interval |t | ≤ π/4. Can you find the solution, by guess-
ing?

5.2. C Use Taylor’s method of order m to approximate the solution of the following
initial-value problems.
(a) y 0 = e x −y , 0 ≤ x ≤ 1; y(0) = 1, with h = 0.25 and m = 2.
(b) y 0 = e x −y , 0 ≤ x ≤ 1; y(0) = 1, with h = 0.5 and m = 3.
sin x − 2xy
(c) y 0 = 2
1 ≤ x ≤ 2; y(1) = 2, with h = 0.5 and m = 4.
x
5.3. (Do not use computer programming for this problem.) Consider the initial-value
problem:
= 1 + (x − y)2 , 2 ≤ x ≤ 3,
 0
y
(5.55)
y(2) = 1,
of which the actual solution is y(x) = x + 1/(1 − x). Use h = 1/2 and a calculator to get
the approximate solution at x = 3 by applying
(a) Euler’s method
(b) RK2
(c) Modified Euler method
(d) RK4
Then, compare their results with the actual value y(3) = 2.5.

5.4. C Now, solve the problem in the preceding exercise, (5.55), by implementing
(a) RK4
(b) Adams-Bashforth-Moulton method
Use h = 0.05 and compare the accuracy.

5.5. C Consider the following system of first-order differential equations:

 0
 u1 = u2 − u3 + t, u1 (0) = 1,
u20 = 3t 2 , u2 (0) = 1, (0 ≤ t ≤ 1) (5.56)
 0 −t
u3 = u2 + e , u3 (0) = −1,

The actual solution is

u1 (t) = −t 5 /20 + t 4 /4 + t + 2 − e −t
u2 (t) = t 3 + 1
u3 (t) = t 4 /4 + t − e −t
202 Chapter 5. Numerical Solution of Ordinary Differential Equations

Use RK4SYSTEM to approximate the solution with h = 0.2, 0.1, 0.05, and compare
the errors to see if you can conclude that the algorithm is a fourth-order method for
systems of differential equations.
Chapter 6

Gauss Elimination and Its Variants

One of the most frequently occurring problems in all areas of scientific

endeavor is that of solving a system of n linear equations in n unknowns.
The main subject of this chapter is to study the use of Gauss elimination
to solve such systems. We will see that there are many ways to organize
this fundamental algorithm.

203
204 Chapter 6. Gauss Elimination and Its Variants

6.1. Systems of Linear Equations

Note: Consider a system of n linear equations in n unknowns



 a11 x1 + a12 x2 + · · · + a1n xn = b1

 a x + a x + ··· + a x = b
21 1 22 2 2n n 2
.
. .
. (6.1)


 . .
an1 x1 + an2 x2 + · · · + ann xn = bn


Given the coefficients aij and the source bi , we wish to find [x1 , x2 , · · · , xn ]
which satisfy the equations.
• Since it is tedious to write (6.1) again and again, we generally prefer
to write it as a single matrix equation
A x = b, (6.2)

where
     
a11 a12 · · · a1n x1 b1
a a · · · a  x  b 
 21 22 2n   2  2
A = . .. ..  , x=.  , and b =  ..  .
 . . . .   ..  .
an1 an2 · · · ann xn bn

Solvers for Linear Systems:

• Direct algebraic solvers

– LU , LL T , LDL T , QR , SVD , SuperLU, · · · (factorization)

– Harder to optimize and parallelize
– Numerically robust, but higher algorithmic complexity
• Iterative algebraic solvers

– Stationary and nonstationary methods

(Jacobi, Gauss-Seidel, SOR, SSOR; CG, MINRES, GMRES, BiCG, QMR, · · · )
– Easier to optimize and parallelize
– Low algorithmic complexity, but may not converge
6.1. Systems of Linear Equations 205

6.1.1. Nonsingular matrices

Definition 6.1. (Definition 1.36) An n × n matrix A is said to be

invertible (nonsingular) if there is an n × n matrix B such that AB =
In = BA , where In is the identity matrix.

Note: In this case, B is the unique inverse of A denoted by A −1 .

(Thus AA −1 = In = A −1 A .)
Theorem 6.2. (Invertible Matrix Theorem; Theorem 1.41) Let A
be an n × n matrix. Then the following are equivalent.
a. A is an invertible matrix.
b. A is row equivalent to the n × n identity matrix.
c. A has n pivot positions.
d. The columns of A are linearly independent.
e. The equation A x = 0 has only the trivial solution x = 0.
f. The equation A x = b has unique solution for each b ∈ Rn .
g. The linear transformation x 7→ A x is one-to-one.
h. The linear transformation x 7→ A x maps Rn onto Rn .
i. There is a matrix C ∈ Rn×n such that CA = I
j. There is a matrix D ∈ Rn×n such that AD = I
k. A T is invertible and (A T )−1 = (A −1 )T .
l. The number 0 is not an eigenvalue of A .
m. det A 6= 0.

Example 6.3. Let A ∈ Rn×n and eigenvalues of A be λi , i = 1, 2, · · · , n.

Show that
n
Y
det (A ) = λi . (6.3)
i=1

Thus we conclude that A is singular if and only if 0 is an eigenvalue of A .

Hint : Consider the characteristic polynomial of A , φ(λ) = det (A − λI), and φ(0). See Remark 1.50.
206 Chapter 6. Gauss Elimination and Its Variants

6.1.2. Numerical solutions of differential equations

Consider the following differential equation:

(a) −uxx + cu = f , x ∈ (ax , bx ),

(b) −ux + β u = g, x = ax , (6.4)
(c) ux + β u = g, x = bx .

where
c ≥ 0 and β ≥ 0 (c + β > 0).

Numerical discretization:

• Select nx equally spaced grid points on the interval [ax , bx ]:

bx − ax
xi = ax + ihx , i = 0, 1, · · · , nx , hx = .
nx

• Let ui = u(xi ), for i = 0, 1, · · · , nx .

• It follows from the Taylor’s series expansion that
−ui −1 + 2ui − ui+1 uxxxx (xi ) 2
−uxx (xi ) = + hx + · · · .
hx2 12
Thus the central second-order finite difference (FD) scheme for uxx at
xi reads
−ui −1 + 2ui − ui+1
−uxx (xi ) ≈ 2
. (6.5)
hx
See also (4.14).
• Apply the FD scheme for (6.4.a) to have

−ui −1 + (2 + hx2 c)ui − ui+1 = hx2 fi , i = 0, 1, · · · , nx . (6.6)

• However, we will meet ghost grid values at the end points. For example,
at the point ax = x0 , the formula becomes

−u−1 + (2 + hx2 c)u0 − u1 = hx2 f0 . (6.7)

Here the value u−1 is not defined and we call it a ghost grid value.
6.1. Systems of Linear Equations 207

• Now, let’s replace the ghost grid value u−1 by using the boundary con-
dition (6.4.b). The central FD scheme for ux at x0 can be formulated as

u 1 − u −1 uxxx (x0 ) 2
ux (x0 ) ≈ , Trunc.Err = − hx + · · · . (6.8)
2hx 6
Thus the equation (6.4.b), −ux + β u = g , can be approximated (at x0 )

u−1 + 2hx β u0 − u1 = 2hx g0 . (6.9)

• Hence it follows from (6.7) and (6.9) that

(2 + hx2 c + 2hx β )u0 − 2u1 = hx2 f0 + 2hx g0 . (6.10)

The same can be considered for the algebraic equation at the point xn .

Scheme 6.4. The problem (6.4) is reduced to finding the solution u

satisfying
A u = b, (6.11)
where A ∈ R(nx +1)×(nx +1) ,
 2

2 + hx c + 2hx β −2

 −1 2 + hx2 c −1 

 ... ... ... 
A =

,

−1 2 + hx2 c −1
 
 
−2 2 + hx2 c + 2hx β

and    
hx2 f0 2hx g0
hx2 f1
   
   0 

b= ..  
+ .. 
.
 .   . 
 2
 hx fnx −1 0
  
  
hx2 fnx 2hx gnx

Definition 6.5. Such a technique of removing ghost grid values is

called outer bordering.
208 Chapter 6. Gauss Elimination and Its Variants

Dirichlet Boundary Condition:

Scheme 6.6. When the boundary values of the DE are known

(Dirichlet boundary condition), the algebraic system does not have
to include rows corresponding to the nodal points.
• However, it is more reusable if the algebraic system incorporates rows
for all nodal points.
• For example, consider

(a) −uxx + cu = f , x ∈ (ax , bx ),

(b) −ux + β u = g, x = ax , (6.12)
(c) u = ud , x = bx .

• Then, the corresponding algebraic system can be formulated as

A 0 u = b0 , (6.13)

where A 0 ∈ R(nx +1)×(nx +1) ,

 2

2 + hx c + 2hx β −2

 −1 2 + hx2 c −1 

... ... ...
A0 = 
 
,
 
−1 2 + hx2 c −1 
 

0 1

and    
hx2 f0 2hx g0
hx2 f1
   
   0 
b0 = 
 ..  
+ .. 
.
 .   . 
 2
 hx fnx −1 0
  
  
ud 0
6.2. Triangular Systems 209

6.2. Triangular Systems

Definition 6.7.

(a) A matrix L = (`ij ) ∈ Rn×n is lower triangular if

`ij = 0 whenever i < j.

(b) A matrix U = (uij ) ∈ Rn×n is upper triangular if

uij = 0 whenever i > j.

Theorem 6.8. Let G be a triangular matrix. Then G is nonsingular

if and only if gii 6= 0 for i = 1, · · · , n.

6.2.1. Lower-triangular systems

Consider the n × n system
L y = b, (6.14)
where L is a nonsingular, lower-triangular matrix (`ii 6= 0). It is easy to see
how to solve this system if we write it in detail:
`11 y1 = b1
`21 y1 + `22 y2 = b2
`31 y1 + `32 y2 + `33 y3 = b3 (6.15)
.. ..
. .
`n1 y1 + `n2 y2 + `n3 y3 + · · · + `nn yn = bn

The first equation involves only the unknown y1 , which can be found as

y1 = b1 /`11 . (6.16)

With y1 just obtained, we can determine y2 from the second equation:

y2 = (b2 − `21 y1 )/`22 . (6.17)

Now with y2 known, we can solve the third equation for y3 , and so on.
210 Chapter 6. Gauss Elimination and Its Variants

Algorithm 6.9. In general, once we have y1 , y2 , · · · , yi −1 , we can solve

for yi using the i th equation:

yi = (bi − `i1 y1 − `i2 y2 − · · · − `i,i −1 yi −1 )/`ii

i −1
1 X  (6.18)
= bi − `ij yj
`ii
j=1

Matlab-code 6.10. (Forward Substitution/Elimination):

for i=1:n
for j=1:i-1
b(i) = b(i)-L(i,j)*b(j)
end (6.19)
if L(i,i)==0, set error flag, exit
b(i) = b(i)/L(i,i)
end

The result is y .
Computational complexity: For each i , the forward substitution re-
quires 2(i − 1) + 1 flops. Thus the total number of flops becomes
n
X n
X
{2(i − 1) + 1} = {2i − 1} = n(n + 1) − n = n2 . (6.20)
i=1 i=1
6.2. Triangular Systems 211

6.2.2. Upper-triangular systems

Consider the system
U x = y, (6.21)
where U = (uij ) ∈ Rn×n is nonsingular, upper-triangular. Writing it out in
detail, we get

u11 x1 + u12 x2 + · · · + u1,n−1 xn−1 + u1,n xn = y1

u22 x2 + · · · + u2,n−1 xn−1 + u2,n xn = y2
.. .. (6.22)
. = .
un−1,n−1 xn−1 + un−1,n xn = yn−1
un,n xn = yn

It is clear that we should solve the system from bottom to top.

Matlab-code 6.11. (Back Substitution):

for i=n:-1:1
if(U(i,i)==0), error(’U: singular!’); end
x(i)=b(i)/U(i,i); (6.23)
b(1:i-1)=b(1:i-1)-U(1:i-1,i)*x(i);
end

Computational complexity: n2 + O(n) flops.

212 Chapter 6. Gauss Elimination and Its Variants

6.3. Gauss Elimination

— a very basic algorithm for solving A x = b

The algorithms developed here produce (in the absence of rounding errors)
the unique solution of A x = b, whenever A ∈ Rn×n is nonsingular.

Strategy 6.12. (Gauss elimination):

• First, transform the system A x = b to an equivalent system Ux = y,
where U is upper-triangular;
• then Further transform the system Ux = y to get x.
– It is convenient to represent A x = b by an augmented matrix [A |b];
each equation in A x = b corresponds to a row of the augmented
matrix.
– Transformation of the system: By means of three elementary
row operations, applied on the augmented matrix.

Definition 6.13. Elementary row operations (EROs).

Replacement: Ri ← Ri + αRj (i 6= j)
Interchange: Ri ↔ Rj (6.24)
Scaling: Ri ← β Ri (β 6= 0)

Proposition 6.14.

(a) If [Â | b̂] is obtained from [A | b] by elementary row operations

(EROs), then systems [A | b] and [Â | b̂] represent the same solution.
(b) Suppose  is obtained from A by EROs. Then  is nonsingular if
and only if A is.
(c) Each ERO corresponds to left-multiple of an elementary matrix.
(d) Each elementary matrix is nonsingular.
(e) The elementary matrices corresponding to “Replacement" and
“Scaling" operations are lower triangular.
6.3. Gauss Elimination 213

6.3.1. The LU factorization/decomposition

The LU factorization is motivated by the fairly common industrial and
business problem of solving a sequence of equations, all with the
same coefficient matrix:

A x = b1 , A x = b2 , · · · , A x = bp . (6.25)

Definition 6.15. Let A ∈ Rm×n . The LU factorization of A is A = LU ,

where L ∈ Rm×m is a unit lower triangular matrix and U ∈ Rm×n is an
echelon form of A (upper triangular matrix):

Figure 6.1

Remark 6.16. Let A x = b be to be solved. Then A x = LU x = b and it

can be solved as (
L y = b,
(6.26)
Ux = y,
each algebraic equation can be solved effectively, via substitutions.
214 Chapter 6. Gauss Elimination and Its Variants
 
1 −2 1
Example 6.17. Let A =  2 −2 3.
 
−3 2 0

a) Reduce it to an echelon matrix, using replacement operations.

b) Express the replacement operations as elementary matrices.
c) Find their inverse.
6.3. Gauss Elimination 215

Algorithm 6.18. (An LU Factorization Algorithm) The derivation

introduces an LU factorization: Let A ∈ Rm×n . Then
A = Im A
= E1−1 E1 A
= E1−1 E2−1 E2 E1 A = (E2 E1 )−1 E2 E1 A
. (6.27)
= ..
= E1−1 E2−1 · · · Ep−1 Ep · · · E2 E1 A = (Ep · · · E2 E1 )−1 Ep · · · E2 E1 A
| {z } | {z }| {z }
an echelon form L U

Remark 6.19. Let E1 and E2 be elementary matrices that correspond

to replacement operations occurred in the LU Factorization Algorithm.
Then E1 E2 = E2 E1 and E1−1 E2−1 = E2−1 E1−1 .

Example 6.20. Find the LU factorization of

 
4 3 −5
A = −4 −5 7.
 
8 8 −7
216 Chapter 6. Gauss Elimination and Its Variants

Theorem 6.21. (LU Decomposition Theorem) The following are

equivalent.
1. All leading principal submatrices of A are nonsingular. (The j th
leading principal submatrix is A (1 : j, 1 : j).)
2. There exists a unique unit lower triangular L and nonsingular
upper-triangular U such that A = LU .

Proof. (2) ⇒ (1): A = LU may also be written

" # " #" # " #
A11 A12 L11 0 U11 U12 L11 U11 L11 U12
= = ,(6.28)
A21 A22 L21 L22 0 U22 L21 U11 L21 U12 + L22 U22
where A11 is a j × j leading principal submatrix. Thus
j
Y
det (A11 ) = det (L11 U11 ) = 1 · det (U11 ) = (U11 )kk 6= 0.
k =1

Here we have used the assumption that U is nonsingular and so is U11 .

(1) ⇒ (2): It can be proved by induction on n.
 
3 −1 1
Example 6.22. Find the LU factorization of A =  9 1 2 .
 
−6 5 −5
Solution. (Practical Implementation):
   
3 −1 1 3 −1 1
 R2 ←R2 −3R1 
A =  9 1 2  −−−−−−−→  3 4 −1 
 
R3 ←R3 +2R1
−6 5 −5 -2 3 −3
 
3 −1 1
R3 ←R3 − 34 R2 
−−−−−−−→  3 4 −1 

-2 3
4
− 49
   
1 0 0 3 −1 1
L =  3 1 0 , U=0 4 −1  .
   
−2 34 1 0 0 − 49
Note: it is easy to verify that A = LU .
6.3. Gauss Elimination 217

Example 6.23. Find the LU factorization of

 
2 −1
6 5
 
A = .

 −10 3 
12 −2
Solution.
     
2 −1 2 −1 2 −1
 6 5

 R2 ←R2 −3R1 
 3 8  R3 ←R3 + 4 R2  3
 1  8 
 −−−−−−−→   −−−−−−1−→ 
 
1
 −10  R3 ←R3 +5R1  −5 −2  R4 ←R4 − 2 R2  −5 − 4 
 
3
R4 ←R4 −6R1
1
12 −2 6 4 6 2

   
1 0 0 0 2 −1
3 1 0 0 0 8
   
L = , U=  .
  
−5 − 14

 1 0   0 0
1
6 2
0 1 4×4
0 0 4×2

Note: U has the same size as A , that is, its size is 4 × 2. L is a square
matrix and is a unit lower triangular matrix.

Remark 6.24. For an n × n dense matrix A (with most entries

nonzero) with n moderately large.
• Computing an LU factorization of A takes about 2n3 /3 flops† (∼ row
reducing [A b]), while finding A −1 requires about 2n3 flops.
• Solving L y = b and Ux = y requires about 2n2 flops, because any n × n
triangular system can be solved in about n2 flops.
• Multiplying b by A −1 also requires about 2n2 flops, but the result may
not as accurate as that obtained from L and U (due to round-off errors
in computing A −1 & A −1 b).
• If A is sparse (with mostly zero entries), then L and U may be sparse,
too. On the other hand, A −1 is likely to be dense. In this case, a
solution of A x = b with LU factorization is much faster than using
A −1 .
†
A flop is a floating point operation by +, −, × or ÷.
218 Chapter 6. Gauss Elimination and Its Variants

6.3.2. Solving linear systems by LU factorization

• The LU factorization can be applied for general m × n matrices:

   
1 0 0 0  ∗ ∗ ∗ ∗
∗ 1 0 0  0  ∗ ∗ ∗
A = (6.29)
   
∗ ∗ 1 0  0 0 0  ∗
  

∗ ∗ ∗ 1 0 0 0 0 0
| {z } | {z }
L U

• Let A ∈ Rn×n be nonsingular. If A = LU , where L is a unit lower-

triangular matrix and U is an upper-triangular matrix, then
(
Ly = b
A x = b ⇐⇒ (LU)x = L (Ux) = b ⇐⇒ (6.30)
Ux = y

In the following couple of examples, LU is given.

   
1 4 −2 −12
Example 6.25. Let A =  2 5 −3 , b = −14;
   
−3 −18 16 64
  
1 0 0 1 4 −2
∆ 
A = LU =  2 1 0   0 −3 1 .
 
−3 2 1 0 0 8

Use the LU factorization of A to solve A x = b.

Solution. From (6.30), we know there are two steps to perform:

(1) Solve L y = b for y;

(2) Solve Ux = y for x.

(1) Solve L y = b for y by row reduction

 ..   .. 
1 0 0 . −12 1 0 0 . −12
.. . .. .
[L . b] =  2 1 0 .. −14  → · · · →  0 1 0 . 10  = [I .. y]. (6.31)
   
. ..
−3 2 1 .. 64 0 0 1 . 8
6.3. Gauss Elimination 219

(2) Solve Ux = y for x by row reduction

. ..
4 −2 .. −12
   
1 1 0 0 . 2
.. . .. .
[U . y] =  0 −3 1 .. 10  → · · · →  0 1 0 . −3  = [I .. x]. (6.32)
   
. ..
0 0 8 .. 8 0 0 1 . 1

Thus, x = [2, −3, 1]T .

   
5 4 −2 −3 −10
2 −10
 15 13
 −29
Example 6.26. Let A =  , b= ;
   
 −5 −1 28 3   30 
10 10 8 −8 −22
  
1 0 0 0 5 4 −2 −3
3 1 0 0 8 −1 
 0 1
∆ 
 
A = LU =  .

−1

 3 1 0  0 0 2 3
2 2 −2 1 0 0 0 6

Use the LU factorization of A to solve A x = b.

Solution.
(1) Solve L y = b for y:
 ..   .. 
1 0 0 0 . −10 1 0 0 0 . −10
 ..   .. 
.  3 1 0 0 . −29  → ··· →  0 1 0 0 . 1 = [I ... y].
[L .. b] = 
 
 −1 3 .. ..
 1 0 . 30 

0 0 1 0
 . 17 
.. ..
2 2 −2 1 . −22 0 0 0 1 . 30

(2) Solve Ux = y for x:

 ..   .. 
5 4 −2 −3 . −10 1 0 0 0 . 3

 0 1 8 −1 ..   .. 
. . 1  → ··· →  0 1 0 0 . −2  = [I ... x].
[U .. y] = 
 
.. ..
0 0 2 3
 . 17 

0 0 1 0
 . 1 
. ..
0 0 0 6 .. 30 0 0 0 1 . 5

Thus, x = [3, −2, 1, 5]T .

220 Chapter 6. Gauss Elimination and Its Variants

6.3.3. Gauss elimination with pivoting

Definition 6.27. A permutation matrix is a matrix that has exactly

one 1 in each row and in each column, all other entries being zero.

Self-study 6.28. Show that if P is permutation matrix, then P T P = PP T =

I. Thus P is nonsingular and

P −1 = P T .

Solution.

Lemma 6.29. Let P and Q be n × n permutation matrices and A ∈

Rn × n . Then
(a) PA is A with its rows permuted
A P is A with its columns permuted.
(b) det (P) = ±1.
(c) PQ is also a permutation matrix.

Example 6.30. Let A ∈ Rn×n , and let Ab be a matrix obtained from scram-
bling the rows of A . Show that there is a unique permutation matrix
P ∈ Rn×n such that A
b = PA .
Hint: Consider the row indices in the scrambled matrix Ab , say {k1 , k2 , · · · , kn }.
(This means that for example, the first row of A
b is the same as the k1 -th
row of A .) Use the index set to define a permutation matrix P .
Proof. (Self-study)
6.3. Gauss Elimination 221

Theorem 6.31. Gauss elimination with partial pivoting, applied

to A ∈ Rn×n , produces a unit lower-triangular matrix L with |`ij | ≤ 1, an
upper-triangular matrix U , and a permutation matrix P such that

A
b = PA = LU (6.33)

or, equivalently,
A = P T LU (6.34)

Note: If A is singular, then so is U .

Algorithm 6.32. Solving A x = b using Gauss elimination with partial

pivoting:
1. Factorize A into A = P T LU , where
P = permutation matrix,
L = unit lower triangular matrix
(i.e., with ones on the diagonal),
U = nonsingular upper-triangular matrix.

2. Solve P T LUx = b
(a) LUx = Pb (permuting b)
(b) Ux = L −1 (Pb) (forward substitution)
(c) x = U −1 (L −1 Pb) (back substitution)

In practice:
) (
A x = b ⇐⇒ P T (LU)x = b L y = Pb
⇐⇒
⇐⇒ L (Ux) = Pb Ux = y
222 Chapter 6. Gauss Elimination and Its Variants

Theorem 6.33. If A is nonsingular, then there exist permutations

P1 and P2 , a unit lower triangular matrix L , and a nonsingular upper-
triangular matrix U such that

P1 AP2 = LU.

Only one of P1 and P2 is necessary.

Remark 6.34. P1 A reorders the rows of A , AP2 reorders the columns,

and P1 AP2 reorders both. Consider
" # " #" #
a11 A12 1 0 u11 U12
P10 AP20 = =
A21 A22 L21 I 0 Ã22
" #
u11 U12
= (6.35)
L21 u11 L21 U12 + Ã22

• We can choose P20 = I and P10 so that a11 is the largest entry in absolute
value in its column, which implies L21 = Aa1121 has entries bounded by 1
in modulus.
• More generally, at step k of Gaussian elimination, where we are com-
puting the k th column of L , we reorder the rows so that the largest
entry in the column is on the pivot. This is called Gaussian elimina-
tion with partial pivoting, or GEPP for short. GEPP guarantees
that all entries of L are bounded by one in modulus.

Remark 6.35. We can choose P1 and P2 so that a11 in (6.35) is the

largest entry in modulus in the whole matrix. More generally, at step k
of Gaussian elimination, we reorder the rows and columns so that the
largest entry in the matrix is on the pivot. This is called Gaussian
elimination with complete pivoting, or GECP for short.
6.3. Gauss Elimination 223
 
3 −1 1
Example 6.36. Find the LU factorization of A =  9 1 2 , which
 
−6 5 −5
is considered in Example 6.22.
Solution. (Without pivoting)
   
3 −1 1 3 −1 1
 R2 ←R2 −3R1 
A =  9 1 2  −−−−−−−→  3 4 −1 
 
R3 ←R3 +2R1
−6 5 −5 -2 3 −3
 
3 −1 1
R3 ←R3 − 34 R2 
−−−−−−−→  3 4 −1 

-2 3
4
− 49
   
1 0 0 3 −1 1
L =  3 1 0 , U=0 4 −1  .
   
−2 34 1 0 0 − 49
(With partial pivoting)
   
3 −1 1 9 1 2
 R1 ↔R2 
A = 9 1 2  −−−−→  3 −1 1
 
−6 5 −5 −6 5 −5
   
9 1 2 9 1 2
R2 ←R2 − 31 R1   R2 ↔R3  2 17
−−−−−−2−→  13 − 43 1
 −−−−→  − 3 3 − 11

3 3 
R3 ←R3 + 3 R1
− 23 173
− 113
1
3
− 43 1
3
 
9 1 2
4
R3 ←R3 + 17 R2  17 R1 ↔R2 R2 ↔R3
−−−−−−−→  − 3 2
− 11 , I −−−−→ E −−−−→ P

3 3 
1 4 9
3
− 17 − 17

PA = LU  
9
   
0 1 0 1 0 0 1 2
17
− 11
 
P = 0 0 1 , L =  − 23 1 0 , U= 0
    
 3 3 
1 4
1 0 0 3
− 17 1 0 0 9
− 17
224 Chapter 6. Gauss Elimination and Its Variants

6.3.4. Calculating A −1

Algorithm 6.37. (The computation of A −1 ):

• The program to solve Ax = b can be used to calculate the inverse of a
matrix. Letting X = A −1 , we have

AX = I. (6.36)

• This equation can be written in partitioned form:

A [x1 x2 · · · xn ] = [e1 e2 · · · en ], (6.37)

where x1 , x2 , · · · , xn and e1 , e2 , · · · , en are columns of X and I, respec-

tively.
• Thus AX = I is equivalent to the n equations

A xi = ei , i = 1, 2, · · · , n. (6.38)

Solving these n systems by Gauss elimination with partial pivoting,

we obtain A −1 .

Computational complexity
• A naive flop count:
2 3
LU -factorization of A : n + O(n2 )
3
Solve for n equations in (6.38): n · 2n2 = 2n3
8 3
Total cost: n + O(n2 )
3
• A modification: The forward-substitution phase requires the solu-
tion of
L yi = ei , i = 1, 2, · · · , n. (6.39)
Some operations can be saved by exploiting the leading zeros in ei .
(For each i , the portion of L to be accessed is triangular.) With these
savings, one can conclude that A −1 can be computed in 2n3 + O(n2 )
flops.
6.3. Gauss Elimination 225

Exercises for Chapter 6

6.1. Solve the equation A x = b by using the LU factorization.

3 −5
   
4 2
A = −4 −5 7 and b =  −4.
8 6 −8 6

(Do not use computer programming for this problem.)  

1/4
Answer: x =  2.
1
6.2. Let L = [`ij ] and M = [mij ] be lower-triangular matrices.

(a) Prove that LM is lower triangular.

(b) Prove that the entries of the main diagonal of LM are

`11 m11 , `22 m22 , · · · , `nn mnn

Thus the product of two unit lower-triangular matrices is unit lower triangular.

6.3. C Consider the system A x = b, where

   
1 −2 −1 3 −12
 1 −2 0 1  −5 
   
A =  and b =  .

 −3 −2 1 7  −14
0 −2 8 5 −7

(a) Perform LU decomposition with partial pivoting for A to show P , L , and U .

(b) Solve the system.

(You may use any built-in functions for this problem.)

6.4. C Consider the finite difference method on uniform meshes to solve

(a) −uxx + u = (π 2 + 1) cos(π x), x ∈ (0, 1),
(6.40)
(b) u(0) = 1 and ux (1) = 0.

(a) Implement a function to construct algebraic systems in the full matrix form, for
general nx ≥ 1.
(b) Use a direct method (e.g., A \b ) to find approximate solutions for nx = 25, 50, 100.
(c) The actual solution for (6.40) is u(x) = cos(π x). Measure the maximum errors for
the approximate solutions.

(This problem is optional for undergraduate students; you will get an extra credit
when you solve it.)
226 Chapter 6. Gauss Elimination and Its Variants
Bibliography

[1] G. D AHLQUIST, A special stability problem for linear multistep methods, BIT, 3 (1963),
pp. 27–43.

[2] F. G OLUB AND C. V. L OAN, Matrix Computations, 3rd Ed., The Johns Hopkins Uni-
versity Press, Baltimore, 1996.

227
228 BIBLIOGRAPHY
Index

(n + 1)-point difference formula, 134 closed formula, 165

`2 -norm, 38 closed Newton-Cotes formulas, 149
k th divided difference, 100 coefficients, 65
x -intercept, 55 cofactor, 19
cofactor expansion, 19
ABM.mw, 192 composite error, 145
absolute error, 25 composite Simpson’s rule, 148
Adams-Bashforth method, 191 composite Simpson’s three-eights rule,
Adams-Bashforth-Moulton method, 191 149
Adams-Moulton method, 191 Composite Trapezoid rule, 151
adaptive mechanism, 104 composite trapezoid rule, 144
adaptive method, 186 computation of A −1 , 224
algorithm, 26 condition number, 24
angle, between two vectors, 13 conditionally stable, 26
antiderivative, 7 continuity, 2
augmented matrix, 14 continuous, 2
augmented system, 14 contractive mapping, 52
average slope, 181 convergence of Newton’s method, 56
back substitution, 211 convergence of oeder α, 27
backward-difference, 132 correction term, 54
Bairstow’s method, 71 cubic spline, 118
Bairstow, Leonard, 72
big Oh, 28, 29 deflation, 75
binary-search method, 40 dense matrix, 217
bisect.m, 45 derivative, 3
bisection method, 40 determinant, 18, 19
Bonnet’s recursion formula, 159 difference formula, (n + 1)-point, 134
difference formula, five-point, 134
cardinal functions, 88 difference formula, three-point, 134
change of variables, 163 difference formula, two-point, 132
characteristic equation, 20 differentiable, 3
characteristic polynomial, 20 direct algebraic solver, 204
Chebyshev, 95 Dirichlet boundary condition, 208
Chebyshev polynomials, 95 distance, 12
clamped cubic spline, 121, 130 divided difference, the k th, 100

229
230 INDEX

divided difference, the first, 100 Gauss integration), 160, 165

divided difference, the second, 100 Gauss-Lobatto integration, 165
divided difference, the zeroth, 100 Gaussian elimination with complete piv-
divided differences, 85, 99 oting, 222
dot product, 12 Gaussian elimination with partial pivot-
ing, 222
eigenvalue, 20 Gaussian quadrature, 157
eigenvector, 20 GECP, 222
elementary matrix, 14, 212 Generalized Rolle’s Theorem, 6
elementary row operations, 14, 212 GEPP, 222
elementary Simpson’s rule, 146, 157 ghost grid value, 206
Euclidean norm, 12, 22 global error, 183, 184
Euler method, 173, 174 guidepoints, 126
Euler.mw, 177
existence and uniqueness of fixed points, Hermite interpolation, 108
48 Hermite Interpolation Theorem, 109
exponential growth of error, 26 Hermite polynomial, 109
extended Newton divided difference, 110 Heun’s method, 183
Extreme Value Theorem, 5 higher-order Taylor methods, 178
Horner’s method, 66, 84
false position method, 63
horner.m, 68
FD scheme, 206
finite difference, 206 induced matrix norm, 23
first divided difference, 100 infinity-norm, 22
first-degree spline accuracy, 114 initial value problem, 170, 173, 181
five-point difference formula, 134 inner product, 12, 13
fixed point, 47 Intermediate Value Theorem, 3
fixed-point iteration, 49, 56 interpolating polynomials in Newton
Fixed-Point Theorem, 51 form, 82
floating point operation, 217 Interpolation Error Theorem, 91, 133
flop, 217 Interpolation Error Theorem, Chebyshev
for loop, 36 nodes, 97
forward elimination, 210 interval-halving method, 40
forward substitution, 210 invertible matrix, 16, 205
forward-difference, 132 invertible matrix theorem, 17, 205
Fourth-order Runge-Kutta method, 184 iteration, 35
Frobenius norm, 23 iterative algebraic solver, 204
FTC, 7 IVP, 170
function, 36 IVT, 3
fundamental period of the motion, 187
Fundamental Theorem of Algebra, 65 Jacobian, 59
Fundamental Theorem of Calculus, 7 Jacobian matrix, 71

Gauss elimination, 212 Kepler’s equation, 76

Gauss elimination with partial pivoting, knots, 113
221 Kronecker delta, 88
INDEX 231

Lagrange form of interpolating polyno- natural cubic splines, optimality theo-

mial, 88 rem, 123
Lagrange interpolating polynomial, 88, nested multiplication, 66, 84, 85
142 Neville’s Method, 104
Lagrange interpolation, 108 Newton form of interpolating polynomi-
Lagrange polynomial, 109, 132 als, 82
leading principal submatrix, 216 Newton form of the Hermite polynomial,
Legendre orthogonal polynomials, 158 109
Legendre polynomials, 159 Newton’s Divided Difference Formula,
length, 12 101
Leonard Bairstow, 72 Newton’s method, 54
limit, 2 Newton-Cotes formula, 142
linear approximation, 10 Newton-Raphson method, 54
linear convergence, 27 newton_horner.m, 69
linear function, 116, 119 NewtonRaphsonSYS.mw, 60
linear growth of error, 26 nodes, 113
linear spline, 113 nonsingular matrix, 16, 205
linear spline accuracy, 114 norm, 12, 22
linspace, in Matlab, 34 normal matrix, 23
Lipschitz condition, 170, 175 NR.mw, 57
little oh, 28, 29 numerical differentiation, 132
local truncation error, 183, 184 numerical discretization, 206
localization of roots, 65 numerical integration, 142
lower-triangular matrix, 209
lower-triangular system, 209 objects, 32
LU decomposition theorem, 216 Octave, 32
LU factorization, 213 open formula, 165
LU factorization algorithm, 215 operator norm, 23
order of convergence, 188, 189
m-step method, 190
orthogonal, 13
maple, 4
orthogonal polynomials, 159
Matlab, 32
outer bordering, 207
matrix norm, 23
maximum-norm, 22
p-norms, 22
Mean Value Theorem, 4, 9
parametric curves, 124
Mean Value Theorem on Integral, 7
partition, 113
mesh points, 173
permutation matrix, 220
method of false position, 63
piecewise cubic Hermite interpolating
method of undetermined coefficients, 156,
polynomial, 130
158, 160
piecewise cubic Hermite polynomial, 125,
Modified Euler method, 183
126
multi-step methods, 190
plot, in Matlab, 33
MVT, 4, 51
Polynomial Interpolation Error Theorem,
mysum.m, 36
91, 102, 129
natural cubic spline, 120, 123, 129 Polynomial Interpolation Theorem, 81
232 INDEX

polynomial of degree n, 65, 70 stable, 26

programming, 32 step-by-step methods, 172
pseudocode, 26 stopping criterion, 44
Pythagorean Theorem, 13 submatrix, 19
subordinate norm, 23
quadratic convergence, 27 super-convergence, 57
quadratic spline, 115, 130 superlinear convergence, 27
quotient, 70 synthetic division, 66
recurrence relation, 72 systems of nonlinear equations, 58
recursive Trapezoid rule, 151, 152 tangent line, 55
reduced echelon form, 14 tangent plane approximation, 10
relative error, 25 Taylor expansion, 135
remainder, 70 Taylor method of order m, 178
Remainder Theorem, 66 Taylor method of order m, 178
repetition, 32, 35
Taylor series, 173
reusability, 36
Taylor’s method of order m, 201
reusable, 32
Taylor’s series expansion, 206
Richardson extrapolation, 137, 138, 153
Taylor’s Theorem, 8, 147
Riemann integral, 6
Taylor’s Theorem for Two Variables, 10
RK2, 182, 183
Taylor’s Theorem with Integral Remain-
RK4, 184
der, 9
RK4.mw, 185
Taylor’s Theorem with Lagrange Remain-
RK4SYS.mw, 196
der, 8, 92
RK4SYSTEM.mw, 199
Taylor’s Theorem, Alternative Form of, 10
RKF45, 186
Taylor-series methods, 173
Rolle’s Theorem, 3, 6
three-point difference formula, 134
Romberg algorithm, 153
three-point endpoint formulas, 134
Romberg integration, 153
three-point midpoint formula, 134
Runge’s phenomenon, 112
three-point midpoint formulas, 137
Runge-Kutta methods, 181
trapezoid rule, 143
Runge-Kutta-Fehlberg method, 186
triangular systems, 209
secant method, 61 two-point difference formula, 132
second divided difference, 100
unique inverse, 16, 205
second-derivative midpoint formula, 135
unit lower triangular matrix, 213
Second-order Runge-Kutta method, 182,
unit lower-triangular matrix, 218
183
unstable, 26
significant digits, 25
upper-triangular matrix, 209, 218
Simpson’s rule, 146
upper-triangular system, 211
Simpson’s three-eights rule, 149
skew-symmetric, 38 vector norm, 22
sparse matrix, 217 volume scaling factor, 18
spline of degree k , 113
spring-mass system, 187 Weierstrass approximation theorem, 80,
square root, 57 112
INDEX 233

Weighted Mean Value Theorem on Inte- WMVT, 7

gral, 7, 143, 147
weighted sum, 142 zeroth divided difference, 100