Numerical Analysis

Lecture Notes

Endre Süli

Mathematical Institute
University of Oxford
2011
2
 3

Overview of the course

Lecture 1. Lagrange interpolation

Lecture 2. Newton–Cotes quadrature

Lecture 3. Newton–Cotes quadrature (continued)

Lecture 4. Gaussian elimination

Lecture 5. LU factorization

Lecture 6. QR factorization

Lecture 7. Matrix eigenvalues

Lecture 8. The symmetric QR algorithm

Lecture 9. The symmetric QR algorithm (continued)

Lecture 10. Best approximation in inner-product spaces

Lecture 11. Least squares approximation

Lecture 12. Orthogonal polynomials

Lecture 13. Gaussian quadrature

Lecture 14. Piecewise polynomial interpolation: splines

Lecture 15. Piecewise polynomial interpolation: splines (continued)

Lecture 16. Richardson extrapolation

Numerical Analysis Hilary Term 2011.
Lecture 1: Lagrange Interpolation.

Notation: Πn = {real polynomials of degree ≤ n}

Setup: given data fi at distinct xi, i = 0, 1, . . . , n, with x0 < x1 < · · · < xn,
can we find a polynomial pn such that pn (xi) = fi ? Such a polynomial is
said to interpolate the data.

E.g.: n = 1,
linear constant
Note: degree ≤ 1 =⇒ pn ∈ Πn
Theorem. ∃pn ∈ Πn such that pn (xi) = fi for i = 0, 1, . . . , n.
Proof. Consider, for k = 0, 1, . . . , n,
(x − x0 ) · · · (x − xk−1)(x − xk+1) · · · (x − xn)
Ln,k (x) = ∈ Πn . (1)
(xk − x0 ) · · · (xk − xk−1)(xk − xk+1) · · · (xk − xn )
Then

Ln,k (xi) = 0 for i = 0, . . . , k − 1, k + 1, . . . , n and Ln,k (xk ) = 1.

So now define n
X
pn (x) = fk Ln,k (x) ∈ Πn (2)
k=0
=⇒ n
X
pn(xi) = fk Ln,k (xi) = fi for i = 0, 1, . . . , n.
k=0 2

The polynomial (2) is the Lagrange interpolating polynomial.

Theorem. The interpolating polynomial of degree ≤ n is unique.
Proof. Consider two interpolating polynomials pn , qn ∈ Πn . Then their
difference dn = pn − qn ∈ Πn satisfies dn (xk ) = 0 for k = 0, 1, . . . , n. i.e., dn
is a polynomial of degree at most n but has at least n + 1 distinct roots.
Algebra =⇒ dn ≡ 0 =⇒ pn = qn. 2

1
Matlab:
% matlab
>> help lagrange
LAGRANGE Plots the Lagrange polynomial interpolant for the
given DATA at the given KNOTS
>> lagrange([1,1.2,1.3,1.4],[4,3.5,3,0]);

3.5

2.5

1.5

0.5

1 1.05 1.1 1.15 1.2 1.25 1.3 1.35 1.4

>> lagrange([0,2.3,3.5,3.6,4.7,5.9],[0,0,0,1,1,1]);

60

50

40

30

20

10

−10

−20

−30

−40
0 1 2 3 4 5 6

2
Data from an underlying smooth function: Suppose that f (x) has
at least n + 1 smooth derivatives in the interval (x0, xn). Let fk = f (xk ) for
k = 0, 1, . . . , n, and let pn be the Lagrange interpolating polynomial for the
data (xk , fk ), k = 0, 1, . . . , n.
Error: how large can the error f (x) − pn (x) be on the interval [x0, xn]?
Theorem. For every x ∈ [x0, xn] there exists ξ = ξ(x) ∈ (x0, xn) such that
def f (n+1)(ξ)
e(x) = f (x) − pn (x) = (x − x0 )(x − x1) · · · (x − xn ) ,
(n + 1)!
where f (n+1) is the n + 1-st derivative of f .
Proof. Trivial for x = xk , k = 0, 1, . . . , n as e(x) = 0 by construction. So
suppose x 6= xk . Let
def e(x)
φ(t) = e(t) − π(t),
π(x)
where
def
π(t) = (t − x0 )(t

− x1) · · · (t − xn )
n
n+1
xi  tn + · · · (−1)n+1x0x1 · · · xn
X
= t − 
i=0
∈ Πn+1.
Now note that φ vanishes at n + 2 points x and xk , k = 0, 1, . . . , n. =⇒
φ′ vanishes at n + 1 points ξ0 , . . . , ξn between these points =⇒ φ′′ vanishes
at n points between these new points, and so on until φ(n+1) vanishes at an
(unknown) point ξ in (x0, xn). But
e(x) (n+1) e(x)
φ(n+1) (t) = e(n+1) (t) − π (t) = f (n+1)(t) − (n + 1)!
π(x) π(x)
since p(n+1)
n (t) ≡ 0 and because π(t) is a monic polynomial of degree n + 1.
The result then follows immediately from this identity since φ(n+1) (ξ) = 0.
2

Example: f (x) = log(1+x) on [0, 1]. Here, |f (n+1)(ξ)| = n!/(1+ξ)n+1 < n!

on (0, 1). So |e(x)| < |π(x)|n!/(n+1)! ≤ 1/(n+1) since |x−xk | ≤ 1 for each
x, xk , k = 0, 1, . . . , n, in [0, 1] =⇒ |π(x)| ≤ 1. This is probably pessimistic
for many x, e.g. for x = 21 , π( 12 ) ≤ 2−(n+1) as | 12 − xk | ≤ 12 .

3
 This shows the important fact that the error can be large at the end
points—there is a famous example due to Runge, where the error from the
interpolating polynomial approximation to f (x) = (1 + x2)−1 for n + 1
equally-spaced points on [−5, 5] diverges near ±5 as n tends to infinity: try
runge from the website in Matlab.
Building Lagrange interpolating polynomials from lower degree
ones.
Notation: Let Qi,j be the Lagrange interpolating polynomial at xk , k =
i, . . . , j.
Theorem.
(x − xi)Qi+1,j (x) − (x − xj )Qi,j−1(x)
Qi,j (x) = (3)
xj − xi
Proof. Let s(x) denote the right-hand side of (3). Because of uniqueness,
we simply wish to show that s(xk ) = fk . For k = i+1, . . . , j−1, Qi+1,j (xk ) =
fk = Qi,j−1(xk ), and hence
(xk − xi )Qi+1,j (xk ) − (xk − xj )Qi,j−1(xk )
s(xk ) = = fk .
xj − xi
We also have that Qi+1,j (xj ) = fj and Qi,j−1(xi) = fi, and hence
s(xi) = Qi,j−1(xi) = fi and s(xj ) = Qi+1,j (xj ) = fj .
2

Comment: this can be used as the basis for constructing interpolating

polynomials. In books: may find topics such as the Newton form and
divided differences.
Generalisation: given data fi and gi at distinct xi, i = 0, 1, . . . , n, with
x0 < x1 < · · · < xn , can we find a polynomial p such that p(xi) = fi and
p′ (xi) = gi ?
Theorem. There is a unique polynomial p2n+1 ∈ Π2n+1 such that p2n+1(xi) =
fi and p′2n+1(xi) = gi for i = 0, 1, . . . , n.
Construction: given Ln,k (x) in (1), let
Hn,k (x) = [Ln,k (x)]2(1 − 2(x − xk )L′n,k (xk ))
and Kn,k (x) = [Ln,k (x)]2(x − xk ).

4
Then n
X
p2n+1(x) = [fk Hn,k (x) + gk Kn,k (x)] (4)
k=0
interpolates the data as required. The polynomial (4) is called the Hermite
interpolating polynomial.
Theorem. Let p2n+1 be the Hermite interpolating polynomial in the case
where fi = f (xi) and gi = f ′(xi) and f has at least 2n+2 smooth derivatives.
Then, for every x ∈ [x0, xn],

f (2n+2)(ξ)
f (x) − p2n+1(x) = [(x − x0)(x − xk−1) · · · (x − xn )]2 ,
(2n + 2)!

where ξ ∈ (x0, xn) and f (2n+2) is the 2n+2-nd derivative of f .

5
Numerical Analysis Hilary Term 2011.
Lecture 2: Newton–Cotes Quadrature.

Terminology: Quadrature ≡ numerical integration

Setup: given f (xk ) at n + 1 equally spaced points xk = x0 + k · h, k =
0, 1, . . . , n, where h = (xn − x0 )/n. Suppose that pn (x) interpolates this
data.
Idea: does Z x Z x
n n
f (x) dx ≈ pn (x) dx? (1)
x0 x0
We investigate the error in such an approximation below, but note that
Z x
n
Z x X
n
n
pn(x) dx = f (xk ) · Ln,k (x) dx
x0 x0 k=0
n
X Z x
n
= f (xk ) · Ln,k (x) dx (2)
x0
k=0
Xn
= wk f (xk ),
k=0

where the coefficients Z x

n
wk = Ln,k (x) dx (3)
x0
k = 0, 1, . . . , n, are independent of f — a formula
Z b n
X
f (x) dx ≈ wk f (xk )
a k=0

with xk ∈ [a, b] and wk independent of f for k = 0, 1, . . . , n is called a

quadrature formula; the coefficients wk are known as weights. The
specific form (1)–(3) is called a Newton–Cotes formula of order n.
Examples:
Trapezium Rule: n = 1:
p1
f
Z x
1 h
f (x) dx ≈ [f (x0) + f (x1)]
x0 2
x0 h x1

1
Proof.
L1,0 (x) L1,1 (x)
z }| { z }| {
Z x
1
Z x
1 x − x1 Z x
1 x − x0
p1(x) dx = f (x0) dx +f (x1) dx
x0 x0 − x1x0 x0 x1 − x0
(x1 − x0) (x1 − x0)
= f (x0) + f (x1)
2 2
Simpson’s Rule: n = 2:
f Z x
2 h
f (x) dx ≈ [f (x0) + 4f (x1) + f (x2)]
p2 x0 3
x0 h x1 h x2

Note: The Trapezium Rule is exact if f ∈ Π1, since if f ∈ Π1 =⇒ p1 = f .

Similarly, Simpson’s Rule is exact if f ∈ Π2, since if f ∈ Π2 =⇒ p2 = f .
The highest degree of polynomial exactly integrated by a quadrature rule
is called the degree of accuracy.
Error: we can use the error in interpolation directly to obtain
Z x
n
Z x
n π(x) (n+1)
[f (x) − pn(x)] dx = f (ξ(x)) dx
x0 x0 (n + 1)!
so that
Z x
n

1 Z x
n
max |f (n+1)(ξ)|
 

 [f (x) − pn (x)] dx ≤ |π(x)| dx, (4)
x0 (n + 1)! ξ∈[x0 ,xn ] x0

which, e.g., for the Trapezium Rule, n = 1, gives

 
Z x
1 (x1 − x0) 
(x1 − x0)3
max |f ′′ (ξ)|.
 

 f (x) dx − [f (x0) + f (x1)] ≤

 x0 2  12 ξ∈[x0 ,x1 ]

In fact, we can prove a tighter result using the Integral Mean-Value Theo-
rem1:
Z x
1 (x1 − x0) (x1 − x0)3 ′′
Theorem. f (x) dx − [f (x0) + f (x1)] = − f (ξ) for
x0 2 12
1
Integral Mean-Value Theorem: if f and g are continuous on [a, b] and g(x) ≥ 0 on this interval,
Z b Z b
then there exits an η ∈ (a, b) for which f (x)g(x) dx = f (η) g(x) dx (see problem sheet).
a a

2
some ξ ∈ (x0, x1).
Proof. See problem sheet. 2

For n > 1, (4) gives pessimistic bounds. But one can prove better results
such as:
Theorem. Error in Simpson’s Rule: if f ′′′′ is continuous on (x0, x2), then
 
Z x
2 (x2 − x0) 
(x2 − x0)5
max |f ′′′′(ξ)|.
 

 f (x) dx − [f (x0) + 4f (x1) + f (x2)] ≤

 x0 6  720 ξ∈[x0 ,x2 ]
Z x
2
Proof. Recall p2 (x) dx = 13 h[f (x0)+4f (x1)+f (x2)], where h = x2 −x1 =
x0
x1 − x0. Consider f (x0) − 2f (x1) + f (x2) = f (x1 − h) − 2f (x1) + f (x1 + h).
Then, by Taylor’s Theorem,
f (x1 − h) f (x1) − hf ′ (x1) + 12 h2 f ′′ (x1) − 16 h3 f ′′′(x1) + 241 h4 f ′′′′(ξ1)
−2f (x1) = −2f (x1) +
+f (x1 + h) f (x1) + hf ′ (x1) + 21 h2 f ′′ (x1) + 16 h3 f ′′′(x1) + 241 h4 f ′′′′(ξ2)
for some ξ1 ∈ (x0, x1) and ξ2 ∈ (x1, x2), and hence
f (x0) − 2f (x1) + f (x2) = h2 f ′′ (x1) + 241 h4 [f ′′′′(ξ1) + f ′′′′(ξ2)]
(5)
= h2 f ′′ (x1) + 121 h4 f ′′′′(ξ3 ),
the last result following from the Intermediate-Value Theorem2 for some
ξ3 ∈ (ξ1, ξ2 ) ⊂ (x0, x2). Now for any x ∈ [x0, x2], we may use Taylor’s
Theorem again to deduce
Z x Z x +h Z x +h
2 1 ′ 1
f (x) dx = f (x1) dx + f (x1) (x − x1 ) dx
x0 x1 −h Z x1 −h
x1 −h Z x +h
1
+ 21 f ′′(x1) (x − x1) dx + 61 f ′′′(x1)
2
(x − x1)3 dx
Z x +h 1 x −h x1 −h
1 ′′′′ 4
+ 24 1
f (η1(x))(x − x1) dx
x1 −h
= 2hf (x1) + 13 h3 f ′′(x1) + 601 h5 f ′′′′(η2)
5 ′′′′
= 3 h[f (x0 ) + 4f (x1 ) + f (x2 )] + 60 h f
1 1
(η2) − 361 h5 f ′′′′(ξ3)
1 x2 − x0 5
Z x !
2
= p2 (x) dx + (3f ′′′′(η2) − 5f ′′′′(ξ3))
x0 180 2
2
Intermediate-Value Theorem: if f is continuous on a closed interval [a, b], and c is any number
between f (a) and f (b) inclusive, then there is at least one number ξ in the closed interval such that
f (ξ) = c. In particular, since c = (df (a) + ef (b))/(d + e) lies between f (a) and f (b) for any positive d and
e, there is a value ξ in the closed interval for which d · f (a) + e · f (b) = (d + e) · f (ξ).

3
where η1 (x) and η2 ∈ (x0, x2), using the Integral Mean-Value Theorem and
(5). Thus, taking moduli,
Z x
2

8
(x2 − x0)5 max |f ′′′′(ξ)|
 
 [f (x) − p2(x)] dx ≤
25

x0 · 180 ξ∈[x0 ,x2 ]

as required. 2

Note: Simpson’s Rule is exact if f ∈ Π3 since then f ′′′′ ≡ 0.

In fact, it is possible to compute a slightly stronger bound.
Theorem. Error in Simpson’s Rule II: if f ′′′′ is continuous on (x0, x2), then
Z x
2 x2 − x0 (x2 − x0)5 ′′′′
f (x) dx = [f (x0) + 4f (x1) + f (x2)] − f (ξ)
x0 6 2880
for some ξ ∈ (x0, x2).
Proof. See Süli and Mayers, Thm. 7.2. 2

4
Numerical Analysis Hilary Term 2011.
Lecture 3: Newton-Cotes Quadrature (continued).

Motivation: we’ve seen oscillations in polynomial interpolation—the

Runge phenomenon–for high-degree polynomials.
Idea: split a required integration interval [a, b] = [x0, xn] into n equal
intervals [xi−1, xi] for i = 1, . . . , n. Then use a composite rule:
Z b Z x
n
n Z xi
X
f (x) dx = f (x) dx = f (x) dx
a x0 i=1 xi−1
Z x
i
in which each f (x) dx is approximated by quadrature.
xi−1
Thus rather than increasing the degree of the polynomials to attain high
accuracy, instead increase the number of intervals.
Trapezium Rule:
Z x
i h h3 ′′
f (x) dx = [f (xi−1) + f (xi)] − f (ξi)
xi−1 2 12
for some ξi ∈ (xi−1, xi)
Composite Trapezium Rule:
h3 ′′ 
 
Z x
n
n
X h
f (x) dx =  [f (xi−1) + f (xi)] − f (ξi)
x0 i=1 2 12
h
= [f (x0) + 2f (x1) + 2f (x2) + · · · + 2f (xn−1) + f (xn)] + eTh
2
where ξi ∈ (xi−1, xi) and h = xi − xi−1 = (xn − x0)/n = (b − a)/n, and the
error eTh is given by
h3 Xn nh3 ′′ h2
eTh = − f ′′(ξi ) = − f (ξ) = −(b − a) f ′′ (ξ)
12 i=1 12 12
for some ξ ∈ (a, b), using the Intermediate-Value Theorem n times. Note
that if we halve the stepsize h by introducing a new point halfway between
each current pair (xi−1, xi), the factor h2 in the error will decrease by four.

1
Another composite rule: if [a, b] = [x0, x2n],
Z b Z x
2n
n Z x2i
X
f (x) dx = f (x) dx = f (x) dx
a x0 i=1 x2i−2
Z x
2i
in which each f (x) dx is approximated by quadrature.
x2i−2
Simpson’s Rule:
Z x
2i h (2h)5 ′′′′
f (x) dx = [f (x2i−1) + 4f (x2i−1) + f (x2i)] − f (ξi )
x2i−2 3 2880
for some ξi ∈ (x2i−2, x2i).
Composite Simpson’s Rule:
(2h)5 ′′′′ 
 
Z x
2n
n
Xh
f (x) dx =  [f (x2i−2) + 4f (x2i−1) + f (x2i)] − f (ξi )
x0 i=1 3 2880
h
= [f (x0) + 4f (x1) + 2f (x2) + 4f (x3) + 2f (x4) + · · ·
3
+ 2f (x2n−2) + 4f (x2n−1) + f (x2n)] + eSh
where ξi ∈ (x2i−2, x2i) and h = xi − xi−1 = (x2n − x0)/2n = (b − a)/2n, and
the error eSh is given by
(2h)5 X
n
′′′′ n(2h)5 ′′′′ h4 ′′′′
eh = −
S
f (ξi) = − f (ξ) = −(b − a) f (ξ)
2880 i=1 2880 180
for some ξ ∈ (a, b), using the Intermediate-Value Theorem n times. Note
that if we halve the stepsize h by introducing a new point half way between
each current pair (xi−1, xi), the factor h4 in the error will decrease by sixteen.
Adaptive procedure: if Sh is the value given by Simpson’s rule with a
stepsize h, then
15
Sh − S 21 h ≈ − eSh .
16
Z b
This suggests that if we wish to compute f (x) dx with an absolute error
a
ε, we should compute the sequence Sh , S 21 h , S 41 h , . . . and stop when the dif-
ference, in absolute value, between two consecutive values is smaller than
16
15 ε. That will ensure that (approximately) |eh | ≤ ε.
S

Sometimes much better accuracy may be obtained: for example, as might

happen when computing Fourier coefficients, if f is periodic with period
b − a so that f (a + x) = f (b + x) for all x.

2
Matlab:

% matlab
>> help adaptive_simpson
ADAPTIVE_SIMPSON Adaptive Simpson’s rule.
S = ADAPTIVE_SIMPSON(F,A,B,NMAX,TOL) computes an approximation
to the integral of F on the interval [A,B]. It will take a
maximum of NMAX steps and will attempt to determine the
integral to a tolerance of TOL.

The function uses an adaptive Simpson’s rule, as described

in lectures.
>> f = inline(’sin(x)’);
>> adaptive_simpson(f,0,pi,8,1.0e-7);
Step 1 integral is 2.0943951024, with error estimate 2.0944.
Step 2 integral is 2.0045597550, with error estimate 0.089835.
Step 3 integral is 2.0002691699, with error estimate 0.0042906.
Step 4 integral is 2.0000165910, with error estimate 0.00025258.
Step 5 integral is 2.0000010334, with error estimate 1.5558e-05.
Step 6 integral is 2.0000000645, with error estimate 9.6884e-07.
Successful termination at iteration 7:
The integral is 2.0000000040, with error estimate 6.0498e-08.
>> g = inline(’sin(sin(x))’);
>> fplot(g,[0,pi])

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
0 0.5 1 1.5 2 2.5 3

3
>> adaptive_simpson(g,0,pi,8,1.0e-7);
Step 1 integral is 1.7623727094, with error estimate 1.7624.
Step 2 integral is 1.8011896009, with error estimate 0.038817.
Step 3 integral is 1.7870879453, with error estimate 0.014102.
Step 4 integral is 1.7865214631, with error estimate 0.00056648.
Step 5 integral is 1.7864895607, with error estimate 3.1902e-05.
Step 6 integral is 1.7864876112, with error estimate 1.9495e-06.
Step 7 integral is 1.7864874900, with error estimate 1.2118e-07.
Successful termination at iteration 8:
The integral is 1.7864874825, with error estimate 7.5634e-09.

4
Numerical Analysis Hilary Term 2011.
Lecture 4: Gaussian Elimination.

Setup: given a square n by n matrix A and vector with n components b,

find x such that
Ax = b.
Equivalently find x = (x1, x2, . . . , xn)T for which

a11 x1 + a12 x2 + · · · + a1n xn = b1

a21 x1 + a22 x2 + · · · + a2n xn = b2
.. (1)
.
an1 x1 + an2x2 + · · · + ann xn = bn.

Lower-triangular matrices: the matrix A is lower triangular if aij = 0

for all 1 ≤ i < j ≤ n. The system (1) is easy to solve if A is lower triangular.
b1
a11 x1 = b1 =⇒ x1 = ⇓
a11
b2 − a21x1
a21 x1 + a22 x2 = b2 =⇒ x2 = ⇓
a22
..
. ⇓
i−1
X
bi − aij xj
j=1
ai1 x1 + ai2x2 + · · · + aii xi = bi =⇒ xi = ⇓
aii
..
. ⇓

This works if, and only if, aii 6= 0 for each i. The procedure is known as
forward substitution.
Computational work estimate: one floating-point operation (flop) is
one multiply (or divide) and possibly add (or subtraction) as in y = a∗x+b,
where a, x, b and y are computer representations of real scalars. Hence the
work in forward substitution is 1 flop to compute x1 plus 2 flops to compute
x2 plus . . . plus i flops to compute xi plus . . . plus n flops to compute xn , or
in total n
i = 12 n(n + 1) = 21 n2 + lower order terms
X

i=1

1
flops. We sometimes write this as 21 n2 + O(n) flops or more crudely O(n2)
flops.
Upper-triangular matrices: the matrix A is upper triangular if aij =
0 for all 1 ≤ j < i ≤ n. Once again, the system (1) is easy to solve if A is
upper triangular.
..
. ⇑
n
X
bi − aij xj
j=i+1
aii xi + · · · + ain−1xn−1 + a1n xn = bi =⇒ xi = ⇑
aii
..
. ⇑
bn−1 − an−1n xn
an−1n−1xn−1 + an−1nxn = bn−1 =⇒ xn−1 = ⇑
an−1n−1
bn
ann xn = bn =⇒ xn = . ⇑
ann
Again, this works if, and only if, aii 6= 0 for each i. The procedure is known
as backward or back substitution. This also takes approximately 21 n2
flops.
For computation, we need a reliable, systematic technique for reducing
Ax = b to U x = c with the same solution x but with U (upper) trian-
gular =⇒ Gauss elimination.
Example     
3 −1 x1 12 
   = .
1 2 x2 11
Multiply first equation by 1/3 and subtract from the second =⇒
    
3 −1 x1 12 

7
  = .
0 3 x2 7

Gauss(ian) Elimination (GE): this is most easily described in terms

of overwriting the matrix A = {aij } and vector b. At each stage, it is a
systematic way of introducing zeros into the lower triangular part of A by
subtracting multiples of previous equations (i.e., rows); such (elementary
row) operations do not change the solution.

2
 for columns j = 1, 2, . . . , n − 1
for rows i = j + 1, j + 2, . . . , n
aij
row i ← row i − ∗ row j
ajj
aij
bi ← bi − ∗ bj
ajj
end
end
Example.
      
 3 −1 2  x1   12   3 −1 2 | 12 
1 2 3 x2 = 11 : represent as 1 2 3 | 11
      
      
      
      
2 −2 −1 x3 2 2 −2 −1 | 2

 
 3 −1 2 | 12 
7 7
=⇒ row 2 ← row 2 − row 1 0 3 | 7
 
1  
3  3 
0 − 43 − 73 | −6
 
row 3 ← row 3 − 23 row 1

 
 3 −1 2 | 12 
7 7
=⇒ 0 3 | 7
 
 
 3 
row 3 ← row 3 + 47 row 2
 
0 0 −1 | −2
Back substitution:
x3 = 2
7 − 73 (2)
x2 = 7 =1
3
12 − (−1)(1) − 2(2)
x1 = = 3.
3
aij
Cost of Gaussian Elimination: note, row i ← row i − ∗ row j is
ajj
for columns k = j + 1, j + 2, . . . , n
aij
aik ← aik − ajk
ajj
end

3
This is approximately n − j flops as the multiplier aij /ajj is calculated
with just one flop; ajj is called the pivot. Overall therefore, the cost of GE
is approximately
n−1 n−1 n(n − 1)(2n − 1) 1 3
(n − j)2 =
X 2
= n + O(n2 )
X
l =
j=1 l=1 6 3
flops. The calculations involving b are
n−1
X n−1
X n(n − 1) 1 2
(n − j) = l= = n + O(n)
j=1 l=1 2 2
flops, just as for the triangular substitution.

4
Numerical Analysis Hilary Term 2011.
Lecture 5: LU Factorization.
The basic operation of Gaussian Elimination, row i ← row i + λ ∗ row j can
be achieved by pre-multiplication by a special lower-triangular matrix
 
 0 0 0
0 λ 0  ← i
 
M(i, j, λ) = I + 

 
0 0 0
↑
j
where I is the identity matrix.
Example: n = 4,
     
 1 0 0 0   a   a 
0 1 0 0  b b 
     
   
M(3, 2, λ) = 


 and M(3, 2, λ)  
 = 
 ,

 0 λ 1 0 
  c



 λb + c 
     
0 0 0 1 d d
i.e., M(3, 2, λ)A performs: row 3 of A ← row 3 of A + λ∗ row 2 of A and
similarly M(i, j, λ)A performs: row i of A ← row i of A + λ∗ row j of A.
So GE for e.g., n = 3 is
M(3, 2, −l32) · M(3, 1, −l31) · M(2, 1, −l21) · A = U = ( ).
a32 a31 a21
l32 = l31 = l21 = (upper triangular)
a22 a11 a11
The lij are the multipliers.
Be careful: each multiplier lij uses the data aij and aii that results from the
transformations already applied, not data from the original matrix. So l32
uses a32 and a22 that result from the previous transformations M(2, 1, −l21)
and M(3, 1, −l31).
Lemma. If i 6= j, (M(i, j, λ))−1 = M(i, j, −λ).
Proof. Exercise.
Outcome: for n = 3, A = M(2, 1, l21) · M(3, 1, l31) · M(3, 2, l32) · U , where
 
 1 0 0 
M(2, 1, l21) · M(3, 1, l31) · M(3, 2, l32) = l21 1 0 = L = ( ).
 
 
 
 
l31 l32 1 (lower triangular)

1
This is true for general n:
Theorem. For any dimension n, GE can be expressed as A = LU , where
U = ( ) is upper triangular resulting from GE, and L = ( ) is unit lower
triangular (lower triangular with ones on the diagonal) with lij = multiplier
used to create the zero in the (i, j)th position.
Most implementations of GE therefore, rather than doing GE as above,

factorize A = LU (takes ≈ 31 n3 flops)

and then solve Ax = b
by solving Ly = b (forward substitution)
and then U x = y (back substitution)

Note: this is much more efficient if we have many different right-hand sides
b but the same A.
Pivoting: GE or LU can fail if the pivot aii = 0, e.g., if
 
0 1
A= ,
1 0

GE will fail at the first step. However, we are free to reorder the equations
(i.e., the rows) into any order we like, e.g., the equations

0 · x1 + 1 · x2 = 1 1 · x1 + 0 · x2 = 2
and
1 · x1 + 0 · x2 = 2 0 · x1 + 1 · x2 = 1

are the same, but their matrices

   
0 1 1 0
  and  
1 0 0 1

have had their rows reordered: GE fails for the first but succeeds for the
second =⇒ better to interchange the rows and then apply GE.
Partial pivoting: when creating the zeros in the j-th column, find |akj | =
max(|ajj |, |aj+1j |, . . . , |anj |), then swap (interchange) rows j and k

2
 e.g.,
   

a11 · a1j−1 a1j · · · a1n  
a11 · a1j−1 a1j · · · a1n 
0 · · · · · · · 0 · · · · · · ·
   
   
   
0 · aj−1j−1 aj−1j · · · aj−1n 0 · aj−1j−1 aj−1j · · · aj−1n
   
   
   
   

0 · 0 ajj · · · ajn  
0 · 0 akj · · · akn 
→
   
   
0 · 0 · · · · · 0 · 0 · · · · ·
   
   
   
0 · 0 akj · · · akn 0 · 0 ajj · · · ajn
   
   
   
   



0 · 0 · · · · · 





0 · 0 · · · · · 


0 · 0 anj · · · ann 0 · 0 anj · · · ann

Property: GE with partial pivoting cannot fail if A is non singular.

Proof. If A is the first matrix above at the j-th stage,
 
a
 jj
· · · ajn 
 · · · · ·
 

 
det[A] = a11 · · · aj−1j−1 · det  akj · · · akn .
 

 
 · · · · ·
 

 
anj · · · ann

Hence det[A] = 0 if ajj = · · · = akj = · · · = anj = 0. Thus if the pivot ak,j

is zero, A is singular. So if all of the pivots are nonzero, A is nonsingular.
(Note, actually ann can be zero and an LU factorization still exist.)
The effect of pivoting is just a permutation (reordering) of the rows, and
hence can be represented by a permutation matrix P .
Permutation matrix: P has the same rows as the identity matrix, but
in the pivoted order. So
P A = LU
represents the factorization—equivalent to GE with partial pivoting. E.g.,
 
 0 1 0
0 0 1  A
 


 
1 0 0

just has the 2nd row of A first, the 3rd row of A second and the 1st row of
A last.

3
% matlab
>> A = rand(6,6)
A =
0.8462 0.6813 0.3046 0.1509 0.4966 0.3420
0.5252 0.3795 0.1897 0.6979 0.8998 0.2897
0.2026 0.8318 0.1934 0.3784 0.8216 0.3412
0.6721 0.5028 0.6822 0.8600 0.6449 0.5341
0.8381 0.7095 0.3028 0.8537 0.8180 0.7271
0.0196 0.4289 0.5417 0.5936 0.6602 0.3093
>> b = A*ones(6,1)
b =
2.8215
2.9817
2.7691
3.8962
4.2491
2.5533
>> x = A \ b
x =
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
>> [L,U,P] = lu(A)
L =
1.0000 0 0 0 0 0
0.6206 -0.0648 0.0183 0.8969 1.0000 0
0.2395 1.0000 0 0 0 0
0.7943 -0.0573 0.9718 0.5673 -0.2248 1.0000
0.9904 0.0519 -0.0113 1.0000 0 0
0.0232 0.6178 1.0000 0 0 0
U =
0.8462 0.6813 0.3046 0.1509 0.4966 0.3420
0 0.6686 0.1205 0.3422 0.7027 0.2593
0 0 0.4602 0.3786 0.2146 0.1412
0 0 0 0.6907 0.2921 0.3765
0 0 0 0 0.3712 -0.2460
0 0 0 0 0 -0.1288

4
P =
1 0 0 0 0 0
0 0 1 0 0 0
0 0 0 0 0 1
0 0 0 0 1 0
0 1 0 0 0 0
0 0 0 1 0 0
>> P*P’
ans =
1 0 0 0 0 0
0 1 0 0 0 0
0 0 1 0 0 0
0 0 0 1 0 0
0 0 0 0 1 0
0 0 0 0 0 1

5
Numerical Analysis Hilary Term 2011.
Lecture 6: QR Factorization.

Definition: a square real matrix Q is orthogonal if QT = Q−1. This is

true if, and only if, QTQ = I = QQT .
Example: the permutation matrices P in LU factorization with partial
pivoting are orthogonal.
Proposition. The product of orthogonal matrices is an orthogonal matrix.
Proof. If S and T are orthogonal, (ST )T = T T S T so
(ST )T(ST ) = T T S T ST = T T (S TS)T = T T T = I.
Definition: The scalar (dot)(inner) product of two vectors
   
x1 y1
x  y 
 2   2 
x=  and y =  
 ·   · 
xn yn
in Rn is n
X
T T
x y=y x= xiyi ∈ R
i=1
Definition: Two vectors x, y ∈ Rn are orthogonal (perpendicular) if
xT y = 0. A set of vectors {u1, u2, . . . , ur } is an orthogonal set if uT
i uj = 0
for all i, j ∈ {1, 2, . . . , r} such that i 6= j.
Lemma. The columns of an orthogonal matrix Q form an orthogonal set,
which is moreover an orthonormal basis for Rn .
.
Proof. Suppose that Q = [q1 q2 .. qn ], i.e., qj is the jth column of Q.
Then    
q1T 1 0 ··· 0
T

 q2T 
 ..

 0 1 ··· 0 

Q Q=I =  [q1 q2 . qn ] =  .. .. ... .. .
 ···   . . . 
qnT 0 0 ··· 1
Comparing the (i, j)th entries yields
(
0 i 6= j
qiT qj =
1 i = j.

1
Note that the columns of an orthogonal matrix are of length 1 as qiT qi = 1, so
they form an orthonormal set ⇐⇒ they are linearly independent (check!)
=⇒ they form an orthonormal basis for Rn as there are n of them. 2

Lemma. If u ∈ Rn , P is n by n orthogonal and v = P u, then uT u = v T v.

Proof. See problem sheet.
Definition: The outer product of two vectors x and y ∈ Rn is
 
x1 y1 x1y2 · · · x1yn
xy xy · · · x2yn 
 2 1 2 2 
xy T =  .. .. ... ..  ,
 . . . 
xn y1 xny2 · · · xn yn

an n by n matrix (notation: xy T ∈ Rn×n ). More usefully, if z ∈ Rn , then

n
!
X
(xy T )z = xy T z = x(y Tz) = yi zi x.
i=1

Definition: For w ∈ Rn , w 6= 0, the Householder matrix H(w) ∈ Rn×n

is the matrix
2
H(w) = I − T wwT .
w w
Proposition. H(w) is an orthogonal matrix.
Proof.
  
2 2
H(w)H(w)T = I− wwT I− wwT
wTw wTw
4 4
= I− wwT + w(wT w)wT .
wTw T
(w w) 2

= I 2

Lemma. Given u ∈ Rn , there exists a w ∈ Rn such that

 
α
 0 
 
H(w)u =  ..  ≡ v,
 . 
0

2
 √
say, where α = ± uTu.
Proof. Take w = γ(u − v), where γ 6= 0. Recall that since H(w) is
orthogonal, uT u = v T v. Then
wT w = γ 2(u − v)T (u − v) = γ 2(uTu − 2uTv + v T v)
= γ 2(uTu − 2uTv + uT u) = 2γuT(γ(u − v))
= 2γwTu.
So
 
2 2wTu 1
H(w)u = I− wwT u = u − w = u − w = u − (u − v) = v.
wTw wTw γ
2

Now if u is the first column of the n by n matrix A,

 
α × ··· ×
 0 
 
H(w)A =  .  , where × = general entry.
 .. B 
0
Similarly for B, we can find ŵ ∈ Rn−1 such that
 
β × ··· ×
 0 
 
H(ŵ)B =  . 
 .. C 
0
and then
 
α × × ··· ×
   
1 0 ··· 0  0 β × ··· × 
   
 0   0 0 
 ..  H(w)A =  .
 . H(ŵ)  
 0 0 

 .. .. C 
0  . . 
0 0
Note " # " #
1 0 0
= H(w2), where w2 = .
0 H(ŵ) ŵ

3
Thus if we continue in this manner for the n − 1 steps, we obtain
 
α × ··· ×
 0 β ··· × 
H(wn−1) · · · H(w3)H(w2)H(w)A =   ( )
 . . . . = .
| {z }  .. .. . . .. 
QT
0 0 ··· γ

The matrix QT is orthogonal as it is the product of orthogonal (House-

holder) matrices, so we have constructively proved that
Theorem. Given any square matrix A, there exists an orthogonal matrix
Q and an upper triangular matrix R such that

A = QR

Notes: 1. This could also be established using the Gram–Schmidt Process.

2. If u is already of the form (α, 0, · · · , 0)T, we just take H = I.
3. It is not necessary that A is square: if A ∈ Rm×n , then we need the
product of (a) m − 1 Householder matrices if m ≤ n =⇒

( ) = A = QR = ( )( )

or (b) n Householder matrices if m > n =⇒

= A = QR = .

Another useful family of orthogonal matrices are the Givens’ rotation

matrices:
 
1
 
 · 
 
 c s  ← ith row
 

 · 


J(i, j, θ) =  −s c  ← jth row

 
 · 
1
↑ ↑
i j

4
where c = cos θ and s = sin θ.
Exercise: Prove that J(i, j, θ)J(i, j, θ)T = I— obvious though, since the
columns form an orthonormal basis.
Note that if x = (x1, x2 , . . . , xn)T and y = J(i, j, θ)x, then

yk = xk for k 6= i, j
yi = cxi + sxj
yj = −sxi + cxj

and so we can ensure that yj = 0 by choosing xi sin θ = xj cos θ, i.e.,

xj xj xi
tan θ = or equivalently s = q and c = q . (1)
xi x2i + x2j x2i + x2j

Thus, unlike the Householder matrices, which introduce lots of zeros by

pre-multiplication, the Givens’ matrices introduce a single zero in a chosen
position by pre-multiplication. Since (1) can always be satisfied, we only
ever think of Givens’ matrices J(i, j) for a specific vector or column with
the angle chosen to make a zero in the jth position, e.g., J(1, 2)x tacitly
implies that we choose θ = tan−1 x2/x1 so that the second entry of J(1, 2)x
is zero. Similarly, for a matrix A ∈ Rm×n , J(i, j)A := J(i, j, θ)A, where
θ = tan−1 aji /aii, i.e., it is the ith column of A that is used to define θ so
that (J(i, j)A)ji = 0.
We shall return to these in a later lecture.

5
Numerical Analysis Hilary Term 2011.
Lecture 7: Matrix Eigenvalues.

Background: first, an important result from analysis (not proved or

examinable!), which will be useful.
Theorem. (Ostrowski) The eigenvalues of a matrix are continuously de-
pendent on the entries. I.e., suppose that {λi , i = 1, . . . , n} and {µi , i =
1, . . . , n} are the eigenvalues of A ∈ Rn×n and A + B ∈ Rn×n respectively.
Given any ε > 0, there is a δ > 0 such that |λi − µi | < ε whenever
maxi,j |bij | < δ, where B = {bij }1≤i,j≤n.
Aim: estimate the eigenvalues of a matrix.
Theorem. Gerschgorin’s theorem: Suppose that A = {aij }1≤i,j≤n ∈
Rn×n , and λ is an eigenvalue of A. Then, λ lies in the union of the Ger-
schgorin discs
 
 

 X n 

Di = z ∈ C |aii − z| ≤ |aij | , i = 1, . . . , n.
 

 j6=i 

j=1

Proof. If λ is an eigenvalue of A ∈ Rn×n , then there exists an eigenvector

x ∈ Rn with Ax = λx, x 6= 0, i.e.,
n
X
aij xj = λxi , i = 1, . . . , n.
j=1

Suppose that |xk | ≥ |xℓ |, ℓ = 1, . . . , n, i.e.,

“xk is the largest entry”. (1)

n
X
Then certainly akj xj = λxk , or
j=1

n
X
(akk − λ)xk = − akj xj .
j6=k
j=1

1
 Dividing by xk , (which, we know, is 6= 0) and taking absolute values,
 
 
X n  X n   X n
 x j   xj 
|akk − λ| =  akj  ≤ |akj |   ≤ |akj |
 j6=k x k xk
 j=1  j6j=1
=k j6=k
j=1

by (1). 2

Example.  
9 1 2
A =  −3 1 1 
 
1 2 −1

−4 −3 −2 −1 0 1 2 3 4 5 6 7 8 9 10 11 12

With Matlab calculate >> eig(A) = 8.6573, -2.0639, 2.4066

Theorem. Gerschgorin’s 2nd theorem: If any union of ℓ (say) discs is
disjoint from the other discs, then it contains ℓ eigenvalues.
Proof. Consider B(θ) = θA + (1 − θ)D, where D = diag(A), the diagonal
matrix whose diagonal entries are those from A. As θ varies from 0 to
1, B(θ) has entries that vary continuously from B(0) = D to B(1) = A.
Hence the eigenvalues λ(θ) vary continuously by Ostrowski’s theorem. The
Gerschgorin discs of B(0) = D are points (the diagonal entries), which are
clearly the eigenvalues of D. As θ increases the Gerschgorin discs of B(θ)
increase in radius about these same points as centres. Thus if A = B(1)
has a disjoint set of ℓ Gerschgorin discs by continuity of the eigenvalues it

2
must contain exactly ℓ eigenvalues (as they can’t jump!). 2
√
Notation: for x ∈ Rn , kxk = xTx is the (Euclidean) length of x.

3
Power Iteration: a simple method for calculating a single (largest) eigen-
value of a square matrix A is: for arbitrary y ∈ Rn , set x0 = y/kyk to
calculate an initial vector, and then for k = 0, 1, . . .
Compute yk = Axk
and set xk+1 = yk /kyk k.
This is the Power Method or Iteration, and computes unit vectors in
the direction of x0, Ax0, A2x0 , A3x0 , . . . , Ak x0.
Suppose that A is diagonalizable so that there is a basis of eigenvectors of
A:
{v1 , v2, . . . , vn}
with Avi = λi vi and kvi k = 1, i = 1, 2, . . . , n, and assume that

|λ1 | > |λ2 | ≥ · · · ≥ |λn |.

Then we can write n

X
x0 = αi vi
i=1
for some αi ∈ R, i = 1, 2, . . . , n, so
n
X n
X
k k
A x0 = A αi vi = αi Ak vi.
i=1 i=1

However, since Avi = λi vi =⇒ A2vi = A(Avi) = λi Avi = λ2i vi , inductively

Ak vi = λki vi. So
n
" n  k #
X X λi
Ak x 0 = αi λki vi = λk1 α1 v1 + αi vi .
i=1 i=2
λ1

Since (λi/λ1 )k → 0 as k → ∞, Ak x0 tends to look like λki α1 v1 as k gets

large. The result is that by normalizing to be a unit vector
Ak x 0 kAk x0k
 k 
 λ1 α1 
→ ±v1 and ≈  = |λ1 |
kAk x0k kAk−1x0k  λk−1 1 α1


as k → ∞, and the sign is identified by looking at, e.g., (Ak x0)1/(Ak−1x0)1.

4
 More usefully, the Power Iteration may be seen to compute yk = βk Ak x0
for some βk . Then, from the above,
yk βk Ak x 0
xk+1 = = · → ±v1 .
kyk k |βk | kAk x0 k
Similarly, yk−1 = βk−1 Ak−1x0 for some βk−1. Thus
βk−1 Ak−1x0 βk−1 Ak x 0
xk = · and hence yk = Axk = · .
|βk−1 | kAk−1x0k |βk−1| kAk−1x0k
Therefore, as above,
kAk x0k
kyk k = ≈ |λ1 |,
kAk−1x0k
and the sign of λ1 may be identified by looking at, e.g., (xk+1)1/(xk )1 .
Hence the largest eigenvalue (and its eigenvector) can be found.
Note: it is possible for a chosen vector x0 that α1 = 0, but rounding errors
in the computation generally introduce a small component in v1, so that in
practice this is not a concern!
This simplified method for eigenvalue computation is the basis for effective
methods, but the current state of the art is the QR Algorithm, which we
consider only in the case when A is symmetric.

5
Numerical Analysis Hilary Term 2011.
Lectures 8–9: The Symmetric QR Algorithm.

We consider only the case where A is symmetric.

Recall: a symmetric matrix A is similar to B if there is a nonsingular ma-
trix P for which A = P −1 BP . Similar matrices have the same eigenvalues,
since if A = P −1 BP ,

0 = det(A − λI) = det(P −1 (B − λI)P ) = det(P −1 ) det(P ) det(B − λI),

so det(A − λI) = 0 if, and only if, det(B − λI) = 0.

The basic QR algorithm is:
Set A1 = A.
for k = 1, 2, . . .
form the QR factorization Ak = Qk Rk
and set Ak+1 = Rk Qk
end
Proposition. The symmetric matrices A1 , A2, . . . , Ak , . . . are all similar
and thus have the same eigenvalues.
Proof. Since
−1
Ak+1 = Rk Qk = (QT T T
k Qk )Rk Qk = Qk (Qk Rk )Qk = Qk Ak Qk = Qk Ak Qk ,

Ak+1 is symmetric if Ak is, and is similar to Ak . 2

This basic QR algorithm works since Ak → a diagonal matrix as k → ∞, the

diagonal entries of which are the eigenvalues. However, a really practical,
fast algorithm is based on some refinements.
Reduction to tridiagonal form: the idea is to apply explicit similarity
transformations QAQ−1 = QAQT, with Q orthogonal, so that QAQT is
tridiagonal.
Note: direct reduction to triangular form would reveal the eigenvalues, but
is not possible.

1
If  
× × ··· ×
 0 × ··· × 
 
H(w)A =  .. .. ... .. 
 . . . 
0 × ··· ×
then H(w)AH(w)T is generally full, i.e., all zeros created by pre-multiplication
are destroyed by the post-multiplication. However, if
" #
T
γ u
A=
u C
(as A = AT ) and
 
" # α
0  0 
 
w= where H(ŵ)u =  .. ,
ŵ  . 
0
it follows that  
T
γ u
 .. 
 α × . × 
H(w)A = 
 .. .. .. .. ,

 . . . . 
..
0 × . ×
i.e., the uT part of the first row of A is unchanged. However, then
 
γ α 0 ··· 0
 
 α 
 
H(w)AH(w)−1 = H(w)AH(w)T = H(w)AH(w) =  0 ,

 .. B 

 . 
0
where B = H(ŵ)CH T(ŵ), as uTH(ŵ)T = (α, 0, · · · , 0); note that
H(w)AH(w)T is symmetric as A is.
Now we inductively apply this to the smaller matrix B, as described for the
QR factorization but using post- as well as pre-multiplications. The result
of n − 2 such Householder similarity transformations is the matrix
H(wn−2) · · · H(w2)H(w)AH(w)H(w2) · · · H(wn−2),

2
which is tridiagonal.
The QR factorization of a tridiagonal matrix can now easily be achieved
with n − 1 Givens rotations: if A is tridiagonal
J(n − 1, n) · · · J(2, 3)J(1, 2)A = R, upper triangular.
| {z }
T
Q
Precisely, R has a diagonal and 2 super-diagonals,
 
× × × 0 0 0 ··· 0
 0 × × × 0 0 ··· 0 
 
 
 0 0 × × × 0 ··· 0 
 . . .. 
 .. .. . 
R=  0 0 0 0 × × × 0 

 
 
 0 0 0 0 0 × × ×
 
 0 0 0 0 0 0 × ×
0 0 0 0 0 0 0 ×
(exercise: check!). In the QR algorithm, the next matrix in the sequence is
RQ.
Lemma. In the QR algorithm applied to a tridiagonal matrix, the tridiag-
onal form is preserved when Givens rotations are used.
Proof. If Ak = QR = J(1, 2)TJ(2, 3)T · · · J(n − 1, n)TR is tridiagonal,
then Ak+1 = RQ = RJ(1, 2)TJ(2, 3)T · · · J(n − 1, n)T. Recall that post-
multiplication of a matrix by J(i, i + 1)T replaces columns i and i + 1
by linear combinations of the pair of columns, while leaving columns j =
1, 2, . . . , i − 1, i + 2, . . . , n alone. Thus, since R is upper triangular, the
only subdiagonal entry in RJ(1, 2)T is in position (2, 1). Similarly, the
only subdiagonal entries in RJ(1, 2)TJ(2, 3)T = (RJ(1, 2)T)J(2, 3)T are in
positions (2, 1) and (3, 2). Inductively, the only subdiagonal entries in
RJ(1, 2)TJ(2, 3)T · · · J(i − 2, i − 1)TJ(i − 1, i)T
= (RJ(1, 2)TJ(2, 3)T · · · J(i − 2, i − 1)T)J(i − 1, i)T
are in positions (j, j − 1), j = 2, . . . i. So, the lower triangular part of Ak+1
only has nonzeros on its first subdiagonal. However, then since Ak+1 is
symmetric, it must be tridiagonal. 2

3
Using shifts. One further and final step in making an efficient algorithm
is the use of shifts:
for k = 1, 2, . . .
form the QR factorization of Ak − µk I = Qk Rk
and set Ak+1 = Rk Qk + µk I
end
For any chosen sequence of values of µk ∈ R, {Ak }∞ k=1 are symmetric and
tridiagonal if A1 has these properties, and similar to A1 .
The simplest shift to use is an,n , which leads rapidly in almost all cases to
" #
Tk 0
Ak = ,
0T λ

where Tk is n − 1 by n − 1 and tridiagonal, and λ is an eigenvalue of A1.

Inductively, once this form has been found, the QR algorithm with shift
an−1,n−1 can be concentrated only on the n − 1 by n − 1 leading submatrix
Tk . This process is called deflation.
The overall algorithm for calculating the eigenvalues of an n by n sym-
metric matrix:
reduce A to tridiagonal form by orthogonal
(Householder) similarity transformations.
for m = n, n − 1, . . . 2
while am−1,m > tol
[Q, R] = qr(A − am,m ∗ I)
A = R ∗ Q + am,m ∗ I
end while
record eigenvalue λm = am,m
A ← leading m − 1 by m − 1 submatrix of A
end
record eigenvalue λ1 = a1,1

4
Numerical Analysis Hilary Term 2011.
Lecture 10: Best Approximation in Inner-Product Spaces.

Best approximation of functions: given a function f on [a, b], find the

“closest” polynomial/piecewise polynomial (see later sections)/ trigonomet-
ric polynomial (truncated Fourier series).
Norms: are used to measure the size of/distance between elements of a
vector space. Given a vector space V over the field R of real numbers, the
mapping k · k : V → R is a norm on V if it satisfies the following axioms:
(i) kf k ≥ 0 for all f ∈ V , with kf k = 0 if, and only if, f = 0 ∈ V ;
(ii) kλf k = |λ|kf k for all λ ∈ R and all f ∈ V ; and
(iii) kf + gk ≤ kf k + kgk for all f, g ∈ V (the triangle inequality).

Examples: 1. For vectors x ∈ Rn , with x = (x1, x2, . . . , xn)T,

1
√
kxk ≡ kxk2 = (x21 + x22 + · · · + x2n) 2 = xT x
is the ℓ2- or vector two-norm.
2. For continuous functions on [a, b],
kf k ≡ kf k∞ = max |f (x)|
x∈[a,b]

is the L∞ - or ∞-norm.
3. For integrable functions on (a, b),
Z b
kf k ≡ kf k1 = |f (x)| dx
a

is the L1- or one-norm.

4. For functions in
Z b
V = L2w (a, b) ≡ {f : [a, b] → R | w(x)[f (x)]2 dx < ∞}
a
for some given weight function w(x) > 0 (this certainly includes continuous
functions on [a, b], and piecewise continuous functions on [a, b] with a finite
number of jump-discontinuities),
Z b  12
kf k ≡ kf k2 = w(x)[f (x)]2 dx
a

1
is the L2 - or two-norm—the space L2 (a, b) is a common abbreviation for
L2w (a, b) for the case w(x) ≡ 1.
Note: kf k2 = 0 =⇒ f = 0 almost everywhere on [a, b]. We say that a certain property
P holds almost everywhere (a.e.) on [a, b] if property P holds at each point of [a, b] except
perhaps on a subset S ⊂ [a, b] of zero measure. We say that a set S ⊂ R has zero measure (or
that it is of measure zero) if for any ε > 0 there exists a sequence {(αi , βi )}∞
i=1 of subintervals
P∞
of R such that S ⊂ ∪∞ i=1 (αi , βi ) and i=1 (βi − αi ) < ε. Trivially, the empty set ∅(⊂ R) has

zero measure. Any finite subset of R has zero measure. Any countable subset of R, such as
the set of all natural numbers N, the set of all integers Z, or the set of all rational numbers Q,
is of measure zero.
Least-squares polynomial approximation: aim to find the best poly-
nomial approximation to f ∈ L2w (a, b), i.e., find pn ∈ Πn for which
kf − pn k2 ≤ kf − qk2 ∀q ∈ Πn .
n
X
Seeking pn in the form pn(x) = αk xk then results in the minimization
k=0
problem " #2
Z b n
X
min w(x) f (x) − αk xk dx.
(α0 ,...,αn ) a k=0
The unique minimizer can be found from the (linear) system
Z b " n
#2
∂ X
w(x) f (x) − αk xk dx = 0 for each j = 0, 1, . . . , n,
∂αj a
k=0

but there is important additional structure here.

Inner-product spaces: a real inner-product space is a vector space V
over R with a mapping h·, ·i : V × V → R (the inner product) for which
(i) hv, vi ≥ 0 for all v ∈ V and hv, vi = 0 if, and only if v = 0;
(ii) hu, vi = hv, ui for all u, v ∈ V ; and
(iii) hαu + βv, zi = αhu, zi + βhv, zi for all u, v, z ∈ V and all α, β ∈ R.

Examples: 1. V = Rn ,
n
X
T
hx, yi = x y = xiyi ,
i=1

2
where x = (x1, . . . , xn)T and y = (y1, . . . , yn)T .
Z b
2. V = L2w (a, b) = {f : (a, b) → R | w(x)[f (x)]2 dx < ∞},
a
Z b
hf, gi = w(x)f (x)g(x) dx,
a

where f, g ∈ L2w (a, b) and w is a weight-function, defined, positive and

integrable on (a, b).
Notes: 1. Suppose that V is an inner product space, with inner product
1
h·, ·i. Then hv, vi 2 defines a norm on V (see the final paragraph on the
last page for a proof). In Example 2 above, the norm defined by the inner
product is the (weighted) L2 -norm.
2. Suppose that V is an inner product space, with inner product h·, ·i, and
1
let k · k denote the norm defined by the inner product via kvk = hv, vi 2 , for
v ∈ V . The angle θ between u, v ∈ V is
 
hu, vi
θ = cos−1 .
kukkvk
Thus u and v are orthogonal in V ⇐⇒ hx, yi = 0.
E.g., x2 and 43 − x are orthogonal in L2 (0, 1) with inner product hf, gi =
Z 1
f (x)g(x) dx as
0
Z 1
x2 3 1 1


4 − x dx = 4 − 4 = 0.
0
3. Pythagoras Theorem: Suppose that V is an inner-product space with
inner product h·, ·i and norm k · k defined by this inner product. For any
u, v ∈ V such that hu, vi = 0 we have
ku ± vk2 = kuk2 + kvk2.
Proof.
ku ± vk2 = hu ± v, u ± vi = hu, u ± vi ± hv, u ± vi [axiom (iii)]
= hu, u ± vi ± hu ± v, vi [axiom (ii)]
= hu, ui ± hu, vi ± hu, vi + hv, vi
= hu, ui + hv, vi [orthogonality]
= kuk2 + kvk2.

3
4. The Cauchy–Schwarz inequality: Suppose that V is an inner-product
space with inner product h·, ·i and norm k · k defined by this inner product.
For any u, v ∈ V ,
|hu, vi| ≤ kukkvk.
Proof. For every λ ∈ R,

0 ≤ hu − λv, u − λvi = kuk2 − 2λhu, vi + λ2 kvk2 = φ(λ),

which is a quadratic in λ. The minimizer of φ is at λ∗ = hu, vi/kvk2, and

thus since φ(λ∗ ) ≥ 0, kuk2 − hu, vi2/kvk2 ≥ 0, which gives the required
inequality. 2

5. The triangle inequality: Suppose that V is an inner-product space

with inner product h·, ·i and norm k · k defined by this inner product. For
any u, v ∈ V ,
ku + vk ≤ kuk + kvk.
Proof. Note that

ku + vk2 = hu + v, u + vi = kuk2 + 2hu, vi + kvk2.

Hence, by the Cauchy–Schwarz inequality,

ku + vk2 ≤ kuk2 + 2kukkvk + kvk2 = (kuk + kvk)2 .

Taking square-roots yields

ku + vk ≤ kuk + kvk.

2
1
Note: The function k · k : V → R defined by kvk := hv, vi 2 on the inner-
product space V , with inner product h·, ·i, trivially satisfies the first two
axioms of norm on V ; this is a consequence of h·, ·i being an inner product
on V . Result 5 above implies that k · k also satisfies the third axiom of
norm, the triangle inequality.

4
Numerical Analysis Hilary Term 2011.
Lecture 11: Least-Squares Approximation.
Z b
For the problem of least-squares approximation, hf, gi = w(x)f (x)g(x) dx
a
and kf k22 = hf, f i where w(x) > 0 on (a, b).
Theorem. If f ∈ L2w (a, b) and pn ∈ Πn is such that
hf − pn , ri = 0 ∀r ∈ Πn, (1)
then
kf − pn k2 ≤ kf − rk2 ∀r ∈ Πn ,
i.e., pn is a best (weighted) least-squares approximation to f on [a, b].
Proof.
kf − pn k22 = hf − pn, f − pn i
= hf − pn, f − ri + hf − pn , r − pn i ∀r ∈ Πn
Since r − pn ∈ Πn the assumption (??) implies that
= hf − pn, f − ri
≤ kf − pn k2kf − rk2 by the Cauchy–Schwarz inequality.
Dividing both sides by kf − pn k2 gives the required result. 2

Remark: the converse is true too (see Problem Sheet 6, Q9).

This gives a direct way to calculate a best approximation: we want to find
n
X
pn (x) = αk xk such that
k=0
n
!
Z b X
w(x) f − αk xk xi dx = 0 for i = 0, 1, . . . , n. (2)
a k=0
[Note that (??) holds if, and only if,
Z b n
! n ! n
X X X
w(x) f − αk xk βi xi dx = 0 ∀q = βi xi ∈ Πn .]
a k=0 i=0 i=0

However, (??) implies that

Xn Z b  Z b
k+i
w(x)x dx αk = w(x)f (x)xi dx for i = 0, 1, . . . , n
k=0 a a

1
which is the component-wise statement of a matrix equation

Aα = ϕ, (3)

to determine the coefficients α = (α0 , α1, . . . , αn )T , where A = {ai,k , i, k =

0, 1, . . . , n}, ϕ = (f0, f1, . . . , fn)T ,
Z b Z b
k+i
ai,k = w(x)x dx and fi = w(x)f (x)xi dx.
a a

The system (??) are called the normal equations.

Example: the best least-squares approximation to ex on [0, 1] from Π1 in
Z b
hf, gi = f (x)g(x) dx. We want
a
Z 1 Z 1
x
[e − (α0 1 + α1 x)]1 dx = 0 and [ex − (α0 1 + α1 x)]x dx = 0.
0 0
⇐⇒ Z 1 Z 1 Z 1
α0 dx + α1 x dx = ex dx
0 0 0

Z 1 Z 1 Z 1
α0 x dx + α1 2
x dx = ex x dx
0 0 0
i.e., " #" # " #
1 1
2 α0 e−1
=
1
2
1
3 α1 1
=⇒ α0 = 4e − 10 and α1 = 18 − 6e, so p1 (x) := (18 − 6e)x + (4e − 10) is
the best approximation.
Proof that the coefficient matrix A is nonsingular will now establish exis-
tence and uniqueness of (weighted) k · k2 best-approximation.
Theorem. The coefficient matrix A is nonsingular.
Proof. Suppose not =⇒ ∃α 6= 0 with Aα = 0 =⇒ αT Aα = 0
n
X n
X n
X
⇐⇒ αi (Aα)i = 0 ⇐⇒ αi aik αk = 0,
i=0 i=0 k=0

2
 Z b
and using the definition aik = w(x)xk xi dx ,
a
n
X n Z
X b 
⇐⇒ αi w(x)xk xi dx αk = 0.
i=0 k=0 a

Rearranging gives
n
! n
! n
!2
Z b X X Z b X
w(x) αi xi αk xk dx = 0 or w(x) αi xi dx = 0
a i=0 k=0 a i=0

n
X
which implies that αi xi = 0 and thus αi = 0 for i = 0, 1, . . . , n. This
i=0
contradicts the initial supposition, and thus A is nonsingular. 2

3
Numerical Analysis Hilary Term 2011.
Lecture 12: Orthogonal Polynomials.

Gram–Schmidt orthogonalization procedure: the solution of the

normal equations Aα = ϕ for best least-squares polynomial approximation
would be easy if A were diagonal. Instead of {1, x, x2, . . . , xn} as a basis for
Xn
Πn , suppose we have a basis {φ0 , φ1, . . . , φn }. Then pn (x) = βk φk (x),
k=0
and the normal equations become
Z b n
!
X
w(x) f (x) − βk φk (x) φi (x) dx = 0 for i = 0, 1, . . . , n,
a k=0
or equivalently
Xn Z b  Z b
w(x)φk (x)φi(x) dx βk = w(x)f (x)φi(x) dx for i = 0, 1, . . . , n,
k=0 a a

i.e.,
Aβ = ϕ, (1)
where β = (β0, β1, . . . , βn)T , ϕ = (f1, f2, . . . , fn)T and now
Z b Z b
ai,k = w(x)φk (x)φi(x) dx and fi = w(x)f (x)φi(x) dx.
a a
So A is diagonal if
(
b
= 0 i 6= k and
Z
hφi , φk i = w(x)φi(x)φk (x) dx
a 6= 0 i = k.
We can create such a set of orthogonal polynomials
{φ0 , φ1, . . . , φn , . . .},
with φi ∈ Πi for each i, by the Gram–Schmidt procedure, which is based
on the following lemma.
Lemma. Suppose that φ0 , φ1, . . . , φk , with φ
Zi ∈ Πi for each i, are orthogonal
b
with respect to the inner product hf, gi = w(x)f (x)g(x) dx. Then,
a
k
X
k+1
φk+1(x) = x − λi φi (x)
i=0

1
satisfies
Z b
hφk+1 , φj i = w(x)φk+1(x)φj (x) dx = 0, j = 0, 1, . . . , k,
a

when
hxk+1, φj i
λj = , j = 0, 1, . . . , k.
hφj , φj i
Proof. For any j, 0 ≤ j ≤ k,
k
X
hφk+1 , φj i = hxk+1, φj i − λi hφi , φj i
i=0
k+1
= hx , φj i − λj hφj , φj i
by the orthogonality of φi and φj , i 6= j,
= 0 by definition of λj . 2

Notes: 1. The Gram–Schmidt procedure does the above for k = 0, 1, . . . , n

successively.
2. φk is always of exact degree k, so {φ0, φ1 , . . . , φℓ} is a basis for Πℓ for
every ℓ ≥ 0.
3. φk can be normalised/scaled to satisfy hφk , φk i = 1 or to be monic, or
...
Examples: 1. The inner product
Z 1
hf, gi = f (x)g(x) dx
−1

has orthogonal polynomials called the Legendre polynomials,

φ0 (x) ≡ 1, φ1 (x) = x, φ2 (x) = x2 − 13 , φ3 (x) = x3 − 35 x, . . .

2. The inner product

1
f (x)g(x)
Z
hf, gi = √ dx
−1 1 − x2
gives orthogonal polynomials, which are the Chebyshev polynomials,

φ0 (x) ≡ 1, φ1 (x) = x, φ2(x) = 2x2 − 1, φ3 (x) = 4x3 − 3x, . . .

2
3. The inner product
Z ∞
hf, gi = e−x f (x)g(x) dx
0

gives orthogonal polynomials, which are the Laguerre polynomials,

φ0 (x) ≡ 1, φ1 (x) = 1 − x, φ2 (x) = 2 − 4x + x2 ,

φ3(x) = 6 − 18x + 9x2 − x3, . . .

Lemma. Suppose that {φ0 , φ1 , . . . , φn , . . .} are orthogonal polynomials for

a given inner product h·, ·i. Then, hφk , qi = 0 whenever q ∈ Πk−1.
k−1
X
Proof. This follows since if q ∈ Πk−1, then q(x) = σiφi (x) for some
i=0
σi ∈ R, i = 0, 1, . . . , k − 1, so
k−1
X
hφk , qi = σi hφk , φi i = 0. 2
i=0
k
X
Remark: note from the above argument that if q(x) = σiφi (x) is of
i=0
exact degree k (so σk 6= 0), then hφk , qi = σk hφk , φk i 6= 0.
Theorem. Suppose that {φ0, φ1 , . . . , φn , . . .} is a set of orthogonal polyno-
mials. Then, there exist sequences of real numbers (αk )∞ ∞ ∞
k=1, (βk )k=1 , (γk )k=1
such that a three-term recurrence relation of the form

φk+1 (x) = αk (x − βk )φk (x) − γk φk−1 (x), k = 1, 2, . . . ,

holds.
Proof. The polynomial xφk ∈ Πk+1, so there exist real numbers

σk,0, σk,1, . . . , σk,k+1

such that
k+1
X
xφk (x) = σk,iφi (x)
i=0

3
as {φ0 , φ1, . . . , φk+1} is a basis for Πk+1. Now take the inner product on
both sides with φj , and note that xφj ∈ Πk−1 if j ≤ k − 2. Thus
Z b Z b
hxφk , φj i = w(x)xφk (x)φj (x) dx = w(x)φk (x)xφj (x) dx = hφk , xφj i = 0
a a

by the above lemma for j ≤ k − 2. In addition

* k+1 + k+1
X X
σk,iφi , φj = σk,ihφi , φj i = σk,j hφj , φj i
i=0 i=0

by the linearity of h·, ·i and orthogonality of φk and φj for k 6= j. Hence

σk,j = 0 for j ≤ k − 2, and so

xφk (x) = σk,k+1φk+1(x) + σk,k φk (x) + σk,k−1φk−1(x).

Taking the inner product with φk+1 reveals that

hxφk , φk+1i = σk,k+1hφk+1, φk+1i,

so σk,k+1 6= 0 by the above remark as xφk is of exact degree k + 1. Thus,

1 σk,k−1
φk+1(x) = (x − σk,k )φk (x) − φk−1(x),
σk,k+1 σk,k+1
which is of the given form, with
1 σk,k−1
αk = , βk = σk,k , γk = , k = 1, 2, . . . .
σk,k+1 σk,k+1
That completes the proof. 2

Example. The inner product

Z ∞
2
hf, gi = e−x f (x)g(x) dx
−∞

has orthogonal polynomials called the Hermite polynomials,

φ0 (x) ≡ 1, φ1 (x) = 2x, φk+1 (x) = 2xφk (x) − 2kφk−1(x) for k ≥ 1.

4
Matlab:

% cat hermite_polys.m

x=linspace(-2.2,2.2,200);
oldH=ones(1,200); plot(x,oldH), hold on
newH=2*x; plot(x,newH)
for n=1:2,...
newnewH=2*x.*newH-2*n*oldH; plot(x,newnewH),...
oldH=newH;newH=newnewH;
end

% matlab
>> hermite_polys

60

40

20

−20

−40

−60
−2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5

5
Numerical Analysis Hilary Term 2011.
Lecture 13: Gaussian Quadrature.

Suppose that w is a weight-function, defined, positive and integrable on the

open interval (a, b) of R.
Lemma. Let {φ0 , φ1 , . . . , φn , . . .} be orthogonal polynomials for the inner
Z b
product hf, gi = w(x)f (x)g(x) dx. Then, for each k = 0, 1, . . ., φk has k
a
distinct roots in the interval (a, b).
Proof. Since φ0 (x) ≡ const. 6= 0, the result is trivially true for k = 0.
Z b
Suppose that k ≥ 1: hφk , φ0 i = w(x)φk (x)φ0(x) dx = 0 with φ0 constant
Z b a

implies that w(x)φk (x) dx = 0 with w(x) > 0, x ∈ (a, b). Thus φk (x)
a
must change sign in (a, b), i.e., φk has at least one root in (a, b).
Suppose that there are ℓ points a < r1 < r2 < · · · < rℓ < b where φk changes
sign for some 1 ≤ ℓ ≤ k. Then
ℓ
Y
q(x) = (x − rj ) × the sign of φk on (rℓ, b)
j=1

has the same sign as φk on (a, b). Hence

Z b
hφk , qi = w(x)φk (x)q(x) dx > 0,
a

and thus it follows from the previous lemma that q, (which is of degree ℓ)
must be of degree ≥ k, i.e., ℓ ≥ k. Therefore ℓ = k, and φk has k distinct
roots in (a, b). 2

Quadrature revisited. The above lemma leads to very efficient quadra-

ture rules since it answers the question: how should we choose the quadra-
ture points x0, x1, . . . , xn in the quadrature rule
Z b n
X
w(x)f (x) dx ≈ wj f (xj ) (1)
a j=0

1
so that the rule is exact for polynomials of degree as high as possible? (The
case w(x) ≡ 1 is the most common.)
Recall: the Lagrange interpolating polynomial
n
X
pn = f (xj )Ln,j ∈ Πn
j=0

is unique, so if f ∈ Πn =⇒ pn ≡ f whatever interpolation points are used,

and moreover
Z b Z b n
X
w(x)f (x) dx = w(x)pn(x) dx = wj f (xj ),
a a j=0

where Z b
wj = w(x)Ln,j (x) dx (2)
a
exactly!
Theorem. Suppose that x0 < x1 < · · · < xn are the roots of the n + 1-st
degree orthogonal polynomial φn+1 with respect to the inner product
Z b
hg, hi = w(x)g(x)h(x) dx.
a

Then, the quadrature formula (1) with weights satisfying (2) is exact when-
ever f ∈ Π2n+1.
Proof. Let p ∈ Π2n+1, then by the Division Algorithm p(x) = q(x)φn+1(x)+
r(x) with q, r ∈ Πn . So
Z b Z b Z b Xn
w(x)p(x) dx = w(x)q(x)φn+1(x) dx + w(x)r(x) dx = wj r(xj )
a a a j=0
(3)
since the integral involving q ∈ Πn is zero by the lemma above and the other
is integrated exactly since r ∈ Πn . Finally p(xj ) = q(xj )φn+1(xj ) + r(xj ) =
r(xj ) for j = 0, 1, . . . , n as the xj are the roots of φn+1. So (3) gives
Z b Xn
w(x)p(x) dx = wj p(xj ),
a j=0

2
where wj is given by (2) whenever p ∈ Π2n+1. 2

These quadrature rules are called Gaussian Quadratures.

• For w(x) ≡ 1, (a, b) = (−1, 1) we have Gauss–Legendre Quadrature.
• For w(x) = (1 − x2)−1/2 and (a, b) = (−1, 1) we have Gauss–Chebyshev
Quadrature.
• For w(x) = e−x and (a, b) = (0, ∞) we have Gauss–Laguerre Quadra-
ture.
2
• For w(x) = e−x and (a, b) = (−∞, ∞) we have Gauss–Hermite Quadra-
ture.
They give better accuracy than Newton–Cotes quadrature for the same
number of function evaluations.
Note that by the simple linear change of variable t = (2x − a − b)/(b − a),
which maps [a, b] → [−1, 1], we can evaluate for example
Z b Z 1  
(b − a)t + b + a b − a
f (x) dx = f dt
a −1 2 2
n  
b−aX b−a b+a
≃ wj f tj + ,
2 j=0 2 2
where ≃ denotes “quadrature” and the tj , j = 0, 1, . . . , n, are the roots of
the n + 1-st degree Legendre polynomial.
Example. 2-point Gauss–Legendre Quadrature: φ2 = x2 − 31 =⇒ t0 = − √13 ,
t1 = √13 and
Z 1 Z 1 √ !
x − √13 3 1
w0 = dx = − x − dx = 1
−1 − √3 − √3 2 2
1 1
−1

with w1 = 1, similarly. So e.g., changing variables x = (t + 3)/2,

Z 2
1 1 1 2 1 1
Z
dx = dt ≃ + = 0.6923077 . . . .
1 x 2 −1 t + 3 3 + √1
3
3 − √1
3

Note that for the Trapezium Rule (also two evaluations of the integrand)
gives Z 2  
1 1 1
dx ≃ + 1 = 0.75,
1 x 2 2
3
 2
1
Z
whereas dx = ln 2 = 0.6931472 . . . .
1 x
Theorem. Error in Gaussian Quadrature: suppose that f (2n+2) is continu-
ous on (a, b). Then,
Z b n n
f (2n+2)(η) b
X Z Y
w(x)f (x) dx = wj f (xj ) + w(x) (x − xj )2 dx
a j=0
(2n + 2)! a j=0

for some η ∈ (a, b).

Proof. The proof is based on Hermite Interpolating Polynomial H2n+1 to
′
f on x0, x1, . . . , xn. [Recall that H2n+1(xj ) = f (xj ) and H2n+1 (xj ) = f ′(xj )
for j = 0, 1, . . . , n.] The error in Hermite interpolation is
n
1 (2n+2)
Y
f (x) − H2n+1(x) = f (η(x)) (x − xj )2
(2n + 2)! j=0

for some η = η(x) ∈ (a, b). Now H2n+1 ∈ Π2n+1, so

Z b Xn n
X
w(x)H2n+1(x) dx = wj H2n+1(xj ) = wj f (xj ),
a j=0 j=0

the first identity because Gaussian Quadrature is exact for polynomials of

this degree and the second by interpolation. Thus
Z b Xn Z b
w(x)f (x) dx − wj f (xj ) = w(x)[f (x) − H2n+1(x)] dx
a j=0 a
b n
1
Z Y
(2n+2)
= f (η(x))w(x) (x − xj )2 dx,
(2n + 2)! a j=0

and hence the required result follows from the Integral Mean Value Theorem
as w(x) nj=0(x − xj )2 ≥ 0.
Q
2

Remark: the “direct” approach of finding Gaussian Quadrature formulae

sometimes works for small n, but is usually hard.
Example. To find the two-point Gauss–Legendre rule w0f (x0) + w1f (x1)
on (−1, 1) with weight function w(x) ≡ 1, we need to be able to integrate

4
any cubic polynomial exactly, so
Z 1
2= 1 dx = w0 + w1 (4)
−1
Z 1
0= x dx = w0 x0 + w1 x1 (5)
−1
Z 1
3 =
2
x2 dx = w0x20 + w1x21 (6)
−1
Z 1
0= x3 dx = w0x30 + w1x31. (7)
−1

These are four nonlinear equations in four unknowns w0 , w1 , x0 and x1.

Equations (5) and (7) give
" #" # " #
x0 x1 w0 0
3 3 = ,
x0 x1 w1 0

which implies that

x0x31 − x1x30 = 0
for w0, w1 6= 0, i.e.,
x0x1(x1 − x0)(x1 + x0) = 0.
If x0 = 0, this implies w1 = 0 or x1 = 0 by (5), either of which contradicts
(6). Thus x0 6= 0, and similarly x1 6= 0. If x1 = x0, (5) implies w1 = −w0,
which contradicts (4). So x1 = −x0 , and hence (5) implies w1 = w0 . But
then (4) implies that w0 = w1 = 1 and (6) gives

x0 = − √13 and x1 = √1
3
,

which are the roots of the Legendre polynomial x2 − 31 .

5
Table 1: Abscissas xj (zeros of Legendre polynomials) and weight factors wj for Gaussian
Z 1 Xn
Quadrature: f (x) dx ≃ wj f (xj ) for n = 0 to 6.
−1 j=0

xj wj
n=0 0.000000000000000e+0 2.000000000000000e+0
n=1 5.773502691896258e−1 1.000000000000000e+0
−5.773502691896258e−1 1.000000000000000e+0
7.745966692414834e−1 5.555555555555556e−1
n=2 0.000000000000000e+0 8.888888888888889e−1
−7.745966692414834e−1 5.555555555555556e−1
8.611363115940526e−1 3.478548451374539e−1
n=3 3.399810435848563e−1 6.521451548625461e−1
−3.399810435848563e−1 6.521451548625461e−1
−8.611363115940526e−1 3.478548451374539e−1
9.061798459386640e−1 2.369268850561891e−1
5.384693101056831e−1 4.786286704993665e−1
n=4 0.000000000000000e+0 5.688888888888889e−1
−5.384693101056831e−1 4.786286704993665e−1
−9.061798459386640e−1 2.369268850561891e−1
9.324695142031520e−1 1.713244923791703e−1
6.612093864662645e−1 3.607615730481386e−1
n=5 2.386191860831969e−1 4.679139345726910e−1
−2.386191860831969e−1 4.679139345726910e−1
−6.612093864662645e−1 3.607615730481386e−1
−9.324695142031520e−1 1.713244923791703e−1
9.491079123427585e−1 1.294849661688697e−1
7.415311855993944e−1 2.797053914892767e−1
4.058451513773972e−1 3.818300505051189e−1
n=6 0.000000000000000e+0 4.179591836734694e−1
−4.058451513773972e−1 3.818300505051189e−1
−7.415311855993944e−1 2.797053914892767e−1
−9.491079123427585e−1 1.294849661688697e−1

6
Table 2: Abscissas xj (zeros of Legendre polynomials) and weight factors wj for Gaussian
Z 1 Xn
Quadrature: f (x) dx ≃ wj f (xj ) for n = 7 to 9.
−1 j=0

xj wj
9.602898564975362e−1 1.012285362903763e−1
7.966664774136267e−1 2.223810344533745e−1
5.255324099163290e−1 3.137066458778873e−1
n=7 1.834346424956498e−1 3.626837833783620e−1
−1.834346424956498e−1 3.626837833783620e−1
−5.255324099163290e−1 3.137066458778873e−1
−7.966664774136267e−1 2.223810344533745e−1
−9.602898564975362e−1 1.012285362903763e−1
9.681602395076261e−1 8.127438836157441e−2
8.360311073266358e−1 1.806481606948574e−1
6.133714327005904e−1 2.606106964029355e−1
3.242534234038089e−1 3.123470770400028e−1
n=8 0.000000000000000e+0 3.302393550012598e−1
−3.242534234038089e−1 3.123470770400028e−1
−6.133714327005904e−1 2.606106964029355e−1
−8.360311073266358e−1 1.806481606948574e−1
−9.681602395076261e−1 8.127438836157441e−2
9.739065285171717e−1 6.667134430868814e−2
8.650633666889845e−1 1.494513491505806e−1
6.794095682990244e−1 2.190863625159820e−1
4.333953941292472e−1 2.692667193099964e−1
n=9 1.488743389816312e−1 2.955242247147529e−1
−1.488743389816312e−1 2.955242247147529e−1
−4.333953941292472e−1 2.692667193099964e−1
−6.794095682990244e−1 2.190863625159820e−1
−8.650633666889845e−1 1.494513491505806e−1
−9.739065285171717e−1 6.667134430868814e−2

7
Numerical Analysis Hilary Term 2011.
Lectures 14–15: Piecewise Polynomial Interpolation: Splines.

Sometimes a ‘global’ approximation like Lagrange Interpolation is not ap-

propriate, e.g., for ’rough’ data.

On the left the Lagrange Interpolant p7 ‘wiggles’ through the points, while
on the right a piecewise linear interpolant (‘join the dots’), or linear spline
interpolant, s appears to represent the data better.
Remark: for any given data s clearly exists and is unique.
Suppose that a = x0 < x1 < · · · < xn = b. Then, s is linear on each interval
[xi−1, xi] for i = 1, . . . , n and continuous on [a, b]. The xi, i = 0, 1, . . . , n,
are called the knots of the linear spline.
Notation: f ∈ Ck [a, b] if f, f ′, . . . , f k exist and are continuous on [a, b].
Theorem. Suppose that f ∈ C2[a, b]. Then,
1
kf − sk∞ ≤ h2 kf ′′k∞
8
where h = max (xi − xi−1) and kf ′′k∞ = max |f ′′(x)|.
1≤i≤n x∈[a,b]
Proof. For x ∈ [xi−1, xi], the error from linear interpolation is
1
f (x) − s(x) = f ′′(η)(x − xi−1)(x − xi )
2
where η = η(x) ∈ (xi−1, xi). However, |(x − xi−1)(x − xi)| = (x − xi−1)(xi −
x) = −x2 + x(xi−1 + xi ) − xi−1xi, which has its maximum value when
2x = xi + xi−1, i.e., when x − xi−1 = xi − x = 12 (xi − xi−1). Thus, for any
x ∈ [xi−1, xi], i = 1, 2, . . . , n, we have
1 1
|f (x) − s(x)| ≤ kf ′′k∞ max |(x − xi−1)(x − xi)| = h2 kf ′′k∞ . 2
2 x∈[xi−1 ,xi ] 8
Note that s may have discontinuous derivatives, but is a locally defined
approximation, since changing the value of one data point affects the ap-
proximation in only two intervals.

1
To get greater smoothness but retain some ‘locality’, we can define cubic
splines s ∈ C2[a, b]. For a given ‘partition’, a = x0 < x1 < · · · < xn = b,
there are (generally different!) cubic polynomials in each interval (xi−1, xi),
i = 1, . . . , n, which are ’joined’ at each knot to have continuity and conti-
nuity of s′ and s′′ . Interpolating cubic splines also satisfy s(xi) = fi for
given data fi, i = 0, 1, . . . , n.
Remark: if there are n intervals, there are 4n free coefficients (four for
each cubic ‘piece’), but 2n interpolation conditions (one each at the ends of
each interval), n − 1 derivative continuity conditions (at x1, . . . , xn−1) and
n − 1 second derivative continuity conditions (at the same points), giving a
total of 4n−2 conditions (which are linear in the free coefficients). Thus the
spline is not unique. So we need to add two extra conditions to generate a
spline that might be unique. There are three common ways of doings this:

(a) specify s′ (x0) = f ′ (x0) and s′ (xn ) = f ′(xn); or

(b) specify s′′ (x0) = 0 = s′′ (xn) — this gives a natural cubic spline; or

(c) enforce continuity of s′′′ at x1 and xn−1 (which implies that the first
two pieces are the same cubic spline, i.e., on [x0, x2], and similarly for
the last two pieces, i.e., on [xn−2, xn], from which it follows that x1 and
xn−1 are not knots! — this is usually described as the ‘not a knot’
end-conditions).

We may describe a cubic spline within the i-th interval as

(
ai x3 + bix2 + ci x + di for x ∈ (xi−1, xi)
si (x) =
0 otherwise

and overall as
 n
X
si (x) for x ∈ [x0, xn] \ {x0, x1, . . . , xn}


s(x) = i=1

f (xi) for x = xi, i = 0, 1, . . . , n.


2
The 4n linear conditions for an interpolating cubic spline s are:

si (x−
i ) = f (xi)
s1(x0) = f (x0) si+1(x+ i ) = f (xi) sn (xn) = f (xn)
s1 (x0) = f ′ (x0) (a)
′
si (xi) − si+1(xi) = 0 s1(xn) = f ′ (xn) (a)
′ ′ ′ (1)
or s′′1 (x0) = 0 (b) s′′i (xi) − s′′i+1(xi) = 0 or s′′n (xn) = 0 (b)
i = 1, . . . , n − 1.

We may write this as Ay = g, with

y = (a1 , b1, c1, d1, a2 , . . . , dn−1, an , bn, cn , dn)T

and the various entries of g are f (xi), i ∈ 0, 1, . . . , n, and f ′(x0), f ′(xn) and
zeros for (a) and zeros for (b).
So if A is nonsingular, this implies that y = A−1g, that is there is a unique
set of coefficients {a1 , b1, c1 , d1, a2 , . . . , dn−1, an , bn, cn , dn}. We now prove
that if Ay = 0 then y = 0, and thus that A is nonsingular for cases (a) and
(b) — it is also possible, but more complicated, to show this for case (c).
Theorem. If f (xi) = 0 at the knots xi, i = 1, . . . , n, and additionally
f ′ (x0) = 0 = f ′ (xn) for case (a), then s(x) = 0 for all x ∈ [x0, xn].
Proof. Consider
Z xn n Z
X xi
′′ 2
(s (x)) dx = (s′′i (x))2 dx
x0 i=1 xi−1
n n Z xi
x
X X
= [s′i (x)s′′i (x)]xii−1 − s′i (x)s′′′
i (x) dx
i=1 i=1 xi−1

using integration by parts. However,

Z xi Z xi
xi
s′i (x)s′′′ ′′′
i (x) dx = si (x) s′i (x) dx = s′′′
i (x) [si (x)]xi−1 = 0
xi−1 xi−1

since s′′′
i (x) is constant on the interval (xi−1, xi) and si (xi−1) = 0 = si (xi ).
Thus, matching first and second derivatives at the knots, telescopic cancel-

3
lation gives
Z xn n
x
X
′′ 2
(s (x)) dx = [s′i (x)s′′i (x)]xii−1
x0 i=1
= s′1(x1)s′′1 (x1) − s′1 (x0)s′′1 (x0)
+ s′2(x2)s′′2 (x2) − s′2 (x1)s′′2 (x1) + · · ·
+ s′n−1(xn−1)s′′n−1(xn−1) − s′n−1 (xn−2)s′′n−1(xn−2)
+ s′n (xn)s′′n(xn) − s′n (xn−1)s′′n (xn−1)
= s′n (xn)s′′n(xn) − s′1 (x0)s′′1 (x0).
However, in case (a), f ′(x0) = f ′(xn) =⇒ s′1 (x0) = 0 = s′n (xn), while in
case (b) s′′1 (x0) = 0 = s′′n (xn). Thus
Z xn
(s′′(x))2 dx = 0,
x0

which implies that s′′i (x) = 0 and thus si (x) = ci x + di . Since s(xi−1) = 0 =
s(xi), s(x) is identically zero on [x0, xn]. 2

Constructing cubic splines. Note that (1) provides a constructive

method for finding an interpolating spline, but generally this is not used.
Motivated by the next result, it is better to find a good basis.
Proposition. The set of natural cubic splines on a given set of knots x0 <
x1 < · · · < xn is a vector space.
Proof. If p, q ∈ C2[a, b] =⇒ αp + βq ∈ C2[a, b] and p, q ∈ Π3 =⇒ αp +
βq ∈ Π3 for every α, β ∈ R. Finally, the natural end-conditions (b) =⇒
(αp + βq)′′(x0) = 0 = (αp + βq)′′(xn) whenever p′′ and q ′′ are zero at x0 and
xn . 2

Best spline bases: the Cardinal splines, Ci, i = 0, 1, . . . , n, defined as

the interpolatory natural cubic splines satisfying
(
1 i=j
Ci(xj ) = δij =
0 i 6= j,
are a basis for which n
X
s(x) = f (xi)Ci(x)
i=0

4
is the interpolatory natural cubic spline to f . These have the disadvantage
that if any xi is changed, all of the Ci s change, — which is clear from writing
down Ci (x) explicitly:
(x − x1 ) · · · (x − xi−1)(x − xi+1) · · · (x − xn )
Ci (x) = .
(xi − x1) · · · (xi − xi−1)(xi − xi+1) · · · (xi − xn)
Preferred are the B-splines (locally) defined by Bi (xi) = 1 for i =
2, 3, . . . , n − 2, Bi (x) ≡ 0 for x ∈/ (xi−2, xi+2), Bi a cubic spline with knots
xj , j = 0, 1, . . . , n, with special definitions for B0 , B1 , Bn−1 and Bn .
Example/construction: Cubic B-spline with knots 0, 1, 2, 3, 4. On [0, 1],

B(x) = ax3

for some a in order that B, B ′ and B ′′ are continuous at x = 0 (recall that

B(x) is required to be identically zero for x < 0). So

B(1) = a, B ′ (1) = 3a, and B ′′(1) = 6a.

On [1, 2], since B is a cubic polynomial, using Taylor’s Theorem,

B ′′ (1)
′
B(x) = B(1) + B (1)(x − 1) + (x − 1)2 + β(x − 1)3
2

= a + 3a(x − 1) + 3a(x − 1)2 + β(x − 1)3

for some β, and since we require B(2) = 1, then β = 1 − 7a. Now, in order
to continue, by symmetry, we must have B ′ (2) = 0, i.e.,

3a + 6a(x − 1)x=2 + 3(1 − 7a)(x − 1)2x=2 = 3 − 12a = 0

and hence a = 14 . So



 0 for x<0
1 3
4x for x ∈ [0, 1]




 − 3 (x − 1)3 + 3 (x − 1)2 + 3 (x − 1) +

1
for x ∈ [1, 2]
4 4 4 4
B(x) = 3 2

 − 4 (3 − x) + 4 (3 − x) + 34 (3 − x) +
3 3 1
4 for x ∈ [2, 3]
3

4 (4 − x) for x ∈ [3, 4]
1





 0 for x > 4.

5
More generally: B-spline on xi = a + hi, where h = (b − a)/n.


 0 for x < xi−2
(x − xi−2)3





3
for x ∈ [xi−2, xi−1]
4h


3 2

 − 3(x − xi−1) + 3(x − xi−1) + 3(x − xi−1) + 1 for x ∈ [xi−1, xi]



Bi (x) = 4h3 3
4h2 2
4h 4
3(x i+1 − x) 3(x i+1 − x) 3(x i+1 − x) 1
− + + + for x ∈ [xi, xi+1]


4h3 4h2 4h 4



(xi+2 − x)3


for x ∈ [xi+1, xi+2]


3



 4h
0 for x > xi+2.


xi−2 xi−1 xi xi+1 xi+2

The ‘end’ B-splines B0, B1 , Bn−1 and Bn are defined analogously by

introducing ‘phantom’ knots x−2, x−1, xn+1 and xn+2. The (cubic) B-spline
basis is only locally affected if some xi is changed.
Spline interpolation: find the natural cubic spline
n
X
s(x) = cj Bj (x),
j=0

which interpolates fi at xi for i = 0, 1, . . . , n. Require

n
X
fi = cj Bj (xi) = ci−1Bi−1(xi) + ci Bi (xi) + ci+1Bi+1(xi).
j=0

For equally-spaced data

fi = 41 ci−1 + ci + 14 ci+1,

6
i.e.,     
1 1
4 c0 f0
1 ...   c1   f1
 1
   
4
 
... ... ...  .   .
  ..  =  ..
 
 .
    
... 1 cn−1   fn−1
 1
   
 4  
1
4 1 cn fn
For linear splines, a similar local basis of ‘hat functions’ or Linear B-
splines φi (x) exist:
 x − xi−1
 x ∈ (xi−1, xi)
xi − xi−1



φi (x) = x − xi+1
x ∈ (xi, xi+1)


 xi − xi+1
0 x∈/ (xi−1, xi+1)


xi−2 xi−1 xi xi+1 xi+2

and provide a C0 piecewise basis.

Matlab:
% cat spline_example.m

% needs the number N of interpolation points to be defined

N=9;

% these will be the knot points and the interpolation points

x=linspace(-4,4,N);

% a vector of the values of the function f=1/(1+x^2) at the interpolation points

ypoints=1./(1+x.^2);
y=[8/(17^2),ypoints,-8/(17^2)];

% calculates the spline interpolant: see help spline

% an extended vector to include the slope at the first and last

7
% interpolation point - this is one of the end-point choices available with
% the matlab command spline (and is what is called option (a) in lectures)
% (f’ = -2x/(1+x^2)^2, so f’(-5) = 10/26^2 and f’(5) = -10/26^2)
s=spline(x,y);

% a fine mesh on which we plot f

fine=linspace(-4,4,200);

%> help ppval

%
% PPVAL Evaluate piecewise polynomial.
% V = PPVAL(PP,XX) returns the value at the points XX of the piecewise
% polynomial contained in PP, as constructed by SPLINE or the spline utility
% MKPP.
%
% See also SPLINE, MKPP, UNMKPP.
plot(fine,ppval(s,fine)),pause

% the function f on the fine mesh

f=1./(1+fine.^2);

% to see the function (in red) and the spline interpolant (in blue) on the
% same figure
hold on
plot(fine,f,’r’),pause

% marks the interpolating values (with black circles)

plot(x,ypoints,’ko’),pause

% To see how the Lagrange interpolating polynomial (in green) does:

p=lagrange(x,ypoints);
plot(fine,polyval(p,fine),’g’),pause

% matlab
>> spline_example

8
 1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
−4 −3 −2 −1 0 1 2 3 4

Error analysis for cubic splines

Theorem. Amongst all functions t ∈ C2[x0, xn] that interpolate f at xi,
i = 0, 1, . . . , n, the unique function that minimizes
Z xn
[t′′ (x)]2 dx
x0

is the natural cubic spline s. Moreover, for any such t,

Z xn Z xn Z xn
[t′′ (x)]2 dx − [s′′(x)]2 dx = [t′′ (x) − s′′ (x)]2 dx.
x0 x0 x0

Proof. See exercises (uses integration by parts and telescopic cancellation,

and is similar to the proof of existence above). 2

We will also need:

Lemma. (Cauchy–Schwarz inequality): if f, g ∈ C[a, b], then
Z b 2 Z b Z b
2
f (x)g(x) dx ≤ [f (x)] dx [g(x)]2 dx.
a a a

Proof. For any λ ∈ R,

Z b Z b Z b Z b
0≤ [f (x) − λg(x)]2 dx = [f (x)]2 dx − 2λ [f (x)g(x)] dx + λ2 [g(x)]2 dx.
a a a a

9
The result then follows directly since the discriminant of this quadratic must
be nonpositive. 2

Theorem. For the natural cubic spline interpolant s of f ∈ C2 [x0, xn] at

x0 < x1 < · · · < xn with h = max1≤i≤n(xi − xi−1), we have that
Z xn  12 Z xn  21
1 3
kf ′ − s′ k∞ ≤ h 2 [f ′′(x)]2 dx and kf − sk∞ ≤ h 2 [f ′′(x)]2 dx .
x0 x0

Proof. Let e := f − s. Take any x ∈ [x0, xn], in which case x ∈ [xj−1, xj ]

for some j ∈ 1, . . . , n. Then e(xj−1) = 0 = e(xj ) as s interpolates f . So by
the Mean-Value Theorem, there is a c ∈ (xj−1, xj ) with e′ (c) = 0. Hence
Z x
′
e (x) = e′′ (t) dt. Then the Cauchy–Schwarz inequality gives that
c
x
Z  Z x 
|e′ (x)|2 ≤   [e′′(t)]2 dt .
   
dt  
c c

However,
Z x  the first required inequality then follows since for x ∈ [xj−1, xj ],
 

 dt ≤ h and because the previous theorem gives that

c
Z x  Z x  Z xn
 [e′′ (t)]2 dt ≤  [f ′′(t)]2 dt ≤ [f ′′(x)]2 dx.
   
   
c c x0

The remaining result follows from Taylor’s Theorem. 2

Theorem. Suppose that f ∈ C4[a, b] and s satisfies end-conditions (a).

Then,
5 4 (4)
kf − sk∞ ≤ h kf k∞
384
and √
9 + 3 3 (4)
kf ′ − s′ k∞ ≤ h kf k∞ ,
216
where h = max1≤i≤n(xi − xi−1).
Proof. Beyond the scope of this course. 2

Similar bounds exist for natural cubic splines and splines satisfying end-
condition (c).

10
Numerical Analysis Hilary Term 2011.
Lecture 16: Richardson Extrapolation.

Extrapolation is based on the general idea that if Th is an approximation

to T , computed by a numerical approximation with (small!) parameter h,
and if there is an error formula of the form
T = Th + K1h + K2 h2 + · · · + O(hn ) (1)
then T = Tk + K1k + K2k 2 + · · · + O(k n ) (2)
for some other value, k, of the small parameter. In this case subtracting
(1) from (2) gives
(k − h)T = kTh − hTk + K2(kh2 − hk 2 ) + · · ·
i.e., the linear combination
kTh − hTk
=T + K kh
| 2{z }
+ ···
k − h
| {z }
2nd order error
“extrapolated formula”
In particular if only even terms arise:
T = Th + K2h2 + K4h4 + · · · + O(h2n )
h2 h4 h2n
and k = 12 h : T = T h + K2 + K4 + · · · + O( 2n )
2 4 16 2
4T h − Th K4 4
then T = 2
− h + · · · + O(h2n ).
3 4
This is the first step of Richardson Extrapolation. Call this new, more
accurate formula
(2) 4T h − Th
Th := 2
,
3
(1)
where Th := Th. Then the idea can be applied again:
(2) (2)
T = Th + K4 h4 + · · · + O(h2n )
4
(2) (2) h
and T = T h + K4 + · · · + O(h2n )
2 16
(2) (2)
16T h − Th (3)
so T = 2
+ K6 h6 + · · · + O(h2n )
| 15
{z }
(3)
Th
1
is a more accurate formula again. Inductively we can define
(j) 1 
j−1 (j−1) (j−1)

Th := j−1 4 Th − Th
4 −1 2

for which
(j)
T = Th + O(h2j )
so long as there are high enough order terms in the error series.
Example: approximation of π by inscribed polygons in unit circle. For a
regular n-gon, the circumference = 2n sin(π/n) ≤ 2π, so let cn = n sin(π/n) ≤
π, or if we put h = 1/n,
1 π 3 h2 π 5 h4
sin(πh) = π −
cn = + + ···
h 6 120
so that we can use Richardson Extrapolation. Indeed c2 = 2 and
q
c2n = 2nr
sin(π/2n) = 2n 1
2 (1 − cos(π/n))
q r q
2
= 2n 1
2 (1 − 1 − sin (π/n)) = 2n 1
2 (1 − 1 − (cn /n)2) .

So1 c4 = 2.8284, c8 = 3.0615, c16 = 3.1214. Extrapolating between c4 and

(2) (2)
c8 we get c4 = 3.1391 and similarly from c8 and c16 we get c8 = 3.1214.
(2) (2) (3)
Extrapolating again between c4 and c8 , we get c4 = 3.141590 . . ..
Example 2: Romberg Integration.
Z b
Consider the Composite Trapezium
Rule for integrating T = f (x) dx:
a
 n

h 2X−1
Th = f (a) + f (b) + 2 f (xj )
2 j=1

with x0 = a, xj = a + jh and h = (b − a)/2n. Recall from Lecture 3 that

2
the error is (b − a) h12 f ′′(ξ) for some ξ ∈ (a, b). If there were an (asymptotic)
error series of the form
Z b
f (x) dx − Th = K2h2 + K4h4 + · · ·
a

we could apply Richardson Extrapolation as above to yield

4T h − Th
T− 2
= K4h4 + · · ·
3
q p
1
This expression is sensitive to roundoff errors, so we rewrite it as c2n = cn / 1
2 + 1
2 1 − (cn /n)2 .

2
There is such as series: the Euler–Maclaurin formula
Z b r
B2k 2k (2k−1)
X
f (x) dx − Th = − h [f (b) − f (2k−1)(a)]
k=1 (2k)!
a
h2r+1B2r+2 (2r+2)
+(b − a) f (ξ)
(2r + 2)!
where ξ ∈ (a, b) and B2k are called the Bernoulli numbers, defined by
x ∞
X xℓ
= Bl
ex − 1 ℓ=0 ℓ!
so that B2 = 1
6 , B4 = − 301 , etc.
Romberg Integration is composite Trapezium for n = 0, 1, 2, 3, and
the repeated application of Richardson Extrapolation. Changing notation
(Th → Tn , h = stepsize, 2n = number of composite steps), we have
b−a
T0 = [f (a) + f (b)] = R0,0
2
b−a
T1 = [f (a) + f (b) + 2f (a + 12 (b + a))]
4
= 12 [R0,0 + (b − a)f (a + 12 (b + a))] = R1,0 .

Extrapolation then gives

4R1,0 − R0,0
R1,1 = .
3
with error O(h4 ). Also
b−a
T2 = [f (a) + f (b) + 2f (a + 12 (b + a))
8
+ 2f (a + 14 (b + a)) + 2f (a + 34 (b + a))]
" #
1 b−a
= R1,0 + [f (a + 4 (b + a)) + f (a + 4 (b + a)) = R2,0.
1 3
2 2
Extrapolation gives
4R2,0 − R1,0
R2,1 =
3
4
with error O(h ). Extrapolation again gives
16R2,1 − R1,1
R2,2 =
15

3
now with error O(h6 ). At the ith stage
" 2i−2 ! ! 
1 b−a X 1 b−a
Ti = Ri,0 = Ri−1,0 + i−2 f a+ j−  .
2 2 j=1 2 2i−1
| {z }
evaluations at new interlacing points
Extrapolate
4j Ri,j−1 − Ri−1,j−1
Ri,j = for j = 1, 2, . . .
4j − 1
This builds a triangular table:
R0,0
R1,0 R1,1
R2,0 R2,1 R2,2
.. .. .. . . .
. . .
Ri,0 Ri,1 Ri,2 . . . Ri,i
Theorem: Composite Composite
Trapezium Simpson
Notes 1. The integrand must have enough derivatives for the Euler–
Maclaurin series to exist (the whole procedure is based on this!).
Z b Z b
2. Rn,n → f (x) dx in general much faster than Rn,0 → f (x) dx.
a a
A final observation: because of the Euler–Maclaurin series, if f ∈
C2n+2[a, b] and is periodic of period b − a, then f (j) (a) = f (j) (b) for j =
0, 1, . . . , 2n − 1, so
Z b h2n+1B2n+2 (2n+2)
f (x) dx − Th = (b − a) f (ξ)
a (2n + 2)!
c.f.,
Z b h2 ′′
f (x) dx − Th = (b − a)
f (ξ)
a 12
for nonperiodic functions! That is, the Composite Trapezium Rule is ex-
tremely accurate for the integration of periodic functions. If f ∈ C∞ [a, b],
then Th → ab f (x) dx faster than any power of h.
R

Example:
Z 2π q
the circumference of an ellipse with semi-axes A and B is
A2 sin2 φ + B 2 cos2 φ dφ. For A = 1 and B = 41 , T8 = 4.2533, T16 =
0
4.2878, T32 = 4.2892 = T64 = · · ·.