Lecture Notes on Numerical Methods

for Engineering (?)

Pedro Fortuny Ayuso

U NIVERSIDAD DE O VIEDO
E-mail address: [email protected]

CC
BY: Copyright
c 20112016 Pedro Fortuny Ayuso
This work is licensed under the Creative Commons Attribution 3.0
License. To view a copy of this license, visit
http://creativecommons.org/licenses/by/3.0/es/
or send a letter to Creative Commons, 444 Castro Street, Suite 900,
Mountain View, California, 94041, USA.
 Contents

Introduction 5
1. Some minor comments 5
Chapter 1. Arithmetic and error analysis 7
1. Exponential notation 7
2. Error, basic definitions 10
3. Bounding the error 16
Chapter 2. Numerical Solutions to Non-linear Equations 19
1. Introduction 19
2. The Bisection Algorithm 20
3. Newton-Raphsons Algorithm 22
4. The Secant Algorithm 24
5. Fixed Points 26
6. Annex: Matlab/Octave Code 32
Chapter 3. Numerical Solutions to Linear Systems of Equations 35
1. Gauss Algorithm and LU Factorization 35
2. Condition Number: behavior of the relative error 40
3. Fixed Point Algorithms 44
4. Annex: Matlab/Octave Code 46
Chapter 4. Interpolation 49
1. Linear (piecewise) interpolation 49
2. Can parabolas be used for this? 50
3. Cubic Splines: Continuous Curvature 51
4. The Lagrange Interpolating Polynomial 57
5. Approximate Interpolation 60
6. Code for some of the Algorithms 64
Chapter 5. Numerical Differentiation and Integration 69
1. Numerical Differentiation 69
2. Numerical IntegrationQuadrature Formulas 71
Chapter 6. Differential Equations 79
1. Introduction 79
3
4 CONTENTS

2. The Basics 81
3. Discretization 82
4. Sources of error: truncation and rounding 85
5. Quadratures and Integration 86
6. Eulers Method: Integrate Using the Left Endpoint 86
7. Modified Euler: the Midpoint Rule 87
8. Heuns Method: the Trapezoidal Rule 89
Chapter 7. Multivariate and higher order ODEs 93
1. A two-variable example 93
2. Multivariate equations: Euler and Heuns methods 96
3. From greater order to order one 98
 Introduction

These notes cover what is taught in the classes of Numerical Meth-

ods for Engineering in the School at Mieres. One should not expect
more than that: a revision of what has been or will be explained dur-
ing the course. For a more correct, deep and thorough explanation,
one should go to the material referenced in the bibliography? There is none.
Simply, stated
These notes are no substitute for a book (or
two).
and even more
What is here may and may not cover
completely the contents of the exam
1. Some minor comments
My aim in these notes is mostly twofold:
To introduce the basic problems tackled by Numerical Cal-
culus in their most simple fashion.
To get the students used to stating algorithms with precision
and to understanding the idea of complexity.
I also would like to be able to make the students aware of the im-
portance of the conditioning of a numerical problem and the need to
verify that the results obtained by a computer program match reality
and fit the problem under study. But there is really no space (much
less time in the course) for this.
We shall refer, along the way, to several computing tools. Specif-
ically,
Matlab: I shall try to make all the code in this notes runnable
on Octave but this text will only speak of Matlab, which is
the software students are used to working with at Mieres.
Wolfram Alpha: This is a powerful and very useful tool with
which a large number of operations (mathematical and not)
can be performed, on a simple web page.
5
6 INTRODUCTION

What may be most interesting is that the students can:

State algorithms with precision, aware that an algorithm must
be clear, finite and finish.
Follow the steps of an algorithm and roughly analyze its
complexity (or at least, compare two of them).
I would also like to stress the importance of discerning if an ap-
proach suits a specific problem or not depending on its good or bad
conditioning but, alas, there is also too little time in the course. . .
It goes without saying that they will have to understand and be
able to follow, step by step, the algorithms stated in the classes (this
is an essential requisite). All the algorithms that will be explained are
of easy memorization because they are either brief (most of them) or
geometrically elemental (so that their verbal expression is easily to
remember). Formal precision is much more important in this course
than geometric ideas because numerical analysis deals with formal
methods of solving specific problems, not with their geometrical or
intuitive expression. This is the main rub of this course. You will
need to memorize.
 CHAPTER 1

Arithmetic and error analysis

Exponential notations, double precision floating point, the no-

tion of error and some common sources of error in computations are
summarily reviewed. The student should at least get a feeling of
the importance of normalization of floating point arithmetic and that
mistakes in its use can be critical.

1. Exponential notation
Real numbers can have a finite or infinite number of digits. When
working with them they have to be approximated using a finite (and
usually small) number of digits. Even more, they should be ex-
pressed in a standard way so that their magnitude and significant
digits are easily noticed at first sight and so that sharing them among
machines is a deterministic and efficient process. These require-
ments have led to what is known as scientific or exponential nota-
tion. We shall explain it roughly; what we intend is to transmit the
underlying idea more than to detail all the rules (they have lots of
particularities and exceptions). We shall use, along these notes, the
following expressions:
D EFINITION 1. An exponential notation of a real number is an ex-
pression of the form
A.B 10C
10C A.B
A.BeC
where A, B and C are natural numbers (possibly 0), is a sign (which
may be elided if +). Any of those expressions refers to the real num-
ber ( A + 0.B) 10C (where 0.B is nought dot B. . . ).
For example:
The number 3.123 is the same 3.123.
The number 0.01e 7 is 0.000000001 (eight zeroes after the
dot and before the 1).
7
8 1. ARITHMETIC AND ERROR ANALYSIS

The number 103 2.3 is 2300.

The number 23.783e 1 is 2.3783.
In general, scientific notation assumes the number to the left of the
decimal point is a single non-zero digit:
D EFINITION 2. The standard exponential notation is the exponential
notation in which A is between 1 and 9.
Finally, machines usually store real numbers in a very specific
way called floating point.
D EFINITION 3. A floating point format is a specification of an ex-
ponential notation in which the length of A plus the length of B is
constant and in which the exponents (the number C) vary in a spe-
cific range.
Thus, a floating point format can only express a finite amount of
numbers (those which can be written following the specifications of
the format).
The blueprint for the floating point standard for microchips (and
for electronic devices in general) is document IEEE-754 (read I-E-
cube seven-five-four). The acronym stands for Institute of Elec-
trical and Electronic Engineers. The last version of the document
dates from 2008.
1.1. The binary double precision format of IEEE-754. The dou-
ble precision format is the IEEE specification for representing real
numbers in sequences of 16, 32, 64, 128 bits (and more cases) and
their decimal representation. The main properties of binary double
precision with 64 bits are roughly explained in this section.
In order to write numbers in double precision, 64 bits are used,
that is, 64 binary digits. The first one stands for the sign (a 0 means
+, a 1 means ). The 11 following ones are used for the exponent
(as will be shown) and the remaining 52 are used for what is called
the mantissa. Thus, a double precision number has three parts: s, the
sign, which can be 0 or 1; e, the exponent, which varies between 0
and 211 1 = 2047; and m, a 52-bit number. Given three values for
s, e and m, the real number represented by them is:
If e 6= 0 and e 6= 2047 (i.e. if the exponent is not one of the
end values), then
N = (1)s 2e1023 1.m,
where 1.m means one-dot-m in binary. Notice and this
is the key that the exponent is not the number represented by
 1. EXPONENTIAL NOTATION 9

the 11 bits of e but that it is shifted to the right. The expo-

nent e = 01010101011, which in decimal is 683 represents,
in double precision format, the number 26831023 = 2340 .
Those e with a starting 0 bit correspond to negative powers
of 2 and those having a starting 1 bit to positive powers (re-
call that 210 = 1024).
If e = 0 then, if m 6= 0 (if there is a mantissa):
N = (1)s 21023 0.m,
where 0.m means zero-dot-m in binary.
If e = 0 and m = 0, the number is either +0 or 0, depend-
ing on the sign s. This means that 0 has a sign in double
precision arithmetic (which makes sense for technical rea-
sons).
The case e = 2047 (when all the eleven digits of e are 1) is
reserved for encoding exceptions like and NaN (not-
a-number-s). We shall not enter into details.
As a matter of fact, the standard is much longer and thorough and
includes a long list of requisites for electronic devices working in
floating point (for example, it specifies how truncations of the main
mathematical operations have to be performed to ensure exactness
of the results whenever possible).
The main advantage of floating point (and of double precision
specifically) is, apart from standardization, that it enables computing
with very small and very large numbers with a single format (the
smallest storable number is 21023 ' 10300 and the largest one is
21023 ' 10300 ). The trade off is that if one works simultaneously with
both types of quantities, the first lose precision and tend to disappear
(a truncation error takes place). However, used carefully, it is a hugely
useful and versatile format.

1.2. Binary to decimal (and back) conversion. ? In this course, Examples?

it is essential to be able to convert a number from binary to decimal
representation and back.
In these notes, we shall use the following notation: the expres-
sion x a means that x is a variable, a is a value (so that it can be
a number or another variable) and that x gets assigned the value of
a. The expression u = c is the conditional statement the value de-
signed by u is the same as that designed by c, which can be either
true or false. We also denote m//n as the quotient of dividing the
natural number m > 0 by the natural number n > 0, and m%n is the
10 1. ARITHMETIC AND ERROR ANALYSIS

remainder of that division. That is,

m = (m//n) n + (m%n).
Finally, if x is a real number x = A.B, the expression { x } means the
fractional part of x, i.e., the number 0.B.
E XAMPLE 1. The following table shows the decimal and binary
expansions of several numbers. The last binary expansion is approx-
imate because it has an infinite number of figures.
Decimal Binary
1 1
2 10
10 1010
0.5 0.1
7.25 111.01
0.1 0.000110011+
Algorithm 1 (in page 11) is a way to convert a decimal number
A.B with a finite number of digits to its binary form. Algorithm 2
performs the reverse conversion. Notice that, as there are numbers
with a finite quantity of decimal digits which cannot be expressed
with a finite quantity of binary digits (the most obvious example
being 0.1), one has to specify a number of fractional digits for the
output (which implies that one does not necessarily obtain the very
same number but a truncation).
Converting from binary to decimal is simpler but requires ad-
ditions on the go: one does not obtain a digit at each step because
one has to add powers of two. This process is described in Algorithm
2. Notice that the number of digits of the output is not specified (it
could be done but it would only make the algorithm more complex).
Finally, in all those steps in which Ai 2i or Bi 2i is added, both
Ai and Bi are either 0 or 1, so that this multiplication just means ei-
ther add or do not add the corresponding power of 2 (this is what a
binary digit means, anyway).

2. Error, basic definitions

Whenever numbers with a finite quantity of digits are used in
computations and whenever real measurements are performed, one
has to acknowledge that errors will be made. This is not grave by it-
self. What matters is having an estimate of their size and knowing that,
as calculations are made, they can propagate. In the end, the best
one can do is to bound the absolute error, that is, to know a reasonable
 2. ERROR, BASIC DEFINITIONS 11

Algorithm 1 Conversion from decimal to binary, without sign.

Input: A.B a number in base 10, with B of finite length, k a positive
integer (which is the desired number of fractional digits after the
binary dot)
Output: a.b (the number x in binary format, truncated to 2k )
if A = 0 then
a 0 and go to the F RACTIONAL PART
end if
? I NTEGRAL PART
i 1, n A
while n > 0 do
i i+1
xi n%2 [remainder]
n n//2 [quotient]
end while
a xi xi1 . . . x0 [the sequence of remainders in reverse
order]
? F RACTIONAL PART
if B = 0 then
b0
return a.b
end if
i 0, n 0.B
while n > 0 and i < k do
i i+1
m 2n
if m 1 then
bi 1
else
bi 0
end if
n {m} [the fractional part of m]
end while
b b1 b2 . . . bi
return a.b

value (the bound) which is larger than the error, in order to assess,
with certainty, how far the real value can be from the computed one.
In what follows, an exact value x is assumed (a constant, a datum,
the exact solution to a problem. . . ) and an approximation will be

denoted x.
12 1. ARITHMETIC AND ERROR ANALYSIS

Algorithm 2 Conversion from binary to decimal, without sign.

Input: A.B, a number in binary format, k a non-negative integer
(the number of the fractional binary digits to be used).
Output: a.b, the decimal expression of the truncation up to preci-
sion 2k of the number A.B
? I NTEGRAL PART
Write A = Ar Ar1 . . . A0 (the binary digits)
a 0, i 0
while i r do
a a + A i 2i
i i+1
end while
? F RACTIONAL PART
if B = 0 then
return a.0
end if
b 0, i 0
while i k do
i i+1
b b + Bi 2i
end while
return a.b

D EFINITION 4. The absolute error incurred when using x instead

of x is the absolute value of their difference | x x |.
But, unless x is 0, one is usually more interested in the order of
magnitude of the error; that is, how relatively large the error is as
compared to the real quantity.
D EFINITION 5. The relative error incurred when using x instead
of x, assuming x 6= 0, is the quotient
| x x |
|x|
(which is always positive).
We are not going to use a specific notation for them (some people
use and , respectively).
E XAMPLE 2. The constant , which is the ratio between the length
of the circumference and its diameter, is, approximately 3.1415926534+
(the trailing + indicates that the real number is larger than the rep-
resentation). Assume one uses the approximation = 3.14. Then
 2. ERROR, BASIC DEFINITIONS 13

The absolute error is | | = 0.0015926534+.

The relative error is |

|
' 104 5.069573.
This last statement means that one is incurring an error of 5 ten-
thousandths per unit (approx. 1/2000) each time 3.14 is used for .
Thus, if one adds 3.14 two thousand times, the error incurred when
using this quantity instead of 2000 will be approximately . This
is the meaning of the relative error: its reciprocal is the number of
times one has to add x in order to get an accrued error equal to the
number being approximated.

Before proceeding with the analysis of errors, let us define the

two most common ways of approximating numbers using a finite
quantity of digits: truncation and rounding. The precise definitions
are too detailed for us to give. Start with a real number (with possi-
bly an infinite quantity of digits):

x = a 1 a 2 . . . a r . a r +1 . . . a n . . .

notice that there are r digits to the left of the decimal point. Define:

D EFINITION 6. The truncation of x to k (significant) digits is:

the number a1 a2 . . . ak 0 . . . 0 (an integer with r digits), if k
r,
the number a1 . . . ar .ar+1 . . . ak if k > r.
That is, one just cuts (i.e. truncates) the numerical expression of x and
adds zeroes to the right if the decimal part has not been reached.

D EFINITION 7. The rounding of x to k (significant) digits is the

following number:
If ak+1 < 5, then the rounding is the same as the truncation.
If 5 ak+1 9, then the rounding is the truncation plus
10rk+1 .
Remark: the rounding described here is called biased to plus infinity
because when the k + 1th digit is greater than or equal to 5, the
approximation is always greater than the true value.

The problem with rounding is that all the digits of the rounded num-
ber can be different from the original value (think of 0.9999 rounded
to 3 significant digits). The great advantage is that the error incurred
when rounding is less than the one incurred when truncating (it can
even be a half of it):
14 1. ARITHMETIC AND ERROR ANALYSIS

E XAMPLE 3. If x = 178.299 and one needs 4 significant1 figures,

then the truncation is x1 = 178.2, whereas the rounding is 178.3. The
absolute error in the former case is 0.099, while it is 0.001 in the latter.
E XAMPLE 4. If x = 999.995 and 5 digits are being used, the trun-
cation is x1 = 999.99, while the rounding is 1000.0. Even though all
the digits are different the absolute error incurred in both cases is the
same: 0.005. This is what matters, not that the digits match.
Why is then truncation important if rounding is always at least
as precise and half the time better? Because when using floating
point, one cannot prevent truncation from happening and it must be
taken into account to bound the global error. As of today (2013) most
of the computer programs working in double precision, perform
their computations internally with many more digits and round their
results to double precision. However, truncations always happen
(there is a limited amount of available digits to the processor).

2.1. Sources of error. Error appears in different ways. On one

hand, any measurement is subject to it (this why any measuring de-
vice is sold with an estimated margin of precision); this is intrinsic
to Nature and one can only take it into account and try to assess
its magnitude (give a bound). On the other hand, computations per-
formed in finite precision arithmetic both propagate these errors and
give rise to new ones precisely because the quantity of available dig-
its is finite.
The following are some of the sources of error:
Measurement error, already mentioned. This is unavoidable.
Truncation error: happens whenever a number (datum or the
result of a computation) has more digits than available and
some of them must be forgotten.
Rounding error: takes place when a number is rounded to a
specified precision.
Loss of significance (also called cancellation error): this appears
when an operation on two numbers increases relative error
substantially more than the absolute error. For example,
when numbers of very similar magnitude are subtracted.
The digits lost to truncation or rounding are too relevant and
the relative error is huge. A classical example is the instabil-
ity of the solution of the quadratic equation.
1We have not explained the meaning of this term, but we shall not do so.
Suffice it to say that significant figures means exact figures in an expression.
 2. ERROR, BASIC DEFINITIONS 15

Accumulation error: which appears when accumulating (with

additions, essentially) small errors of the same sign a lot of
times. This is what happened with the Patriot Missile Sta-
tions in 1991, during the Desert Storm operation2.
All the above errors may take place when working with finite arith-
metic. The following rules (which describe the worst-case behaviors)
apply:
When adding numbers of the same sign, the absolute error can
be up to the sum of the absolute errors and the same can
happen to the relative error.
When adding numbers of different sign, the absolute error be-
haves like in the previous case but the relative error may behave
wildly: 1000.2 1000.1 has only a significant digit, so that the
relative error can be up to 10%, but the relative error in the
operands was at most 1 104 .
When multiplying, the absolute error is of the same magni-
tude as the largest factor times the absolute error in the other
factor. If both factors are of the same magnitude, the abso-
lute error can be up to double the maximum absolute error
times the maximum factor. The relative error is of the same
magnitude as the maximum relative error in the factors (and
is the sum if both factors are of the same magnitude).
When dividing by a number greater than or equal to 1, the ab-
solute error is approximately the absolute error in the nu-
merator divided by the denominator and the relative error
is the relative error in the numerator (more or less like in
multiplication). When dividing by numbers near 0, absolute
precision is lost and later operations with numbers of the
same magnitude will probably give rise to large cancellation
errors. Example 5 is illustrative of this fact: the first line is
assumed to be an exact computation, the second one an ap-
proximation using a truncation to 6 significant figures in the
divisor of the quotient. One can see that the approxima-
tion is totally useless.
An example of the bad behavior of division and addition is the
following:

2One can read a summary at

http://www.cs.utexas.edu/downing/papers/PatriotA1992.pdf
and the official report is also available:
http://www.fas.org/spp/starwars/gao/im92026.htm.
16 1. ARITHMETIC AND ERROR ANALYSIS

E XAMPLE 5. Consider the following two computations, the first

one is assumed to be correct while the second one is an approxi-
mation:
33
26493 = 0.256 (the exact result) .
0.0012456
33
26493 = 8.2488 (approximation)
0.001246
The relative error in the rounding of the denominator is only 3
104 (less than half a thousandth of a unit), but the relative error in
the final result is 33.2 (which means that the obtained result is 33
times larger in absolute value than it should, so it is worthless as
an approximation). Notice that not even the sign is correct. This
is (essentially) the main problem of resolution methods involving
divisions (like Gauss and, obviously, Cramers). When explaining
Gauss method, convenient ways to choose an adequate denomina-
tor will be shown (these are called pivoting strategies). As a general
rule, the larger the denominator, the better.

3. Bounding the error

As we have already said, one does not seek an exact knowledge
of the error incurred during a measurement or the solution to a prob-
lem, as this will likely be impossible. Rather, one aims to have an idea
of it and, especially, to know a good bound of it. That is, to be able to
assess the maximum value that the absolute error can take and this
being a useful piece of information. Knowing that 2.71 is an approx-
imation of e with an error less than 400 is worthless. Knowing that
the error is less than 0.01 is useful.
In general, the only possibility is to estimate a reasonably small
number larger than the incurred error. This is bounding the error.

3.1. Some bounds. If one knows that a quantity is between two

values, which is usually expressed in the form x = a e for some
e > 0, then the absolute error incurred when using a instead of x is
unknown but is bounded by e. So that the relative error is, at most, this
e divided by the smallest of the possible absolute values of x. Notice
that this is tricky because if a e < 0 and a + e > 0 then there is no
way to bound the relative error because x can be 0 and then there is no
relative error (it is not defined for x = 0).
 3. BOUNDING THE ERROR 17

E XAMPLE 6. If = 3.14 0.01, then the maximum absolute error

incurred is 0.01, so that the relative error is at most
0.01
' .003194
|3.13|
(more or less 1/300).
Notice that in order to get an upper bound of a quotient, the de-
nominator must be bounded from below (the lesser the denominator, the
greater the quotient).
The rules in page 15 are essential in order to bound the error of
a sequence of arithmetic operations. One has to be especially careful
when dividing by numbers less than one, as this may lead to useless
results like the result is 7 with a relative error of 23.
 CHAPTER 2

Numerical Solutions to Non-linear Equations

The problem of solving the non-linear equation f ( x ) = 0 given f

and some initial conditions is treated in this chapter.
Each of the algorithms which will be explained in this chapter
has its own advantages and disadvantages; one should not discard
anyone a priori just for its slowness for example, bisection. We
shall see that the best one namely, Newton-Raphsons has two
drawbacks: it may converge far from the initial condition, becoming
useless for finding a root near a specific point and it requires the
computation of the value of the derivative of f at each step, which
may be too costly.
Notice that all the algorithms we present include stopping con-
ditions which depend on the value of f ( x ) and are of Cauchy-type.
This is done in order to simplify the exposition. If a specific quan-
tity of exact figures is required, one needs to perform a much deeper
study of the problem being solved, of the derivative of f and other
elements, which are out of the scope of these notes.

1. Introduction
Computing roots of functions, and especially of polynomials, is
one of the classical problems of Mathematics. It used to be believed
that any polynomial could be explicitly solved like the quadratic
equation, via a formula involving radicals (roots of numbers). Galois
Theory, developed in the beginning of the XIX Century, proved that
this is not the case at all and that, as a matter of fact, most polynomi-
als of degree greater than 4 are not solvable using radicals.
However, the search for a closed formula for solving equations is
just a way of putting off the real problem. In the end, and this is
what matters:
The only computations that can always be performed exactly are addition,
subtraction and multiplication.
19
20 2. NUMERICAL SOLUTIONS TO NON-LINEAR EQUATIONS

It is not even possible to divide and get exact results1.

From the beginning, people sought ways to quickly find approx-
imate solutions to nonlinear equations involving the four rules: ad-
dition, subtraction, multiplication and division.
Two different kind of algorithms are explained in this chapter:
those with a geometric meaning and the fixed point method, which
requires the notion of contractivity.
Before proceeding, let us remark two essential requirements in
any root-computing algorithm:
The specification of a precision, that is an e > 0 such that if
| f (c)| < e, then c is taken as an approximate root of f . This is
needed because, as non-exact computations are performed,
expecting a result f (c) = 0 is unrealistic, so that this equality
is useless as a stopping condition.
A second stopping condition unrelated to the value of f . It
may well happen that either the algorithm does not con-
verge or that f has no roots, so that the statement | f (c)| < e
is never true. In any case, the algorithm must finish at some
point. Otherwise, it is not an algorithm. This implies the need
for a condition unrelated to f . This takes usually the form
perform no more than n steps, where n is a counter.

2. The Bisection Algorithm

Assume f is a continuous function on a closed interval [ a, b].
Bolzanos Theorem may be useful if its conditions hold:
T HEOREM (Bolzano). Let f : [ a, b] R be a continuous function on
[ a, b] such that f ( a) f (b) < 0 (i.e. it changes sign between a and b). Then
there is c ( a, b) with f (c) = 0.
This statement asserts that if the sign of f changes on [ a, b] then
there is at least a root of f in it. One could sample the interval using
small sub-intervals (say of width (b a)/10i ) and seek, among this
sub-intervals, one where the sign changes, thus nearing a root at each
step. Actually, dividing the interval into two segments turns out to
be much simpler.
Start with a function f , continuous on the interval [ a, b] with val-
ues f ( a) and f (b). Specify a desired precision e > 0 (so that if

1One could argue that using rational numbers solves this problem but, again,
there is a point at which decimal expansions are needed.
 2. THE BISECTION ALGORITHM 21

| f (c)| < e then c is accepted as an approximate root) and a maxi-

mum number of iterations N > 0 of the process. With all these data,
one proceeds taking advantage of Bolzanos Theorem, as follows:
If f ( a) f (b) < 0, then there is a root in ( a, b). Take c as the mid-
a+b
point2 of the interval, that is c = . There are three possibilities
2
now:
Either | f (c)| < e and one takes c as an approximate root and
finishes the algorithm.
Or f ( a) f (c) < 0 and one substitutes c for b and repeats.
Or f (c) f (b) < 0 and one substitutes c for a and repeats.
The iteration is done at most N times (so that one obviously has to
keep track of their number). If after N iterations no approximate root
has been found, the process ends with an error message.
The formal statement of the process just described is Algorithm
3. Notice that the expression a b is used, meaning that a takes (or
is given) the value b.
E XAMPLE 7. Take the function f ( x ) = cos(e x ), which is contin-
uous in the interval [0, 1]. Obviously, f (0) f (1) < 0 (why is this ob-
vious?), so that Bolzanos Theorem applies. Setting a = 0, b = 1,
Algorithm 3 gives the following sequence: each assignment to a or b
means that the corresponding endpoint of the interval is substituted
for that value.
octave > Bisec(@(x) cos(exp(x)), 0, 1, .001, 10)
c = 0.50000
b = 0.50000 % new interval: [0, 0.50000]
c = 0.25000
a = 0.25000 % new interval: [0.25000, 0.50000]
c = 0.37500
a = 0.37500 % new interval: [0.37500, 0.50000]
c = 0.43750
a = 0.43750 % new interval: [0.43750, 0.50000]
c = 0.46875
b = 0.46875 % new interval: [0.43750, 0.46875]
c = 0.45312
b = 0.45312 % new interval: [0.43750, 0.45312]
c = 0.44531
a = 0.44531 % new interval: [0.44531, 0.45312]
c = 0.44922
a = 0.44922 % new interval: [0.44922, 0.45312]
c = 0.45117
octave> f(c) ans = 6.4520e-04

2This is what gives the algorithm the alternative name midpoint rule.
 22 2. NUMERICAL SOLUTIONS TO NON-LINEAR EQUATIONS

Algorithm 3 Bisection Algorithm.

Input: A function f ( x ), a pair of real numbers a, b with f ( a) f (b) <
0, a tolerance e > 0 and a limit of iterations N > 0
Output: either an error message or a real number c between a and
b such that | f (c)| < e (i.e. an approximate root)
? P RECONDITION
if f ( a) 6 R of f (b) 6 R or a + b 6 R [overflow] then
return ERROR
end if
? S TART
i0
a+b
c [both NaN and can happen]
2
while | f (c)| e and i N do
[Never ever compare signs multiplying]
if f ( a) f (c) < 0 then
b c [interval [ a, c]]
else
a c [interval [c, b]]
end if
i i+1
a+b
c [middle point (overflow possible)]
2
end while
if i > N then
return ERROR
end if
return c

As one can see, 0.45117 is an approximate root of cos(e x ), in the sense

that | cos(e0.45117 )| < 0.001.

3. Newton-Raphsons Algorithm
A classical geometric idea (which also appears in approximation
theory) is to use the best3 linear approximation to f in order to compute
Examples an approximate solution to f ( x ) = 0. This best linear? approxima-
tion is, of course, the tangent line to f , which is computed using the
derivative f 0 ( x ). So, instead of trying to solve f ( x ) = 0 directly, one
draws the tangent to the graph of f at ( x, f ( x )) and finds the meeting
3From the point of view of infinitesimal analysis and polynomial
approximation.
 3. NEWTON-RAPHSONS ALGORITHM 23

point of this line with the OX axis. Obviously, this will most likely
not be a root of f but in sufficiently general conditions, it is expected to
approach one. If the process is repeated enough times, it get nearer
a root of f . This is the idea of Newton-Raphsons method.
Recall that the equation of the line passing through ( x0 , y0 ) with
slope b is:
Y = b ( X x0 ) + y0
so that the equation of the tangent line to the graph of f at ( x0 , f ( x0 ))
is (assuming f has a derivative at x0 ):
Y = f 0 ( x0 )( X x0 ) + f ( x0 ).
The meeting point between this line and OX is
 
f ( x0 )
( x1 , y1 ) = x0 0 ,0
f ( x0 )
assuming it exists (i.e. f 0 ( x0 ) 6= 0).
If x1 is not an approximate root of f with the desired precision,
one proceeds in the same way at the point ( x1 , f ( x1 )). After having
performed n steps, the next point xn+1 takes the form:
f ( xn )
x n +1 = x n .
f 0 ( xn )
One carries on until the desired precision or a bound in the number
of iterations is reached. This is Algorithm 4. In its formal expression,
we only specify a possible error place in order to keep it clear but one
has to take into account that each time a computation is performed a
floating-point error might occur.
E XAMPLE 8. Take the function f ( x ) = e x + x, which is easily seen
to have a unique zero on the whole real line (why is this so and why
is it easy to see?). Its derivative is f 0 ( x ) = e x + 1. Let us use x0 = 1
as the seed. An implementation of Newton-Raphsons algorithm in
Octave might show the following steps:
octave> c=newtonF(f, fp, 1, .0001, 5)
xn = 0
xn = -0.500000000000000
xn = -0.566311003197218
xn = -0.567143165034862
xn = -0.567143290409781
xn = -0.567143290409784
c = -0.567143290409784
octave> f(c)
ans = -1.11022302462516e-16
24 2. NUMERICAL SOLUTIONS TO NON-LINEAR EQUATIONS

Algorithm 4 Newton-Raphson.
Input: A differentiable function f ( x ), a seed x0 R, a tolerance
e > 0 and a limit for the number of iterations N > 0
Output: Either an error message or a real number c such that
| f (c)| < e (i.e. an approximate root)
? S TART
i0
while | f ( xi )| e and i N do
f (x )
x i +1 x i 0 i
f ( xi )
i i+1
end while
if i > N then
return ERROR
end if
c xi
return c

The convergence speed (which will be studied in detail in 5.5) is clear

in the sequence of xn, as is the very good approximation to 0 of f (c)
for the approximate root after just 5 iterations.
E XAMPLE 9. However, the students are suggested to try Newton-
Raphsons algorithm for the function cos(e x ) in Example 7. A strange
phenomenon takes place: the algorithm is almost never convergent
and no approximation to any root happens. Is there a simple expla-
nation for this fact?
E XAMPLE 10. Newton-Raphsons method can behave quite strangely
when f 0 (c) = 0, if c is a root of f . We shall develop some examples
in the exercises and in the practicals.

4. The Secant Algorithm

f (x )
Newton-Raphsons algorithm contains the evaluation of f 0 ( xn ) for
n
which one has to compute not only f ( xn ) but also f 0 ( xn ), which may
be too costly. Moreover, there are cases in which one does not have
true information on f 0 ( x ), so that assuming it can be computed may
be utopic.
The simplest solution of this problem is to approximate the value
of the derivative using the geometric idea that the tangent line is
the limit of the secants; instead of computing the tangent line one
 4. THE SECANT ALGORITHM 25

approximates it by means of two points near each other. Recall that

the derivative of f ( x ) at c is (if it exists) the limit
f (c + h) f (c)
lim .
h 0 h
When applying Newton-Raphons algorithm, if instead of using a
single point xn , one makes use of two: xn and xn1 , then the deriva-
tive f 0 ( xn ) can be approximated by means of
f ( x n ) f ( x n 1 )
(1) f 0 ( xn ) ' ,
x n x n 1
so that the formula for computing xn+1 becomes, using this approx-
imation,
x n x n 1
x n +1 = x n f ( x n ) ,
f ( x n ) f ( x n 1 )
and the Secant Algorithm is obtained. Notice that, in order to start
it, two seeds are required, instead of one. The coefficient of f ( xn ) in
the iteration is just the reciprocal of approximation (1).

Algorithm 5 The Secant Algorithm.

Input: A function f ( x ), a tolerance e > 0, a bound for the number
of iterations N > 0 and two seeds x1 , x0 R
Output: Either a real number c R with | f (c)| < e or an error
message
? S TART
i0
while | f ( xi )| e and i N do
x i x i 1
x i +1 x i f ( x i )
f ( x i ) f ( x i 1 )
i i+1
end while
if i > N then
return ERROR
end if
c xi
return c

It is worthwhile, when implementing this method, keeping in

memory not just xn and xn1 but also the values f ( xn ) and f ( xn1 )
so as not to recompute them.
26 2. NUMERICAL SOLUTIONS TO NON-LINEAR EQUATIONS

E XAMPLE 11. Take the same function as in example 8, that is

f ( x ) = e x + x. Let us use x0 = 1 and x1 = 0.999 as the two seeds. An
implementation of the secant algorithm in Octave might show the
following steps:
octave> c=secant(f, 1, 0.999, .0001, 6)
xn = -3.65541048223395e-04
xn = -0.368146813789761
xn = -0.544505751846815
xn = -0.566300946933390
xn = -0.567139827650291
c = -0.567139827650291
octave> f(c)
ans = 5.42664370639656e-06

It needs an iteration more than Newton-Raphsons method to achieve

the same precision.
E XAMPLE 12. The secant algorithm has the same problem with
the function of Example 7 because it is essentially an approximation
to Newton-Raphsons, so that if this fails, it is expected for the secant
method to fail as well.

5. Fixed Points
Fixed point algorithms which, as we shall see later, include the
previous ones indirectly are based on the notion of contractivity,
which reflects the idea that a function (a transformation) may map
pairs of points in such a way that the images are always nearer than
the original two (i.e. the function shrinks, contracts the initial space).
This idea, linked to differentiability, leads to that of fixed point of an
iteration and, by means of a little artifact, to the approximate solu-
tion of general equations using just iterations of a function.
5.1. Contractivity and equations g( x ) = x. Let g be a real-valued
function of one real variable, differentiable at c. That is, for any in-
finitesimal o, there exists another one o1 such that
g(c + o) = g(c) + g0 (c)o + oo1 ,
which means that near c, the function g is very similar to its linear ap-
proximation.
Assume that o is the width of a small interval centered at c.
Removing the supralinear error (the term oo1 ) for approximating, one
might think that (c o, c + o) is mapped into ( g(c) g0 (c)o, g(c) +
g0 (c)o): that is, an interval of radius o is mapped into one of radius
g0 (c)o (it dilates or shrinks by a factor of g0 (c)). This is essentially
what motivates the Jacobian Theorem in integration: the derivative
 5. FIXED POINTS 27

measures the dilation or contraction which the real line undergoes at

a point when transformed by g. If | g0 (c)| < 1, the real line contracts
near c. As a matter of fact, one usually reads the following
D EFINITION 8. A map f : [ a, b] R is contractive if | f ( x )
f (y)| < | x y| for any two x, y [ a, b].
Notice that if f is everywhere differentiable and | f 0 ( x )| < 1 for
all x [ a, b] then f is contractive.
Assume for simplicity that g0 is continuous on [ a, b] and that | g0 ( x )| <
1 for all x [ a, b] i.e., g shrinks everywhere along [ a, b]. There
are several facts to notice. First of all, the conditions imply that
there exists < 1 such that | g0 ( x )| < , by Weierstrass Theorem
applied to g0 ( x ). Moreover, by the Mean Value Theorem, for any
x1 , x2 [ a, b], the following inequality holds:
| g( x1 ) g( x2 )| | x1 x2 |,
which means: the distance between the images of two arbitrary points is
less than times the original distance between the points, and, as < 1,
what really happens is that the distance between the images is always less
than the original distance. That is, the map g is shrinking the interval
[ a, b] everywhere. It is as if [ a, b] were a piece of cotton cloth and it
was being washed with warm water: it shrinks everywhere.
The last paragraph contains many emphasis but they are neces-
sary for understanding the final result: the existence of a fixed point,
a point which gets mapped to itself.
From all the explanation above, one infers that the width of g([ a, b])
is less than or equal to (b a). If moreover g([ a, b]) [ a, b] (that is,
if g mapped [ a, b] into itself) then one could also compute g( g([ a, b])),
which would be at most of width (b a) = 2 (b a). Now
one could repeat the iteration indefinitely and g g g g =
gn ([ a, b]) would have width less than n (b a). As < 1, this
width tends to 0 and using the Nested Intervals Principle, there must
be a point [ a, b] for which g() = . This is, by obvious reasons,
a fixed point. Even more, the fact that < 1 implies that is unique.
There is one and only one fixed point for g in [ a, b]. We have just
given a plausible argument for the next result:
T HEOREM 1. Let g : [ a, b] [ a, b] be a map of [ a, b] into itself, which
is continuous and differentiable on [ a, b]. If there is a positive < 1 such
that for any x [ a, b], | g0 ( x )| , then there exists one and only one
[ a, b] for which g() = . Even more, for any x0 [ a, b], if one
defines
xn = g( xn1 ) for n > 0
28 2. NUMERICAL SOLUTIONS TO NON-LINEAR EQUATIONS

then
lim xn = .
n

This means that under suitable conditions, the equation g( x ) = x

has a single solution in the interval [ a, b]. The explanation before
the statement shows how to approximate this solution: take any x
[ a, b] and compute g( x ), g( g( x )), . . . . The limit (which exists under
those conditions) is always , regardless of x.
Hence,
solving equations of the form g( x ) = x for contractive g is utterly simple:
just iterate g.
5.2. Application to General Equations f ( x ) = 0. In real life
(if this happens to relate to real life at all, anyway), nobody comes
across an equation of the form g( x ) = x. One always encounters
problems like f ( x ) = 0 or, more generally f ( x ) = c, c being a con-
stant, which are easily turned into f ( x ) = 0.
But this is not a problem because
f (x) = 0 f (x) + x = x
so that, searching for a root of f ( x ) is the same thing as searching for a
fixed point of g( x ) = f ( x ) + x. Or, for the same reason, of x f ( x ).
R EMARK (Skip in a first reading). As a matter of fact, if ( x ) is
a nowhere zero function, then searching for a root of f ( x ) is the same
as searching for a fixed point of g( x ) = x ( x ) f ( x ). This allows,
for example, to scale f so that its derivative is approximately 1 and
g0 is small in order to accelerate convergence. Or one can just take
g( x ) = x c f ( x ) for a suitable c which makes the derivative of g
relatively small in absolute value.
5.3. The Algorithm. This is probably the easiest algorithm to im-
plement, as it only requires computing the value of g each time. As
any other, it requires a tolerance e and a maximum number of itera-
tions N. The drawback is that the algorithm can be useless if [ a, b] is
not mapped into itself. This has to be checked beforehand.
R EMARK 1. Let g : [ a, b] R be a map. If a fixed point of g in
[ a, b] is to be found using contractivity, it is necessary:
If g is differentiable4 in [ a, b], there must exist R such
that 0 < < 1 and for which | g0 ( x )| for all x [ a, b].
4If g it is not, then the requirement the Lipschitz condition is much harder
to test.
 5. FIXED POINTS 29

Assumed both checks have been performed, the method for find-
ing a fixed point of g : [ a, b] [ a, b] is stated in Algorithm 6:

Algorithm 6 Fixed Point.

Input: A function g (contractive etc. . . ), a seed x0 [ a, b], a toler-
ance e > 0 and a maximum number of iterations N > 0
Output: either c [ a, b] such that |c g(c)| < e or an error mes-
sage
? S TART
i 0, c x0
while |c g(c)| e and i N do
c g(c)
i i+1
end while
if i > N then
return ERROR
end if
return c

E XAMPLE 13. For the fixed point algorithm we shall use the same
function as for the bisection method, that is f ( x ) = cos(e x ). In order
to find a root of this function, we need to turn the equation
cos(e x ) = 0
into a fixed-point problem. This is always done in the same (or sim-
ilar) way: the above equation is obviously equivalent to
cos(e x ) + x = x,
which is a fixed-point problem. Let us call g( x ) = cos(e x ) + x, whose
fixed point we shall try to find.
In order to apply the fixed-point theorem, one needs an interval
[ a, b] which is mapped into itself. It is easily shown that g( x ) de-
creases near x = 0.5 and, as a matter of fact, that it does so in the
interval I = [0.4, 0.5]. Moreover, g(0.4) ' 0.478+ while g(0.5) '
0.4221+, which implies that the interval I is mapped into itself by
g (this is probably the most difficult part of a fixed-point problem:
finding an appropriate interval which gets mapped into itself). The
derivative of g is g0 ( x ) = e x sin(e x ) + 1, whose absolute value is,
in that interval, less than 0.8 (it is less than 1 in the whole interval
[0, 1], as a matter of fact). This is the second condition to be verified
in order to apply the theorem.
30 2. NUMERICAL SOLUTIONS TO NON-LINEAR EQUATIONS

Now we can be certain that there is a fixed point for g( x ) in

[0.4, 0.5]. Matlab would give a sequence like
octave> c=fixed_point(g, 0.5, .0001, 10)
xn = 0.422153896052972
xn = 0.467691234530252
xn = 0.442185884541927
xn = 0.456876713444163
xn = 0.448538950613744
xn = 0.453312782246289
xn = 0.450592834887592
xn = 0.452146949739883
xn = 0.451260386650212
xn = 0.451766601977469
c = 0.451766601977469
octave> f(c)
ans = -2.88890800240215e-04

The convergence speed does not look too good, which is usual in
fixed-point problems.
R EMARK (Skip in first reading). The fixed point algorithm can be
used, as explained in 5.2, for finding roots of general equations using
a suitable factor; to this end, if the equation is f ( x ) = 0, one can use
any function g( x ) of the form
g( x ) = x k f ( x )
where k R is an adequate number. This transformation is per-
formed so that the derivative of g is around 0 near the root and so
that if c is the (unknown) root, then g defines a contractive map on
an interval of the form [c , c + ] (which will be the [ a, b] used in
the algorithm).
The hard part is to check that g is a contractive map of [ a, b] onto
itself.
5.4. Convergence Speed of the Fixed Point Method. If the ab-
solute value of the derivative is bounded by < 1, the convergence
speed of the fixed point algorithm is easily bounded because [ a, b] is
mapped onto a sub-interval of width at most (b a) and this con-
traction is repeated at each step. So, after i iterations, the width of
the image set is at most i (b a). If < 101 and (b a) < 1, for
example, then one can guarantee that each iteration gives an extra
digit of precision in the root. However, tends to be quite large (like
0.9) and the process is usually slow.
E XAMPLE 14. If [ a, b] = [0, 1] and | g0 ( x )| < 0.1, then after each it-
eration there is an exact decimal digit more in x as an approximation
to the fixed point c, regardless of the initial value of the seed x.
 5. FIXED POINTS 31

5.5. Newton-Raphsons convergence speed. As a matter of fact,

Newton-Raphsons algorithm, under suitable conditions, is just a
fixed point algorithm and the analysis of the convergence speed for
these can be applied to it in this case. Notice that the expression:
f ( xn )
x n +1 = x n
f 0 ( xn )
corresponds to the search of a fixed point of
1
g( x ) = x 0 f (x)
f (x)
which, as explained above, is a way of turning the equation f ( x ) = 0
into a fixed-point problem. The derivative of g is, in this case:
0 f 0 ( x )2 f ( x ) f 00 ( x )
g (x) = 1
f 0 ( x )2
which, at a root c of f (that is, f (c) = 0) gives g0 (c) = 0. That is, the
derivative of g is 0 at the fixed point of g. This makes convergence
very fast when x is near c (and other conditions hold).
In fact, the following strong result (which is usually stated as
Newton-Raphson has quadratic convergence) can be proved:
T HEOREM 2. Assume f is a twice-differentiable function on an inter-
val [r e, r + e], that r is a root of f and that
The second derivative of f is bounded from above: | f 00 ( x )| < K
for x [r e, r + e],
The first derivative of f is bounded from below: | f 0 ( x )| > L > 0
for x [r e, r + e]
then, if xk [r e, r + e], the following term in Newton-Raphsons itera-
tion is also in that interval and
K
|r xk+1 | < | ||r xk |2 .
2L
And, as a corollary:
C OROLLARY 1 (Duplication of exact digits in Newton-Raphsons
metod). Under the conditions of Theorem 2, if e < 0.1 and K < 2L then,
for n k, each iteration xn+1 of Newton-Raphson approximates c with
twice the number of exact digits as xn .
P ROOF. This happens because if k = 0 then x0 has at least one
exact digit. By the Theorem, | x1 r | is less than 0.12 = .01. And
from this point on, the number of zeroes in the decimal expansion of
| xn r | gets doubled each time. 
32 2. NUMERICAL SOLUTIONS TO NON-LINEAR EQUATIONS

In order to use Theorem 2 or its corollary, one has to:

Be certain that a root is near. The usual way to check this is
using Bolzanos Theorem (i.e. checking that f changes sign
on a sufficiently small interval).
Bound the width of the interval (for example, by 1/10).
Bound f 00 ( x ) from above and f 0 ( x ) from below on the inter-
val (this is usually the hardest part).

6. Annex: Matlab/Octave Code

Some code with correct implementations of the algorithms of
this chapter are included here. They should run both on Matlab and
Octave. However, notice that the code here does not check for floating-
point exceptions. For example, if one uses the function log( x ) with
any of this programs and it is evaluated at a negative number, the
programs will happily continue but the results will probably be ei-
ther absurd or not real. The decision not to include those checks has
been made for simplicity.
6.1. The Bisection Algorithm. The code in Listing 2.1 implements
the bisection algorithm for Matlab/Octave. The input parameters
are:
f: and anonymous function,
a: the left endpoint of the interval,
b: the right endpoint of the interval,
epsilon: a tolerance (by default, eps),
n: a bound for the number of iterations (by default, 50).
The output may be null (if there is no sign change, with an appro-
priate message) or an approximate root (to the tolerance) or a warn-
ing message together with the last computed value (if the tolerance
has not been reached in the specified number of iterations). The for-
mat of the output is a pair [z, N] where z is the approximate root
(or the last computed value) and N is the number of iterations per-
formed.
% Bisection Algorithm with tolerance and stopping condition
% Notice that at any evaluation f(...) an error might take place
% which is NOT checked.
function [c, N] = Bisec(f, a, b, epsilon = eps, n = 50)
N = 0;
if(f(a)*f(b)>0)
warning(no sign change)
return
end
% store values in memory
 6. ANNEX: MATLAB/OCTAVE CODE 33

fa = f(a);
fb = f(b);
if(fa == 0)
c = a;
return
end
if(fb == 0)
c = b;
return
end
c = (a+b)/2
fc = f(c);
while(abs(fc) >= epsilon & N < n)
N = N + 1;
% multiply SIGNS, not values
if(sign(fc)*sign(fa) < 0)
b = c;
fb = fc;
else
a = c;
fa = fc;
end
% An error might happen here
c = (a+b)/2;
fc = f(c);
end
if(N >= n)
warning("Tolerance not reached.")
end
end

L ISTING 2.1. Code for the Bisection Algorithm.

6.2. Newton-Raphsons Algorithm. Newton-Raphsons algorithm

is easier to implement (always without checking for floating point
exceptions) but requires more complex input data: the derivative of
f , another anonymous function. The code in Listing 2.2 does not
perform symbolic computations so that this derivative must be pro-
vided by the user.
The input is, then:
f: an anonymous function,
fp: another anonymous function, the derivative of f,
x0: the seed,
epsilon: the tolerance (by defect eps),
N: the maximum number of iterations (by default 50).
The format of the output, in order to facilitate its study, is a pair
[xn,N], where xn is the approximate root (or the last computed
value) and N is the number of iterations performed.
34 2. NUMERICAL SOLUTIONS TO NON-LINEAR EQUATIONS

% Newton-Raphson implementation
function [z n] = NewtonF(f, fp, x0, epsilon = eps, N = 50)
n = 0;
xn = x0;
% Both f and fp are anonymous functions
fn = f(xn);
while(abs(fn) >= epsilon & n <= N)
n = n + 1;
fn = f(xn); % memorize to prevent recomputing
% next iteration
xn = xn - fn/fp(xn); % an exception might take place here
end
z = xn;
if(n == N)
warning(Tolerance not reached.);
end
end

L ISTING 2.2. Code for Newton-Raphsons Algorithm

 CHAPTER 3

Numerical Solutions to Linear Systems of Equations

Some of the main classical algorithms for (approximately) solv-

ing systems of linear equations are discussed in this chapter. We
begin with Gauss reduction method (and its interpretation as the
LU factorization). The notion of condition number of a matrix and its
relation with relative error is introduced (in a very simplified way,
without taking into account the errors in the matrix). Finally, the
fixed-point algorithms are introduced and two of them (Jacobis and
Gauss-Seidels) are explained.
All three algorithms are relevant by themselves but knowing the
scope of each one is as important, at least from the theoretical stand-
point; for example, the requirement that the matrix be convergent (the
analogue to contractivity). We shall not speak about the spectrum
nor use the eigenvalues; these are of the utmost importance but the
students have no true knowledge of them. We prefer to insist on the
necessity of convergence (and the student will be able to apply it to
each case when need comes).
In this chapter, the aim is to (approximately) solve a system of
linear equations of the form

(2) Ax = b

where A is a square matrix of order n and b is a vector in Rn . We

always assume unless explicitely stated that A is non-singular, so that
the system is consistent and has a unique solution.

1. Gauss Algorithm and LU Factorization

Starting from system (2), Gauss method transforms A and b by
means of simple operations into an upper triangular matrix A and
a vector b such that the new system Ax = b is solvable by regres-
sive substitution (i.e. xn is directly solvable and each variable xk is
solvable in terms of xk+1 , . . . , xn ). Obviously, in order that this trans-
formation make sense, the main requirement is that the solution to
35
36 3. NUMERICAL SOLUTIONS TO LINEAR SYSTEMS OF EQUATIONS

the new system be the same as that of the original1. To this end, only
the following operation is permitted:
Any equation Ei (the i th row of A together with the i th
element of b) may be substituted by a linear combination of
the form Ei + Ek for some k < i and R. In this case, bi
is substituted with bi + bk .
The fact that Ei appears with coefficient 1 in the substituting expres-
sion (Ei + Ek ) is what guarantees that the new system has the same
solution as the original one.
Let A be the augmented matrix A = ( A|b).

L EMMA 1. In order to transform a matrix A into a matrix A using the

above operation, it is enough to multiply A on the left by the matrix Lik ()
whose elements are:
If m = n, then ( Lik ())mn = 1 (diagonal with 1).
If m = i, n = k, then ( Lik ())mn = (the element (i, k) is ).
Any other element is 0.
E XAMPLE 15. Starting with the following A

3 2 1 4 1
0 1 4 2 3
A= 6 1 2

5 0
1 4 3 2 4
and combining row 3 with row 1 times 2 (in order to make a zero
at the 6), then one has to multiply by L31 (2)

1 0 0 0 3 2 1 4 1 3 2 1 4 1
0 1 0 0 0 1 4 2 3
= 0 1 4 2 3

.
2 0 1 0 6 1 2 5 0 0 5 4 3 2
0 0 0 1 1 4 3 2 4 1 4 3 2 4
Algorithm 7 is a simplified statement of Gauss reduction method.
The line with a comment [*] 7 is precisely the multiplication of A
which is upper triangular, is
on the left by L ji (m ji ). In the end, A,
a product of these matrices:
(3) A = Ln,n1 (mn,n1 ) Ln,n2 (mn,n2 ) L2,1 (m2,1 ) A = LA

1However, it is important to realize that no numerical algorithm intends to

find the exact solutions to any equation or system of equations. So this sameness
of the solutions relates only to the theoretical algorithm
 1. GAUSS ALGORITHM AND LU FACTORIZATION 37

and L is a lower triangular matrix with 1s on the diagonal (this is a

simple exercise to verify). It is easy to check (although this looks a
bit like magic) that
L EMMA . The inverse matrix of L in (3) is the lower triangular matrix
whose (i, j) entry is mij . That is,

( Ln,n1 (mn,n1 ) Ln,n2 (mn,n2 ) L2,1 (m2,1 ))1 =

1 0 0 ... 0

m21 1 0 . . . 0
m31 m32 1 . . . 0

. .. .. . . ..
.. . . . .
mn1 mn2 mn3 . . . 1

Algorithm 7 Gauss Algorithm for linear systems

Input: A square matrix A and a vector b, of order n
Output: Either an error message or a matrix A and a vector b such
= b has the same
that A is upper triangular and the system Ax
solutions as Ax = b
? S TART
A A, b b, i 1
while i < n do
if A ii = 0 then
return ERROR [division by zero]
end if
[combine rows underneath i with row i]
j i+1
while j n do
m ji A ji / A ii
[Next line is an operation on a row]
A j A j m ji A i [*]
b j b j m ji b i
j j+1
end while
i i+1
end while
return A, b

We have thus proved the following result:

38 3. NUMERICAL SOLUTIONS TO LINEAR SYSTEMS OF EQUATIONS

T HEOREM 3. If in the Gauss reduction process no element of the di-

agonal is 0 at any step, then there is a lower triangular matrix L whose
elements are the corresponding coefficients at their specific place and an
upper triangular matrix U such that
A = LU
and such that the system Ux = b = L1 b is equivalent to the initial one
Ax = b.
With this result, a factorization of A is obtained which simplifies
the resolution of the original system, as Ax = b can be rewritten
LUx = b and one can proceed step by step:
First the system Ly = b is solved by direct substitution that
is, from top to bottom, without even dividing.
Then the system Ux = y is solved by regressive substitution
from bottom to top; divisions will be needed at this point.
This solution method only requires storing the matrices L and U in
memory and is very fast.

1.1. Pivoting Strategies and the LUP Factorization. If during

Gaussian reduction a pivot appears (the element which determines
the multiplication) of value approximately 0, either the process can-
not be continued or one should expect large errors to take place,
due to rounding and truncation2. This problem can be tackled by
means of pivoting strategies, either swapping rows or both rows and
columns. If only rows are swapped, the operation is called partial
pivoting. If swapping both rows and columns is allowed, total pivot-
ing takes place.
D EFINITION 9. A permutation matrix is a square matrix whose en-
tries are all 0 but for each row and column, in each of which there is
exactly a 1.
Permutation matrices can be built from the identity matrix by
permuting rows (or columns).
A permutation matrix P has, on each row, a single 1 and the rest
entries are 0. If on row i the only 1 is on column j, the multiplication
PA means swap rows j and i of A.
Obviously, the determinant of a permutation matrix is not zero
(it is either 1 or 1). It is not so easy to show that the inverse of a
2Recall Example 5 of Chapter 1, where a tiny truncation error in a denominator
became a huge error in the final result.
 1. GAUSS ALGORITHM AND LU FACTORIZATION 39

permutation matrix is another permutation matrix and, as a matter

of fact, that P1 = P T (the transpose) if P is a permutation matrix.
L EMMA 2. If A is an n m matrix and P is a permutation matrix of
order n having only 2 non-zero elements out of the diagonal, say (i, j) and
( j, i ), then PA is the matrix obtained from A by swapping rows i and j. On
the other hand, if P is of order m, then AP is the matrix A with columns i
and j swapped.
D EFINITION 10. A pivoting strategy is followed in Gauss reduc-
tion process if the pivot element chosen at step i is the one with great-
est absolute value.
In order to perform Gauss algorithm with partial pivoting, one
can just follow the same steps but choosing, at step i, the row j > i in
which the pivot has the greatest absolute value and swapping rows
i and j.
Pivoting strategies give rise to a factorization which differs from
LU in that permutation matrices are allowed as factors.
L EMMA 3. Given the linear system of equations Ax = b with A non-
singular, there is a permutation matrix P and two matrices L, U, the former
lower triangular (with only 1 on the diagonal) and the latter upper trian-
gular such that
PA = LU.
This result is proved indirectly by recurrence (we are not going
to do it).
Both Matlab and Octave include the function lu which, given A,
returns three values: L, U and P.
For example, if

1 2 3 4
1 2 5 6
A= 1 2 3 7

0 12 7 8
then

1 0 0 0 1 2 3 4 1 0 0 0
0 1 0 0 0 12 7 8 0 0 0 1
L=
1
U=
0 0 8 10 , P = 0 1

0 1 0 0 0
1 0 0 1 0 0 0 11 0 0 1 0
In order to compute L, U and P one only needs to follow Gauss
algorithm and whenever a swap of rows i and j is performed, the same
swap has to be performed on L up to the i 1 column (that is, if a
40 3. NUMERICAL SOLUTIONS TO LINEAR SYSTEMS OF EQUATIONS

swap is needed on A at step i then the rows i and j are swapped but
only the columns 1 to i 1 are involved). Also, the P computed up to
step i has to be multiplied on the left by Pij (the permutation matrix
for i and j).
This can be stated as Algorithm 8.

Algorithm 8 LUP Factorization for a matrix A

Input: A matrix A of order n
Output: Either an error or three matrices: L lower triangular with
1 on the diagonal, U upper triangular and a permutation matrix P
such that LU = PA
Comment: Pip is the permutation matrix permuting rows i and p.
? S TART
L Idn , U Idn , P Idn , i 1
while i < n do
p row index such that |U pi | is maximum, with p i
if U pi = 0 then
return ERROR [division by zero]
end if
[swap rows i and p]
P Pip P
U Pip U
[on L swap only rows i and p on the submatrix
n (i 1) at the left, see the text]
L L
[combine rows on U and keep track on L]
j i+1
while j <= n do
m ji Uji /Uii
Uj Uj mij Ui
L ji m ji
j j+1
end while
i i+1
end while
return L, U, P

2. Condition Number: behavior of the relative error

We now deal briefly with the question of the stability of the meth-
ods for solving systems of linear equations. This means: if instead
 2. CONDITION NUMBER: BEHAVIOR OF THE RELATIVE ERROR 41

of starting with a vector b (which might be called the initial condi-

how much does the
tion), one starts with a slight modification b,
solution to the new system differ from the original one? The proper
way to ask this question uses the relative error, not the absolute one.
Instead of system (2), consider the modified one
Ay = b + b
where b is a small vector. The new solution will have the form x + x ,
for some x (which one expects to be small as well).
E XAMPLE 16. Consider the system of linear equations
    
2 3 x1 2
=
4 1 x2 1

whose solution is ( x1 , x2 ) = (0.1, 0.6). If we take b = (2.1, 1.05) then

b = (0.1, 0.05) and the solution of the new system
    
2 3 x1 2.1
=
4 1 x2 1.05
is ( x1 , x2 ) = (0.105, 0.63) so that x = (0.005, 0.03). In this case, a
small increment b gives rise to a small increment x but this needs
not be so.
However, see Example 17 for a system with a very different be-
haviour.
The size of vectors is measured using norms (the most common
one being length, in Euclidean space, but we shall not make use of
this one). As x is a solution, one has
A( x + x ) = b + b ,
so that
Ax = b ,
but, as we want to compare x with b , we can write

x = A1 b
so that, taking sizes (i.e. norms, which are denoted with kk), we get

kx k = k A1 b k.
Recall that we are trying to asses the relative displacement, not the
absolute one. To this end we need to include k x k in the left hand
42 3. NUMERICAL SOLUTIONS TO LINEAR SYSTEMS OF EQUATIONS

side of the equation. The available information is that Ax = b, from

which k Ax k = kbk. Hence,
kx k k A1 b k
(4) = ,
k Ax k kbk
but this is not very useful (it is obvious, as it stands). However,
assume there is some kind of matrix norm k Ak which satisfies that
k Ax k k Akk x k. Then, performing this substitution in Equation
(4), one gets
k A1 b k kx k kx k
=
kbk k Ax k k Akk x k
and, applying the same reasoning to the right hand side of (4), one
gets
k A1 kkb k k A1 b k
,
kbk kbk
and combining everything,
k A1 kkb k kx k

kbk k Akk x k
so that, finally, one gets an upper bound for the relative displacement
of x:
kx k k k
k Akk A1 k b
kxk kbk
The matrix norm which was mentioned above does exist. In fact,
there are many of them. We shall only make use of the following one
in these notes:
D EFINITION 11. The infinity norm of a square matrix A = ( aij ) is
the number
n
k Ak = max |aij |,
1 i n j =1

that is, the maximum of the sums of absolute values of each row.
The following result relates the infinity norms of matrices and
vectors:
L EMMA 4. The infinity norm is such that, for any vector x, k Ax k
k Ak k x k , where k x k is the norm given by the maximum of the abso-
lute values of the coordinates of x.
 2. CONDITION NUMBER: BEHAVIOR OF THE RELATIVE ERROR 43

This means that, if one measures the size of a vector by its largest
coordinate (in absolute value), and one calls it k x k , then
kx k k k
k A k k A 1 k b .
k x k kbk
The product k Ak k A1 k is called the condition number of A for the
infinity norm, is denoted ( A) and is a bound for the maximum pos-
sible displacement of the solution when the initial vector gets dis-
placed. The greater the condition number, the greater (to be ex-
pected) the displacement of the solution when the initial condition
(independent term) changes a little.
The condition number also bounds from below the relative dis-
placement:
L EMMA 5. Let A be a nonsingular matrix of order n and x a solution
of Ax = b. Let b be a displacement of the initial conditions and x the
corresponding displacement in the solution. Then:
1 kb k kx k k k
( A) b .
( A) kbk kxk kbk
So that the relative displacement (or error) can be bounded using the rela-
tive residue (the number kb k/kbk).
The following example shows how large condition numbers are
usually an indicator that solutions may be strongly dependent on the
initial values.
E XAMPLE 17. Consider the system
0.853x + 0.667y = 0.169
(5)
0.333x + 0.266y = 0.067
whose condition number for the infinity norm is 376.59, so that a rel-
ative change of a thousandth of unit in the initial conditions (vector
b) is expected to give rise to a relative change of more than 37% in
the solution. The exact solution of (5) is x = 0.055+, y = 0.182+.
However, the size of the condition number is a tell-tale sign that
a small perturbation on the system will modify the solutions greatly.
If, instead of b = (0.169, 0.067), one uses b = (0.167, 0.067) (which is
a relative displacement of just 1.1% in the first coordinate), the new
solution is x = 0.0557+, y = 0.321+, for which x has not even the
same sign as in the original problem and y is displaced 76% from its
original value. This is clearly unacceptable. If the equations describe
a static system, for example, and the coefficients have been measured
44 3. NUMERICAL SOLUTIONS TO LINEAR SYSTEMS OF EQUATIONS

with up to 3 significant digits, then the system of equations is useless,

as one cannot be certain that the measuring errors are meaningful.

3. Fixed Point Algorithms

Gauss reduction method is a first attempt at a fast algorithm for
solving linear systems of equations. It has two drawbacks: it is quite
complex (in the technical sense, which means it requires a lot of op-
erations to perform) and it depends greatly on the accuracy of divi-
sions (and when small divisors do appear, large relative errors are
expected to come up). However, the first issue (complexity) is the
main one. A rough approximation shows that for a system with n
variables, Gauss algorithm requires approximately n3 operations.
For n in the tens of thousands this soon becomes impractical.
Fixed point algorithms tackle this problem by not trying to find
an exact solution (which is the aim of Gauss method) but to ap-
proximate one using the techniques explained in Chapter 2.
Start with a system like (2), of the form Ax = b. One can trans-
form it into a fixed point problem by means of a decomposing of
matrix A into two: A = N P, where N is some invertible matrix.
This way, Ax = b can be written ( N P) x = b, i.e.

( N P) x = b Nx = b + Px x = N 1 b + N 1 Px.
If one calls c = N 1 b and M = N 1 P, then one obtains the following
fixed point problem:
x = Mx + c
which can be solved (if at all) in the very same way as in Chapter 2:
start with a seed x0 and iterate
xn = Mxn1 + c,
until a sufficient precision is reached.
In what follows, the infinity norm k k is assumed whenever the
concept of convergence appears3.
One needs the following results:
T HEOREM 4. Assume M is a matrix of order n and that k Mk < 1.
Then the equation x = Mx + c has a unique solution for any c and the
iteration xn = Mxn1 + c converges to it for any initial value x0 .

3However, all the results below apply to any matrix norm.

 3. FIXED POINT ALGORITHMS 45

T HEOREM 5. Given M with k Mk < 1, and given a seed x0 for the

iterative method of Theorem 4, if s is the solution to x = Mx + c, then the
following bound holds:
k Mkn
k xn sk k x1 x0 k .
1 k M k
Recall that for vectors, the infinity norm k x k is the maximum of the ab-
solute values of the coordinates of x.
We now proceed to explain the two basic iterative methods for
solving linear systems of equations: Jacobis and Gauss-Seidel. Both
rely on decomposing A in different ways: Jacobi takes N as the
diagonal of A and Gauss-Seidel as the lower triangular part of A,
including its diagonal.
3.1. Jacobis Algorithm. If each coordinate xi is solved explicitly
in terms of the others, for the system Ax = b, then something like
what follows appears:
1
xi = (bi ai1 x1 aii1 xi1 aii+1 xi+1 ain xn ),
aii
in matrix form,
x = D 1 ( b ( A D ) x ),
where D is the diagonal matrix whose nonzero elements are the di-
agonal of A. One can write,
x = D 1 b D 1 ( A D ) x,
which is a fixed point problem.
If D 1 ( A D ) satisfies the conditions of Theorem 4, then the
iterations corresponding to the above expression converge for any
choice of seed x0 and the bound of Theorem 5 holds. This is called
Jacobis method. In order to verify the necessary conditions, the
computation of D 1 ( A D ) is required, although there are other
sufficient conditions (especially Lemma 6).
3.2. Gauss-Seidels Algorithm. If instead of using the diagonal
of A one takes its lower triangular part (including the diagonal) then
one gets a system of the form
x = T 1 b T 1 ( A T ) x,
which is also a fixed point problem.
If T 1 ( A T ) satisfies the conditions of Theorem 4, then the cor-
responding iteration (called the Gauss-Seidel iteration) converges for
any initial seed x0 and the bound of Theorem 5 holds. In order to
46 3. NUMERICAL SOLUTIONS TO LINEAR SYSTEMS OF EQUATIONS

verify the conditions one should compute T 1 ( A T ), but there are

other sufficient conditions.
3.3. Sufficient conditions for convergence. We shall use two re-
sults which guarantee convergence for the iterative methods explained
above. One requires a technical definition, the other applies to posi-
tive definite matrices.
D EFINITION 12. A matrix A = ( aij ) is strictly diagonally dominant
by rows if
| aii | > | aij |
j 6 =i
for any i from 1 to n. That is, if the elements on the diagonal are
greater in absolute value than the sum of the rest of the elements of
their rows in absolute value.
For these matrices, the convergence of both Jacobis and Gauss-
Seidels methods is guaranteed:
L EMMA 6. If A is a strictly diagonally dominant matrix by rows
then both Jacobis and Gauss-Seidels methods converge for any system of
the form Ax = b and any seed.
For Gauss-Seidel, the following also holds:
L EMMA 7. If A is a positive definite symmetric matrix, then Gauss-
Seidels method converges for any system of the form Ax = b.

4. Annex: Matlab/Octave Code

Code for some of the algorithms explained is provided; it should
work both in Matlab and Octave.
4.1. Gauss Algorithm without Pivoting. Listing 3.1 implements
Gauss reduction algorithm for a system Ax = b, returning L, U and
the new b vector, assuming the multipliers are never 0; if some is 0, then
the process terminates with a message.
Input:
A: a square matrix (if it is not square, the output gives the tri-
angulation under the principal diagonal),
b: a vector with as many rows as A.
The output is a trio [L, At, bt], as follows:
L: the lower triangular matrix (with the multipliers),
At: the transformed matrix (which is U in the LU-factorization)
and which is upper triangular.
 4. ANNEX: MATLAB/OCTAVE CODE 47

bt: the transformed initial vector.

The new system to be solved is, hence, At x = bt.
function [L, At, bt] = gauss(A,b)
n = size(A);
m = size(b);
if(n(2) = m(1))
warning(The sizes of A and b do not match);
return;
end
At=A; bt=b; L=eye(n);
k=1;
while (k<n(1))
l=k+1;
if(At(k,k) == 0)
warning(There is a 0 on the diagonal);
return;
end
% careful with rows & columns:
% L(l,k) means ROW l, COLUMN k
while(l<=n)
L(l,k)=At(l,k)/At(k,k);
% Combining rows is easy in Matlab
At(l,k:n) = [0 At(l,k+1:n) - L(l,k) * At(k,k+1:n)];
bt(l)=bt(l)-bt(k)*L(l,k);
l=l+1;
end
k=k+1;
end
end
L ISTING 3.1. Gauss Reduction Algorithm

4.2. LUP Factorization. Gauss Reduction depends on the non-

appearing of zeros on the diagonal and it can also give rise to large
rounding errors if pivots are small. Listing 3.2 implements LUP fac-
torization, which provides matrices L, U and P such that LU = PA,
L and U being respectively lower and upper triangular and P a per-
mutation matrix. Its input is
A: a square matrix of order n.
b: An n-row vector.
The output is a vector [L, At, P, bt] where, L, At, P and
bt, are three matrices and a vector corresponding to L, U, P and the
according to the algorithm.
transformed vector b,
function [L, At, P, bt] = gauss_pivotaje(A,b)
n = size(A);
m = size(b);
if(n(2) = m(1))
warning(Dimensions of A and b do not match);
48 3. NUMERICAL SOLUTIONS TO LINEAR SYSTEMS OF EQUATIONS

return;
end
At=A;
bt=b;
L=eye(n);
P=eye(n);
i=1;
while (i<n)
j=i+1;
% beware nomenclature:
% L(j,i) is ROW j, COLUMN i
% the pivot with greatest absolute value is sought
p = abs(At(i,i));
pos = i;
for c=j:n
u = abs(At(c,i));
if(u>p)
pos = c;
p = u;
end
end
if(u == 0)
warning(Singular system);
return;
end
% Swap rows i and p if i != p
% in A and swap left part of L
% This is quite easy in Matlab, there is no need
% for temporal storage
P([i pos],:) = P([pos i], :);
if(i = pos)
At([i pos], :) = At([pos i], :);
L([i pos], 1:i-1) = L([pos i], 1:i-1);
b([i pos], :) = b([pos i], :);
end
while(j<=n)
L(j,i)=At(j,i)/At(i,i);
% Combining these rows is easy
% They are 0 up to column i
% And combining rows is easy as above
At(j,i:n) = [0 At(j,i+1:n) - L(j,i)*At(i,i+1:n)];
bt(j)=bt(j)-bt(i)*L(j,i);
j=j+1;
end
i=i+1;
end
end
L ISTING 3.2. LUP Factorization
 CHAPTER 4

Interpolation

Given a set of data generally a cloud of points on a plane, the

human temptation is to use them as source of knowledge and fore-
casting. Specifically, given a list of coordinates associated to some
kind of event (say, an experiment or a collection of measurements)
( xi , yi ), the natural thing to do is to use it for deducing or predicting
the value y would take if x took some other value not in the list. This
is the interpolating and extrapolating tendency of humans. There is no
helping it. The most that can be done is studying the most reason-
able ways to perform those forecasts.
We shall use, along this whole chapter, a list of n + 1 pairs ( xi , yi )
x x 0 x 1 . . . x n 1 x n
(6)
y y 0 y 1 . . . y n 1 y n
which is assumed ordered on the x coordinates, which are all differ-
ent, as well: xi < xi+1 . The aim is to find functions which somehow
have a relation (a kind of proximity) to that cloud of points.

1. Linear (piecewise) interpolation

The first, simplest and useful idea is to use a piecewise defined
function from x0 to xn consisting in joining each point to the next
one by a straight line. This is called piecewise linear interpolation or
linear spline we shall define spline in general later on.
D EFINITION 13. The piecewise linear interpolating function for
list (6) is the function f : [ x0 , xn ] R defined as follows:
y y i 1
f (x) = i ( x xi1 ) + yi1 if x [ xi1 , xi ]
x i x i 1
that is, the piecewise defined function whose graph is the union of
the linear segments joining ( xi1 , yi1 ) to ( xi , yi ), for i = 1, . . . , n.
Piecewise linear interpolation has a set of properties which make
it quite interesting:
It is very easy to compute.
It passes through all the data points.
49
50 4. INTERPOLATION

f (x)
linear interpolation
4

4 6 8 10

F IGURE 1. Piecewise linear interpolation of a 9-point cloud.

It is continuous.
That is why it is frequently used for drawing functions (it is what
Matlab does by default): if the data cloud is dense, the segments are
short and corners will not be noticeable on a plot.
The main drawback of this technique is, precisely, the corners
which appear anywhere the cloud of points does not correspond to
a straight line. Notice also that this (and the following splines which
we shall explain) are interpolation methods only, not suitable for ex-
trapolation: they are used to approximate values between the endpoints
x0 , xn , never outside that interval.

2. Can parabolas be used for this?

Linear interpolation is deemed to give rise to corners whenever
the data cloud is not on a straight line. In general, interpolation re-
quirements do not only include the graph to pass through the points
in the cloud but also to be reasonably smooth (this is not just for aes-
thetic reasons but also because reality usually behaves this way).
One could try and improve linear interpolation with higher degree
functions, imposing that the tangents of these functions match at the
intersection points. For example, one could try with parabolic seg-
ments (polynomials of degree two). As they have three degrees of
freedom, one could make them not only pass through ( xi1 , yi1 )
and ( xi , yi ), but also have the same derivative at xi . This may seem
reasonable but has an inherent undesirable property: it is intrinsi-
cally asymmetric (we shall not explain why, the reader should be
 3. CUBIC SPLINES: CONTINUOUS CURVATURE 51

f (x)
quadratic spline
4

4 6 8 10

F IGURE 2. Quadratic interpolation of a 9-point cloud.

Compare with the original function.

able to realize this after some computations). Without delving into

the details, one can verify that this quadratic spline is not optimal al-
though it approximates the data cloud without corners.
It has also some other problems (it tends to get far from the data
cloud if there are points whose x-coordinates are near but whose y-
coordinates differ in different directions). It is almost never used
except where the cloud of points is known a priori to resemble a
parabola.
The next case, that of cubic splines is by far the most used: as
a matter of fact, computer aided design (CAD) software uses cubic
splines any time it needs to draw a smooth curve, not exactly using
a table like (6) but with two tables, because curves are parametrized
as ( x (t), y(t)).

3. Cubic Splines: Continuous Curvature

In order to approximate clouds of points with polynomials of de-
gree 3, the following definition is used:
D EFINITION 14. A cubic spline for a table like (6) is a function
f : [ x0 , xn ] R such that:
It matches a degree 3 polynomial on every segment [ xi1 , xi ],
for i = 1, . . . , n.
It is two times differentiable with continuity at every xi , for
i = 1, . . . , n 1 (the inner points).
52 4. INTERPOLATION

f (x)
cubic spline
4

4 6 8 10

F IGURE 3. Cubic spline for a 9-point cloud. Compare

with the original function.

From this follows that, if Pi is the polynomial of degree 3 match-

ing f on [ xi1 , xi ], then Pi0 ( xi ) = Pi0+1 ( xi ) and Pi00 ( xi ) = Pi00+1 ( xi );
these facts, together with Pi ( xi ) = yi and Pi ( xi+1 ) = yi+1 give 4 con-
ditions for each Pi , except for the first and the last ones, for which
there are only 3 (this symmetry of behavior at the endpoints is what
quadratic splines lack). Thus, the conditions for being a cubic spline
almost determine the polynomials Pi (and hence f ). Another condi-
tion for P1 and Pn is needed for a complete specification. There are
several alternatives, some of which are:
That the second derivative at the endpoints be 0. This gives
rise to the natural cubic spline, but needs not be the best op-
tion for a problem. The corresponding equations are P100 ( x0 ) =
0 and Pn00 ( xn ) = 0.
That the third derivative matches at the almost extremal
points: P1000 ( x1 ) = P2000 ( x1 ) and Pn000 ( xn1 ) = Pn0001 ( xn1 ). This
is the extrapolated spline.
A periodicity condition: P10 ( x0 ) = Pn0 ( xn ) and the same for
the second derivative: P100 ( x0 ) = Pn00 ( xn ). This may be rele-
vant if, for example, one is interpolating a periodic function.

3.1. Computing the Cubic Spline: Tridiagonal Matrices. Before

proceeding with the explicit computation of the cubic spline, let us
introduce some useful notation which will allow us to simplify most
of the expressions that will appear.
 3. CUBIC SPLINES: CONTINUOUS CURVATURE 53

We shall denote by Pi the polynomial corresponding to the inter-

val [ xi1 , xi ] and we shall always write it relative to xi1 , as follows:
(7) Pi ( x ) = ai + bi ( x xi1 ) + ci ( x xi1 )2 + di ( x xi1 )3 .
We wish to compute ai , bi , ci and di for i = 1, . . . , n from the cloud of
points (x, y) above (6).
The next normalization is to rename xi xi1 and use
hi = xi xi1 , for i = 1, . . . , n
(that is, use the width of the n intervals instead of the coordinates
xi ).
Finally, in an attempt to clarify further, known data will be writ-
ten in boldface.
We reason as follows:
The values ai are directly computable, because Pi (xi1 ) = ai
and, by hypothesis, this must equal yi1 . Hence,
ai = yi1 for i = 1, . . . , n
As Pi (xi ) = yi , using now that ai yi1 on the right hand side
of the equality (7), one gets
(8) bi h i + c i h i 2 + d i h i 3 = y i y i 1 .
for i = 1, . . . , n, which gives n linear equations.
The condition on the continuity of the derivative is Pi0 (xi ) =
Pi0+1 (xi ), so that

(9) bi + 2ci hi + 3di hi 2 = bi+1 ,

for i = 1, . . . n 1. This gives other n 1 equations.
Finally, the second derivatives must match at the intermedi-
ate points, so that
(10) 2ci + 6di hi = 2ci+1 ,
for i = 1, . . . , n 1, which gives other n 1 equations.
Summing up, there are (apart from the known ai ), 3n 2 linear equa-
tions for 3n unknowns (all of the bi , ci and di ). We have already re-
marked that one usually imposes two conditions at the endpoints in
order to get 3n linear equations.
However, before including the other conditions, one simplifies
and rewrites all the above equations in order to obtain a more intel-
ligible system. What is done is to solve the ds and the bs in terms of
the cs, obtaining a system in which there are only cs.
54 4. INTERPOLATION

First of all, Equations (10) are used to solve the di :

c i +1 c i
(11) di =
3hi
and substituting in (8), one gets (solving for bi ):
yi yi1 c + 2ci
(12) bi = h i i +1 ;
hi 3
on the other hand, substituting di in (8) and computing until solving
bi , one gets
(13) bi = bi 1 + h i 1 ( c i + c i 1 ) .
for i = 2, . . . , n. Now one only needs to use Equations (12) for i and
i 1 and introduce them into (13), so that there are only cs. After
some other elementary calculations, one gets
(14)
yi yi1 yi1 yi2
 
hi1 ci1 + (2hi1 + 2hi )ci + hi ci+1 = 3
hi hi1
for i = 2, . . . , n 1.
This is a system of the form Ac = , where A is the n 2 by n
matrix
(15)
h1 2( h1 + h2 ) h2 0 ... 0 0 0

0 h2 2( h2 + h3 ) h3 ... 0 0 0
A= ... .. .. .. .. .. .. ..
. . . . . . .
0 0 0 0 . . . h n 1 2 ( h n 1 + h n ) h n
and c is the column vector (c1 , . . . , cn )t , whereas is

2
3
.
..
n 1
with
y i y i 1 y i 1 y i 2
 
i = 3 .
hi h i 1
It is easy to see that this is a consistent system (with infinite so-
lutions, because there are two missing equations). The equations
which are usually added were explained above. If one sets, for ex-
ample, c1 = 0 and cn = 0, so that A is completed above with a row
(1 0 . . . 0)and below with (0 0 0 . . . 1), whereas gets a top and
 3. CUBIC SPLINES: CONTINUOUS CURVATURE 55

bottom 0. From these n equations, one computes all the ci and, using
(11) and (12), one gets all the bs and ds.
The above system (15), which has only nonzero elements on the
diagonal and the two adjacent lines is called tridiagonal. These sys-
tems are easily solved using LU factorization and one can even com-
pute the solution directly, solving the cs in terms of the s and hs.
One might as well use iterative methods but for these very simple
systems, LU factorization is fast enough.
3.2. The Algorithm. We can now state the algorithm for com-
puting the interpolating cubic spline for a data list x, y of length
n + 1, x = ( x0 , . . . , xn ), y = (y0 , . . . , yn ), in which xi < xi+1 (so
that all the values in x are different). This is Algorithm 9.
3.3. Bounding the Error. The fact that the cubic spline is graphi-
cally satisfactory does not mean that it is technically useful. As a matter
of fact, it is much more useful than what it might seem. If a function
is well behaved on the fourth derivative, then the cubic spline is a
very good approximation to it (and as the intervals get smaller, the
better the approximation is). Specifically, for clamped cubic splines, we
have:
T HEOREM 6. Let f : [ a, b] R be a 4 times differentiable function
with | f 4) ( x )| < M for x [ a, b]. Let h be the maximum of xi xi1 for
i = 1, . . . , n. If s( x ) is a the clamped cubic spline for ( xi , f ( xi )), then
5M 4
|s( x ) f ( x )| h .
384
This result can be most useful for computing integrals and bound-
ing the error or for bounding the error when interpolating values
of solutions of differential equations. Notice that the clamped cu-
bic spline for a function f is such that s0 ( x0 ) = f 0 ( x0 ) and s0 ( xn ) =
f 0 ( xn ), that is, the first derivative at the endpoints is given by the
first derivative of the interpolated function.
Notice that this implies, for example, that if h = 0.1 and M <
60 (which is rather common: a derivative greater than 60 is huge),
then the distance at any point between the original function f and the
interpolating cubic spline is less than 104 .
3.4. General Definition of Spline. We promised to give the gen-
eral definition of spline:
D EFINITION 15. Given a data list as (6), an interpolating spline of
degree m for it, (for m > 0), is a function f : [ x0 , xn ] R such that
56 4. INTERPOLATION

Algorithm 9 Computation of the Cubic Spline

Input: a data table x, y as specified in (6) and two conditions, one
on the first and one on the last node, usually.
Output: an interpolating cubic spline for the table. Specifically,
a n lists of four numbers ( ai , bi , ci , di ) such that the polynomials
Pi ( x ) = ai + bi ( x xi1 ) + ci ( x xi1 )2 + di ( x xi1 )3 give an
interpolating cubic spline for the table, satisfying the two extra
conditions of the input.
? S TART
ai yi1 for i from 1 to n
hi xi xi1 for i from 1 to n
i1
[Build the tridiagonal system:]
while i n do
if i > 1 and i < n then
Fi (0 0 hi1 2(hi1 + hi ) hi 0 0)
i = 3(yi yi1 )/hi 3(yi1 yi2 )/hi1
else
Fi the row corresponding to the equation for Pi
i the coefficient corresponding to the equation for Pi
end if
i i+1
end while
M the matrix whose rows are Fi for i = 1 to n
c M1 [solve the system Mc = ]
i1
[Solve the b0 s and d0 s:]
while i < n do
bi (yi yi1 )/hi hi (ci+1 + 2ci )/3
di (ci+1 ci )/(3hi )
i i+1
end while
bn bn 1 + h n 1 ( c n + c n 1 )
dn (yn yn1 bn hn cn h2n )/h3n
return ( a, b, c, d)

It passes through all the points: f ( xi ) = yi ,

Is m 1 times differentiable
On every interval [ xi , xi+1 ] it coincides with a polynomial of
degree at most m.
 4. LAGRANGE INTERPOLATION 57

That is, a piecewise polynomial function passing through all the

points and m 1 times differentiable (where 0 times differentiable
means continuity).
The most used are linear splines (degree 1) and cubic splines (de-
gree 3). Notice that there is not a unique spline of degree m for m 2.

4. The Lagrange Interpolating Polynomial

Splines, as has been explained, are piecewise polynomial functions
which pass through a given set of points, with some differentiability
conditions.
One could state a more stringent problem: finding a true poly-
nomial of the least degree which passes through each of the points
given by (6). That is, given n + 1 points, find a polynomial of degree
at most n which passes through each and every point. This is the
Lagrange interpolation problem, which has a solution:
T HEOREM 7 (Lagranges interpolation polynomial). Given a data
list as (6) (recall that xi < xi+1 ), there is a unique polynomial of degree at
most n passing through each ( xi , yi ) for i = 0, . . . , n.
The proof is elementary. First of all, one starts with a simpler
version of the problem: given n + 1 values x0 < < xn , is there a
polynomial of degree at most n whose value is 0 at every x j for j =
0, . . . , n but for xi where its value is 1? This is easy because there is a
simple polynomial of degree n whose value is 0 at each x j for j 6= i:
namely i ( x ) = ( x x0 )( x x1 ) . . . ( x xi1 )( x xi+1 ) . . . ( x xn ).
Now one only needs multiply i ( x ) by a constant so that its value at
xi is 1. The polynomial i ( x ) has, at xi the value
i ( xi ) = ( xi x1 ) . . . ( xi xi1 )( xi xi+1 ) . . . ( xi xn ) = ( x i x j ),
j 6 =i

so that the following polynomial pi( x) takes the value 1 at xi and 0 at

any x j for j 6= i.

j 6 =i ( x x j )
(16) pi ( x ) = .
j 6 =i ( xi x j )
These polynomials p0 ( x ), . . . , pn ( x ) are called the Lagrange basis poly-
nomials (there are n + 1 of them, one for each i = 0, . . . , n). The collec-
tion { p0 ( x ), . . . , pn ( x )} can be viewed as the basis of the vector space
Rn+1 . From this point of view, now the vector P( x ) = (y0 , y1 , . . . , yn )
58 4. INTERPOLATION

f (x)
5 cubic spline
Lagrange
4

4 6 8 10

F IGURE 4. Comparison between a cubic spline and the

Lagrange interpolating polynomial for the same f ( x )
as above. Notice the relatively large spike of the La-
grange polynomial at the last interval.

is to be expressed as a linear combination of them, but there is only

one way to do so:
n
P ( x ) = y0 p0 ( x ) + y1 p1 ( x ) + + y n p n ( x ) = yi pi ( x )
i =0
(17) n j 6 =i ( x x j )
= yi .
i =0 j 6 = i i
(x xj )

One verifies easily that this P( x ) passes through all the points ( xi , yi )
for i = 0, . . . , n.
The fact that there is only one polynomial of the same degree
as P( x ) satisfying that condition can proved as follows: if there ex-
isted Q( x ) of degree less than or equal to n passing through all those
points, the difference P( x ) Q( x ) between them would be a poly-
nomial of degree at most n with n + 1 zeros and hence, would be 0.
So, P( x ) Q( x ) would be 0, which implies that Q( x ) equals P( x ).
Compare the cubic spline interpolation with the Lagrange inter-
polating polynomial for the same function as before in figure 4.
The main drawbacks of Lagranges interpolating polynomial are:
There may appear very small denominators, which may give
rise to (large) rounding errors.
It is too twisted.
 4. LAGRANGE INTERPOLATION 59

0.8

0.6

0.4

0.2
1
f ( x ) = 1+12x 2
Lagrange
0
1 0.5 0 0.5 1

F IGURE 5. Runges Phenomenon: the Lagrange in-

terpolating polynomial takes values very far away
from the original function if the nodes are evenly dis-
tributed. There is a thin blue line which corresponds
to the cubic spline, which is indistinguishable from the
original function.

The first problem is intrinsic to the way we have computed it1.

The second one depends on the distribution of the xi in the seg-
ment [ x0 , xn ]. A remarkable example is given by Runges phenome-
non: whenever a function with large derivatives is approximated by
a Lagrange interpolating polynomial with the x coordinates evenly
distributed, then this polynomial gets values which differ greatly
from those of the original function (even though it passes through
the interpolation points). This phenomenon is much less frequent in
splines.
One may try to avoid this issue (which is essentially one of cur-
vature) using techniques which try to minimize the maximum value
taken by the polynomial

R( x ) = ( x x0 )( x x1 ) . . . ( x xn ),

that is, looking for a distribution of the xi which solves a minimax

problem. We shall not get into details. Essentially, one has:

1There are different ways, but we shall not describe them.

60 4. INTERPOLATION

0.8

0.6

0.4

0.2 1
f ( x ) = 1+12x 2
Lagrange
0
1 0.5 0 0.5 1

F IGURE 6. Approximation of Runges function using

the Lagrange polynomial and Chebychevs nodes. The
maximum error is much less than in 5.
L EMMA 8. The points x0 , . . . , xn minimizing the largest maximum
value of R( x ) on the interval [1, 1] are given by the formula
 
2k + 1
xi = cos
n+1 2
So that, the corresponding points for [ a, b], for a, b R, are
( b a ) xi + ( a + b )
xi = .
2
The points xi in the lemma are called Chebychevs nodes for the in-
terval [ a, b]. They are the ones to be used if one wishes to interpolate
a function using Lagranges polynomial and get a good approxima-
tion.
5. Approximate Interpolation
In experimental and statistical problems, data are always consid-
ered inexact, so that trying to find a curve fitting them all exactly
makes no sense at all. This gives rise to a different interpolation con-
cept: one is no longer interested in making a curve pass through
each point in a cloud but in studying what curve in a family resem-
bles that cloud in the best way. In our case, when one has a data table
like (6), this resemblance is measured using the minimal quadratic
distance from f ( xi ) to yi , where f is the interpolating function, but
this is not the only way to measure the nearness (however, it is the
one we shall use).
 5. APPROXIMATE INTERPOLATION 61

1
1 0.5 0 0.5 1

F IGURE 7. Least squares interpolation of a cloud of

points using functions of the form y = ax + b.

5.1. Least Squares Interpolation. The most common approxi-

mate interpolating method is least squares interpolation. One starts
with a cloud of points x, y, where x e y are arbitrary vectors of length
N. This is different from (6): now we enumerate xi and yi from 1 to
N (notice also that there may be repeated x values and they need
not be sorted). Given f : R R, one defines
D EFINITION . The quadratic error of f at xi (one of the coordinates
of x)) is the number ( f ( xi ) yi )2 . The total quadratic error of f on the
cloud (x, y) is the sum
N
( f ( x i ) y i )2 .
i =1

The least squares linear interpolation problem consists in, given the
cloud of points ( xi , yi ) and a family of functions, to find the func-
tion among those in the family which minimizes the total quadratic
error. This family is assumed to be a finite-dimensional vector space
(whence the term linear interpolation).
We shall give an example of nonlinear interpolation problem, just
to show the difficulties inherent to its lack of vector structure.

5.2. Linear Least Squares. Assume one has to approximate a

cloud of points of size N with a function f belonging to a vector
space V of dimension n and that a basis of V is known: f 1 , . . . , f n .
We always assume N > n (it is actually much greater, in general).
62 4. INTERPOLATION

That is, we try to find a function f V such that

N
( f ( x i ) y i )2
i =1

is minimal. As V is a vector space, this f must be a linear combina-

tion of the elements of the basis:
f = a1 f 1 + a2 f 2 + + a n f n .
And the problem consists in finding the coefficients a1 , . . . , an mini-
mizing the value of the nvariable function
N
F ( a1 , . . . , a n ) = ( a1 f 1 ( x i ) + + a n f n ( x i ) y i )2 ,
i =1

which is an ndimensional optimization problem. It is obvious that

F is differentiable function, so that the minimum will annul all the
partial derivatives (notice that the variables are the ai !). The follow-
ing system, then, needs to be solved (the critical values of F):
F F F
= 0, = 0, . . . , = 0.
a1 a2 an
The expansion of the partial derivative of F with respect to a j is
N
F
= 2 f j ( xi )( a1 f 1 ( xi ) + + an f n ( xi ) yi ) = 0.
a j i =1

Writing
N
yj = f j ( xi ) yi ,
i =1
and stating the equations in matrix form, one gets the system
f 1 ( x1 ) f 1 ( x2 ) ... f1 (x N ) f 1 ( x1 ) f 2 ( x1 ) ... f n ( x1 ) a1

f 2 ( x1 ) f 2 ( x2 ) f2 (x N )
... f 1 ( x2 ) f 2 ( x2 ) ... f n ( x2 )
a2

.. .. =

. .. .. .. .. .. ..
.. . . . . . . . .
f n ( x1 ) f n ( x2 ) ... fn (xN ) f1 (x N ) f2 (x N ) ... fn (xN ) an
(18)
f 1 ( x1 ) f 1 ( x2 ) ... f1 (x N ) y1 y1

f 2 ( x1 ) f 2 ( x2 ) f2 (x N )
... y2 y 2

= . .. .. = .. ,

.. ..
.. . . . . .
f n ( x1 ) f n ( x2 ) ... fn (xN ) yN yn

which is of the form

XX t a = Xy
 5. APPROXIMATE INTERPOLATION 63

where X is the matrix whose row i is the list of values of f i at x1 , . . . , x N

and y is the column vector (y1 , y2 , . . . , y N )t , that is:
f 1 ( x1 ) f 1 ( x2 ) . . . f 1 ( x N ) y1

f 2 ( x1 ) f 2 ( x2 ) . . . f 2 ( x N ) y1
X= ... .. .. .. , y= ...

. . .
f n ( x1 ) f n ( x2 ) . . . f n ( x N ) yN
System (18) is consistent system and has a unique solution if there
are at least as many points as the dimension of V and the functions
f i generate linearly independent rows (i.e. if the rank of X is n).
However, system (18) will very likely be ill-conditioned.

5.3. Non-linear Least Squares... Danger. There are many cases

in which the interpolating family of functions is not a vector space.
A well-known example is the set given by the functions:
2
(19) f ( x ) = aebx .
These are Gaussian bells for a, b > 0. They obviously do not form a
vector space and the techniques of last section do not apply.
One could try to transform the problem into a linear one, per-
form the least squares approximation on the transformed problem
and revert the result. This may be catastrophic.
Taking logarithms on both sides of (19), one gets:
log( f ( x )) = log( a) + bx2 = a0 + bx2 ,
2
and, if instead of considering functions aebx , one considers c + dx2 ,
and tries to approach the cloud of points x, log(y), then one has
a classical linear interpolation problem, for which the techniques
above apply. If one obtains a solution a0 = log( a0 ), b0 = b0 , then
one could think that
2
g( x ) = a0 eb0 x
would be a good approximation to the original cloud of points (x, y).
It might be. But it might not be. One has to be aware that this approx-
imation may not minimize the total quadratic error.
Notice that, when taking logarithms, yvalues near 0 are trans-
formed into values near , and these will have a much greater im-
portance than they had before the transformation. This is because,
when taking logarithms, absolute and relative errors behave in a to-
tally unrelated way. Moreover, if one of yvalues is 0, there is no
way to take logarithms and it would have to be discarded.
64 4. INTERPOLATION

3 data
2
3e2x
linear interp.
2

0
2 1 0 1 2

F IGURE 8. Non-linear interpolation using logarithms:

the cloud of points resembles the function f ( x ), but the
linear interpolation taking logarithms and performing
the inverse transform afterwards (the low bell) is no-
ticeably bad. This is because there are yvalues too
near to 0.

What is at stake here is, essentially, the fact that log( x ) is not a
linear function of x, that is: log( a + b) 6= log( a) + log(b). Then, per-
forming a linear computation on the logarithms and then computing
the exponential is not the same as performing the linear operation on
the initial data.

6. Code for some of the Algorithms

6.1. Cubic Splines. Matlab computes not-a-knot splines by de-
fault (which arise from imposing a condition on the third derivative
at the second and last-but-one points) and natural splines can only be
computed using a toolbox. Listing 4.1 implements the computation
of natural cubic splines for a cloud of points. Input is:
x: the list of x coordinates
y: the list of ycoordinates
It returns a Matlab/Octave object which implements a piecewise
polynomial. There is no need to completely understand this structure,
only that if one wants to compute its value at a point, one uses the
function ppval.
% natural cubic spline: second derivative at both
% endpoints is 0. Input is a pair of lists describing
% the cloud of points.
 6. CODE FOR SOME OF THE ALGORITHMS 65

function [f] = spline_cubico(x, y)

% safety checks
n = length(x)-1;
if(n<=1 | length(x) = length(y))
warning(Wrong data)
f= [];
return
end

% variables and coefficients for the linear system,

% these are the ordinary names. Initialization
a = y(1:n);
h = diff(x);
F = zeros(n);
alpha = zeros(n, 1);

% true coefficients (and independent terms) of the linear system

for k=1:n
if(k> 1& k < n)
F(k,[k-1 k k+1]) = [h(k-1), 2*(h(k-1) + h(k)), h(k)] ;
alpha(k) = 3*(y(k+1)-y(k))/h(k) - 3*(y(k) - y(k-1))/h(k-1);
else
% these two special cases are the natural condition
% (second derivatives at endpoints = 0)
F(k,k) = 1;
alpha(k) = 0;
end
k=k+1;
end

% These are the other coefficients of the polynomials

% (a + bx + cx2 + dx3)... Initialization
c = (F\alpha);
b = zeros(1,n);
d = zeros(1,n);
% unroll all the coefficients as in the theory
k = 1;
while(k<n)
b(k) = (y(k+1)-y(k))/h(k) - h(k) *(c(k+1)+2*c(k))/3;
k=k+1;
end
d(1:n-1) = diff(c)./(3*h(1:n-1));

% the last b and d have explicit expressions:

b(n) = b(n-1) + h(n-1)*(c(n)+c(n-1));
d(n) = (y(n+1)-y(n)-b(n)*h(n)-c(n)*h(n)2)/h(n)3;

% finally, build the piecewise polynomial (a Matlab function)

% we might implement it by hand, though
f = mkpp(x,[d; c; b ;a ]);
end
 66 4. INTERPOLATION

L ISTING 4.1. Natural cubic spline computation using

Matlab/Octave
The following lines are an example of the usage of Listing 4.1 for
approximating the graph of the sine function.
> x = linspace(-pi, pi, 10);
> y = sin(x);
> f = spline_cubico(x, y);
> u = linspace(-pi, pi, 400);
> plot(u, ppval(f, u)); hold on;
> plot(u, sin(u), r);

6.2. The Lagrange Interpolating Polynomial. The computation

of the Lagrange Interpolating Polynomial in Matlab/Octave is rather
simple, as one can confirm reading listing 4.2.
Input is:
x: The list of xcoordinates of the cloud of points
y: The list of ycoordinates of the cloud of points
Output is a polynomial in vector form, that is, a list of the coeffi-
cients an , an1 , . . . , a0 such that P( x ) = an x n + + a1 x + a0 is the
Lagrange interpolating polynomial for (x, y).
1 % Lagrange interpolation polynomial
2 % A single base polynomial is computed at
3 % each step and then added (multiplied by
4 % its coefficient) to the final result.
5 % input is a vector of x coordinates and
6 % a vector (of equal length) of y coordinates
7 % output is a polynomial in vector form (help poly).
8 function [l] = lagrange(x,y)
9 n = length(x);
10 l = 0;
11 for m=1:n
12 b = poly(x([1:m-1 m+1:n]));
13 c = prod(x(m)-x([1:m-1 m+1:n]));
14 l = l + y(m) * b/c;
15 end
16 end
L ISTING 4.2. Code for computing the Lagrange
interpolating polynomial

6.3. Linear Interpolation. For linear interpolation in Matlab/Oc-

tave, one has to use a special object, a cell array: the list of functions
which comprise the basis of the vector space V has to be input be-
tween curly braces. See Listing 4.3.
Input:
x: The list of xcoordinates of the cloud
 6. CODE FOR SOME OF THE ALGORITHMS 67

y: The list of ycoordinates of the cloud

F: A cell array of anonymous functions. Each elements is one
of the functions of the basis of the linear space used for in-
terpolation.
The output is a vector c with coordinates (c1 , . . . , cn ), such that
c1 F1 + + cn Fn is the least squares interpolating function in V for
the cloud (x, y).
1 % interpol.m
2 % Linear interpolation.
3 % Given a cloud (x,y), and a Cell Array of functions F,
4 % return the coefficients of the least squares linear
5 % interpolation of (x,y) with the base F.
6 #
7 % Input:
8 % x: vector of scalars
9 % y: vector of scalars
10 % F: Cell array of anonymous functions
11 #
12 % Outuput:
13 % c: coefficients such that
14 % c(1)*F{1,1} + c(2)*F{1,2} + ... + c(n)*F{1,n}
15 % is the LSI function in the linear space <F>.
16
17 function [c] = interpol(x, y, F)
18 n = length(F);
19 m = length(x);
20 X = zeros(n, m);
21 for k=1:n
22 X(k,:) = F{1,k}(x);
23 end
24 A = X*X.;
25 b = X*y.;
26 c = (A\b);
27 end
L ISTING 4.3. Code for least squares interpolation
Follows an example of use, interpolating with trigonometric func-
tions
> f1=@(x) sin(x); f2=@(x) cos(x); f3=@(x) sin(2*x); f4=@(x) cos(2*x);
> f5=@(x)sin(3*x); f6=@(x)cos(3*x);
> F={f1, f2, f3, f4, f5, f6};
> u=[1,2,3,4,5,6];
> r=@(x) 2*sin(x)+3*cos(x)-4*sin(2*x)-5*cos(3*x);
> v=r(u)+rand(1,6)*.01;
> interpol(u,v,F)
ans =
1.998522 2.987153 -4.013306 -0.014984 -0.052338 -5.030067
 CHAPTER 5

Numerical Differentiation and Integration

Numerical differentiation is explained in these notes using the

symmetrical increments formula and showing how this can be gen-
eralized for higher order derivatives. A brief glimpse into the insta-
bility of the problem is given.
Integration is dealt with in deeper detail (because it is easier and
much more stable) but also briefly. A simple definition of quadrature
formula is given and the easiest ones (the trapezoidal and Simpsons
rules) are explained.

1. Numerical Differentiation
Sometimes (for example, when approximating the solution of a
differential equation) one has to approximate the value of the deriv-
ative of a function at point. A symbolic approach may be unavailable
(either because of the software or because of its computational cost)
and a numerical recipe may be required. Formulas for approximat-
ing the derivative of a function at a point are available (and also for
higher order derivatives) but one also needs, in general, bounds for
the error incurred. In these notes we shall only show the symmetric
rule and explain why the naive approximation is suboptimal.? instability?

1.1. The Symmetric Formula for the Derivative. The first (sim-
ple) idea for approximating the derivative of f at x is to use the def-
inition of derivative as a limit, that is, the following formula:
f ( x + h) f ( x )
(20) f 0 (x) ' ,
h
where h is a small increment of the x variable.
However, the very expression in formula (20) shows its weak-
ness: should one take h positive or negative? This is not irrelevant.
Assume f ( x ) = 1/x and try to compute its derivative at x = 2. We
shall take |h| = .01. Obviously f 0 (0.5) = 0.25. Using the natural
approximation one has, for h > 0:
1
f ( x + .01) f ( x ) 2.01 12
= = 0.248756+
.01 .01
69
70 5. NUMERICAL DIFFERENTIATION AND INTEGRATION

f ( x0 + h )

f ( x0 )

f ( x0 h )

x0 h x0 x0 + h

F IGURE 1. Approximate derivative: on the right & left

(dotted) and symmetric (dashed). The symmetric one
is much similar to the tangent (solid straight line).

whereas, for h < 0:

1 1
f ( x .01) f ( x )
= 1.99 2
= 0.251256+
.01 .01
which are already different at the second decimal digit. Which of
the two is to be preferred? There is no abstract principle leading to
choosing one or the other.
Whenever there are two values which approximate a third one
and there is no good reason for preferring one to the other, it is rea-
sonable to expect that the mean value should be a better approxima-
tion than either of the two. In this case:
1 f ( x + h) f ( x ) f ( x h) f ( x )
 
+ = 0.2500062+
2 h h
which is an approximation to the real value 0.25 to five significant
figures.
This is, actually, the correct way to state the approximation prob-
lem to numerical differentiation: to use the symmetric difference
around the point and divide by the double of the width of the inter-
val. This is exactly the same as taking the mean value of the right
and left hand derivative.
T HEOREM 8. The naive approximation to the derivative has order 1
precision, whereas the symmetric formula has order 2.
 2. NUMERICAL INTEGRATIONQUADRATURE FORMULAS 71

P ROOF. Let f be a three times differentiable function on [ x

h, x + h]. Using the Taylor polynomial of degree 1, one has
f 00 ( ) 2
f ( x + h) = f ( x ) + f 0 ( x )h + h ,
2
for some in the interval, so that, solving f 0 ( x ), one gets
f ( x + h) f ( x ) f 00 ( )
f 0 (x) = h,
h 2
which is exactly the meaning of having order 1. Notice that the
rightmost term cannot be eliminated.
However, for the symmetric formula one can use the Taylor poly-
nomial of degree 2, twice:
f 00 ( x ) 2 f 3) ( ) 3
f ( x + h) = f ( x ) + f 0 ( x )h + h + h ,
2 6
f 00 ( x ) 2 f 3) ( ) 3
f ( x h) = f ( x ) f 0 ( x )h + h h
2 6
for some [ x h, x + h] and [ x h, x + h]. Subtracting:
f ( x + h) f ( x h) = 2 f 0 ( x )h + K (, )h3
where K is a sum of the degree 3 coefficients in the previous equa-
tion, so that
f ( x + h) f ( x h) K (, ) 2
f 0 (x) = + h ,
2h 2
which is the meaning of having order 2. 

2. Numerical IntegrationQuadrature Formulas

Numerical integration, being a problem in which errors accumu-
late (a global problem) is more stable than differentiation, surpris-
ing as it may seem (even though taking derivatives is much simpler,
symbolically, than integration).
In general, one wishes to state an abstract formula for perform-
ing numerical integration: an algebraic expression such that, given
a function f : [ a, b] R whose integral is to be computed, one can
substitute f in the expression and get a value the approximated in-
tegral.
D EFINITION 16. A simple quadrature formula for an interval [ a, b] is
a family of points x1 < x2 < < xn+1 [ a, b] and coefficients (also
72 5. NUMERICAL DIFFERENTIATION AND INTEGRATION

f (x)
2

0
2 3 3.5 4 5

F IGURE 2. Midpoint rule: the area under f is approxi-

mated by the rectangle.

called weights) a1 , . . . , an+1 . The approximate integral of a function f on

[ a, b] using that formula is the expression
a 1 f ( x 1 ) + a 2 f ( x 2 ) + + a n +1 f ( x n +1 ).
That is, a quadrature formula is no more than a way to approx-
imate an integral using intermediate points and weights. The for-
mula is closed if x1 = a and xn+1 = b; if neither a nor b are part of the
intermediate points, then it is open.
If the xi are evenly spaced, then the formula is a Newton-Coates
formula. These are the only ones we shall explain in this course.
Obviously, one wants the formulas which best approach the inte-
grals of known functions.
D EFINITION 17. A quadrature formula with coefficients a1 , . . . , an+1
is of order m if for any polynomial of degree m, one has
Z b
P( x ) dx = a1 P( x1 ) + a2 P( x2 ) + + an+1 P( xn+1 ).
a

That is, if the formula is exact for polynomials of degree m.

The basic quadrature formulas are: the midpoint formula (open,
n = 0), the trapezoidal rule (closed, two points, n = 1) and Simp-
sons formula (closed, three points, n = 2).
We first show the simple versions and then generalize them to
their composite versions.
 2. NUMERICAL INTEGRATIONQUADRATURE FORMULAS 73

2.1. The Midpoint Rule. A coarse but quite natural way to ap-
proximate an integral is to multiply the value of the function at the
midpoint by the width of the interval. This is the midpoint formula:
D EFINITION 18. The midpoint quadrature formula corresponds to
x1 = ( a + b)/2 and a1 = (b a). That is, the approximation
Z b  
a+b
f ( x ) dx ' (b a) f .
a 2
given by the area of the rectangle having a horizontal side at f ((b
a)/2).
One checks easily that the midpoint rule is of order 1: it is exact
for linear polynomials but not for quadratic ones.

2.2. The Trapezoidal Rule. The next natural approximation (which

is not necessarily better) is to use two points. As there are two al-
ready given (a and b), the naive idea is to use them.
Given a and b, one has the values of f at them. One could in-
terpolate f linearly (using a line) and approximate the value of the
integral with the area under the line. Or one could use the mean
value between two rectangles (one with height f ( a) and the other
with height f (b)). The fact is that both methods give the same value.
D EFINITION 19. The trapezoidal rule for [ a, b] corresponds to x1 =
a, x2 = b and weights a1 = a2 = (b a)/2. That is, the approxima-
tion
Z b
ba
f ( x ) dx ' ( f ( a) + f (b))
a 2
for the integral of f using the trapeze with a side on [ a, b], joining
( a, f ( a)) with (b, f (b)) and parallel to OY.
Even though it uses one point more than the midpoint rule, the
trapezoidal rule is also of order 1.

2.3. Simpsons Rule. The next natural step involves 3 points in-
stead of 2 and using a parabola instead of a straight line. This method
is remarkably precise (it has order 3) and is widely used. It is called
Simpsons Rule.
D EFINITION 20. Simpsons rule is the quadrature formula corre-
sponding to the nodes x1 = a, x2 = ( a + b)/2 and x3 = b, and
the weights, corresponding to the correct interpolation of a degree
74 5. NUMERICAL DIFFERENTIATION AND INTEGRATION

f (x)
2

0
2 3 4 5

F IGURE 3. trapezoidal rule: approximate the area un-

der f by that of the trapeze.

4( b a )
2 polynomial. That is1, a1 = b6 a , a2 = 6 and a3 = b6 a . Hence, it
is the approximation of the integral of f by
Z b
ba
   
a+b
f ( x ) dx ' f ( a) + 4 f + f (b) ,
a 6 2
which is a weighted mean of the areas of three intermediate rectan-
gles. This rule must be memorized: one sixth of the length times the
values of the function at the endpoints and midpoint with weights
1, 4, 1.
The remarkable property of Simpsons rule is that it has order 3:
even though a parabola is used, the rule integrates correctly poly-
nomials of degree up to 3. Notice that one does not need to know the
equation of the parabola: the values of f and the weights (1/6, 4/6, 1/6)
are enough.
2.4. Composite Formulas. Composite quadrature formulas are
no more than applying the simple ones in subintervals. For ex-
ample, instead of using the trapezoidal rule for approximating an
integral, like
Z b
ba
f ( x ) dx ' ( f ( a) + f (b))
a 2
one subdivides [ a, b] into subintervals and performs the approxima-
tion on each of these.
1This is an easy computation which the reader should perform by himself.
 2. NUMERICAL INTEGRATIONQUADRATURE FORMULAS 75

f (x)
2

0
2 3 3.5 4 5

F IGURE 4. Simpsons rule: approximating the area un-

der f by the parabola passing through the endpoints
and the midpoint (black).

f (x)
2

0
2 2.5 3 3.5 4 4.5 5

F IGURE 5. Composite trapezoidal rule: just repeat the

simple one on each subinterval.

One might do this mechanically, one subinterval after another.

The fact is that, due to their additive nature, it is always simpler to
apply the general formula than to apply the simple one step by step,
for closed Newton-Coates formulas (not for open ones, for which
the computations are the same one way or the other).
2.4.1. Composite Trapezoidal Rule. If there are two consecutive in-
tervals [ a, b] and [b, c] of equal length (that is, b a = c b) and the
76 5. NUMERICAL DIFFERENTIATION AND INTEGRATION

trapezoidal rule is applied on both in order to approximate the inte-

gral of f on [ a, c], one gets:
Z c
ba cb
f ( x ) dx =
( f (b) + f ( a)) + ( f (c) + f (b)) =
a 2 2
h
( f ( a) + 2 f (b) + f (c)) ,
2
where h = b a is the width of each subinterval, either [ a, b] or [b, c]:
the formula just a weighted mean of the values of f at the endpoints
and the midpoint. In the general case, when there are more than two
subintervals, one gets an analogous formula:
D EFINITION 21 (Composite trapezoidal rule). Given an interval
[ a, b] and n + 1 nodes, the composite trapezoidal rule for [ a, b] with n + 1
nodes is given by the nodes x0 = a, x1 = a + h, . . . , xn = b, the value
h = (b a)/n and the approximation
Z c
h
f ( x ) dx '
( f ( a) + 2 f ( x2 ) + 2 f ( x3 ) + + 2 f ( xn ) + f (b)) .
a 2
That is, half the width of the subintervals times the sum of the values
at the endpoints and the double of the values at the interior points.
2.4.2. Composite Simpsons Rule. In a similar way, the composite
Simpsons rule is just the application of Simpsons rule in a sequence
of subintervals (which, for the Newton-Coates formula, have all the
same width). As before, as the left endpoint of each subinterval is
the right endpoint for the next one, the whole composite formula is
somewhat simpler to implement than the mere addition of the sim-
ple formula on each subinterval.
D EFINITION 22 (Composite Simpsons rule). Given an interval
[ a, b], divided into n subintervals (so, given 2n + 1 evenly distributed
nodes), Simpsons composite rule for [ a, b] with 2n + 1 nodes is given by
the nodes x0 = a, x1 = a + h, . . . , x2n = b, (where h = (b a)/(2n))
and the approximation
Z c
ba
f ( x ) dx ' ( f ( a ) + 4 f ( x2 ) + 2 f ( x3 ) + 4 f ( x3 ) + . . .
a 6n
+ 2 f ( x2n1 ) + 4 f ( x2n ) + f (b)).
This means that the composite Simpsons rule is the same as the
simple version taking into account that one multiplies all by h/6,
where h is the width of each subinterval and the coefficients are 1
for the two endpoints, 4 for the inner midpoints and 2 for the inner
endpoints.
 2. NUMERICAL INTEGRATIONQUADRATURE FORMULAS 77

80 f (x)

60

40

20

0
2 2.75 3.5 4.25 5

F IGURE 6. Composite Simpsons rule: conceptually, it

is just the repetition of the simple rule on each subin-
terval.
 CHAPTER 6

Differential Equations

There is little doubt that the main application of numerical meth-

ods is for integrating differential equations (which is the technical term
for solving them numerically).

1. Introduction
A differential equation is a special kind of equation: one in which
one of the unknowns is a function. We have already studied some:
any integral is a differential equation (it is the simplest kind). For
example,
y0 = x
is an equation in which one seeks a function y( x ) whose derivative
is x. It is well-known that the solution is not unique: there is an
integration constant and the general solution is written as
x2
y= + C.
2
This integration constant can be better understood graphically.
When computing the primitive of a function, one is trying to find
a function whose derivative is known. A function can be thought
of as a graph on the X, Y plane. The integration constant specifies
at which height the graph is. This does not change the derivative,
obviously. On the other hand, if one is given the specific problem
y0 = x, y(3) = 28,
then one is trying to find a function whose derivative is x, with a con-
dition at a point: that its value at 3 be 28. Once this value is fixed, there
is only one graph having that shape and passing through (3, 28). The con-
dition y(3) = 28 is called an initial condition. One imposes that the
graph of f passes through a point and then there is only one f solv-
ing the integration problem, which means there is only one suitable
constant C. As a matter of fact, C can be computed by substitution:
28 = 32 + C C = 19.
79
80 6. DIFFERENTIAL EQUATIONS

The same idea gives rise to the term initial condition for a differential
equation.
Consider the equation
y0 = y
(whose general solution should be known). This equation means:
find a function y( x ) whose derivative is equal to the same function
y( x ) at every point. One tends to think of the solution as y( x ) = e x
but. . . is this the only possible solution? A geometrical approach may
be more useful; the equation means the function y( x ) whose deriv-
ative is equal to the height y( x ) at each point. From this point of
view it seems obvious that there must be more than one solution to
the problem: at each point one should be able to draw the corre-
sponding tangent, move a little to the right and do the same. There
is nothing special on the points ( x, e x ) for them to be the only solu-
tion to the problem. Certainly, the general solution to the equation
is
y( x ) = Ce x ,
where C is an integration constant. If one also specifies an initial con-
dition, say y( x0 ) = y0 , then necessarily

y0 = Ce x0

so that C = y0 /e x0 is the solution to the initial value problem (notice

that the denominator is not 0).
This way, in order to find a unique solution to a differential equa-
tion, one needs to specify at least one initial condition. As a matter
of fact, the number of these must be the same as the order of the
equation.
Consider
y00 = y,
whose solutions should also be known: the functions of the form
y( x ) = a sin( x ) + b cos( x ), for two constants a, b R. In order for
the solution to be unique, two initial conditions must be specified.
They are usually stated as the value of y at some point, and the value
of the derivative at that same place. For example, y(0) = 1 and
y0 (0) = 1. In this case, the solution is y( x ) = sin( x ) + cos( x ).
This chapter deals with approximate solutions to differential equa-
tions. Specifically, ordinary differential equations (i.e. with functions
y( x ) of a single variable x).
 2. THE BASICS 81

2. The Basics
The first definition is that of differential equation
D EFINITION 23. An ordinary differential equation is an equality A =
B in which the only unknown is a function of one variable whose de-
rivative of some order appears explicitly.
The adjective ordinary is the condition on the unknown of being
a function of a single variable (there are no partial derivatives).
E XAMPLE 18. We have shown some examples above. Differential
equations can get many forms:
y0 = sin( x )
xy = y0 1
(y0 )2 2y00 + x2 y = 0
y0
xy = cos(y)
y
etc.
In this chapter, the unknown in the equation will always be de-
noted with the letter y. The variable on which it depends will usually
be either x or t.
D EFINITION 24. A differential equation is of order n if n is the
highest order derivative of y appearing in it.
The specific kind of equations we shall study in this chapter are
the solved ones (which does not mean that they are already solved,
but that they are written in a specific way):
y0 = f ( x, y).
D EFINITION 25. An initial value problem is a differential equation
together with an initial condition of the form y( x0 ) = y0 , where
x0 , y0 R.
D EFINITION 26. The general solution to a differential equation E
is a family of functions f ( x, c), where c is one (or several) constants
such that:
Any solution of E has the form f ( x, c) for some c.
Any expression f ( x, c) is a solution of E,
except for possibly a finite number of values of c.
82 6. DIFFERENTIAL EQUATIONS

If integrating functions of a real variable is already a complex

problem, the exact integration a differential equation is, in general,
impossible. That is, the explicit computation of the symbolic solution
to a differential equation is a problem which is usually not tackled.
What one seeks is to know an approximate solution and a reasonably
good bound for the error incurred when using that approximation
instead of the true solution.
Notice that, in reality, most of the numbers appearing in the equa-
tion describing a problem will already be inexact, so trying to get an
exact solution is already a mistake.

3. Discretization
We shall assume a two-variable function f ( x, y) is given, which
is defined on a region x [ x0 , xn ], y [ a, b], and which satisfies the
following condition (which the reader is encouraged to forget):
D EFINITION 27. A function f ( x, y) defined on a set X R2 satis-
fies Lipschitzs condition if there exists K > 0 such that
| f ( x1 ) f ( x2 )| K | x1 x2 |
for any x1 , x2 , X, where | | denotes the absolute value of a number.
This is a kind of strong continuity condition (i.e. it is easier for
a function to be continuous than to be Lipschitz). What matters is
that this condition has a very important consequence for differential
equations: it guarantees the uniqueness of the solution. Let X be
a set [ x0 , xn ] [ a, b] (a strip, or a rectangle) and f ( x, y) : X R a
function on X which satisfies Lipschitzs condition. Then
T HEOREM 9 (Cauchy-Kovalevsky). Under the conditions above, any
differential equation y0 = f ( x, y) with an initial condition y( x0 ) = y0 for
y0 ( a, b) has a unique solution y = y( x ) defined on [ x0 , x0 + t] for some
t R greater than 0.
Lipschitzs condition is not so strange. As a matter of fact, poly-
nomials and all the analytic functions (exponential, logarithms,
trigonometric functions, etc. . . ) and their inverses (where they are
defined and continuous) satisfy it. An example which does not is
f ( x ) = x on an interval containing 0, because f at that point has
a vertical tangent line. The reader should not worry about this
condition (only if he sees a derivative becoming infinity or a point of
discontinuity, but we shall not discuss them in these notes). We give
just an example:
 3. DISCRETIZATION 83

E XAMPLE 19 (Bourbaki Functions of a Real Variable, Ch. 4, 1).

0
p
The differential equation y = 2 |y| with initial condition y(0) = 0
has an infinite number of solutions. For example, any of the follow-
ing, for a, b > 0:
(1) y(t) = 0 for any interval (b, a),
(2) y(t) = (t + b)2 for t b,
(3) y(t) = (t a)2 for t a
is a solution of that
p equation. This is because the function on the
right hand side, |y|, is not Lipschitz near y = 0. (The reader is
suggested to verify both assertions).

In summary, any normal initial value problem has a unique

solution. What is difficult is finding this.
And what about an approximation?

3.1. The derivative as an arrow. One is usually told that the de-
rivative of a function of a real variable is the slope of its graph at the
corresponding point. However, a more useful idea for the present
chapter is to think of it as the Y coordinate of the velocity vector of the
graph.
When plotting a function, one should imagine that one is draw-
ing a curve with constant horizontal speed (because the OX axis is
homogeneous, one goes from left to right at uniform speed). This
way, the graph of f ( x ) is actually the plane curve ( x, f ( x )). Its tan-
gent vector at any point is (1, f 0 ( x )): the derivative f 0 ( x ) of f ( x ) is
the vertical component of this vector.
From this point of view, a differential equation in solved form
0
y = f ( x, y) can be interpreted as the statement find a curve ( x, y( x ))
such that the velocity vector at each point is (1, f ( x, y)). One can
then draw the family of velocity vectors on the plane ( x, y) given
by (1, f ( x, y)) (the function f ( x, y) is known, remember). This visu-
alization, like in Figure 1, already gives an idea of the shape of the
solution.
Given the arrows the vectors (1, f ( x, y)) on a plane, draw-
ing a curve whose tangents are those arrows should not be too hard.
Even more, if what one needs is just an approximation, instead of
drawing a curve, one could draw little segments going in the di-
rection of the arrows. If these segments have a very small x coor-
dinate, one reckons that an approximation to the solution will be
obtained.
This is exactly Eulers idea.
84 6. DIFFERENTIAL EQUATIONS

F IGURE 1. Arrows representing vectors of a solved

differential equation y0 = f ( x, y). Notice how the hor-
izontal component is constant while the vertical one
is not. Each arrow corresponds to a vector (1, f ( x, y))
where f ( x, y) ' x cos(y).

Given a plane filled with arrows indicating the velocity vectors

at each point, the simplest idea for drawing the corresponding curve
that:
Passes through a specified point (initial condition y( x0 ) =
y0 )
Whose yvelocity component is f ( x, y) at each point. . .
consists in discretizing the x coordinates. Starting at x0 , one as-
sumes that OX is quantized in intervals of width h (constant, by
now) small enough. Now, instead of drawing a smooth curve, one
approximates it by small steps (of width h) on the x variable.
As the solution has to verify y( x0 ) = y0 , one needs only
Draw the point ( x0 , y0 ) (the initial condition).
Compute f ( x0 , y0 ), the vertical value of the velocity vector of
the solution at the initial point.
As the x coordinate is quantized, the next point of the ap-
proximation will have x coordinate equal to x0 + h.
As the velocity at ( x0 , y0 ) is (1, f ( x0 , y0 )), the simplest ap-
proximation to the displacement of the curve in an interval of
width h on the x coordinate is (h, h f ( x0 , y0 )). Let x1 = x0 + h
and y1 = y0 + h f ( x0 , y0 ).
Draw the segment from ( x0 , y0 ) to ( x1 , y1 ): this is the first
approximate segment of the solution.
 4. SOURCES OF ERROR: TRUNCATION AND ROUNDING 85

At this point one is in the same situation as at the beginning

but with ( x1 , y1 ) instead of ( x0 , y0 ). Repeat.
And so on as many times as one desires. The above is Eulers algo-
rithm for numerical integration of differential equations.

4. Sources of error: truncation and rounding

It is obvious that the solution to an equation y0 = f ( x, y) will
never (or practically so) be a line composed of straight segments.
When using Eulers method, there is an intrinsic error which can
be analyzed, for example, with Taylors formula. Assume f ( x, y)
admits a sufficient number of partial derivatives. Then y( x ) will be
differentiable also and
h2
y( x0 + h) = y( x0 ) + hy0 ( x0 ) + y00 ( )
2
for some [ x0 , x0 + h]. What Eulers method does is to remove the
last term, so that the error incurred is exactly that: a term of order 2
in h in the first iteration. This error, which arises from truncating the
Taylor expansion is called truncation error in this setting.
In general, one assumes that the interval along which the inte-
gration is performed has width of magnitude h1 . Usually, starting
from an interval [ a, b] one posits a number of subintervals n (or inter-
mediate points, n 1) and takes x0 = a, xn = b and h = (b a)/n, so
that going from x0 to xn requires n iterations and the global truncation
error has order h1O(h2 ) ' O(h).
However, truncation is not the only source of error. Floating-
point operations incur always rounding errors.
4.1. Details on the truncation and rounding errors. Specifically,
if using IEEE-754, one considers that the smallest significant quantity
(what is called, the epsilon) is e = 252 ' 2.2204 1016 . Thus,
the rounding error in each operation is less than e. If operation
means a step in the Euler algorithm (which may or may not be the
case), then each step from xi to xi+1 incurs an error of at most e and
hence, the global rounding error is bounded by e/h. This is very tricky
because it implies that the smaller the interval h, the larger the rounding
error incurred, which is counterintutive. So, making h smaller does
not necessarily improve the accuracy of the method!
Summing up, the addition of the truncation and rounding errors
can be approximated as
e
E(h) ' + h,
h
86 6. DIFFERENTIAL EQUATIONS

which grows both when h becomes larger and when it gets smaller.
The minimum E(h) is given by h ' e: which means that there is
no point in using intervals of width less than e in the Euler method
(actually, ever). Taking smaller intervals may perfectly lead to huge
errors.
This explanation about truncation and rounding errors is rele-
vant for any method, not just Eulers. One needs to bound both for
every method and know how to choose the best h. There are even
problems for which a single h is not useful and it has to be modified
during the execution of the algorithm. We shall not deal with these
problems here (the interested reader should look for the term stiff
differential equation).

5. Quadratures and Integration

Assume that in the differential equation y0 = f ( x, y), the func-
tion f depends only on the variable x. Then, the equation would be
written
y 0 = f ( x ),
expression which means y is the function whose derivative is f ( x ). That
is, the problem is that of computing a primitive. We have already
dealt with it in Chapter 5 but it is inherently related to what we are
doing for differential equations.
When f ( x, y) depends also on y, the relation between the equa-
tion and a primitive is harder to perceive but we know (by the Fun-
damental Theorem of Calculus) that, if y( x ) is the solution to the
initial value problem y0 = f ( x, y) with y( x0 ) = y0 , then, integrating
both sides, one gets
Z x
y( x ) = f (t, y(t)) dt + y0 ,
x0

which is not a primitive but looks like it. In this case, there is no way to
approximate the integral using the values of f at intermediate points
because one does not know the value of y(t). But one can take a similar
approach.

6. Eulers Method: Integrate Using the Left Endpoint

One can state Eulers method as in Algorithm 10. If one reads
carefully, the gist of each step is to approximate each value yi+1 as the
previous one yi plus f evaluated at ( xi , yi ) times the interval width.
 7. MODIFIED EULER: THE MIDPOINT RULE 87

That is:
Z x i +1
f (t, y(t)) dt ' ( xi+1 xi ) f ( xi , yi ) = h f ( xi , yi ).
xi
If f ( x, y) were independent of y, then one would be performing the
following approximation (for an interval [ a, b]):
Z b
f (t) dt = (b a) f ( a),
a
which, for lack of a better name, could be called the left endpoint rule:
the integral is approximated by the area of the rectangle of height
f ( a) and width (b a).

Algorithm 10 Eulers algorithm assuming exact arithmetic.

Input: A function f ( x, y), an initial condition ( x0 , y0 ), an interval
[ a, b] = [ x0 , xn ] and a step h = ( xn x0 )/n
Output: A family of values y0 , y1 , . . . , yn (which approximate the
solution to y0 = f ( x, y) on the net x0 , . . . , xn )
? S TART
i0
while i n do
y i +1 y i + h f ( x i , y i )
i i+1
end while
return (y0 , . . . , yn )

One might try (as an exercise) to solve the problem using the
right endpoint: this gives rise to what are called the implicit meth-
ods which we shall not study (but which usually perform better than
the explicit ones we are going to explain).
7. Modified Euler: the Midpoint Rule
Instead of using the left endpoint of [ xi , xi+1 ] for integrating and
computing yi+1 , one might use (and this would be better, as the
reader should verify) the midpoint rule somehow. As there is no
way to know the intermediate values of y(t) further than x0 , some
kind of guesswork has to be done. The method goes as follows:
Use a point near Pi = ( xi , yi ) whose x coordinate is the
midpoint of [ xi , xi+1 ].
For lack of a better point, the first approximation is done us-
ing Eulers algorithm and one takes a point Qi = [ xi+1 , yi +
h f ( xi , yi )].
88 6. DIFFERENTIAL EQUATIONS

Compute the midpoint of the segment Pi Qi , which is ( xi +

h/2, yi + h/2 f ( xi , yi )). Let k be its ycoordinate.
Use the value of f at that point in order to compute yi+1 : this
gives the formula yi+1 = yi + h f ( xi + h/2, k).
If, as above, f ( x, y) did not depend on y, one verifies easily that
the corresponding approximation for the integral is
Z b
a+b
f ( x ) dx ' (b a) f (
),
a 2
which is exactly the midpoint rule of numerical integration.
This method is called modified Eulers method and its order is 2, the
same as Eulers, which implies that the accrued error at xn is O(h).
It is described formally in Algorithm 11.

yi + k1
z2 + k 2

z2
y i +1

yi

xi xi + h x i +1
2

F IGURE 2. Modified Eulers Algorithm. Instead of

using the vector (h, h f ( xi , yi )) (dashed), one sums at
( xi , yi ) the dotted vector, which is the tangent vector at
the midpoint.

As shown in Figure 2, this method consist in first using Eulers in

order to get an initial guess, then computing the value of f ( x, y) at
the midpoint between ( x0 , y0 ) and the guess and using this value of
f as the approximate slope of the solution at ( x0 , y0 ). There is still an
error but, as the information is gathered a bit to the right, it is less
than the one of Eulers method, analogue to the trapezoidal rule.
The next idea is, instead of using a single vector, compute two of
them and use a mean value: as a matter of fact, the mean between
the vector at the origin and the vector at the end of Eulers method.
 8. HEUNS METHOD: THE TRAPEZOIDAL RULE 89

Algorithm 11 Modified Eulers algorithm, assuming exact arith-

metic.
Input: A function f ( x, y), an initial condition ( x0 , y0 ), an interval
[ x0 , xn ] and a step h = ( xn x0 )/n
Output: A family of values y0 , y1 , . . . , yn (which approximate the
solution of y0 = f ( x, y) on the net x0 , . . . , xn )
? S TART
i0
while i n do
k 1 f ( xi , yi )
z2 yi + 2h k1
k2 f ( xi + 2h , z2 )
yi+1 yi + hk2
i i+1
end while
return (y0 , . . . , yn )

8. Heuns Method: the Trapezoidal Rule

Instead of using the midpoint of the segment of Eulers method,
one can take the vector corresponding to Eulers method and also
the vector at the endpoint of Eulers method and compute the mean
of both vectors and use this mean for approximating. This way, one
is using the information at the point and some information later
on. This improves Eulers method and is called accordingly improved
Eulers method or Heuns method. It is described in Algorithm 12.
At each step one has to perform the following operations:
Compute k1 = f ( xi , yi ).
Compute z2 = y j + hk1 . This is the coordinate yi+1 in Eulers
method.
Compute k2 = f ( xi+1 , z2 ). This would be the slope at ( xi+1 , yi+1 )
with Eulers method.
Compute the mean of k1 and k2 : k1 +2 k2 and use this value as
slope. That is, set yi+1 = yi + 2h (k1 + k2 ).
Figure 3 shows a graphical representation of Heuns method.
Figures 4 and 5 show the approximate solutions to the equations
y0 = y and y0 = y + cos( x ), respectively, using the methods ex-
plained in the text, together with the exact solution.
90 6. DIFFERENTIAL EQUATIONS

yi + hk1

y i +1

yi + hk2

yi
xi x i +1 x i +1 + h

F IGURE 3. Heuns method. At ( xi , yi ) one uses the

mean displacement (solid) between the vectors at
( xi , yi ) (dashed) and at the next point ( xi+1 , yi + hk1 )
in Eulers method (dotted).
Algorithm 12 Heuns algorithm assuming exact arithmetic.
Input: A function f ( x, y), an initial condition ( x0 , y0 ), an interval
[ x0 , xn ] and a step h = ( xn x0 )/n
Output: A family of points y0 , y1 , . . . , yn ( which approximate the
solution to y0 = f ( x, y) on the net x0 , . . . , xn )
? S TART
i0
while i n do
k 1 f ( xi , yi )
z2 yi + hk1
k2 f ( xi + h, z2 )
yi+1 yi + 2h (k1 + k2 )
i i+1
end while
return (y0 , . . . , yn )
 8. HEUNS METHOD: THE TRAPEZOIDAL RULE 91

5 Euler
Modified
4 Solution

2 1 0 1 2

F IGURE 4. Comparison between Eulers and Modified

Eulers methods and the true solution to the ODE y0 =
y for y(2) = 101 .

Euler
Modified
1 Heun
Solution
0.5

0.5

1.5
0 1 2 3 4 5 6

F IGURE 5. Comparison between Eulers, Modified Eu-

lers and Heuns methods and the true solution to the
ODE y0 = y + cos( x ) for y(0) = 1.
 CHAPTER 7

Multivariate and higher order ODEs

In Chapter 6 we dealt with the simplest case of Ordinary Differ-

ential Equations: those of a single dependent variable. As a matter of
fact, in most cases, the number of dependent variables is more than
one and, as we shall see later, any ODE of order greater than one can
be transformed into a system of equations of order one with more
than one dependent variable. This chapter gives a summary intro-
duction to the numerical solution of these problems. The expectation
is that the student gets acquainted with problems of order greater
than one and understands their transformation into order one equa-
tions.

1. A two-variable example
Let us consider a problem in two variables. Usually and we
shall follow the convention in this chapter the independent vari-
able is called t, reflecting the common trait of ODEs of being equa-
tions which define a motion in terms of time.
Consider a dimensionless body B a mass point moving with
velocity v in the plane. Hence, if the coordinates of B at time t are
B(t) = ( x (t), y(t)), we shall denote v = ( x (t), y (t)). Assume we
know, for whatever reason, that the velocity vector satisfies some
condition F which depends on the position of B in the plane. This
can be expressed as v = F ( x, y). The function F is then a vector
function of two variables, F ( x, y) = ( F1 ( x, y), F2 ( x, y)), and we can
write the condition on v as:

x (t) = F1 ( x (t), y(t))
(21)
y (t) = F2 ( x (t), y(t))
which means, exactly, that the x component of the velocity depends
on the position at time t as the value of F1 at the point and that the
ycomponent depends as the value of F2 . If we want, writing B(t) =
( x (t), y(t)) as above, expression 21 can be written, in a compact way
as
v(t) = F ( B(t))
93
94 7. MULTIVARIATE AND HIGHER ORDER ODES

which reflects the idea that what we are doing is just the same as in
Chapter 6, only with more coordinates. This is something that has to be
clear from the beginning: the only added difficulty is the number of
computations to be carried out.
Notice that in the example above, F depends only on x and y
not on t. However, it might as well depend on t (because the be-
haviour of the system may depend on time), so that in general, we
should write
v(t) = F (t, B(t)).
Just for completeness, if F does not depend on t, the system is called
autonomous, whereas if it does depend on t, it is called autonomous.

1.1. A specific example. Consider, to be more precise, the same

autonomous problem as above but with F ( x, y) = (y, x ). This
gives the following equations
x (t) = y(t)

(22)
y (t) = x (t).
It is clear that, in order to have an initial value problem, we need an
initial condition. Because the equation is of order one, we need just
the starting point of the motion, that is: two cordinates. Let us take
x (0) = 1 and y(0) = 0 as the initial position for time t = 0. The
initial value problem is, then
x (t) = y(t)

(23) with ( x (0), y(0)) = (1, 0)
y (t) = x (t).
which, after some scaling (so that the plot looks more or less nice) de-
scribes the vector field depicted in Figure 1. It may be easy to guess
that a body whose velocity is described by the arrows in the dia-
gram follows a circular motion. This is what we are going to prove,
actually.
The initial value problem (23) means, from a symbolic point of
view, that the functions x (t) and y(t) satisfy the following condi-
tions:
First, the derivative of x (t) with respect to t is y(t).
Then, the derivative of y(t) with respect to t is x (t).
And finally, the initial values are 1 and 0, for t = 0, respec-
tively.
The first two conditions imply that x (t) = x (t) and y (t) = y(t).
This leads one to think of trigonometric functions and, actually, it is
 1. A TWO-VARIABLE EXAMPLE 95

F IGURE 1. Vector field representing (after scaling) the

ODE of Equation (23). The initial condition (with the
vector at that point) is marked with a black dot.

easy to check that x (t) = cos(t), y(t) = sin(t) verify the conditions
above. That is, the solution to the initial value problem (23) is

x (t) = cos(t)
(24)
y(t) = sin(t)
which, as the reader will have alredy realized, is a circular trajectory
around the origin, passing through (1, 0) at time t = 0. Any other
initial condition ( x (0), y(0)) = ( a, b) gives rise to a circular trajectory
starting at ( a, b).
However, our purpose, as we have stated repeatedly in these
notes, is not to find a symbolic solution to any problem, but to ap-
proximate it using the tools at hand. In this specific case, which is of
dimension two, one can easily describe the generalization of Eulers
method of Chapter 6 to the problem under study. Let us fix a dis-
cretization of t, say in steps of size h. Then, a rough approximation
to a solution would be:
(1) We start with x0 = 1, y0 = 0.
(2) At that point, the differential equation means that the veloc-
ity v(t) = ( x (t), y (t)) is
x (0) = 0, y (0) = 1.
(3) Because t moves in steps of size h, the trajectory ( x (t), y(t))
can only be approximated by moving as much as the vector
at t = 0 says multiplied by the timespan h, hence the next
96 7. MULTIVARIATE AND HIGHER ORDER ODES

F IGURE 2. Approximation to the solution of Problem

(23) using Eulers method. Notice the (obvious) error.

approximate position, ( x1 , y1 ) is given by

( x1 , y1 ) = ( x0 , y0 ) + h ( x (0), y (0)) = (1, 0) + (h 0, h 1) = (0, h).
(4) Now, at the point ( x1 , y1 ) = (0, h), we carry out the anal-
ogous computations, taking into account that at this point,
v = ( cos(h), sin(0)). . .
If we follow the steps just described, we get a picture like Figure
2, in which the step is h = 0.1 and the error incurred is easily noticed.
The fact that this method, for this example, gives what looks like
a bad approximation needs not make one reject it straightway. There
is a clear reason for this, which we shall not delve into but which has
to deal with the solutions being always convex i.e. the trajecto-
ries describing the true solutions are always curved in the same way.
This makes the Euler method accumulate the errors always, instead
of having some positive errors and some negative ones.

2. Multivariate equations: Euler and Heuns methods

The example with two coordinates above is easily generalized to
n coordinates. Consider an initial value problem


x 1 = f 1 (t, x1 , . . . , xn ), x1 (0) = a1

x = f (t, x , . . . , x ), x (0) = a
2 2 1 n 2 2
(25) .

.
.

x n = f n (t, x1 , . . . , xn ), xn (0) = an

 2. MULTIVARIATE EQUATIONS: EULER AND HEUNS METHODS 97

where f 1 (t, x1 , . . . , xn ), . . . , f n (t, x1 , . . . , xn ) are continuous functions

of n + 1 variables time, so to say, and position. We do not
assume the system is autonomous. Our aim is to compute an ap-
proximate solution using increments of the independent variable t
of size h. The generalization of Eulers method is just a copy of what
was described above:
(1) Start with i = 0 and let x0,1 = a1 , . . . , x0,n = an . Notice that
the first subindex indicates the number of the iteration and
the second is the coordinate.
(2) Set ti = h i.
(3) Compute the coordinates of the derivative: that is, compute
v1 = f 1 (ti , xi,1 (ti ), . . . , xi,n (ti )), . . . , vn = f n (ti , xi,1 (ti ), . . . , xi,n (ti )).
This is much simpler than it seems.
(4) Compute the next point: xi+1,1 = xi,1 + h v1 , xi,2 = xi,2 + h
v2 , . . . , xi+1,n = xi,n + h vn .
(5) Increase i = i + 1 and goto step 2 until one stops.
These steps are formally stated in Algorithm 13.

Algorithm 13 Eulers algorithm for n coordinates. Notice that bold-

face elements denote vectors.
Input: n functions of n + 1 variables:
f 1 (t, x1 , . . . , xn ), . . . , f n (t, x1 , . . . , xn ), an initial condition
( a1 , a2 , . . . , an ), an interval [ A, B] and a step h = ( B A)/N
Output: A family of vector values x0 , . . . , xN (which approxi-
mate the solution to the corresponding initial value problem for
t [ A, B])
? S TART
i 0, t0 A, x0 ( a1 , . . . , an )
while i < N do
v ( f 1 (ti , xi ), . . . , f n (ti , xi ))
x i +1 x i + h v i
i i+1
end while
return x0 , . . . , x N

As we did in Chapter 6, we could state a modified version of Eu-

lers method but we prefer to go straightway for Heuns. It is pre-
cisely stated in Algorithm 14.
98 7. MULTIVARIATE AND HIGHER ORDER ODES

Algorithm 14 Heuns algorithm for n coordinates. Notice that bold-

face elements denote vectors.
Input: n functions of n variables:
f 1 (t, x1 , . . . , xn ), . . . , f n (t, x1 , . . . , xn ), an initial condition
( a1 , a2 , . . . , an ), an interval [ A, B] and a step h = ( B A)/N
Output: A family of vector values x0 , . . . , xN (which approxi-
mate the solution to the corresponding initial value problem for
t [ A, B])
? S TART
i 0, t0 A, x0 ( a1 , . . . , an )
while i < N do
v ( f 1 (ti , xi ), . . . , f n (ti , xi ))
z xi + h vi
w ( f 1 (ti , zi ), . . . , f n (ti , zi ))
xi+1 xi + 2h (v + w)
i i+1
end while
return x0 , . . . , x N

In Figure 3 a plot of the approximate solution to the initial value

problem of Equation 23 is shown. Notice that the approximate tra-
jectory is on the printed page indistinguishable from a true cir-
cle (we have plotted more than a single loop to show how the ap-
proximate solution overlaps to the naked eye). This shows the im-
proved accuracy of Heuns algorithm although it does not behave
so nicely in the general case.

3. From greater order to order one

Ordinary differential equations are usually of order greater than
1. For instance, one of the most usual settings is a mechanical prob-
lem in which forces act on a body to determine its motion. The equa-
tion that governs this system is, as the reader should know, New-
tons second law:
~a = m ~F
which is stated as acceleration is mass times force. If the position
of the body at time t is ( x1 (t), . . . , xn (t)), then the acceleration vec-
tor is ~a = ( x1 (t), . . . , x n (t)) and if the resultant force is ( F1 , . . . , Fn )
(which would depend on ( x1 , . . . , xn )) then Newtons Law becomes,
 3. FROM GREATER ORDER TO ORDER ONE 99

F IGURE 3. Approximation to the solution of Problem

(23) using Heuns method. The error is unnoticeable
on the printed page.

in coordinates:


x1 (t) = F1 ( x1 (t), . . . , xn (t)) m
x (t) = F ( x (t), . . . , x (t)) m

2 2 1 n
.
..


x n (t) = Fn ( x1 (t), . . . , xn (t)) m

which is an ordinary differential equation but has order greater than

one so that we cannot apply the methods explained above straight-
away. Notice that the system is autnomous because Newtons second
law is time-invariant (it depends only on the positions of the bodies,
not on time).
However, one can look at a body in motion under the influence
of forces as a system whose state is not just the position that
is, the set of coordinates ( x1 (t), . . . , xn (t)) but as a system whose
state is described by position and velocity. As a matter of fact, every-
body knows by experience that the trajectory of a body (say, a stone)
influenced by gravity depends not only on its initial position (from
where it is thrown away) but also on its initial velocity (how fast it
is thrown away). So, in mechanics, velocities are intrinsic elements
of the system of coordinates. Thus, the body B under study has 2n
coordinates to account for its state: its position ( x1 , . . . , xn ) and its
velocity (v1 , . . . , vn ). By their own nature, these 2n coordinates sat-
isfy the following equations:
x 1 (t) = v1 (t), x 2 (t) = v2 (t), . . . , x n (t) = vn (t).
100 7. MULTIVARIATE AND HIGHER ORDER ODES

And now Newtons law can be expressed as a true ordinary differ-

ential equation of order one


x 1 (t) = v1 (t)
v 1 (t) = F1 ( x1 (t), . . . , xn (t)) m





x 2 (t) = v2 (t)



(26) v 2 (t) = F2 ( x1 (t), . . . , xn (t)) m


.
..


x n (t) = vn (t)





v n (t) = Fn ( x1 (t), . . . , xn (t)) m

to which the methods explained above can be applied. Notice that

one is usually only interested in knowing the positions ( x1 (t), . . . , xn (t))
but one must solve the system for both positions and velocities. Af-
ter computing both values, one may discard the latter, if it is truly
unnecessary.
For an initial value problem, one needs (as can be seen from
Equation 26) not only the position at time t0 but also the velocity
at that time.
3.1. The general case is similar. In general, one starts with an
ordinary differential equation of some order k the maximum or-
der of derivation that appears for some variable and transforms the
problem having n variables and the derivatives up to order k 1 into
one having kn variables by adding (k 1) new variables.
One may state the general case as a differential equation

d k x1
= F1 (t, x, x0 , . . . , xk1) )





dt k

d k x2


(27) k
= F2 (t, x, x0 , . . . , xk1) )
dt
..
.




dk xn


= Fn (t, x, x0 , . . . , xk1) )



dtk
where the functions F1 , . . . , Fn depend on t and the variables x1 , . . . , xn
and their derivatives with respect to t up to order k 1. It is easier
to understand than to write down.
In order to turn it into a system of equations of order one, one
just defines k 1 new systems of variables, which we enumerate
with superindices
(u11 , . . . , u1n ), . . . , (u1k1 , . . . , ukn1 )
 3. FROM GREATER ORDER TO ORDER ONE 101

and specify that each system corresponds to the derivative of the

previous one (and the first one is the derivative of the coordinates),

1
x 1 = u1
1 2
u 1 = u1
u 1k2 = u1k1
.. , ..
. ,..., ..
. .
1
u kn2 = ukn1

1 2

xn = un u n = un

Using this technique, one ends up with a (very long) ordinary

differential equation of order one in the variables

( x1 , . . . , xn , u11 , . . . , u1n , . . . , u1k1 , . . . , ukn1 ).

3.2. The two-body problem. As an example, let us consider a

typical problem of newtonian mechanics, the motion of a system of
two bodies which is, again, autonomous. Let B1 and B2 be two
dimensionless objects with masses m1 and m2 which we suppose are
placed on the same plane (so that we only need two coordinates to
describe their positions). The usual approach is to consider one of
the bodies fixed and the other movable but we want to show how
both bodies do move and how they do interact with each other. If
( x1 , y1 ) and ( x2 , y2 ) denote the coordinates of each body, and dots
indicate derivation with respect to the time variable, Newtons equa-
tions of motion state that, respectively:

m2
( x1 , y1 ) = G 3/2
( x2 x1 , y2 y1 )
(28)
(( x2 x1 )2 + (y2 y1 )2 )
m1
( x2 , y2 ) = G 3/2
( x1 x2 , y1 y2 )
(( x2 x1 )2 + (y2 y1 )2 )

where G is the gravitational constant. We shall use units in which

G = 1 in order to simplify the exposition. There are 4 equations in
(28), one for each second derivative. Notice that the first derivative
of each xi , yi does not appear explicitely but this is irrelevant: the
problem is of order two and to turn it into one of order one, they
have to be made explicit. Given that there are four coordinates, we
need another four variables, one for each first derivative. Let us call
them (u x , uy ) and (v x , vy ). Writing all the equations one after the
 102 7. MULTIVARIATE AND HIGHER ORDER ODES

other for G = 1, we get

x 1 = u x
m2
u x = ( x x1 )
3/2 2
(( x2 x1 )2 + (y2 y1 )2 )
y 1 = uy
m2
u y = ( y y1 )
3/2 2
(( x2 x1 )2 + ( y2 y1 )2 )
(29)
x2 = v x
m1
v x = ( x x2 )
3/2 1
(( x2 x1 )2 + (y2 y1 )2 )
y2 = vy
m1
v y = ( y y2 )
3/2 1
(( x2 x1 )2 + (y2 y1 )2 )
which is a standard ordinary differential equation of the first order.
An initial value problem would require both a pair of initial po-
sitions (for ( x1 , y1 ) and ( x2 , y2 )) and a pair of initial velocities (for
(u x , uy ) and (v x , vy )): this gives eight values, apart from the masses,
certainly.
In Listing 7.1 we show an implementation of Heuns method for
the two-body problem (with G = 1). In order to use it, the function
twobody must receive the values of the masses, a vector of initial
conditions as ( x1 (0), y1 (0), x2 (0), y2 (0), u x (0), uy (0), v x (0), vy (0))), a
step size h and the number of steps to compute.
1 % two-body problem with Heun
2 % x is the initial value vector:
3 % (x1, y1, x2, y2, v1x, v1y, v2x, v2y)
4 function s = twobody (m1, m2, x, h, steps)
5 % use next line for plotting to pngs (prevents plotting on screen)
6 % set(0, defaultfigurevisible, on);
7 % hold on
8 s = [x(1) x(2) x(3) x(4)];
9 for k = 1:steps
10 d = ((x(3) - x(1))2 + (x(4) - x(2))2)(1/2);
11 F1 = m2/d3*[x(3)-x(1) x(4)-x(2)];
12 F2 = -m1/(d3*m2)*F1;
13 w1 = [x(5:end) F1 F2];
14 z = x + h*[x(5:end) F1 F2];
15 dd = ((z(3) - z(1))2 + (z(4) - z(2))2)(1/2);
16 FF1 = m2/dd3*[z(3)-z(1) z(4)-z(2)];
17 FF2 = -m1/(dd3*m2)*FF1;
18 w2 = [z(5:end) FF1 FF2];
19 v = (w1 + w2)/2;
20 x = x + h*v;
21 s = [s; x(1:4)];
22 % next lines for filming, if desired
 3. FROM GREATER ORDER TO ORDER ONE 103

F IGURE 4. Approximation to the solution of the two-

body problem with initial values as in the text. One of
the bodies (in red) is more massive than the other.

23 %plot(x1,y1,r, x2,y2,b);
24 %axis([-2 2 -2 2]);
25 %filename=sprintf(pngs/%05d.png,k);
26 %print(filename);
27 %clf;
28 end
29 endfunction
30 % interesting values:
31 % r=twobody(4,400,[-1,0,1,0,0,14,0,-0.1],.01,40000); (pretty singular)
32 % r=twobody(1,1,[-1,0,1,0,0,0.3,0,-0.5],.01,10000); (strange->loss of
precision!)
33 % r=twobody(1,900,[-30,0,1,0,0,2.25,0,0],.01,10000); (like sun-earth)
34 % r=twobody(1, 333000, [149600,0,0,0,0,1,0,0], 100, 3650); (earth-sun)...
L ISTING 7.1. An implementation of the two-body
problem with G = 1 using Heuns method.
In Figure 4, a plot for the run of the function twobbody for masses
4, 400 and initial conditions (1, 0), (1, 0), (0, 14), (0, 0.1) is given,
using h = 0.01 and 40000 steps. The difference in mass is what makes
the trajectories so different from one another, more than the differ-
ence in initial speed. Notice that the blue particle turns elliptically
around the red one.
Finally, in Figure 5, we see both the plot of the motion of two par-
ticles and, next to it, the relative motion of one of them with respect
to the other, in order to give an idea of:
The fact that trajectories of one body with respect to the
other are ellipses.
The error incurred in the approximation (if the solution were
exact, the plot on the right would be a closed ellipse, not the
open trajectory shown).
104 7. MULTIVARIATE AND HIGHER ORDER ODES

F IGURE 5. Absolute and relative motion of two parti-

cles. On the left, the absolute trajectories, on the right,
the relative ones.